NetXMS Support Forum

Hi,

Something wierd is happing after I upgraded my 1.2.8 netxms on a VM running centos 5.

The server is running up to one day and then the process is killed by the kernel "out of memory", do you have an idea what is this happeing? before on version 1.2.8 I've never had this issue.

The network discovery is disable. Please find below the logs from the VM.

Thanks,


Christian Diaz


Jun 12 07:30:55 dashboard kernel: netxmsd invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0
Jun 12 07:30:55 dashboard kernel: netxmsd cpuset=/ mems_allowed=0
Jun 12 07:30:55 dashboard kernel: Pid: 24776, comm: netxmsd Not tainted 2.6.32-358.14.1.el6.x86_64 #1
Jun 12 07:30:55 dashboard kernel: Call Trace:
Jun 12 07:30:55 dashboard kernel: [<ffffffff810cb561>] ? cpuset_print_task_mems_allowed+0x91/0xb0
Jun 12 07:30:55 dashboard kernel: [<ffffffff8111cd80>] ? dump_header+0x90/0x1b0
Jun 12 07:30:55 dashboard kernel: [<ffffffff8121d2ac>] ? security_real_capable_noaudit+0x3c/0x70
Jun 12 07:30:55 dashboard kernel: [<ffffffff8111d202>] ? oom_kill_process+0x82/0x2a0
Jun 12 07:30:55 dashboard kernel: [<ffffffff8111d141>] ? select_bad_process+0xe1/0x120
Jun 12 07:30:55 dashboard kernel: [<ffffffff8111d640>] ? out_of_memory+0x220/0x3c0
Jun 12 07:30:55 dashboard kernel: [<ffffffff8112c2ec>] ? __alloc_pages_nodemask+0x8ac/0x8d0
Jun 12 07:30:55 dashboard kernel: [<ffffffff811609ea>] ? alloc_pages_current+0xaa/0x110
Jun 12 07:30:55 dashboard kernel: [<ffffffff8111a167>] ? __page_cache_alloc+0x87/0x90
Jun 12 07:30:55 dashboard kernel: [<ffffffff81119b4e>] ? find_get_page+0x1e/0xa0
Jun 12 07:30:55 dashboard kernel: [<ffffffff8111b127>] ? filemap_fault+0x1a7/0x500
Jun 12 07:30:55 dashboard kernel: [<ffffffff81143124>] ? __do_fault+0x54/0x530
Jun 12 07:30:55 dashboard kernel: [<ffffffff811436f7>] ? handle_pte_fault+0xf7/0xb50
Jun 12 07:30:55 dashboard kernel: [<ffffffff81063360>] ? wake_up_state+0x10/0x20
Jun 12 07:30:55 dashboard kernel: [<ffffffff810a9ae0>] ? wake_futex+0x40/0x60
Jun 12 07:30:55 dashboard kernel: [<ffffffff810aa35e>] ? futex_wake+0x10e/0x120
Jun 12 07:30:55 dashboard kernel: [<ffffffff8114438a>] ? handle_mm_fault+0x23a/0x310
Jun 12 07:30:55 dashboard kernel: [<ffffffff810474e9>] ? __do_page_fault+0x139/0x480
Jun 12 07:30:55 dashboard kernel: [<ffffffff81435fd5>] ? sys_sendto+0x185/0x190
Jun 12 07:30:55 dashboard kernel: [<ffffffff8150dd80>] ? thread_return+0x4e/0x76e
Jun 12 07:30:55 dashboard kernel: [<ffffffff815137be>] ? do_page_fault+0x3e/0xa0
Jun 12 07:30:55 dashboard kernel: [<ffffffff81510b75>] ? page_fault+0x25/0x30
Jun 12 07:30:55 dashboard kernel: Mem-Info:
Jun 12 07:30:55 dashboard kernel: Node 0 DMA per-cpu:
Jun 12 07:30:55 dashboard kernel: CPU    0: hi:    0, btch:   1 usd:   0
Jun 12 07:30:55 dashboard kernel: Node 0 DMA32 per-cpu:
Jun 12 07:30:55 dashboard kernel: CPU    0: hi:  186, btch:  31 usd:  68
Jun 12 07:30:55 dashboard kernel: active_anon:63386 inactive_anon:63461 isolated_anon:0
Jun 12 07:30:55 dashboard kernel: active_file:265 inactive_file:334 isolated_file:0
Jun 12 07:30:55 dashboard kernel: unevictable:0 dirty:6 writeback:1 unstable:0
Jun 12 07:30:55 dashboard kernel: free:13251 slab_reclaimable:2690 slab_unreclaimable:13761
Jun 12 07:30:55 dashboard kernel: mapped:8752 shmem:8570 pagetables:3855 bounce:0
Jun 12 07:30:55 dashboard kernel: Node 0 DMA free:8348kB min:332kB low:412kB high:496kB active_anon:3132kB inactive_anon:3268kB active_file:52kB inactive_file:224kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15304kB mlocked:0kB dirty:0kB writeback:4kB mapped:1300kB shmem:1252kB slab_reclaimable:8kB slab_unreclaimable:72kB kernel_stack:416kB pagetables:128kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:288 all_unreclaimable? no
Jun 12 07:30:55 dashboard kernel: lowmem_reserve[]: 0 2004 2004 2004
Jun 12 07:30:55 dashboard kernel: Node 0 DMA32 free:44656kB min:44720kB low:55900kB high:67080kB active_anon:250412kB inactive_anon:250576kB active_file:1008kB inactive_file:1112kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:2052256kB mlocked:0kB dirty:24kB writeback:0kB mapped:33708kB shmem:33028kB slab_reclaimable:10752kB slab_unreclaimable:54972kB kernel_stack:3176kB pagetables:15292kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:2816 all_unreclaimable? no
Jun 12 07:30:55 dashboard kernel: lowmem_reserve[]: 0 0 0 0
Jun 12 07:30:55 dashboard kernel: Node 0 DMA: 23*4kB 18*8kB 21*16kB 7*32kB 2*64kB 2*128kB 2*256kB 3*512kB 1*1024kB 2*2048kB 0*4096kB = 8348kB
Jun 12 07:30:55 dashboard kernel: Node 0 DMA32: 10656*4kB 2*8kB 2*16kB 2*32kB 2*64kB 0*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 44656kB
Jun 12 07:30:55 dashboard kernel: 18191 total pagecache pages
Jun 12 07:30:55 dashboard kernel: 9001 pages in swap cache
Jun 12 07:30:55 dashboard kernel: Swap cache stats: add 5526230, delete 5517229, find 13659110/14099219
Jun 12 07:30:55 dashboard kernel: Free swap  = 0kB
Jun 12 07:30:55 dashboard kernel: Total swap = 4128760kB
Jun 12 07:30:55 dashboard kernel: 524272 pages RAM
Jun 12 07:30:55 dashboard kernel: 43681 pages reserved
Jun 12 07:30:55 dashboard kernel: 57899 pages shared
Jun 12 07:30:55 dashboard kernel: 453313 pages non-shared
Jun 12 07:30:55 dashboard kernel: [ pid ]   uid  tgid total_vm      rss cpu oom_adj oom_score_adj name
Jun 12 07:30:55 dashboard kernel: [  429]     0   429     2766        1   0     -17         -1000 udevd
Jun 12 07:30:55 dashboard kernel: [  997]     0   997    23293       11   0     -17         -1000 auditd
Jun 12 07:30:55 dashboard kernel: [ 1013]     0  1013    62367        7   0       0             0 rsyslogd
Jun 12 07:30:55 dashboard kernel: [ 1025]    81  1025     7911        2   0       0             0 dbus-daemon
Jun 12 07:30:55 dashboard kernel: [ 1063]     0  1063    16563        3   0     -17         -1000 sshd
Jun 12 07:30:55 dashboard kernel: [ 1071]    38  1071     7540       36   0       0             0 ntpd
Jun 12 07:30:55 dashboard kernel: [ 1180]     0  1180    20216       23   0       0             0 master
Jun 12 07:30:55 dashboard kernel: [ 1189]    89  1189    20279       18   0       0             0 qmgr
Jun 12 07:30:55 dashboard kernel: [ 1190]     0  1190    29312       23   0       0             0 crond
Jun 12 07:30:55 dashboard kernel: [ 1202]     0  1202    15628      213   0       0             0 miniserv.pl
Jun 12 07:30:55 dashboard kernel: [ 1210]     0  1210     1014        2   0       0             0 mingetty
Jun 12 07:30:55 dashboard kernel: [ 1212]     0  1212     1014        2   0       0             0 mingetty
Jun 12 07:30:55 dashboard kernel: [ 1214]     0  1214     1014        2   0       0             0 mingetty
Jun 12 07:30:55 dashboard kernel: [ 1216]     0  1216     1014        2   0       0             0 mingetty
Jun 12 07:30:55 dashboard kernel: [ 1218]     0  1218     1014        2   0       0             0 mingetty
Jun 12 07:30:55 dashboard kernel: [ 1220]     0  1220     1014        2   0       0             0 mingetty
Jun 12 07:30:55 dashboard kernel: [ 1229]     0  1229     3095        1   0     -17         -1000 udevd
Jun 12 07:30:55 dashboard kernel: [ 1455]     0  1455    11713      175   0       0             0 openvpn
Jun 12 07:30:55 dashboard kernel: [ 1459]     0  1459     2765        1   0     -17         -1000 udevd
Jun 12 07:30:55 dashboard kernel: [27796]    26 27796    53961      270   0     -17         -1000 postmaster
Jun 12 07:30:55 dashboard kernel: [27798]    26 27798    44714       23   0       0             0 postmaster
Jun 12 07:30:55 dashboard kernel: [27800]    26 27800    54027     8439   0       0             0 postmaster
Jun 12 07:30:55 dashboard kernel: [27801]    26 27801    53961       48   0       0             0 postmaster
Jun 12 07:30:55 dashboard kernel: [27802]    26 27802    54187      189   0       0             0 postmaster
Jun 12 07:30:55 dashboard kernel: [27803]    26 27803    44926      166   0       0             0 postmaster
Jun 12 07:30:55 dashboard kernel: [29741]     0 29741   145458      575   0       0             0 nxagentd
Jun 12 07:30:55 dashboard kernel: [24680]     0 24680  1328356    76557   0       0             0 netxmsd
Jun 12 07:30:55 dashboard kernel: [24681]    26 24681    54756     5044   0       0             0 postmaster
Jun 12 07:30:55 dashboard kernel: [24682]    26 24682    54784     8936   0       0             0 postmaster
Jun 12 07:30:55 dashboard kernel: [24683]    26 24683    54463     5255   0       0             0 postmaster
Jun 12 07:30:55 dashboard kernel: [24684]    26 24684    54319     2435   0       0             0 postmaster
Jun 12 07:30:55 dashboard kernel: [24685]    26 24685    54326     1270   0       0             0 postmaster
Jun 12 07:30:55 dashboard kernel: [24686]    26 24686    54297      333   0       0             0 postmaster
Jun 12 07:30:55 dashboard kernel: [24806]    26 24806    85327    32101   0       0             0 postmaster
Jun 12 07:30:55 dashboard kernel: [24894]    26 24894    54494     8758   0       0             0 postmaster
Jun 12 07:30:55 dashboard kernel: [24895]    26 24895    54837     9009   0       0             0 postmaster
Jun 12 07:30:55 dashboard kernel: [19022]    89 19022    20236       19   0       0             0 pickup
Jun 12 07:30:55 dashboard kernel: [20255]     0 20255    18737     5391   0       0             0 miniserv.pl
Jun 12 07:30:55 dashboard kernel: [20279]     0 20279     2824       47   0       0             0 sh
Jun 12 07:30:55 dashboard kernel: [20281]     0 20281    19113     1052   0       0             0 rpm
Jun 12 07:30:55 dashboard kernel: [20295]    26 20295    54363      667   0       0             0 postmaster
Jun 12 07:30:55 dashboard kernel: Out of memory: Kill process 24680 (netxmsd) score 692 or sacrifice child
Jun 12 07:30:55 dashboard kernel: Killed process 24680, UID 0, (netxmsd) total-vm:5313424kB, anon-rss:305708kB, file-rss:520kB

I have the same problem. I just updated from version 1.2.8 to 1.2.4 on a VM running Centos 6.

Please find the debug message after trying to modify an object tool:

[10-Jun-2014 13:06:57.880] [ERROR] SQL query failed (Query = "DELETE FROM object_tools_acl WHERE tool_id=?"): 08P01 ERROR:  bind message supplies 9 parameters, but prepared statement "netxms_stmt_0x7fbef4055630_1402420017" requires 1

I could modify the objects by doing it manually on the Database.