NetXMS Support Forum

English Support => General Support => Topic started by: envinet on October 27, 2015, 09:22:00 AM

Title: NetXMS server crashed by SNMP trap
Post by: envinet on October 27, 2015, 09:22:00 AM
Hello.
My environment is NetXMS 2.0-RC2 Ubuntu server 14.04 LTS.
I have the problem that netxmsd crashes after some time.
In syslog is:
NetXMS kernel: [248886.914205] netxmsd[24807]: segfault at 18 ip 00007fa108d8d4d5 sp 00007fa0e88b7660 error 4 in libnxcore.so.2.0.0[7fa108c84000+1c9000]

I tried, export database and reinstall NetXMS. But it is still same.
I using zones and proxy for SNMP, NetXMS agent and ping.


root@NetXMS:/var/log# tail syslog
Oct 26 12:08:24 NetXMS snmpd[3917]: Connection from UDP: [10.161.193.46]:58751->[10.161.193.46]:161
Oct 26 12:08:25 NetXMS snmpd[3917]: Connection from UDP: [127.0.0.1]:51545->[127.0.0.1]:161
Oct 26 12:08:26 NetXMS snmpd[3917]: Connection from UDP: [10.161.193.46]:58751->[10.161.193.46]:161
Oct 26 12:08:27 NetXMS snmpd[3917]: Connection from UDP: [127.0.0.1]:51545->[127.0.0.1]:161
Oct 26 12:08:28 NetXMS snmpd[3917]: Connection from UDP: [10.161.193.46]:58751->[10.161.193.46]:161
Oct 26 12:08:29 NetXMS snmpd[3917]: Connection from UDP: [127.0.0.1]:51545->[127.0.0.1]:161
Oct 26 12:08:33 NetXMS snmpd[3917]: message repeated 2 times: [ Connection from UDP: [127.0.0.1]:51545->[127.0.0.1]:161]
Oct 26 12:08:35 NetXMS snmpd[3917]: Connection from UDP: [127.0.0.1]:54935->[127.0.0.1]:161
Oct 26 12:17:01 NetXMS CRON[4895]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Oct 26 13:09:40 NetXMS kernel: [254401.868480] netxmsd[3466]: segfault at 18 ip 00007f0b26c3c4d5 sp 00007f0af66d5660 error 4 in libnxcore.so.2.0.0[7f0b26b33000+1c9000]
root@NetXMS:/var/log# tail netxmsd
[26-Oct-2015 13:09:25.710] [DEBUG] StatusPoll(HALA-SW2L2 [364]): boot time set to 1426642695 from SNMP
[26-Oct-2015 13:09:25.710] [DEBUG] StatusPoll(HALA-SW2L2 [364]): unable to get agent uptime
[26-Oct-2015 13:09:25.711] [DEBUG] Finished status poll for node HALA-SW2L2 (ID: 364)
[26-Oct-2015 13:09:30.052] [DEBUG] Starting status poll for node HALA-SW1L2 (ID: 309)
[26-Oct-2015 13:09:31.862] [DEBUG] StatusPoll(HALA-SW1L2 [309]): boot time set to 1426642501 from SNMP
[26-Oct-2015 13:09:31.862] [DEBUG] StatusPoll(HALA-SW1L2 [309]): unable to get agent uptime
[26-Oct-2015 13:09:31.862] [DEBUG] Finished status poll for node HALA-SW1L2 (ID: 309)
[26-Oct-2015 13:09:40.517] [DEBUG] AgentConnectionEx::onSnmpTrap(): Received SNMP trap message from agent at 10.161.181.13, node ID 733
[26-Oct-2015 13:09:40.517] [DEBUG] AgentConnectionEx::onSnmpTrap(): SNMP trapID is valid
[26-Oct-2015 13:09:40.517] [DEBUG] Received SNMP INFORM-REQUEST .1.3.6.1.2.1.17.0.2 from 10.161.181.195
root@NetXMS:/var/log#


After disable SNMP in NetXMS server config, server is stable.
"Name","Value","Restart"
"EnableCheckPointSNMP","0","No"
"EnableSNMPTraps","0","Yes"
"LogAllSNMPTraps","0","Yes"
"SNMPRequestTimeout","2000","Yes"
"SNMPTrapLogRetentionTime","90","No"
"SNMPTrapPort","162","Yes"
"UseSNMPTrapsForDiscovery","0","Yes"



Title: Re: NetXMS server crashed by SNMP trap
Post by: Victor Kirhenshtein on October 29, 2015, 10:30:27 PM
Hi,

do you have core file left? If yes, please provide stack trace from it. Alternatively, try tu run netxmsd under debugger, wait for crash, and send us output. Here is the instruction: https://wiki.netxms.org/wiki/Running_NetXMS_under_debugger (https://wiki.netxms.org/wiki/Running_NetXMS_under_debugger).

Best regards,
Victor
Title: Re: NetXMS server crashed by SNMP trap
Post by: envinet on October 30, 2015, 09:26:00 AM
This fault is only when is set  in nxagetd.conf on Proxy client for NetXMS.
EnableSNMPTrapProxy = yes

Starting program: /usr/bin/netxmsd -D2
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Using configuration file "/etc/netxmsd.conf"
[30-Oct-2015 08:20:57.556] [DEBUG] Code page set to ASCII
[30-Oct-2015 08:20:57.574] [INFO ] Platform subagent "/usr/lib/x86_64-linux-gnu/netxms/linux.nsm" successfully loaded
[30-Oct-2015 08:20:57.575] [DEBUG] Debug callback set for DB library
[New Thread 0x7ffff087b700 (LWP 11759)]
[Thread 0x7ffff087b700 (LWP 11759) exited]
[30-Oct-2015 08:20:57.699] [INFO ] Database driver "mysql.ddr" loaded and initialized successfully
[30-Oct-2015 08:20:57.764] [DEBUG] Successfully connected to database [email protected]
[New Thread 0x7ffff7fd0700 (LWP 11760)]
[30-Oct-2015 08:20:58.130] [DEBUG] Database Connection Pool initialized
[30-Oct-2015 08:20:58.131] [DEBUG] Database Connection Pool maintenance thread started
[30-Oct-2015 08:20:58.180] [DEBUG] Server ID 2B16E871C8BB601F
[30-Oct-2015 08:20:58.326] [INFO ] Stalled database lock removed
[30-Oct-2015 08:20:58.470] [DEBUG] Data directory set to /var/lib/netxms
[30-Oct-2015 08:20:58.515] [DEBUG] Global configuration loaded
[30-Oct-2015 08:20:58.520] [DEBUG] Validating ciphers
[30-Oct-2015 08:20:58.521] [DEBUG]    AES-256 enabled
[30-Oct-2015 08:20:58.521] [DEBUG]    Blowfish-256 enabled
[30-Oct-2015 08:20:58.521] [DEBUG]    IDEA disabled (config)
[30-Oct-2015 08:20:58.521] [DEBUG]    3DES enabled
[30-Oct-2015 08:20:58.521] [DEBUG]    AES-128 enabled
[30-Oct-2015 08:20:58.521] [DEBUG]    Blowfish-128 enabled
[30-Oct-2015 08:20:58.522] [DEBUG] Crypto library initialized
[30-Oct-2015 08:20:58.537] [DEBUG] Creating thread pools
[New Thread 0x7fffebdf3700 (LWP 11761)]
[New Thread 0x7fffebcf2700 (LWP 11762)]
[New Thread 0x7fffebbf1700 (LWP 11763)]
[New Thread 0x7fffebaf0700 (LWP 11764)]
[New Thread 0x7fffeb9ef700 (LWP 11765)]
[New Thread 0x7fffeb8ee700 (LWP 11766)]
[New Thread 0x7fffeb7ed700 (LWP 11767)]
[New Thread 0x7fffeb6ec700 (LWP 11768)]
[New Thread 0x7fffeb5eb700 (LWP 11769)]
[30-Oct-2015 08:20:58.552] [DEBUG] Thread pool MAIN initialized (min=8, max=256)
[30-Oct-2015 08:20:58.750] [DEBUG] ID table created
[30-Oct-2015 08:20:58.803] [DEBUG] Script Filter::SNMP added to library
[30-Oct-2015 08:20:58.804] [DEBUG] Script Filter::Agent added to library
[30-Oct-2015 08:20:58.804] [DEBUG] Script Filter::AgentOrSNMP added to library
[30-Oct-2015 08:20:58.804] [DEBUG] Script DCI::SampleTransform added to library
[New Thread 0x7fffeb4ea700 (LWP 11770)]
[30-Oct-2015 08:20:58.808] [DEBUG] SMTP mailer thread started
[New Thread 0x7fffeb3e9700 (LWP 11771)]
[30-Oct-2015 08:20:58.813] [DEBUG] User account status update thread started
[30-Oct-2015 08:20:58.815] [DEBUG] User accounts loaded
[New Thread 0x7fffeb2e8700 (LWP 11772)]
[30-Oct-2015 08:20:59.355] [DEBUG] Loading network device drivers from /usr/lib/x86_64-linux-gnu/netxms/ndd
[30-Oct-2015 08:20:59.382] [INFO ] Network device driver "CISCO-SB" loaded successfully
[30-Oct-2015 08:20:59.392] [INFO ] Network device driver "UBNT" loaded successfully
[30-Oct-2015 08:20:59.402] [INFO ] Network device driver "PING3" loaded successfully
[30-Oct-2015 08:20:59.411] [INFO ] Network device driver "H3C" loaded successfully
[30-Oct-2015 08:20:59.420] [INFO ] Network device driver "DELL-PWC" loaded successfully
[30-Oct-2015 08:20:59.430] [INFO ] Network device driver "CATALYST-2900XL" loaded successfully
[30-Oct-2015 08:20:59.439] [INFO ] Network device driver "NETSCREEN" loaded successfully
[30-Oct-2015 08:20:59.448] [INFO ] Network device driver "MIKROTIK" loaded successfully
[30-Oct-2015 08:20:59.459] [INFO ] Network device driver "HPSW" loaded successfully
[30-Oct-2015 08:20:59.468] [INFO ] Network device driver "PROCURVE" loaded successfully
[30-Oct-2015 08:20:59.479] [INFO ] Network device driver "DLINK" loaded successfully
[30-Oct-2015 08:20:59.491] [INFO ] Network device driver "ERS8000" loaded successfully
[30-Oct-2015 08:20:59.504] [INFO ] Network device driver "CATALYST-GENERIC" loaded successfully
[30-Oct-2015 08:20:59.516] [INFO ] Network device driver "AT" loaded successfully
[30-Oct-2015 08:20:59.525] [INFO ] Network device driver "CISCO-ESW" loaded successfully
[30-Oct-2015 08:20:59.537] [INFO ] Network device driver "AIRESPACE" loaded successfully
[30-Oct-2015 08:20:59.549] [INFO ] Network device driver "QTECH-OLT" loaded successfully
[30-Oct-2015 08:20:59.561] [INFO ] Network device driver "TB" loaded successfully
[30-Oct-2015 08:20:59.573] [INFO ] Network device driver "BAYSTACK" loaded successfully
[30-Oct-2015 08:20:59.587] [INFO ] Network device driver "NTWS" loaded successfully
[30-Oct-2015 08:20:59.599] [INFO ] Network device driver "SYMBOL-WS" loaded successfully
[30-Oct-2015 08:20:59.600] [DEBUG] 21 network device drivers loaded
[30-Oct-2015 08:20:59.617] [DEBUG] Built-in objects created
[New Thread 0x7fffe838c700 (LWP 11773)]
[30-Oct-2015 08:20:59.619] [DEBUG] Loading container categories...
[30-Oct-2015 08:20:59.620] [DEBUG] Apply template thread started
[30-Oct-2015 08:20:59.629] [DEBUG] Loading built-in object properties...
[30-Oct-2015 08:20:59.704] [DEBUG] Loading zones...
[30-Oct-2015 08:20:59.759] [DEBUG] Loading conditions...
[30-Oct-2015 08:20:59.760] [DEBUG] Loading subnets...
[30-Oct-2015 08:21:00.199] [DEBUG] Loading racks...
[30-Oct-2015 08:21:00.199] [DEBUG] Loading mobile devices...
[30-Oct-2015 08:21:00.199] [DEBUG] Loading nodes...
[30-Oct-2015 08:21:03.115] [DEBUG] Loading access points...
[30-Oct-2015 08:21:03.115] [DEBUG] Loading interfaces...
[30-Oct-2015 08:21:08.710] [DEBUG] Loading network services...
[30-Oct-2015 08:21:08.711] [DEBUG] Loading VPN connectors...
[30-Oct-2015 08:21:08.711] [DEBUG] Loading clusters...
[New Thread 0x7fffe828b700 (LWP 11776)]
[30-Oct-2015 08:21:08.713] [DEBUG] Loading templates...
[30-Oct-2015 08:21:08.714] [DEBUG] Started caching of DCI values
[30-Oct-2015 08:21:08.715] [DEBUG] Finished caching of DCI values
[Thread 0x7fffe828b700 (LWP 11776) exited]
[30-Oct-2015 08:21:08.784] [DEBUG] Loading agent policies...
[30-Oct-2015 08:21:08.784] [DEBUG] Loading network maps...
[30-Oct-2015 08:21:08.815] [DEBUG] Loading containers...
[30-Oct-2015 08:21:08.851] [DEBUG] Loading template groups...
[30-Oct-2015 08:21:08.851] [DEBUG] Loading policy groups...
[30-Oct-2015 08:21:08.851] [DEBUG] Loading map groups...
[30-Oct-2015 08:21:08.857] [DEBUG] Loading dashboards...
[30-Oct-2015 08:21:08.894] [DEBUG] Loading business services...
[30-Oct-2015 08:21:08.894] [DEBUG] Loading node links...
[30-Oct-2015 08:21:08.894] [DEBUG] Loading service checks...
[30-Oct-2015 08:21:08.894] [DEBUG] Linking objects...
[New Thread 0x7fffe828b700 (LWP 11777)]
[30-Oct-2015 08:21:08.900] [DEBUG] Objects loaded and initialized
[30-Oct-2015 08:21:08.900] [DEBUG] Map update thread started
[30-Oct-2015 08:21:08.900] [DEBUG] Situations loaded and initialized
[30-Oct-2015 08:21:08.901] [DEBUG] Helpdesk link disabled
[30-Oct-2015 08:21:08.902] [DEBUG] Loading performance data storage drivers
[30-Oct-2015 08:21:08.902] [DEBUG] 0 performance data storage drivers loaded
[New Thread 0x7fffe818a700 (LWP 11778)]
[New Thread 0x7fffdbfff700 (LWP 11779)]
[New Thread 0x7fffdbefe700 (LWP 11780)]
[New Thread 0x7fffdbdfd700 (LWP 11781)]
[New Thread 0x7fffdbcfc700 (LWP 11782)]
[New Thread 0x7fffdbbfb700 (LWP 11783)]
[New Thread 0x7fffdbafa700 (LWP 11784)]
[New Thread 0x7fffdb9f9700 (LWP 11785)]
[New Thread 0x7fffdb8f8700 (LWP 11786)]
[New Thread 0x7fffdb7f7700 (LWP 11787)]
[New Thread 0x7fffdb6f6700 (LWP 11788)]
[New Thread 0x7fffdb5f5700 (LWP 11789)]
[New Thread 0x7fffdb4f4700 (LWP 11790)]
[New Thread 0x7fffdb3f3700 (LWP 11791)]
[New Thread 0x7fffdb2f2700 (LWP 11792)]
[New Thread 0x7fffdb1f1700 (LWP 11793)]
[New Thread 0x7fffdb0f0700 (LWP 11794)]
[New Thread 0x7fffdafef700 (LWP 11795)]
[New Thread 0x7fffdaeee700 (LWP 11796)]
[New Thread 0x7fffdaded700 (LWP 11797)]
[New Thread 0x7fffdacec700 (LWP 11798)]
[New Thread 0x7fffdabeb700 (LWP 11799)]
[New Thread 0x7fffdaaea700 (LWP 11800)]
[New Thread 0x7fffda9e9700 (LWP 11801)]
[New Thread 0x7fffda8e8700 (LWP 11802)]
[New Thread 0x7fffda7e7700 (LWP 11803)]
[New Thread 0x7fffda6e6700 (LWP 11804)]
[New Thread 0x7fffda5e5700 (LWP 11805)]
[30-Oct-2015 08:21:08.962] [DEBUG] 0 mapping tables loaded
[30-Oct-2015 08:21:08.963] [DEBUG] DCI cache loader thread started
[New Thread 0x7fffda4e4700 (LWP 11806)]
[New Thread 0x7fffda3e3700 (LWP 11807)]
[30-Oct-2015 08:21:08.973] [DEBUG] Node poller started
[New Thread 0x7fffda2e2700 (LWP 11808)]
[30-Oct-2015 08:21:08.984] [DEBUG] Job Manager worker thread started
[New Thread 0x7fffda1e1700 (LWP 11809)]
[New Thread 0x7fffda0e0700 (LWP 11810)]
[New Thread 0x7fffd9fdf700 (LWP 11811)]
[30-Oct-2015 08:21:08.997] [ERROR] SQL query failed (Query = "SELECT idata_value,idata_timestamp FROM idata_2090 WHERE item_id=608 ORDER BY idata_timestamp DESC LIMIT 1"): Lost connection to MySQL server during query
[New Thread 0x7fffd9ede700 (LWP 11812)]
[30-Oct-2015 08:21:09.001] [ERROR] Unable to establish connection with database (Lost connection to MySQL server at 'reading initial communication packet', system error: 4)
[New Thread 0x7fffd9ddd700 (LWP 11813)]
[30-Oct-2015 08:21:09.008] [DEBUG] Housekeeper: wakeup time is 02:00
[New Thread 0x7fffd9cdc700 (LWP 11814)]
[30-Oct-2015 08:21:09.033] [DEBUG] Syncer thread started, sync_interval = 60
[New Thread 0x7fffd9bdb700 (LWP 11815)]
[New Thread 0x7fffd9ada700 (LWP 11816)]
[New Thread 0x7fffd99d9700 (LWP 11817)]
[New Thread 0x7fffd98d8700 (LWP 11818)]
[New Thread 0x7fffd97d7700 (LWP 11819)]
[New Thread 0x7fffd96d6700 (LWP 11820)]
[New Thread 0x7fffd95d5700 (LWP 11821)]
[New Thread 0x7fffd94d4700 (LWP 11822)]
[New Thread 0x7fffd93d3700 (LWP 11823)]
[New Thread 0x7fffd92d2700 (LWP 11824)]
[New Thread 0x7fffd91d1700 (LWP 11825)]
[30-Oct-2015 08:21:09.066] [DEBUG] Thread pool POLLERS initialized (min=10, max=250)
[New Thread 0x7fffd90d0700 (LWP 11826)]
[New Thread 0x7fffd8fcf700 (LWP 11827)]
[New Thread 0x7fffd8ece700 (LWP 11828)]
[New Thread 0x7fffd8dcd700 (LWP 11829)]
[New Thread 0x7fffd8ccc700 (LWP 11830)]
[New Thread 0x7fffc75f5700 (LWP 11852)]
[New Thread 0x7fffc76f6700 (LWP 11851)]
[New Thread 0x7fffc77f7700 (LWP 11850)]
[New Thread 0x7fffc78f8700 (LWP 11849)]
[New Thread 0x7fffc79f9700 (LWP 11848)]
[New Thread 0x7fffc7afa700 (LWP 11847)]
[New Thread 0x7fffc7bfb700 (LWP 11846)]
[New Thread 0x7fffc7cfc700 (LWP 11845)]
[New Thread 0x7fffc7dfd700 (LWP 11844)]
[New Thread 0x7fffc7efe700 (LWP 11843)]
[New Thread 0x7fffc7fff700 (LWP 11842)]
[New Thread 0x7fffd81c1700 (LWP 11841)]
[New Thread 0x7fffd82c2700 (LWP 11840)]
[New Thread 0x7fffd83c3700 (LWP 11839)]
[New Thread 0x7fffd84c4700 (LWP 11838)]
[New Thread 0x7fffd85c5700 (LWP 11837)]
[New Thread 0x7fffd86c6700 (LWP 11836)]
[New Thread 0x7fffd87c7700 (LWP 11835)]
[New Thread 0x7fffd88c8700 (LWP 11834)]
[New Thread 0x7fffd89c9700 (LWP 11833)]
[New Thread 0x7fffd8aca700 (LWP 11832)]
[New Thread 0x7fffd8bcb700 (LWP 11831)]
[Thread 0x7fffd8ccc700 (LWP 11830) exited]
[Thread 0x7fffd8bcb700 (LWP 11831) exited]
[30-Oct-2015 08:21:10.045] [DEBUG] Event storm detector thread stopped because event storm detection is off
[Thread 0x7fffc76f6700 (LWP 11851) exited]
[New Thread 0x7fffd8ccc700 (LWP 11854)]
[New Thread 0x7fffd8bcb700 (LWP 11853)]
[Thread 0x7fffd83c3700 (LWP 11839) exited]
[New Thread 0x7fffc76f6700 (LWP 11855)]
[Thread 0x7fffd85c5700 (LWP 11837) exited]
[New Thread 0x7fffd83c3700 (LWP 11856)]
[Thread 0x7fffd87c7700 (LWP 11835) exited]
[New Thread 0x7fffd85c5700 (LWP 11857)]
[Thread 0x7fffc7bfb700 (LWP 11846) exited]
[Thread 0x7fffd8dcd700 (LWP 11829) exited]
[New Thread 0x7fffc7bfb700 (LWP 11858)]
[Thread 0x7fffd8ece700 (LWP 11828) exited]
[New Thread 0x7fffd8dcd700 (LWP 11859)]
[New Thread 0x7fffd8ece700 (LWP 11860)]
[New Thread 0x7fffd87c7700 (LWP 11861)]
[New Thread 0x7fffc74f4700 (LWP 11862)]
[Thread 0x7fffd83c3700 (LWP 11856) exited]
[Thread 0x7fffc7efe700 (LWP 11843) exited]
[Thread 0x7fffc75f5700 (LWP 11852) exited]
[Thread 0x7fffd8dcd700 (LWP 11859) exited]
[Thread 0x7fffd82c2700 (LWP 11840) exited]
[New Thread 0x7fffd8dcd700 (LWP 11863)]
[Thread 0x7fffd9cdc700 (LWP 11814) exited]
[New Thread 0x7fffc75f5700 (LWP 11864)]
[New Thread 0x7fffd82c2700 (LWP 11865)]
[New Thread 0x7fffc7efe700 (LWP 11866)]
[Thread 0x7fffd87c7700 (LWP 11861) exited]
[New Thread 0x7fffd83c3700 (LWP 11867)]
[Thread 0x7fffd88c8700 (LWP 11834) exited]
[30-Oct-2015 08:21:10.198] [INFO ] Listening for syslog messages on UDP socket 0.0.0.0:514
[30-Oct-2015 08:21:10.198] [INFO ] Listening for syslog messages on UDP socket [::]:514
[Thread 0x7fffc7fff700 (LWP 11842) exited]
[New Thread 0x7fffd88c8700 (LWP 11868)]
[New Thread 0x7fffd87c7700 (LWP 11869)]
[Thread 0x7fffd8bcb700 (LWP 11853) exited]
[New Thread 0x7fffc7fff700 (LWP 11870)]
[Thread 0x7fffc75f5700 (LWP 11864) exited]
[New Thread 0x7fffd8bcb700 (LWP 11871)]
[30-Oct-2015 08:21:10.227] [ERROR] SQL query failed (Query = "SELECT var_value FROM config_clob WHERE var_name=?"): Lost connection to MySQL server during query
[New Thread 0x7fffc73f3700 (LWP 11872)]
[Thread 0x7fffc74f4700 (LWP 11862) exited]
[New Thread 0x7fffc75f5700 (LWP 11873)]
[Thread 0x7fffd8dcd700 (LWP 11863) exited]
[New Thread 0x7fffc72f2700 (LWP 11874)]
[New Thread 0x7fffd8dcd700 (LWP 11875)]
[Thread 0x7fffd8aca700 (LWP 11832) exited]
[New Thread 0x7fffc74f4700 (LWP 11876)]
[Thread 0x7fffc77f7700 (LWP 11850) exited]
[New Thread 0x7fffd8aca700 (LWP 11877)]
[New Thread 0x7fffc77f7700 (LWP 11878)]
[New Thread 0x7fffc71f1700 (LWP 11879)]
[Thread 0x7fffd89c9700 (LWP 11833) exited]
[New Thread 0x7fffc70f0700 (LWP 11880)]
[Thread 0x7fffd8ece700 (LWP 11860) exited]
[New Thread 0x7fffd89c9700 (LWP 11881)]
[Thread 0x7fffc76f6700 (LWP 11855) exited]
[New Thread 0x7fffc76f6700 (LWP 11882)]
[New Thread 0x7fffd8ece700 (LWP 11883)]
[Thread 0x7fffc71f1700 (LWP 11879) exited]
[New Thread 0x7fffc6fef700 (LWP 11885)]
[New Thread 0x7fffc71f1700 (LWP 11884)]
[New Thread 0x7fffc6eee700 (LWP 11886)]
[New Thread 0x7fffc6ded700 (LWP 11887)]
[Thread 0x7fffd81c1700 (LWP 11841) exited]
[Thread 0x7fffc76f6700 (LWP 11882) exited]
[New Thread 0x7fffd81c1700 (LWP 11888)]
[Thread 0x7fffd8aca700 (LWP 11877) exited]
[New Thread 0x7fffc76f6700 (LWP 11889)]
[Thread 0x7fffc7cfc700 (LWP 11845) exited]
[Thread 0x7fffc7fff700 (LWP 11870) exited]
[Thread 0x7fffc7afa700 (LWP 11847) exited]
[New Thread 0x7fffc7cfc700 (LWP 11890)]
[Thread 0x7fffd87c7700 (LWP 11869) exited]
[New Thread 0x7fffc7fff700 (LWP 11891)]
[Thread 0x7fffd85c5700 (LWP 11857) exited]
[Thread 0x7fffc70f0700 (LWP 11880) exited]
[New Thread 0x7fffd87c7700 (LWP 11893)]
[Thread 0x7fffc73f3700 (LWP 11872) exited]
[New Thread 0x7fffc7afa700 (LWP 11895)]
[New Thread 0x7fffd85c5700 (LWP 11894)]
[Thread 0x7fffc71f1700 (LWP 11884) exited]
[Thread 0x7fffd86c6700 (LWP 11836) exited]
[Thread 0x7fffc72f2700 (LWP 11874) exited]
[Thread 0x7fffd84c4700 (LWP 11838) exited]
[New Thread 0x7fffc72f2700 (LWP 11896)]
[Thread 0x7fffd83c3700 (LWP 11867) exited]
[Thread 0x7fffc78f8700 (LWP 11849) exited]
[Thread 0x7fffd8ece700 (LWP 11883) exited]
[Thread 0x7fffc7bfb700 (LWP 11858) exited]
[Thread 0x7fffc79f9700 (LWP 11848) exited]
[New Thread 0x7fffc7bfb700 (LWP 11897)]
[30-Oct-2015 08:21:10.598] [DEBUG] Thread pool SCHEDULER initialized (min=1, max=64)
[30-Oct-2015 08:21:10.602] [DEBUG] Syslog receiver thread started
[30-Oct-2015 08:21:10.602] [DEBUG] Syslog writer thread started
[New Thread 0x7fffc79f9700 (LWP 11898)]
[Thread 0x7fffc7efe700 (LWP 11866) exited]
[New Thread 0x7fffc7efe700 (LWP 11899)]
[Thread 0x7fffd87c7700 (LWP 11893) exited]
[Thread 0x7fffc7dfd700 (LWP 11844) exited]
[Thread 0x7fffd89c9700 (LWP 11881) exited]
[New Thread 0x7fffd87c7700 (LWP 11901)]
[New Thread 0x7fffc7dfd700 (LWP 11900)]
[Thread 0x7fffc74f4700 (LWP 11876) exited]
[Thread 0x7fffd85c5700 (LWP 11894) exited]
[New Thread 0x7fffc74f4700 (LWP 11902)]
[New Thread 0x7fffd85c5700 (LWP 11903)]
[Thread 0x7fffc7fff700 (LWP 11891) exited]
[Thread 0x7fffc72f2700 (LWP 11896) exited]
[New Thread 0x7fffc7fff700 (LWP 11904)]
[New Thread 0x7fffd89c9700 (LWP 11906)]
[New Thread 0x7fffc72f2700 (LWP 11905)]
[Thread 0x7fffc74f4700 (LWP 11902) exited]
[Thread 0x7fffc7cfc700 (LWP 11890) exited]
[Thread 0x7fffd82c2700 (LWP 11865) exited]
[New Thread 0x7fffc7cfc700 (LWP 11907)]
[30-Oct-2015 08:21:10.825] [ERROR] SQL query failed (Query = "INSERT INTO idata_756 (item_id,idata_timestamp,idata_value) VALUES (?,?,?)"): Lost connection to MySQL server during query
[30-Oct-2015 08:21:10.825] [ERROR] SQL query failed (Query = "INSERT INTO tdata_rows_733 (row_id,column_id,value) VALUES (?,?,?)"): Lost connection to MySQL server during query
[Thread 0x7fffd8dcd700 (LWP 11875) exited]
[New Thread 0x7fffd82c2700 (LWP 11910)]
[Thread 0x7fffd85c5700 (LWP 11903) exited]
[New Thread 0x7fffd8dcd700 (LWP 11911)]
[New Thread 0x7fffd85c5700 (LWP 11912)]
[New Thread 0x7fffd8ece700 (LWP 11914)]
[New Thread 0x7fffc74f4700 (LWP 11913)]
[30-Oct-2015 08:21:10.883] [INFO ] Listening for client connections on TCP socket :::4701
[30-Oct-2015 08:21:10.884] [ERROR] SQL query failed (Query = "SELECT var_value FROM config WHERE var_name=?"): Lost connection to MySQL server during query
[30-Oct-2015 08:21:10.885] [INFO ] Listening for client connections on TCP socket 0.0.0.0:4701
[30-Oct-2015 08:21:10.885] [ERROR] SQL query failed (Query = "UPDATE raw_dci_values SET raw_value=?,transformed_value=?,last_poll_time=? WHERE item_id=?"): Lost connection to MySQL server during query
[Thread 0x7fffc7efe700 (LWP 11899) exited]
[New Thread 0x7fffc78f8700 (LWP 11915)]
[30-Oct-2015 08:21:10.901] [ERROR] SQL query failed (Query = "SELECT var_value FROM config WHERE var_name=?"): Lost connection to MySQL server during query
[30-Oct-2015 08:21:10.902] [INFO ] Listening for mobile device connections on TCP socket 0.0.0.0:4747
[30-Oct-2015 08:21:10.903] [ERROR] SQL query failed (Query = "UPDATE raw_dci_values SET raw_value=?,transformed_value=?,last_poll_time=? WHERE item_id=?"): Lost connection to MySQL server during query
[30-Oct-2015 08:21:10.909] [DEBUG] LIBDIR: /usr/lib/x86_64-linux-gnu/netxms
[30-Oct-2015 08:21:10.909] [DEBUG] UptimeCalculator thread started
[30-Oct-2015 08:21:10.922] [INFO ] Listening for mobile device connections on TCP socket :::4747
[30-Oct-2015 08:21:10.926] [DEBUG] Server initialization completed
[New Thread 0x7fffc7efe700 (LWP 11917)]
[30-Oct-2015 08:21:10.933] [INFO ] NetXMS Server started

NetXMS Server V2.0-RC2 Build 8046 (UNICODE) Ready
Enter "help" for command list or "down" for server shutdown
System Console

netxmsd:
Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffc7bfb700 (LWP 11897)]
0x00007ffff7ad04d5 in ProcessTrap(SNMP_PDU*, InetAddress const&, int, SNMP_Transport*, SNMP_Engine*, bool) () from /usr/lib/x86_64-linux-gnu/libnxcore.so.2
(gdb) bt
#0  0x00007ffff7ad04d5 in ProcessTrap(SNMP_PDU*, InetAddress const&, int, SNMP_Transport*, SNMP_Engine*, bool) () from /usr/lib/x86_64-linux-gnu/libnxcore.so.2
#1  0x00007ffff7a27ad6 in AgentConnectionEx::onSnmpTrap(NXCPMessage*) ()
   from /usr/lib/x86_64-linux-gnu/libnxcore.so.2
#2  0x00007ffff77b29ca in AgentConnection::receiverThread() ()
   from /usr/lib/x86_64-linux-gnu/libnxsrv.so.2
#3  0x00007ffff77b2ad9 in AgentConnection::receiverThreadStarter(void*) ()
   from /usr/lib/x86_64-linux-gnu/libnxsrv.so.2
#4  0x00007ffff50fd182 in start_thread ()
   from /lib/x86_64-linux-gnu/libpthread.so.0
#5  0x00007ffff422f47d in clone () from /lib/x86_64-linux-gnu/libc.so.6
Title: Re: NetXMS server crashed by SNMP trap
Post by: Victor Kirhenshtein on November 06, 2015, 12:31:52 PM
Hi,

can you please repeat this test with debug level set to 6? Also, are you using SNMPv3 for sending traps?

Best regards,
Victor
Title: Re: NetXMS server crashed by SNMP trap
Post by: envinet on November 09, 2015, 11:06:25 AM
No, we using only all with 2c SNMP.
I had to reconfigure NetXMS with only one zone without netxms proxy.
I try if is will be the same problem in this configuration.
Now debug running  2 hours without crashing.