NetXMS Support Forum

...and it crashed again.

[New Thread 0x7fffa78f8700 (LWP 11125)]
[New Thread 0x7fffa7fff700 (LWP 11130)]
[New Thread 0x7fff9c040700 (LWP 11132)]
[New Thread 0x7fff9fb7b700 (LWP 11139)]
[09-Mar-2016 10:36:52.579] [DEBUG] [CLSN-1] Received message CMD_GET_SERVER_INFO
[09-Mar-2016 10:36:52.579] [DEBUG] [CLSN-1] Server time zone: CET+01CEST
[09-Mar-2016 10:36:52.579] [DEBUG] [CLSN-1] Sending message CMD_REQUEST_COMPLETED

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fff9b535700 (LWP 12974)]
0x00007ffff43039f0 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
(gdb) bt
#0  0x00007ffff43039f0 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007ffff68d78b7 in AbstractMessageReceiver::readMessage(unsigned int, MessageReceiverResult*) () from /usr/lib/x86_64-linux-gnu/libnetxms.so.2
#2  0x00007ffff7aad0c7 in ClientSession::readThread() () from /usr/lib/x86_64-linux-gnu/libnxcore.so.2
#3  0x00007ffff7aad8d9 in ClientSession::readThreadStarter(void*) () from /usr/lib/x86_64-linux-gnu/libnxcore.so.2
#4  0x00007ffff5106b50 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#5  0x00007ffff42b930d in clone () from /lib/x86_64-linux-gnu/libc.so.6
#6  0x0000000000000000 in ?? ()

Same problem with other commands.

I'm using deb packages.

I kept gdb running after the last crash and tried running rest of commands:

(gdb) frame 1
#1  0x00007ffff68d78b7 in AbstractMessageReceiver::readMessage(unsigned int, MessageReceiverResult*) () from /usr/lib/x86_64-linux-gnu/libnetxms.so.2
(gdb) print *this
No symbol table is loaded.  Use the "file" command.
(gdb) frame 2
#2  0x00007ffff7aad0c7 in ClientSession::readThread() () from /usr/lib/x86_64-linux-gnu/libnxcore.so.2
(gdb) print *this
No symbol table is loaded.  Use the "file" command.

I've now restarted, as per your instructions.

Happened sooner than I expected.

[New Thread 0x7fffb4ccc700 (LWP 10037)]
[New Thread 0x7fffaad2d700 (LWP 10067)]
[New Thread 0x7fffaac2c700 (LWP 10095)]

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffaf777700 (LWP 25348)]
0x00007ffff43039f0 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
(gdb) bt
#0  0x00007ffff43039f0 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007ffff68d78b7 in AbstractMessageReceiver::readMessage(unsigned int, MessageReceiverResult*) () from /usr/lib/x86_64-linux-gnu/libnetxms.so.2
#2  0x00007ffff7aad0c7 in ClientSession::readThread() () from /usr/lib/x86_64-linux-gnu/libnxcore.so.2
#3  0x00007ffff7aad8d9 in ClientSession::readThreadStarter(void*) () from /usr/lib/x86_64-linux-gnu/libnxcore.so.2
#4  0x00007ffff5106b50 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#5  0x00007ffff42b930d in clone () from /lib/x86_64-linux-gnu/libc.so.6
#6  0x0000000000000000 in ?? ()
(gdb)

OK, it's running under gdb. What should I do should crash occur? Anything other than bt?

[18476348.206574] netxmsd[1485]: segfault at 10680c2c0 ip 00007f61b406a050 sp 00007f6164180978 error 4 in libc-2.13.so[7f61b3f44000+181000]
[19513061.150308] netxmsd[19644]: segfault at f563b5e0 ip 00007ff538c4c9f0 sp 00007ff4deeed978 error 4 in libc-2.13.so[7ff538b24000+184000]
[19677206.312203] netxmsd[867]: segfault at b6b7cd50 ip 00007f92c1d499f0 sp 00007f927ca89978 error 4 in libc-2.13.so[7f92c1c21000+184000]
[20116663.843012] netxmsd[30912]: segfault at e7a81c78 ip 00007f2c047679f0 sp 00007f2bb8685978 error 4 in libc-2.13.so[7f2c0463f000+184000]

since some time netxmsd crashes every few days like above. Everything else on this machine works fine. Any ideas?

OK then, I'll upgrade and see what happens

Just a note, that this issue still exists in 2.0-RC1, and changelog doesn't seem to indicate that anything is changed in RC2.

I already have similar script for some other stuff, but never got around to adapting it to netxms. I suppose I'll do it now that you've shown how to do it - I still would very much like to have it fixed in NetXMS tough.

Linux ofc.

I'm using ejabberd, not that it matters. Since everybody using XMPP seeem to have this problem, I strongly believe that main problem with XMPP support in NetXMS is Victor not using XMPP

Yeah, that's sad. People use pointless things like google/facebook chat, and these aren't xmpp-compatible because 'why not'. I'm only missing http://xmpp.org/extensions/xep-0280.html - next to no client implements it.

I'll probably get to debugging with gdb this weekend, but if you could do it sooner then go ahead. Details are in Victor's post.

Just a note, that sending XMPP messages holds up the event processing thread. 7 events with xmpp action held up the queue for 10 minutes and 'show queues' on server console shown 206 waiting events.

tcpdump shows no communication with xmpp server until the exact moment the message is actually sent, so it looks like the problem is internal to netxms.

Oh, and it may be worth mentioning, that server console over xmpp works just fine, since above mentioned 'show queues' results were gathered via xmpp, so it would seem, like the library itself isn't the culprit.

Yeah, I know it's free. But I like it. I like showing my appreciation, or my company's appreciation for that matter, so Victor keeps working on it