NetXMS Support Forum

Hi mstjohn1974,

Please read https://www.netxms.org/documentation/adminguide/
This will give you a basic understanding on how netxms works.
It also tells you how to monitor and how to configure it.

Regards,
Edward

any news on this?

I'm open for any ideas to speed up the resolve of this issue.

include some more info in the code?
reinstall system? move to other OS?

Thanks,
Edward

Hi,

Here is the full log.
It is huge, so I have compressed it.

Hope this helps.

Hello,

Any progress on this?
Server is still crashing several times...

Would it be an option to downgrade to 1.2.14?
This was our previous version.

Thanks,
Edward

and here is another one:

==6816== Thread 66:
==6816== Conditional jump or move depends on uninitialised value(s)
==6816==    at 0x4CC549E: ClientSession::onAlarmUpdate(unsigned int, NXC_ALARM*) (session.cpp:4989)
==6816==    by 0x4C631ED: EnumerateClientSessions(void (*)(ClientSession*, void*), void*) (client.cpp:337)
==6816==    by 0x4C5F278: AlarmManager::newAlarm(char*, char*, int, int, unsigned int, unsigned int, Event*, unsigned int) (alarm.cpp:384)
==6816==    by 0x4C81704: EPRule::generateAlarm(Event*) (epp.cpp:531)
==6816==    by 0x4C81D4A: EPRule::processEvent(Event*) (epp.cpp:468)
==6816==    by 0x4C81F4B: EventPolicy::processEvent(Event*) (epp.cpp:814)
==6816==    by 0x4C862C9: EventProcessor(void*) (evproc.cpp:225)
==6816==    by 0x313D407850: start_thread (in /lib64/libpthread-2.12.so)
==6816==    by 0x313CCE811C: clone (in /lib64/libc-2.12.so)


Hope this helps finding the crash cause.

Here is a part from an error from Valgrind:

==6816== Thread 7:
==6816== Invalid read of size 4
==6816==    at 0x4C5CDD0: AlarmManager::watchdogThread() (alarm.cpp:1105)
==6816==    by 0x4C5CF28: WatchdogThreadStarter(void*) (alarm.cpp:204)
==6816==    by 0x313D407850: start_thread (in /lib64/libpthread-2.12.so)
==6816==    by 0x313CCE811C: clone (in /lib64/libc-2.12.so)
==6816==  Address 0x2c8c57bc is 52,620 bytes inside a block of size 243,984 free'd
==6816==    at 0x4A0610F: realloc (vg_replace_malloc.c:525)
==6816==    by 0x4C5EF8A: AlarmManager::newAlarm(char*, char*, int, int, unsigned int, unsigned int, Event*, unsigned int) (alarm.cpp:359)
==6816==    by 0x4C81704: EPRule::generateAlarm(Event*) (epp.cpp:531)
==6816==    by 0x4C81D4A: EPRule::processEvent(Event*) (epp.cpp:468)
==6816==    by 0x4C81F4B: EventPolicy::processEvent(Event*) (epp.cpp:814)
==6816==    by 0x4C862C9: EventProcessor(void*) (evproc.cpp:225)
==6816==    by 0x313D407850: start_thread (in /lib64/libpthread-2.12.so)
==6816==    by 0x313CCE811C: clone (in /lib64/libc-2.12.so)


Hello,
Unfortunately it crashed again. Same error
I'm running it now under valgrind, but that takes a lot of resources. I run it with default settings
Did not have a crash yet

it looks like this issue is coming from pre 1.2.16 alarms.
We had about 6 old alarms with a sticky acknowledge set on it.
I was looking at the alarm.cpp code to see if I could find anything.
I saw that piece of code was added last release.
So, I deleted(terminated) all of the old alarms and we did not have a crash yet.
We have had new alarms coming in with no error.

regards,
Edward

of course we can.
any options you want to give valgrind to check?

Hi Victor,

Did you have a chance to look at the trace?
it keeps crashing several times a day....

Some additional info:
in the nxmc log I have this info:

!ENTRY org.netxms.ui.eclipse.console 4 0 2014-08-27 10:57:26.046
!MESSAGE Exception in ConsoleJob
!STACK 0
org.netxms.client.NXCException: Input/Output error
at org.netxms.client.NXCSession.renderReport(NXCSession.java:7474)
at org.netxms.ui.eclipse.reporter.widgets.ReportExecutionForm$11.runInternal(ReportExecutionForm.java:496)
at org.netxms.ui.eclipse.jobs.ConsoleJob.run(ConsoleJob.java:85)
at org.eclipse.core.internal.jobs.Worker.run(Worker.java:54)

!ENTRY org.netxms.ui.eclipse.reporter 4 16 2014-08-27 10:57:26.051
!MESSAGE Cannot render report IP Inventory (job ID 7c3839e0-9543-4f29-881d-ee7313b57b61): Input/Output error


in my temp folder there is a file created: nxc323266819402099091data
If I rename this file to .pdf I can open the report

Hope this will help finding the issue.
Regards,

Edward

the iligalArgumentExeption is coming from the Arrays.copyOfRange
Arrays.copyOfRange should have start and end instead of position and length.
After changing this to the following code this works.


private void sendFileData(final long requestId, final byte[] data) throws IOException
{
NXCPMessage msg = new NXCPMessage(NXCPCodes.CMD_FILE_DATA, requestId);
msg.setBinaryMessage(true);

for(int start = 0; start < data.length; start += 16384)
{
int end = start+16384;
if(end>data.length) end=data.length;
msg.setBinaryData(Arrays.copyOfRange(data, start, end));
sendMessage(msg);
}
}




After deploying it I can execute a report and the results are shown in the console.
Now when rendering I do not get any error, but the console is executing the job for a very long time.
After that I get the following error in the console:
Cannot render report IP Inventory (job ID 5277760b-41b4-4520-ac0a-95144c78367b): Input/Output error

here it is:


Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffef576700 (LWP 19648)]
0x00007ffff7d04da7 in AlarmManager::watchdogThread (this=0x7ffff7ff7740) at alarm.cpp:1090
1090                            if ((m_pAlarmList.dwTimeout > 0) &&
Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.80.el6_3.6.x86_64 keyutils-libs-1.4-4.el6.x86_64 krb5-libs-1.10.3-10.el6.x86_64 libcom_err-1.41.12-14.el6.x86_64 libgcc-4.4.6-4.el6.x86_64 libselinux-2.0.94-5.3.el6.x86_64 libstdc++-4.4.6-4.el6.x86_64 mysql-libs-5.1.66-2.el6_3.x86_64 nss-softokn-freebl-3.12.9-11.el6.x86_64 openssl-1.0.0-27.el6.x86_64 zlib-1.2.3-29.el6.x86_64
(gdb) bt
#0  0x00007ffff7d04da7 in AlarmManager::watchdogThread (this=0x7ffff7ff7740) at alarm.cpp:1090
#1  0x00007ffff7d04f29 in WatchdogThreadStarter (pArg=<value optimized out>) at alarm.cpp:204
#2  0x000000313d407851 in start_thread () from /lib64/libpthread.so.0
#3  0x000000313cce811d in clone () from /lib64/libc.so.6


regards,
Edward

Hi Alex,

There seems a problem in the table script with the column reportId.
As I can see from the source that is is an hybernate uuid binary column I think it should be binary(16) instead of binary(255).
I have tried it (recreated the table) and now it is showing up in the console.

Only thing is that I can't render the results. I'm getting the following error in the nxreporting log:
Exception in thread "Thread-9" java.lang.IllegalArgumentException: 32768 > 16384
        at java.util.Arrays.copyOfRange(Arrays.java:2549)
        at com.radensolutions.reporting.infrastructure.impl.TcpConnector$SessionWorker.sendFileData(TcpConnector.java:232)
        at com.radensolutions.reporting.infrastructure.impl.TcpConnector$SessionWorker.run(TcpConnector.java:174)
        at java.lang.Thread.run(Thread.java:722)

I am already running it under gdb, but it did not crash yet...
As soon as I have a trace I will send it to you.

regards,
Edward