NetXMS Support Forum

Hi all!

We just published new patch release for version 3.8 (3.8.267). Only server and web API are affected. Changes since release 3.8.262:

- Fixed server crash during scheduled task processing
- Fixed issues:
        NX-2009 (Add notification channel driver "Shell command")
        NX-2035 (Instance discovery using ICMP.Targets list crashes server)
        NX-2037 (Save IP address of client connecting via WebAPI to audit log)

Best regards,
Victor

Hi,

most likely you have hit RAM limit when allow thread pool to grow to 4000 threads. Could also be other memory-related issue (memory leak in server, slow database causing DB writer queues to grow, etc. Do you have pre-crash data on number of threads, queues, and memory consumption?

Also, 4000 nodes is not that much, it is quite strange that 500 threads is not enough. Such high load on poller pool indicates that there are lot of polls taking long time to complete.  Could it be that you have many nodes that timeout when polled?

Best regards,
Victor

Hi all!

New patch release for version 3.8 - 3.8.262 - is just published. Changes since version 3.8.250:

- Fixed server crash when accessing NXSL class metadata
- Fixed agent crash when parsing log files
- Fixed file handle leak in log parser subagent
- DCI Editor filter now search by description, parameter, and DCI ID
- Long polling can be disabled in Telegram driver
- Fixed Log Parser Policy save with empty agent action field
- Fixed issues:
        NX-1770 (Add ability to filter by DCI name in DCI list)
        NX-2033 (Incorrect interface name displayed in MAC address table for Mikrotik routers)
        NX-2036 (Filter in DCI editor should search both description and parameter)

Best regards,
Victor

Hi,

we likely found the issue. New patch release will be available in a next few days, please re-test this issue then.

Best regards,
Victor

В списке файлов очень много раз встречается /var/log/auth.log.1, /var/log/mail.log.1, и /var/log/apcupsd.events. На агенте не настроен мониторинг этих файлов через logwatch?

Hi,

that one looks like PostgreSQL problem - server drops connection for some reason. Check in Postgres or system log if it was restarted.

Best regards,
Victor

Hi,

looks like bug in nxdbmgr. Somebody from dev team will check.

Best regards,
Victor

Default path for source install is /usr/local/bin and /use/local/lib. It looks like old version is in your path, so command

which nxdbmgr

should show it's location.

Best regards,
Victor

Hi,

looks like you have older version of NetXMS (3.4.178) somewhere in the path as well. Make sure that you don't have old version installed from sources for example.

Best regards,
Victor

Hi,

looks like call to Cipher.getInstance("AES/CBC/PKCS5Padding") sometimes is very slow. To test it, please create the following Java file:


import javax.crypto.Cipher;

class CipherTest
{
   public static final void main(String[] args)
   {
      long start = System.currentTimeMillis();
      try
      {
         Cipher.getInstance("AES/CBC/PKCS5Padding");
      }
      catch(Exception e)
      {
         e.printStackTrace();
      }
      long elapsed = System.currentTimeMillis() - start;
      System.out.println("Elapsed time " + elapsed + " milliseconds");
   }
}


call it CipherTest.java, then compile as

javac CipherTest.java


and run as

java CipherTest


It should print elapsed time for cipher initialization. On my machine it is around 60 milliseconds. If it will take many seconds during some run then this is problem with local JVM (likely with entropy collection, maybe with /dev/random or /dev/urandom).

Best regards,
Victor

Там где агент падает - поставьте пожалуйста еще -dbg пакеты (netxms-agent-dbg, netxms-dbdrv-sqlite-dbg, и т.д.) - тогда дамп будет сильно более полезным.

По поводу дескрипторов - пришлите пожалуйста вывод lsof -p для процесса агента.

Hi,

are those nodes in default zone or in remote zones? If yes and zone proxy cannot resolve host name then server will not attempt to resolve it directly unless server configuration parameter Objects.Nodes.FallbackToLocalResolver set to true. This change was introduced in version 3.5.

Best regards,
Victor

Hi!

Looks like Java side issue. Could you please try to do the following:

1. Download nxshell-3.8.250.jar and the following files:
https://repo1.maven.org/maven2/ch/qos/logback/logback-core/1.2.3/logback-core-1.2.3.jar
https://repo1.maven.org/maven2/ch/qos/logback/logback-classic/1.2.3/logback-classic-1.2.3.jar
into same directory.
2. Create file logback.xml in the same directory with the following content:

<configuration>
   <appender name="Console" class="ch.qos.logback.core.ConsoleAppender">
      <layout class="ch.qos.logback.classic.PatternLayout">
         <Pattern>%d{HH:mm:ss.SSS} [%t] %-5level %logger{36} - %msg%n</Pattern>
      </layout>
   </appender>
   <root level="debug">
      <appender-ref ref="Console"/>
   </root>
</configuration>

3. Run nxshell as following:

java -Dnetxms.server=127.0.0.1 -Dnetxms.login=admin -Dnetxms.password=your_password -cp logback-classic-1.2.3.jar:logback-core-1.2.3.jar:.:nxshell-3.8.250.jar org.netxms.Shell

until it hangs and send me console output.

Best regards,
Victor

Hi,

when you do search for IP address, server will find node with that IP address, and show you MAC address of interface where this IP address is configured. You get specific MAC because you are looking for specific IP. When looking for a node by it's ID you have to somehow decide what interface to use. You can use primary IP address for example to determine what interface and MAC address to show.

Best regards,
Victor

Hi,

please verify with command show flags in server debug console if flag AF_RESOLVE_IP_FOR_EACH_STATUS_POLL is actually set.

Best regards,
Victor