NetXMS Support Forum

Hi,

most likely you have hit RAM limit when allow thread pool to grow to 4000 threads. Could also be other memory-related issue (memory leak in server, slow database causing DB writer queues to grow, etc. Do you have pre-crash data on number of threads, queues, and memory consumption?

Also, 4000 nodes is not that much, it is quite strange that 500 threads is not enough. Such high load on poller pool indicates that there are lot of polls taking long time to complete.  Could it be that you have many nodes that timeout when polled?

Best regards,
Victor

Hi all!

New patch release for version 3.8 - 3.8.262 - is just published. Changes since version 3.8.250:

- Fixed server crash when accessing NXSL class metadata
- Fixed agent crash when parsing log files
- Fixed file handle leak in log parser subagent
- DCI Editor filter now search by description, parameter, and DCI ID
- Long polling can be disabled in Telegram driver
- Fixed Log Parser Policy save with empty agent action field
- Fixed issues:
        NX-1770 (Add ability to filter by DCI name in DCI list)
        NX-2033 (Incorrect interface name displayed in MAC address table for Mikrotik routers)
        NX-2036 (Filter in DCI editor should search both description and parameter)

Best regards,
Victor

Hi,

we likely found the issue. New patch release will be available in a next few days, please re-test this issue then.

Best regards,
Victor

В списке файлов очень много раз встречается /var/log/auth.log.1, /var/log/mail.log.1, и /var/log/apcupsd.events. На агенте не настроен мониторинг этих файлов через logwatch?

Hi,

that one looks like PostgreSQL problem - server drops connection for some reason. Check in Postgres or system log if it was restarted.

Best regards,
Victor

Hi,

looks like bug in nxdbmgr. Somebody from dev team will check.

Best regards,
Victor

Default path for source install is /usr/local/bin and /use/local/lib. It looks like old version is in your path, so command

which nxdbmgr

should show it's location.

Best regards,
Victor

Hi,

looks like you have older version of NetXMS (3.4.178) somewhere in the path as well. Make sure that you don't have old version installed from sources for example.

Best regards,
Victor

Hi,

looks like call to Cipher.getInstance("AES/CBC/PKCS5Padding") sometimes is very slow. To test it, please create the following Java file:


import javax.crypto.Cipher;

class CipherTest
{
   public static final void main(String[] args)
   {
      long start = System.currentTimeMillis();
      try
      {
         Cipher.getInstance("AES/CBC/PKCS5Padding");
      }
      catch(Exception e)
      {
         e.printStackTrace();
      }
      long elapsed = System.currentTimeMillis() - start;
      System.out.println("Elapsed time " + elapsed + " milliseconds");
   }
}


call it CipherTest.java, then compile as

javac CipherTest.java


and run as

java CipherTest


It should print elapsed time for cipher initialization. On my machine it is around 60 milliseconds. If it will take many seconds during some run then this is problem with local JVM (likely with entropy collection, maybe with /dev/random or /dev/urandom).

Best regards,
Victor

Там где агент падает - поставьте пожалуйста еще -dbg пакеты (netxms-agent-dbg, netxms-dbdrv-sqlite-dbg, и т.д.) - тогда дамп будет сильно более полезным.

По поводу дескрипторов - пришлите пожалуйста вывод lsof -p для процесса агента.

Hi,

are those nodes in default zone or in remote zones? If yes and zone proxy cannot resolve host name then server will not attempt to resolve it directly unless server configuration parameter Objects.Nodes.FallbackToLocalResolver set to true. This change was introduced in version 3.5.

Best regards,
Victor

Hi!

Looks like Java side issue. Could you please try to do the following:

1. Download nxshell-3.8.250.jar and the following files:
https://repo1.maven.org/maven2/ch/qos/logback/logback-core/1.2.3/logback-core-1.2.3.jar
https://repo1.maven.org/maven2/ch/qos/logback/logback-classic/1.2.3/logback-classic-1.2.3.jar
into same directory.
2. Create file logback.xml in the same directory with the following content:

<configuration>
   <appender name="Console" class="ch.qos.logback.core.ConsoleAppender">
      <layout class="ch.qos.logback.classic.PatternLayout">
         <Pattern>%d{HH:mm:ss.SSS} [%t] %-5level %logger{36} - %msg%n</Pattern>
      </layout>
   </appender>
   <root level="debug">
      <appender-ref ref="Console"/>
   </root>
</configuration>

3. Run nxshell as following:

java -Dnetxms.server=127.0.0.1 -Dnetxms.login=admin -Dnetxms.password=your_password -cp logback-classic-1.2.3.jar:logback-core-1.2.3.jar:.:nxshell-3.8.250.jar org.netxms.Shell

until it hangs and send me console output.

Best regards,
Victor

Hi,

when you do search for IP address, server will find node with that IP address, and show you MAC address of interface where this IP address is configured. You get specific MAC because you are looking for specific IP. When looking for a node by it's ID you have to somehow decide what interface to use. You can use primary IP address for example to determine what interface and MAC address to show.

Best regards,
Victor

Hi,

please verify with command show flags in server debug console if flag AF_RESOLVE_IP_FOR_EACH_STATUS_POLL is actually set.

Best regards,
Victor

Hi all!

We just published next patch release for NetXMS 3.8 - version 3.8.350 (3.8.251 for Linux agents). Changes since previous patch release:

- Added notification channel driver for SNMP traps
- Event processing macro %K (alarm key) also works in rules following one that triggered alarm
- Improved Data Collection Editor and Policy Editor usability
- DCI instance used for custom attribute lookup when expanding %{} macro
- Fixed sorting for "Object query" dashboard element
- Fixed memory leak in server related to "Object query" dashboard element
- Fixed issues:
        NX-2022 (Rack with custom height is incorrectly displayed)
        NX-2028 (Use instance name for custom attribute search when expanding %{} macro)
        NX-2029 (Line Chart General Time Period keeps reverting to "Back from now" when trying to use "Fixed time frame" option)
        NX-2031 (HTTP proxy parameter configured in nxmc properties has no effect)

Best regards,
Victor