NetXMS Support Forum

Dear support team,
i observing strange behavior in times when netxms service starting, maibe service trying to cache some data from database, but this takes a lot of memory and if memory goes empty system begins to swap some process out and in this time netxms process release a lot of memory and then starts and runs ok...its strange.

My environment is debian 10, database is mariadb.

Server version 3.8.267-1

Monitored nodes 297
Collectible DCIs  22487
Active alarms 180012

Mariadb has 60Gb of data.

System has 2vCPU 10GB RAM, usually memory is at 75 percent and everything runs ok, I see problems only at startup.

You can see a few minutes delay in startup log, in this time memory usage goes to swap, and then drops to normal.


2021.04.27 16:14:32.298 *I* [logger             ] Log file opened (rotation policy 2, max size 16777216)
2021.04.27 16:14:32.298 *D* [                   ] LIB directory set to /usr/lib/x86_64-linux-gnu/netxms
2021.04.27 16:14:32.298 *I* [                   ] Code page set to UTF8
2021.04.27 16:14:32.301 *I* [                   ] Platform subagent "/usr/lib/x86_64-linux-gnu/netxms/linux.nsm" successfully loaded
2021.04.27 16:14:32.302 *I* [db.drv             ] Database driver "mariadb.ddr" loaded and initialized successfully
2021.04.27 16:14:32.303 *D* [                   ] Local administration interface listener started
2021.04.27 16:14:32.303 *D* [                   ] Successfully connected to database [email protected]
2021.04.27 16:14:32.312 *D* [db.cpool           ] Database Connection Pool initialized
2021.04.27 16:14:32.312 *D* [db.cpool           ] Database Connection Pool maintenance thread started
2021.04.27 16:14:32.312 *I* [                   ] Server ID 3007CF4CB43E2255
2021.04.27 16:14:32.316 *D* [db.lock            ] Database lock set
2021.04.27 16:14:32.322 *I* [db.writer          ] DBWriter/Housekeeper interlock is OFF
2021.04.27 16:14:32.322 *D* [core               ] Data directory set to /var/lib/netxms
2021.04.27 16:14:32.323 *D* [                   ] Global configuration loaded
2021.04.27 16:14:32.325 *D* [crypto             ] Validating ciphers
2021.04.27 16:14:32.325 *D* [crypto             ]    AES-256 enabled
2021.04.27 16:14:32.325 *D* [crypto             ]    Blowfish-256 enabled
2021.04.27 16:14:32.325 *D* [crypto             ]    IDEA disabled (config)
2021.04.27 16:14:32.325 *D* [crypto             ]    3DES enabled
2021.04.27 16:14:32.325 *D* [crypto             ]    AES-128 enabled
2021.04.27 16:14:32.325 *D* [crypto             ]    Blowfish-128 enabled
2021.04.27 16:14:32.325 *I* [crypto             ] Crypto library initialized (OpenSSL 1.1.1d  10 Sep 2019)
2021.04.27 16:14:32.325 *I* [crypto.cert        ] Server certificate not set
2021.04.27 16:14:32.334 *D* [threads.pool       ] Thread pool MAIN initialized (min=8, max=256)
2021.04.27 16:14:32.334 *D* [threads.pool       ] Thread pool AGENT initialized (min=4, max=256)
2021.04.27 16:14:32.363 *D* [watchdog           ] Watchdog thread started
2021.04.27 16:14:32.364 *D* [db.writer          ] Using 1 DCI data write queues
2021.04.27 16:14:32.364 *D* [db.writer          ] Raw DCI data flush interval is 30 seconds
2021.04.27 16:14:32.364 *D* [threads.pool       ] Thread pool NPE initialized (min=1, max=1024)
2021.04.27 16:14:32.364 *D* [                   ] SMTP mailer thread started
2021.04.27 16:14:32.365 *D* [                   ] SQLite version 3.27.2
2021.04.27 16:14:32.365 *I* [db.drv             ] Database driver "sqlite.ddr" loaded and initialized successfully
2021.04.27 16:14:34.605 *D* [                   ] SQLite version 3.27.2
2021.04.27 16:14:34.605 *I* [db.drv             ] Database driver "sqlite.ddr" loaded and initialized successfully
here something eats my memory few minutes
2021.04.27 16:17:06.738 *D* [nc                 ] Loading notification channel drivers from /usr/lib/x86_64-linux-gnu/netxms/ncdrv
2021.04.27 16:17:06.906 *D* [nc                 ] 18 notification channel drivers loaded
2021.04.27 16:17:06.906 *D* [nc                 ] 0 notification channels added
2021.04.27 16:17:06.907 *D* [ndd                ] Loading network device drivers from /usr/lib/x86_64-linux-gnu/netxms/ndd
2021.04.27 16:17:06.908 *I* [ndd                ] Network device driver CAMBIUM-CNPILOT loaded successfully
2021.04.27 16:17:06.908 *I* [ndd                ] Network device driver CAMBIUM-EPMP loaded successfully
2021.04.27 16:17:06.924 *I* [ndd                ] Network device driver SAF-INTEGRA-B loaded successfully
2021.04.27 16:17:06.925 *I* [ndd                ] Network device driver SYMBOL-WS loaded successfully
2021.04.27 16:17:06.925 *I* [ndd                ] Network device driver WESTERSTRAND loaded successfully
2021.04.27 16:17:06.926 *I* [ndd                ] Network device driver UBNT loaded successfully
2021.04.27 16:17:06.927 *I* [ndd                ] Network device driver AT loaded successfully
2021.04.27 16:17:06.928 *I* [ndd                ] Network device driver CATALYST-GENERIC loaded successfully
2021.04.27 16:17:06.928 *I* [ndd                ] Network device driver CATALYST-2900XL loaded successfully
2021.04.27 16:17:06.928 *I* [ndd                ] Network device driver CISCO-ESW loaded successfully
2021.04.27 16:17:06.928 *I* [ndd                ] Network device driver CISCO-NEXUS loaded successfully
2021.04.27 16:17:06.928 *I* [ndd                ] Network device driver CISCO-SB loaded successfully
2021.04.27 16:17:06.928 *I* [ndd                ] Network device driver CISCO-WLC loaded successfully
2021.04.27 16:17:06.928 *I* [ndd                ] Network device driver CISCO-GENERIC loaded successfully
2021.04.27 16:17:06.928 *I* [ndd                ] Network device driver OPTIX loaded successfully
....
2021.04.27 16:17:12.453 *D* [                   ] 0 startup scripts processed
2021.04.27 16:17:12.453 *D* [                   ] Server initialization completed in 160155 milliseconds
2021.04.27 16:17:12.454 *D* [statcoll           ] Server statistic collector thread started
2021.04.27 16:17:12.454 *I* [                   ] NetXMS Server started



After server started all working ok.
HTop images attached from time after Netxms server is up and responding to console.

Any idea what can cause this peak ?

Hi!

Try to add the following option to netxmsd.conf:

DBCacheConfigurationTables = no

Best regards,
Victor

Quote from: Victor Kirhenshtein on April 28, 2021, 08:47:51 AM
Hi!

Try to add the following option to netxmsd.conf:

DBCacheConfigurationTables = no

Best regards,
Victor

Thank You Victor, after configuration change memory footprint is much better now, memory peak is much smaller and server have enough RAM.
No swaping occurs at start.