Dear support team,
i observing strange behavior in times when netxms service starting, maibe service trying to cache some data from database, but this takes a lot of memory and if memory goes empty system begins to swap some process out and in this time netxms process release a lot of memory and then starts and runs ok...its strange.
My environment is debian 10, database is mariadb.
Server version 3.8.267-1
Monitored nodes 297
Collectible DCIs 22487
Active alarms 180012
Mariadb has 60Gb of data.
System has 2vCPU 10GB RAM, usually memory is at 75 percent and everything runs ok, I see problems only at startup.
You can see a few minutes delay in startup log, in this time memory usage goes to swap, and then drops to normal.
2021.04.27 16:14:32.298 *I* [logger ] Log file opened (rotation policy 2, max size 16777216)
2021.04.27 16:14:32.298 *D* [ ] LIB directory set to /usr/lib/x86_64-linux-gnu/netxms
2021.04.27 16:14:32.298 *I* [ ] Code page set to UTF8
2021.04.27 16:14:32.301 *I* [ ] Platform subagent "/usr/lib/x86_64-linux-gnu/netxms/linux.nsm" successfully loaded
2021.04.27 16:14:32.302 *I* [db.drv ] Database driver "mariadb.ddr" loaded and initialized successfully
2021.04.27 16:14:32.303 *D* [ ] Local administration interface listener started
2021.04.27 16:14:32.303 *D* [ ] Successfully connected to database [email protected]
2021.04.27 16:14:32.312 *D* [db.cpool ] Database Connection Pool initialized
2021.04.27 16:14:32.312 *D* [db.cpool ] Database Connection Pool maintenance thread started
2021.04.27 16:14:32.312 *I* [ ] Server ID 3007CF4CB43E2255
2021.04.27 16:14:32.316 *D* [db.lock ] Database lock set
2021.04.27 16:14:32.322 *I* [db.writer ] DBWriter/Housekeeper interlock is OFF
2021.04.27 16:14:32.322 *D* [core ] Data directory set to /var/lib/netxms
2021.04.27 16:14:32.323 *D* [ ] Global configuration loaded
2021.04.27 16:14:32.325 *D* [crypto ] Validating ciphers
2021.04.27 16:14:32.325 *D* [crypto ] AES-256 enabled
2021.04.27 16:14:32.325 *D* [crypto ] Blowfish-256 enabled
2021.04.27 16:14:32.325 *D* [crypto ] IDEA disabled (config)
2021.04.27 16:14:32.325 *D* [crypto ] 3DES enabled
2021.04.27 16:14:32.325 *D* [crypto ] AES-128 enabled
2021.04.27 16:14:32.325 *D* [crypto ] Blowfish-128 enabled
2021.04.27 16:14:32.325 *I* [crypto ] Crypto library initialized (OpenSSL 1.1.1d 10 Sep 2019)
2021.04.27 16:14:32.325 *I* [crypto.cert ] Server certificate not set
2021.04.27 16:14:32.334 *D* [threads.pool ] Thread pool MAIN initialized (min=8, max=256)
2021.04.27 16:14:32.334 *D* [threads.pool ] Thread pool AGENT initialized (min=4, max=256)
2021.04.27 16:14:32.363 *D* [watchdog ] Watchdog thread started
2021.04.27 16:14:32.364 *D* [db.writer ] Using 1 DCI data write queues
2021.04.27 16:14:32.364 *D* [db.writer ] Raw DCI data flush interval is 30 seconds
2021.04.27 16:14:32.364 *D* [threads.pool ] Thread pool NPE initialized (min=1, max=1024)
2021.04.27 16:14:32.364 *D* [ ] SMTP mailer thread started
2021.04.27 16:14:32.365 *D* [ ] SQLite version 3.27.2
2021.04.27 16:14:32.365 *I* [db.drv ] Database driver "sqlite.ddr" loaded and initialized successfully
2021.04.27 16:14:34.605 *D* [ ] SQLite version 3.27.2
2021.04.27 16:14:34.605 *I* [db.drv ] Database driver "sqlite.ddr" loaded and initialized successfully
here something eats my memory few minutes
2021.04.27 16:17:06.738 *D* [nc ] Loading notification channel drivers from /usr/lib/x86_64-linux-gnu/netxms/ncdrv
2021.04.27 16:17:06.906 *D* [nc ] 18 notification channel drivers loaded
2021.04.27 16:17:06.906 *D* [nc ] 0 notification channels added
2021.04.27 16:17:06.907 *D* [ndd ] Loading network device drivers from /usr/lib/x86_64-linux-gnu/netxms/ndd
2021.04.27 16:17:06.908 *I* [ndd ] Network device driver CAMBIUM-CNPILOT loaded successfully
2021.04.27 16:17:06.908 *I* [ndd ] Network device driver CAMBIUM-EPMP loaded successfully
2021.04.27 16:17:06.924 *I* [ndd ] Network device driver SAF-INTEGRA-B loaded successfully
2021.04.27 16:17:06.925 *I* [ndd ] Network device driver SYMBOL-WS loaded successfully
2021.04.27 16:17:06.925 *I* [ndd ] Network device driver WESTERSTRAND loaded successfully
2021.04.27 16:17:06.926 *I* [ndd ] Network device driver UBNT loaded successfully
2021.04.27 16:17:06.927 *I* [ndd ] Network device driver AT loaded successfully
2021.04.27 16:17:06.928 *I* [ndd ] Network device driver CATALYST-GENERIC loaded successfully
2021.04.27 16:17:06.928 *I* [ndd ] Network device driver CATALYST-2900XL loaded successfully
2021.04.27 16:17:06.928 *I* [ndd ] Network device driver CISCO-ESW loaded successfully
2021.04.27 16:17:06.928 *I* [ndd ] Network device driver CISCO-NEXUS loaded successfully
2021.04.27 16:17:06.928 *I* [ndd ] Network device driver CISCO-SB loaded successfully
2021.04.27 16:17:06.928 *I* [ndd ] Network device driver CISCO-WLC loaded successfully
2021.04.27 16:17:06.928 *I* [ndd ] Network device driver CISCO-GENERIC loaded successfully
2021.04.27 16:17:06.928 *I* [ndd ] Network device driver OPTIX loaded successfully
....
2021.04.27 16:17:12.453 *D* [ ] 0 startup scripts processed
2021.04.27 16:17:12.453 *D* [ ] Server initialization completed in 160155 milliseconds
2021.04.27 16:17:12.454 *D* [statcoll ] Server statistic collector thread started
2021.04.27 16:17:12.454 *I* [ ] NetXMS Server started
After server started all working ok.
HTop images attached from time after Netxms server is up and responding to console.
Any idea what can cause this peak ?
Quote from: Victor Kirhenshtein on April 28, 2021, 08:47:51 AM
Hi!
Try to add the following option to netxmsd.conf:
DBCacheConfigurationTables = no
Best regards,
Victor
Thank You Victor, after configuration change memory footprint is much better now, memory peak is much smaller and server have enough RAM.
No swaping occurs at start.