NetXMS Support Forum

Hello,

Problem description:
- tuneled agent crashes after 0s..45min after start (repeatedly)

Environment description:
- LXC container running (bookworm) on Proxmox running (bookworm) all up2date @04.16.2024
- kernel 6.5.11-7-pve
- DEB Package: netxms-agent, Version: 4.5.6-1

--------------------------------
:~# gdb /usr/bin/nxagentd
GNU gdb (Debian 13.1-3) 13.1
Copyright (C) 2023 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/bin/nxagentd...
Reading symbols from /usr/lib/debug/.build-id/95/2e3f9104195351fe4deab7db75a2be62a439f2.debug...
(gdb) set args -f -D 4
--------------------------------
--------------------------------
(gdb) run
Starting program: /usr/bin/nxagentd -f -D 4
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Error reading additional configuration files from "/etc/nxagentd.conf.d"
2024.04.16 07:20:04.009 *I* [logger             ] Log rotation policy set to 2 (size=16777216, count=4)
2024.04.16 07:20:04.010 *I* [startup            ] Core agent version 4.5.6 (build tag 4.5-213-g58d2aa16a8)
2024.04.16 07:20:04.010 *I* [config             ] Additional configuration files was loaded from /etc/nxagentd.conf.d
2024.04.16 07:20:04.010 *I* [logger             ] Debug level set to 4
2024.04.16 07:20:04.010 *I* [config             ] Main configuration file: /etc/nxagentd.conf
2024.04.16 07:20:04.010 *I* [config             ] Configuration tree:
2024.04.16 07:20:04.010 *I* [config             ] config
2024.04.16 07:20:04.010 *I* [config             ]  +- CORE
2024.04.16 07:20:04.010 *I* [config             ]      +- ServerConnection
2024.04.16 07:20:04.010 *I* [config             ]           value: 172.16.xx.xx
2024.04.16 07:20:04.010 *I* [startup            ] Data directory: /var/lib/netxms
2024.04.16 07:20:04.010 *I* [startup            ] File store: /tmp
2024.04.16 07:20:04.010 *D* [startup            ] Effective user ID 0
2024.04.16 07:20:04.010 *D* [startup            ] Effective group ID 0
2024.04.16 07:20:04.010 *D* [startup            ] Configuration policy directory: /var/lib/netxms/config_ap/
2024.04.16 07:20:04.010 *D* [startup            ] Log parser policy directory: /var/lib/netxms/logparser_ap/
2024.04.16 07:20:04.010 *D* [startup            ] User agent policy directory: /var/lib/netxms/useragent_ap/
2024.04.16 07:20:04.011 *D* [startup            ] Certificate directory: /var/lib/netxms/certificates/
2024.04.16 07:20:04.011 *D* [startup            ] CRL directory: /var/lib/netxms/crl/
2024.04.16 07:20:04.014 *D* [crypto             ] Validating ciphers
2024.04.16 07:20:04.014 *D* [crypto             ]    AES-256 enabled
2024.04.16 07:20:04.014 *D* [crypto             ]    Blowfish-256 disabled (validation failed)
2024.04.16 07:20:04.014 *D* [crypto             ]    IDEA disabled (config)
2024.04.16 07:20:04.014 *D* [crypto             ]    3DES enabled
2024.04.16 07:20:04.014 *D* [crypto             ]    AES-128 enabled
2024.04.16 07:20:04.014 *D* [crypto             ]    Blowfish-128 disabled (validation failed)
2024.04.16 07:20:04.014 *I* [crypto             ] Crypto library initialized (OpenSSL 3.0.11 19 Sep 2023)
[New Thread 0x7ffff649b6c0 (LWP 33983)]
[New Thread 0x7ffff5bff6c0 (LWP 33984)]
[New Thread 0x7ffff5afe6c0 (LWP 33985)]
2024.04.16 07:20:04.016 *D* [threads.pool       ] Thread pool TIMER initialized (min=2, max=16)
2024.04.16 07:20:04.016 *D* [startup            ] Subagent API initialized
2024.04.16 07:20:04.035 *D* [db.drv.sqlite      ] SQLite version 3.40.1
2024.04.16 07:20:04.035 *I* [db.drv             ] Database driver "sqlite.ddr" loaded and initialized successfully
2024.04.16 07:20:04.036 *D* [db.conn            ] New DB connection opened: handle=0x7ffff61641c0
2024.04.16 07:20:04.038 *D* [db.query           ] DB Library: global long running query threshold set to 250
2024.04.16 07:20:04.045 *I* [db.agent           ] Local database opened successfully
2024.04.16 07:20:04.046 *I* [startup            ] Agent ID is 6334d94b-6ab3-46ba-8515-891f843f5907
2024.04.16 07:20:04.047 *I* [startup            ] Local host name is "d001-spooler01.local"
2024.04.16 07:20:04.047 *I* [startup            ] Using system name "d001-spooler01"
2024.04.16 07:20:04.047 *D* [comm               ] Maximum number of sessions set to 32
[New Thread 0x7ffff645a6c0 (LWP 33986)]
[New Thread 0x7ffff561e6c0 (LWP 33987)]
[New Thread 0x7ffff551d6c0 (LWP 33988)]
[New Thread 0x7ffff541c6c0 (LWP 33989)]
[New Thread 0x7ffff531b6c0 (LWP 33990)]
[New Thread 0x7ffff521a6c0 (LWP 33991)]
[New Thread 0x7ffff51196c0 (LWP 33992)]
[New Thread 0x7ffff50186c0 (LWP 33993)]
[New Thread 0x7ffff4f176c0 (LWP 33994)]
2024.04.16 07:20:04.049 *D* [threads.pool       ] Thread pool COMM initialized (min=8, max=512)
[New Thread 0x7ffff4e166c0 (LWP 33995)]
[New Thread 0x7ffff4dd56c0 (LWP 33996)]
[New Thread 0x7ffff4cd46c0 (LWP 33997)]
[New Thread 0x7ffff4bd36c0 (LWP 33998)]
[New Thread 0x7ffff4ad26c0 (LWP 33999)]
2024.04.16 07:20:04.049 *D* [threads.pool       ] Thread pool WEBSVC initialized (min=4, max=64)
2024.04.16 07:20:04.050 *D* [tunnel             ] Added server tunnel 172.16.xx.xx
2024.04.16 07:20:04.050 *I* [startup            ] Arbitrary command execution disabled
2024.04.16 07:20:04.064 *D* [smbios             ] BIOS read failed
[New Thread 0x7ffff49256c0 (LWP 34001)]
2024.04.16 07:20:05.084 *D* [linux              ] Using /sys/block to distinguish devices from partitions
2024.04.16 07:20:05.084 *D* [linux              ] CPU usage collector thread started
[New Thread 0x7ffff48246c0 (LWP 34002)]
[New Thread 0x7ffff47236c0 (LWP 34003)]
2024.04.16 07:20:05.089 *I* [subagents          ] Subagent "Linux" (/usr/lib/x86_64-linux-gnu/netxms/linux.nsm) loaded successfully (version 4.5.6)
2024.04.16 07:20:05.089 *I* [startup            ] System hardware ID is unknown
2024.04.16 07:20:05.090 *I* [linux              ] Unable to parse /proc/drbd, DRBD data collector will not start
2024.04.16 07:20:05.090 *D* [linux              ] ParseIoStat(): new device added (name=loop0 isRealDevice=1)
2024.04.16 07:20:05.090 *D* [linux              ] ParseIoStat(): new device added (name=loop1 isRealDevice=1)
2024.04.16 07:20:05.090 *D* [linux              ] ParseIoStat(): new device added (name=loop2 isRealDevice=1)
2024.04.16 07:20:05.091 *D* [linux              ] ParseIoStat(): new device added (name=loop3 isRealDevice=1)
2024.04.16 07:20:05.091 *D* [linux              ] ParseIoStat(): new device added (name=loop4 isRealDevice=1)
2024.04.16 07:20:05.091 *D* [linux              ] ParseIoStat(): new device added (name=loop5 isRealDevice=1)
2024.04.16 07:20:05.091 *D* [linux              ] ParseIoStat(): new device added (name=loop6 isRealDevice=1)
2024.04.16 07:20:05.092 *D* [linux              ] ParseIoStat(): new device added (name=loop7 isRealDevice=1)
2024.04.16 07:20:05.092 *D* [linux              ] ParseIoStat(): new device added (name=nvme0n1 isRealDevice=1)
2024.04.16 07:20:05.092 *D* [linux              ] ParseIoStat(): new device added (name=nvme0n1p1 isRealDevice=0)
2024.04.16 07:20:05.092 *D* [linux              ] ParseIoStat(): new device added (name=nvme0n1p2 isRealDevice=0)
2024.04.16 07:20:05.092 *D* [linux              ] ParseIoStat(): new device added (name=nvme0n1p3 isRealDevice=0)
2024.04.16 07:20:05.093 *D* [linux              ] ParseIoStat(): new device added (name=nvme0n1p4 isRealDevice=0)
2024.04.16 07:20:05.093 *D* [linux              ] ParseIoStat(): new device added (name=nvme0n1p128 isRealDevice=0)
2024.04.16 07:20:05.093 *D* [linux              ] ParseIoStat(): new device added (name=zd0 isRealDevice=1)
2024.04.16 07:20:05.093 *D* [linux              ] ParseIoStat(): new device added (name=zd16 isRealDevice=1)
2024.04.16 07:20:05.093 *D* [linux              ] ParseIoStat(): new device added (name=zd16p1 isRealDevice=0)
2024.04.16 07:20:05.094 *D* [linux              ] ParseIoStat(): new device added (name=zd16p14 isRealDevice=0)
2024.04.16 07:20:05.094 *D* [linux              ] ParseIoStat(): new device added (name=zd16p15 isRealDevice=0)
2024.04.16 07:20:05.094 *D* [linux              ] ParseIoStat(): new device added (name=zd32 isRealDevice=1)
2024.04.16 07:20:05.094 *D* [linux              ] ParseIoStat(): new device added (name=zd48 isRealDevice=1)
2024.04.16 07:20:05.095 *D* [linux              ] ParseIoStat(): new device added (name=zd64 isRealDevice=1)
2024.04.16 07:20:05.095 *D* [linux              ] ParseIoStat(): new device added (name=zd80 isRealDevice=1)
2024.04.16 07:20:05.095 *D* [linux              ] ParseIoStat(): new device added (name=zd96 isRealDevice=1)
2024.04.16 07:20:05.095 *D* [linux              ] ParseIoStat(): new device added (name=zd112 isRealDevice=1)
2024.04.16 07:20:05.096 *D* [linux              ] ParseIoStat(): new device added (name=zd112p1 isRealDevice=0)
2024.04.16 07:20:05.096 *D* [linux              ] ParseIoStat(): new device added (name=zd112p14 isRealDevice=0)
2024.04.16 07:20:05.096 *D* [linux              ] ParseIoStat(): new device added (name=zd112p15 isRealDevice=0)
2024.04.16 07:20:05.096 *D* [linux              ] ParseIoStat(): new device added (name=zd128 isRealDevice=1)
2024.04.16 07:20:05.096 *D* [linux              ] ParseIoStat(): new device added (name=zd144 isRealDevice=1)
2024.04.16 07:20:05.097 *D* [linux              ] ParseIoStat(): new device added (name=zd144p1 isRealDevice=0)
2024.04.16 07:20:05.097 *D* [linux              ] ParseIoStat(): new device added (name=zd144p14 isRealDevice=0)
2024.04.16 07:20:05.097 *D* [linux              ] ParseIoStat(): new device added (name=zd144p15 isRealDevice=0)
2024.04.16 07:20:05.097 *D* [linux              ] ParseIoStat(): new device added (name=zd160 isRealDevice=1)
2024.04.16 07:20:05.097 *D* [linux              ] ParseIoStat(): new device added (name=zd160p1 isRealDevice=0)
2024.04.16 07:20:05.097 *D* [linux              ] ParseIoStat(): new device added (name=zd160p2 isRealDevice=0)
2024.04.16 07:20:05.098 *D* [linux              ] ParseIoStat(): new device added (name=zd160p3 isRealDevice=0)
2024.04.16 07:20:05.098 *D* [linux              ] ParseIoStat(): new device added (name=zd160p4 isRealDevice=0)
2024.04.16 07:20:05.098 *D* [linux              ] ParseIoStat(): new device added (name=zd160p5 isRealDevice=0)
[Thread 0x7ffff47236c0 (LWP 34003) exited]
2024.04.16 07:20:05.099 *D* [linux              ] ParseIoStat(): new device added (name=zd160p6 isRealDevice=0)
2024.04.16 07:20:05.099 *D* [linux              ] ParseIoStat(): new device added (name=zd160p7 isRealDevice=0)
2024.04.16 07:20:05.099 *D* [linux              ] ParseIoStat(): new device added (name=zd160p8 isRealDevice=0)
[New Thread 0x7ffff39ff6c0 (LWP 34004)]
[New Thread 0x7ffff39be6c0 (LWP 34005)]
2024.04.16 07:20:06.091 *D* [threads.pool       ] Thread pool PROCEXEC initialized (min=1, max=16)
2024.04.16 07:20:06.091 *D* [notifications      ] StartNotificationProcessor: Loaded 1 servers
[New Thread 0x7ffff38bd6c0 (LWP 34006)]
2024.04.16 07:20:06.092 *D* [notifications      ] Notification processor started
2024.04.16 07:20:06.092 *D* [notifications      ] Notification housekeeper started
2024.04.16 07:20:06.092 *D* [notifications      ] Notification housekeeper execution completed
[New Thread 0x7ffff37bc6c0 (LWP 34007)]
[New Thread 0x7ffff377b6c0 (LWP 34008)]
[New Thread 0x7ffff31ff6c0 (LWP 34009)]
[New Thread 0x7ffff30fe6c0 (LWP 34010)]
[New Thread 0x7ffff2ffd6c0 (LWP 34011)]
2024.04.16 07:20:06.093 *D* [threads.pool       ] Thread pool DATACOLL initialized (min=4, max=64)
[New Thread 0x7ffff2efc6c0 (LWP 34012)]
2024.04.16 07:20:06.094 *D* [dc                 ] Data collection scheduler thread started
[New Thread 0x7ffff2dfb6c0 (LWP 34013)]
[New Thread 0x7ffff27ff6c0 (LWP 34014)]
[New Thread 0x7ffff26fe6c0 (LWP 34015)]
2024.04.16 07:20:06.094 *D* [dc                 ] Data sender thread started
[New Thread 0x7ffff25fd6c0 (LWP 34016)]
[New Thread 0x7ffff1dff6c0 (LWP 34017)]
[New Thread 0x7ffff1cfe6c0 (LWP 34018)]
2024.04.16 07:20:06.095 *D* [                   ] Session agent connector disabled
2024.04.16 07:20:06.096 *D* [comm.listener      ] SocketListener/ProxyHeartbeat: Trying to bind on 0.0.0.0:4700/udp
2024.04.16 07:20:06.096 *D* [comm.listener      ] SocketListener/ProxyHeartbeat: Trying to bind on [::]:4700/udp
2024.04.16 07:20:06.096 *D* [comm               ] Trying to bind on 0.0.0.0:4700
2024.04.16 07:20:06.096 *D* [                   ] Trying to bind on [::]:4700
2024.04.16 07:20:06.096 *I* [comm               ] Listening on socket 0.0.0.0:4700
2024.04.16 07:20:06.096 *I* [comm               ] Listening on socket :::4700
2024.04.16 07:20:06.096 *D* [dc                 ] Database writer thread started
2024.04.16 07:20:06.096 *D* [dc                 ] Data reconciliation thread started (block size 1024, timeout 60000 ms)
[New Thread 0x7ffff19fd6c0 (LWP 34019)]
[New Thread 0x7ffff18fc6c0 (LWP 34020)]
2024.04.16 07:20:06.107 *D* [                   ] NamedPipeListener(nxagentd.push): waiting for connection
2024.04.16 07:20:06.107 *D* [                   ] NamedPipeListener(nxagentd.events): waiting for connection
2024.04.16 07:20:06.107 *I* [comm               ] Control connector is disabled
[New Thread 0x7ffff17fb6c0 (LWP 34021)]
2024.04.16 07:20:06.108 *D* [tunnel             ] Tunnel manager started
2024.04.16 07:20:06.114 *D* [tunnel             ] 172.16.xx.xx: Certificate and private key loaded
2024.04.16 07:20:06.126 *D* [tunnel             ] 172.16.xx.xx: Server certificate subject is CN=d001-mgmt01
2024.04.16 07:20:06.127 *D* [tunnel             ] 172.16.xx.xx: Server certificate issuer is CN=d001
2024.04.16 07:20:06.127 *D* [tunnel             ] 172.16.xx.xx: Server certificate verification is disabled
2024.04.16 07:20:06.127 *D* [tunnel             ] 172.16.xx.xx: Server certificate pinning is disabled
[New Thread 0x7ffff16fa6c0 (LWP 34022)]
2024.04.16 07:20:06.171 *I* [                   ] Tunnel with 172.16.xx.xx established
2024.04.16 07:20:06.171 *D* [tunnel             ] 172.16.xx.xx: Tunnel is active
2024.04.16 07:20:07.109 *I* [filemon            ] Path list for file monitor is empty
2024.04.16 07:20:07.109 *I* [                   ] NetXMS Agent started
Agent running. Press Ctrl+C to shutdown.
2024.04.16 07:20:13.988 *D* [comm.cs.2          ] Session registered (control=false, master=false)
[New Thread 0x7ffff15f96c0 (LWP 34023)]
2024.04.16 07:20:13.989 *D* [comm.cs.2          ] Using protocol version 5
2024.04.16 07:20:13.994 *D* [comm.cs.2          ] Server capabilities: IPv6: yes; bulk reconciliation: yes; compression: yes
2024.04.16 07:20:13.994 *D* [comm.cs.2          ] Server ID set to 319612E4EC7036B6
2024.04.16 07:20:13.995 *D* [comm.cs.2          ] Server ID set to 319612E4EC7036B6
2024.04.16 07:20:14.005 *D* [dc                 ] 0 SNMP targets received from server 319612E4EC7036B6
2024.04.16 07:20:14.005 *D* [dc                 ] 0 proxies received from server 319612E4EC7036B6
2024.04.16 07:20:14.005 *D* [dc                 ] 0 data collection elements received from server 319612E4EC7036B6 (extended data: YES)
2024.04.16 07:20:14.005 *D* [dc                 ] Data collection for server 319612E4EC7036B6 reconfigured
2024.04.16 07:20:14.009 *D* [notifications      ] RegisterSessionForNotifications: starting background sync for server 319612E4EC7036B6
2024.04.16 07:20:14.009 *D* [notifications      ] Notification synchronization started
2024.04.16 07:20:14.009 *D* [notifications      ] Vacuum local database
2024.04.16 07:20:14.016 *D* [notifications      ] Notification synchronization finished
2024.04.16 07:48:09.800 *D* [comm.cs.3          ] Session registered (control=false, master=false)
[New Thread 0x7ffff11f86c0 (LWP 34029)]
2024.04.16 07:48:09.808 *D* [comm.cs.3          ] Using protocol version 5

Thread 42 "nxagentd" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffff11f86c0 (LWP 34029)]
0x00007ffff7a330a2 in ?? () from /lib/x86_64-linux-gnu/libcrypto.so.3
-------------------------------------------------
-------------------------------------------------
(gdb) bt
#0  0x00007ffff7a330a2 in ?? () from /lib/x86_64-linux-gnu/libcrypto.so.3
#1  0x00007ffff7a315af in OPENSSL_LH_doall_arg () from /lib/x86_64-linux-gnu/libcrypto.so.3
#2  0x00007ffff7a332ab in ?? () from /lib/x86_64-linux-gnu/libcrypto.so.3
#3  0x00007ffff7a24af6 in ?? () from /lib/x86_64-linux-gnu/libcrypto.so.3
#4  0x00007ffff7ec8fe0 in RSAPublicEncrypt (outlen=4096, padding=4, rsa=0x7ffff321e0e0, out=0x7ffff11ed990 "0\202\001\n\002\202\001\001", inlen=32, in=0x7ffff320b0a0) at ../../include/nxcrypto.h:211
#5  SetupEncryptionContext (msg=msg@entry=0x7ffff328b000, ppCtx=ppCtx@entry=0x7ffff11eea20, ppResponse=ppResponse@entry=0x7ffff11eea30, privateKey=privateKey@entry=0x0, nxcpVersion=<optimized out>) at ./src/libnetxms/crypto.cpp:474
#6  0x00005555555a8dfe in CommSession::readThread (this=0x7ffff1e61310) at ./src/agent/core/session.cpp:340
#7  0x00005555555aa9a2 in ThreadCreate_SharedPtr_Wrapper_0<CommSession> (arg=0x7ffff1e22080) at ../../../include/nms_threads.h:795
#8  0x00007ffff7357134 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#9  0x00007ffff73d77dc in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
(gdb)
-------------------------------------------------
thank you,

Br, Aleš

Hello,

I apologize for late reply. Excluding potential infrastructure issues took longer than anticipated.
After analysis of all crashes, issue was found to be associated with single debian/bullseye package
update:

old: lxcfs=4.0.12-pve1
new: lxcfs=5.0.3-pve1

Further analysis was not done.

Upgrade of package causes segmentation fault, general protection faults in multiple nxagentd linked libraries
that unfortunately manifest in multiple nxagentd failures, consequential local sqlite database corruption
and associated 2nd level of errors. We observed crashes in: libnxsqlite.so, libc-2.31.so, libnetxms.so.43...

Downgrading proxmox v7 lxcfs package to 4.0.12-pve1 resolved issues.

thank you for your help and apologies for any inconvenience caused.

Br, Aleš

Hi,

Thank you.

We will update agent and monitor systems after release. This error was only occurrence of failure that was
repeatable 8/10 cases. I did not want to report issues with agent from deb repository as we are noticing
multiple errors (segfaults) after v4.0.x -> v4.3.0 + debian (stretch,buster,bullseye) latest updates.
We are still unable to isolate/repeat most of them. (this is not critc, just did not want to throw everything
at once and clear any issues that could be due our infrastructure beforehand)

Br,

Aleš

Hello,

Problem description:
- agent crashes immediately after trying to vacuum local sqlite database

Environment description:
- LXC container running (buster) on Proxmox running (bullseye) all up2date @04.04.2023
- kernel v6.2.6
- agent nxagent-4.3.3-linux-x86_64-static

Etc:
- sqlite database was created by nxagent-4.3.3-linux-x86_64-static
- filesystem zfs

---
# /opt/netxms/bin/nxagentd -v
NetXMS Core Agent Version 4.3.3 Build 4.3-416-g5911d89 (NON-UNICODE)

# sha256sum /opt/netxms/bin/nxagentd
2c208fb7eb7d8b4a99babad8bf2a11717d666652d8a0a5edb78c1e9ff62f1568  /opt/netxms/bin/nxagentd

# gdb /opt/netxms/bin/nxagentd

GNU gdb (Debian 8.2.1-2+b3) 8.2.1
...
(gdb) run -f -D 9
Starting program: /opt/netxms/bin/nxagentd -f -D 9
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
...
2023.04.04 08:59:26.011 *I* [config            ] Main configuration file: /etc/nxagentd.conf
2023.04.04 08:59:26.011 *I* [config            ] Configuration tree:
2023.04.04 08:59:26.011 *I* [config            ] config
2023.04.04 08:59:26.011 *I* [config            ]  +- CORE
2023.04.04 08:59:26.011 *I* [config            ]      +- ServerConnection
2023.04.04 08:59:26.011 *I* [config            ]      |    value: <removed>
2023.04.04 08:59:26.011 *I* [config            ]      +- Servers
2023.04.04 08:59:26.011 *I* [config            ]      |    value: <removed>
2023.04.04 08:59:26.011 *I* [config            ]      +- ControlServers
2023.04.04 08:59:26.011 *I* [config            ]      |    value: <removed>
2023.04.04 08:59:26.011 *I* [config            ]      +- MasterServers
2023.04.04 08:59:26.011 *I* [config            ]          value: <removed>
...
2023.04.04 08:59:26.014 *D* [crypto            ] Validating ciphers
2023.04.04 08:59:26.014 *D* [crypto            ]    AES-256 enabled
2023.04.04 08:59:26.014 *D* [crypto            ]    Blowfish-256 enabled
2023.04.04 08:59:26.014 *D* [crypto            ]    IDEA enabled
2023.04.04 08:59:26.014 *D* [crypto            ]    3DES enabled
2023.04.04 08:59:26.014 *D* [crypto            ]    AES-128 enabled
2023.04.04 08:59:26.014 *D* [crypto            ]    Blowfish-128 enabled
2023.04.04 08:59:26.014 *I* [crypto            ] Crypto library initialized (OpenSSL 1.1.1i  8 Dec 2020)
2023.04.04 08:59:26.014 *D* [dload              ] DLOpen: file="(null)", module=0x7ffff7ffe190
2023.04.04 08:59:26.014 *D* [dload              ] DLGetSymbolAddr: module=0x7ffff7ffe190, symbol=drvAPIVersion, address=0xbae490
2023.04.04 08:59:26.014 *D* [dload              ] DLGetSymbolAddr: module=0x7ffff7ffe190, symbol=drvName, address=0xbae488
2023.04.04 08:59:26.015 *D* [dload              ] DLGetSymbolAddr: module=0x7ffff7ffe190, symbol=drvCallTable, address=0xbae480
2023.04.04 08:59:26.015 *D* [db.drv.sqlite      ] SQLite version 3.36.0
2023.04.04 08:59:26.015 *I* [db.drv            ] Database driver ":self:" loaded and initialized successfully
2023.04.04 08:59:26.015 *D* [db.conn            ] DBConnect: server=(null) db=/opt/netxms/var/lib/netxms/nxagentd.db login=(null) schema=(null)
2023.04.04 08:59:26.015 *D* [db.conn            ] New DB connection opened: handle=0xc4afe0
2023.04.04 08:59:26.016 *D* [db.query          ] DB Library: global long running query threshold set to 250
2023.04.04 08:59:26.020 *I* [db.agent          ] Local database opened successfully
...
[New Thread 0x7ffff7856700 (LWP 1161774)]
.
2023.04.04 08:59:28.122 *D* [db.query          ] COMMIT TRANSACTION successful (level 0)
...
2023.04.04 08:59:33.123 *D* [db.query          ] BEGIN TRANSACTION successful (level 1)
2023.04.04 08:59:33.124 *D* [db.query          ] COMMIT TRANSACTION successful (level 0)
2023.04.04 08:59:33.124 *D* [dc                ] Database writer: 61 records inserted
...
2023.04.04 08:59:54.533 *D* [dc                ] 0 SNMP targets received from server <removed>
2023.04.04 08:59:54.533 *D* [dc                ] 0 proxies received from server <removed>
2023.04.04 08:59:54.533 *D* [dc                ] 62 data collection elements received from server <removed> (extended data: YES)
2023.04.04 08:59:54.534 *D* [dc                ] Data collection for server <removed> reconfigured
2023.04.04 08:59:54.534 *D* [comm.cs.64        ] Sending message CMD_REQUEST_COMPLETED (ID 6; size 32; uncompressed)
2023.04.04 08:59:54.534 *D* [comm.cs.64        ] Outgoing message dump:
  ** 000000 | 00 1D 50 00 00 00 00 20 00 00 00 06 00 00 00 01 | ..P.... ........
  ** 000010 | 00 00 00 1C 00 00 00 00 00 00 00 00 00 00 00 00 | ................
  ** code=0x001D (CMD_REQUEST_COMPLETED) version=5 flags=0x0000 id=6 size=32 numFields=1
  ** 000000: [    28] INT32      0
...
2023.04.04 08:59:54.564 *D* [comm.cs.64        ] Message dump:
  ** 000000 | 00 BA 50 00 00 00 00 10 00 00 00 07 00 00 00 00 | ..P.............
  ** code=0x00BA (CMD_ENABLE_AGENT_TRAPS) version=5 flags=0x0000 id=7 size=16 numFields=0

2023.04.04 08:59:54.564 *D* [comm.cs.64        ] Received message CMD_ENABLE_AGENT_TRAPS (7)
2023.04.04 08:59:54.565 *D* [notifications      ] RegisterSessionForNotifications: starting background sync for server <removed>
2023.04.04 08:59:54.565 *D* [comm.cs.64        ] Sending message CMD_REQUEST_COMPLETED (ID 7; size 32; uncompressed)
2023.04.04 08:59:54.565 *D* [comm.cs.64        ] Outgoing message dump:
  ** 000000 | 00 1D 50 00 00 00 00 20 00 00 00 07 00 00 00 01 | ..P.... ........
  ** 000010 | 00 00 00 1C 00 00 00 00 00 00 00 00 00 00 00 00 | ................
  ** code=0x001D (CMD_REQUEST_COMPLETED) version=5 flags=0x0000 id=7 size=32 numFields=1
  ** 000000: [    28] INT32      0

2023.04.04 08:59:54.565 *D* [notifications      ] Notification synchronization started
2023.04.04 08:59:54.565 *D* [notifications      ] Vacuum local database
free(): invalid pointer

Thread 5 "$COMM/WRK" received signal SIGABRT, Aborted.
[Switching to Thread 0x7ffff7856700 (LWP 1161774)]
__GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
50      ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) thread
[Current thread is 5 (Thread 0x7ffff7856700 (LWP 1161774))]
(gdb) backtrace
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1  0x00007ffff7ad6535 in __GI_abort () at abort.c:79
#2  0x00007ffff7b2d648 in __libc_message (action=action@entry=do_abort, fmt=fmt@entry=0x7ffff7c372a0 "%s\n") at ../sysdeps/posix/libc_fatal.c:181
#3  0x00007ffff7b33d6a in malloc_printerr (str=str@entry=0x7ffff7c3544e "free(): invalid pointer") at malloc.c:5359
#4  0x00007ffff7b3559c in _int_free (av=<optimized out>, p=<optimized out>, have_lock=<optimized out>) at malloc.c:4180
#5  0x0000000000547068 in sqlite3_free (p=0x2) at sqlite3.c:28180
#6  0x000000000056dc48 in sqlite3DbFree (p=<optimized out>, db=0x0) at sqlite3.c:28239
#7  sqlite3DeleteColumnNames (db=0x0, pTable=<optimized out>, pTable=<optimized out>) at sqlite3.c:47566
#8  0x000000000056f9b2 in deleteTable (db=0x0, pTable=0xc90cc8) at sqlite3.c:113163
#9  0x000000000056fb19 in sqlite3DeleteTable (db=<optimized out>, pTable=<optimized out>) at sqlite3.c:113180
#10 0x00000000005701ab in sqlite3SchemaClear (p=0xc3d618) at sqlite3.c:118313
#11 0x00000000005702bb in sqlite3ResetAllSchemasOfConnection (db=0xc3cd18) at sqlite3.c:113070
#12 0x00000000005c95c3 in sqlite3RunVacuum (pzErrMsg=pzErrMsg@entry=0xc47390, db=db@entry=0xc3cd18, iDb=<optimized out>, pOut=0x0) at sqlite3.c:143149
#13 0x00000000005a644b in sqlite3VdbeExec (p=<optimized out>) at sqlite3.c:93837
#14 0x00000000005ae200 in sqlite3Step (p=<optimized out>) at sqlite3.c:84869
#15 sqlite3_step (pStmt=0xc472e8) at sqlite3.c:19390
#16 0x00000000005aef85 in sqlite3_exec (db=0xc3cd18, zSql=<optimized out>, xCallback=0x0, pArg=0x0, pzErrMsg=0x0) at sqlite3.c:126198
#17 0x000000000052ea20 in QueryInternal (conn=conn@entry=0xc3d1b0, query=query@entry=0x7fffd4002af0 "VACUUM", errorText=errorText@entry=0x7ffff7854890 L"") at sqlite/sqlite.cpp:261
#18 0x000000000052eab3 in Query (connection=0xc3d1b0, query=<optimized out>, errorText=0x7ffff7854890 L"") at sqlite/sqlite.cpp:287
#19 0x00000000004b0412 in DBQueryEx (hConn=hConn@entry=0xc4afe0, query=query@entry=0x7e189e "VACUUM", errorText=errorText@entry=0x7ffff78558d0 "") at session.cpp:245
#20 0x00000000004b067f in DBQuery (hConn=hConn@entry=0xc4afe0, query=query@entry=0x7e189e "VACUUM") at session.cpp:291
#21 0x0000000000495098 in NotificationSync (session=std::shared_ptr<class CommSession> (use count 3, weak count 1) = {...}) at nproc.cpp:256
#22 0x0000000000495a0c in __ThreadPoolExecute_SharedPtr_Wrapper<CommSession> (arg=0x7fffc8015050) at ../../../include/nms_threads.h:993
#23 0x00000000004e4c4d in ProcessSerializedRequests (data=0x7fffc8079400) at tp.cpp:487
#24 0x00000000004e4eb9 in WorkerThread (threadInfo=0xc63de0) at tp.cpp:199
#25 0x00000000004e6b2a in ThreadCreate_Wrapper_1<WorkerThreadInfo*> (context=0xc63e00) at ../../include/nms_threads.h:539
#26 0x00007ffff7fa1fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#27 0x00007ffff7bad06f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
(gdb)

thank you,

Br, Aleš

If running nxagent as non root user within LXC container src/agent/subagents/linux/hypervisor.cpp will fail to detect container=lxc as it can not read /proc/1/environ.

Would it be possible to:

OR

1) extend src/agent/subagents/linux/hypervisor.cpp to implement methods to detect hypervisor and node type
    example: https://github.com/systemd/systemd/blob/main/src/basic/virt.c#L657
 
2) use systemd-detect-virt executable directly

3) try to read /run/systemd/container

4) enable/explain option to manually set:
    Node type: Container
    Hypervisor type: LXC

thank you,

Ales

Hello,

We are using ldap server that provides group name in cn attribute vs user name in uid attribute.
Would it be possible to split configuration parameter LdapMappingName to LdapUserMappingName
and LdapGroupMappingName and fix logic in src/server/core/ldap.cpp?

thank you,

Br, Ales

Hello,

You can solve issue by:
1. unbind template with SLA DCI from node
2. delete SLA custom attributes from node
3. on node create DCI described Status collecting Status parameter from origin Internal
4. apply template with SLA DCI to node

or:
1. manually populate SLA custom attributes with values
2. on node create DCI described Status collecting Status parameter from origin Internal

Br, A

Solved in v2.0.0.

Br, Ales

Environment:

-----------------------------------
os: CentOS release 6.7 (Final)
arch: x86_64
netxms: netxms-2.0-RC2
openldap: 2.4.40-6.el6_7
-----------------------------------

Ldap server:
-----------------------------------
MS AD
ldap policy: Show Values

Policy                          Current(New)

MaxPoolThreads        4
MaxDatagramRecv     4096
MaxReceiveBuffer      10485760
InitRecvTimeout        120
MaxConnections         5000
MaxConnIdleTime      900
MaxPageSize             20000
MaxQueryDuration     120
MaxTempTableSize    10000
MaxResultSetSize      262144
MinResultSets                0
MaxResultSetsPerConn    0
MaxNotificationPerConn   5
MaxValRange                 25000
ThreadMemoryLimit        0
SystemMemoryLimitPercent 0
-----------------------------------

-----------------------------------
netxmsd: debug 4
netxmsd: ldapsync

returns:
[DEBUG] LDAPConnection::initLDAP(): Connecting to LDAP server
[DEBUG] LDAPConnection::syncUsers(): LDAP could not get search results. Error code: Operations error
[DEBUG] LDAPConnection::closeLDAPConnection(): Disconnect form LDAP server
-----------------------------------

Query using command line ldapsearch works on server running netxmsd.

TcpDump shows:
-----------------------------------
1. bind was successful
2. search filter works and was applied correctly
3. search results were returned
-----------------------------------

tried:
- TLS (on/off) same results
- removal of all special characters on all places (password/bind user/ldap path)
- filter on/off
- ...


thank you for your help,

Br, Ales

Environment:

CentOS v5.11, x64

/etc/nxagentd.conf:
----
Servers = <removed>
ControlServers = <removed>
MasterServers = <removed>
ListenPort = 4700
LogFile = /var/log/netxms/nxagentd.log
FileStore = /var/nxagentd
EnableProxy = yes
EnableSNMPProxy = yes
----

Existing agent:
----
v1.2.2
install location: /opt/netxms
----

Fail (CentOS-5.11-x64_1.2.2-1.2.17.log):
----
sysinfo.cpp:370: warning: use of 'h' length modifier with 's' type character
  CXX    tools.o
  CXX    trap.o
  CXX    upgrade.o
  CXX    watchdog.o
  CXXLD  nxagentd
../../../src/libnetxms/.libs/libnetxms.so: undefined reference to `tre_regwexec'
../../../src/libnetxms/.libs/libnetxms.so: undefined reference to `tre_regwcomp'
collect2: ld returned 1 exit status
gmake[4]: *** [nxagentd] Error 1
gmake[4]: Leaving directory `/tmp/nxinst.z19411/inst/netxms-1.2.17/src/agent/core'
gmake[3]: *** [all-recursive] Error 1
gmake[3]: Leaving directory `/tmp/nxinst.z19411/inst/netxms-1.2.17/src/agent'
gmake[2]: *** [all-recursive] Error 1
gmake[2]: Leaving directory `/tmp/nxinst.z19411/inst/netxms-1.2.17/src'
gmake[1]: *** [all-recursive] Error 1
gmake[1]: Leaving directory `/tmp/nxinst.z19411/inst/netxms-1.2.17'
gmake: *** [all] Error 2
build failed, restarting old agent
----

Win update using procedure (CentOS-5.11-x64_1.2.17.log):
----
# service nxagentd stop
mv /opt/netxms /opt/_netxms
# cd /var/nxagentd
./nxagent-1.2.17.apkg
----

Thank you for your support

Br, Ales

- we are using configuration with zones
- when server receives trap following lines are logged:

[DD-MMM-YYYY HH:MM:SS] AgentConnectionEx::onTrap(): Received trap message from agent at a.b.c.d
[DD-MMM-YYYY HH:MM:SS] Cannot find node for IP address a.b.c.d

* this is the only trace on server that it received agent trap

- is there any known workaround for server in zone A to receive trap from agent in zone B

thank you,

Ales

What would be advised way to handle server to agent connection in case that server address
is source ip network address translated and translated IP is chosen dynamically from pool of
ip addresses. (static translation is not possible, not technical reason)

Practically:
SERVER (ip a.a.a.a/24) <-> SNAT (pool b.b.b.b/26) <- VPN -> agent (ip c.c.c.c/24)

Server translated address at time of connection could be b.b.b.1-b.b.b.127.

Solutions  (parameters Servers =, ControlServers =, MasterServers =)
* define all possible ip address in Servers = ...
* create dns A record with all possible ip addressees and use dns name as parameter value
* create feature request for parameters to accept CIDR notation

Possible solution:
* create feature request to add logic to client:
   if none of  Servers =, ControlServers =, MasterServers =  are defined
   and required authentication+encryption is configured
   allow configuration value 0.0.0.0
   as Servers =, ControlServers =, MasterServers = parameter value.
 
thank you,

Ales

https://www.netxms.org/forum/configuration/clustering-with-snmp-proxy/

thank you

Ales

Hello,

I tested the button. You can provide feedback by adding little timeout
to ServiceCheck.Custom(<insert arguments here>) as additional parameter
so that check would give back value when TCP port is not reachable

And one more suggestion if I can (pushing my luck) evaluate feature
requests, create table put value preset pay pall button in each row in that
table.

thank you,

Ales

Thank you,

BTW I am still searching for pay pal donations button
For those of us that appreciate hard work but can not
afford professional support.

Ales