NetXMS Support Forum

Hello,

Problem description:
- tuneled agent crashes after 0s..45min after start (repeatedly)

Environment description:
- LXC container running (bookworm) on Proxmox running (bookworm) all up2date @04.16.2024
- kernel 6.5.11-7-pve
- DEB Package: netxms-agent, Version: 4.5.6-1

--------------------------------
:~# gdb /usr/bin/nxagentd
GNU gdb (Debian 13.1-3) 13.1
Copyright (C) 2023 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/bin/nxagentd...
Reading symbols from /usr/lib/debug/.build-id/95/2e3f9104195351fe4deab7db75a2be62a439f2.debug...
(gdb) set args -f -D 4
--------------------------------
--------------------------------
(gdb) run
Starting program: /usr/bin/nxagentd -f -D 4
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Error reading additional configuration files from "/etc/nxagentd.conf.d"
2024.04.16 07:20:04.009 *I* [logger             ] Log rotation policy set to 2 (size=16777216, count=4)
2024.04.16 07:20:04.010 *I* [startup            ] Core agent version 4.5.6 (build tag 4.5-213-g58d2aa16a8)
2024.04.16 07:20:04.010 *I* [config             ] Additional configuration files was loaded from /etc/nxagentd.conf.d
2024.04.16 07:20:04.010 *I* [logger             ] Debug level set to 4
2024.04.16 07:20:04.010 *I* [config             ] Main configuration file: /etc/nxagentd.conf
2024.04.16 07:20:04.010 *I* [config             ] Configuration tree:
2024.04.16 07:20:04.010 *I* [config             ] config
2024.04.16 07:20:04.010 *I* [config             ]  +- CORE
2024.04.16 07:20:04.010 *I* [config             ]      +- ServerConnection
2024.04.16 07:20:04.010 *I* [config             ]           value: 172.16.xx.xx
2024.04.16 07:20:04.010 *I* [startup            ] Data directory: /var/lib/netxms
2024.04.16 07:20:04.010 *I* [startup            ] File store: /tmp
2024.04.16 07:20:04.010 *D* [startup            ] Effective user ID 0
2024.04.16 07:20:04.010 *D* [startup            ] Effective group ID 0
2024.04.16 07:20:04.010 *D* [startup            ] Configuration policy directory: /var/lib/netxms/config_ap/
2024.04.16 07:20:04.010 *D* [startup            ] Log parser policy directory: /var/lib/netxms/logparser_ap/
2024.04.16 07:20:04.010 *D* [startup            ] User agent policy directory: /var/lib/netxms/useragent_ap/
2024.04.16 07:20:04.011 *D* [startup            ] Certificate directory: /var/lib/netxms/certificates/
2024.04.16 07:20:04.011 *D* [startup            ] CRL directory: /var/lib/netxms/crl/
2024.04.16 07:20:04.014 *D* [crypto             ] Validating ciphers
2024.04.16 07:20:04.014 *D* [crypto             ]    AES-256 enabled
2024.04.16 07:20:04.014 *D* [crypto             ]    Blowfish-256 disabled (validation failed)
2024.04.16 07:20:04.014 *D* [crypto             ]    IDEA disabled (config)
2024.04.16 07:20:04.014 *D* [crypto             ]    3DES enabled
2024.04.16 07:20:04.014 *D* [crypto             ]    AES-128 enabled
2024.04.16 07:20:04.014 *D* [crypto             ]    Blowfish-128 disabled (validation failed)
2024.04.16 07:20:04.014 *I* [crypto             ] Crypto library initialized (OpenSSL 3.0.11 19 Sep 2023)
[New Thread 0x7ffff649b6c0 (LWP 33983)]
[New Thread 0x7ffff5bff6c0 (LWP 33984)]
[New Thread 0x7ffff5afe6c0 (LWP 33985)]
2024.04.16 07:20:04.016 *D* [threads.pool       ] Thread pool TIMER initialized (min=2, max=16)
2024.04.16 07:20:04.016 *D* [startup            ] Subagent API initialized
2024.04.16 07:20:04.035 *D* [db.drv.sqlite      ] SQLite version 3.40.1
2024.04.16 07:20:04.035 *I* [db.drv             ] Database driver "sqlite.ddr" loaded and initialized successfully
2024.04.16 07:20:04.036 *D* [db.conn            ] New DB connection opened: handle=0x7ffff61641c0
2024.04.16 07:20:04.038 *D* [db.query           ] DB Library: global long running query threshold set to 250
2024.04.16 07:20:04.045 *I* [db.agent           ] Local database opened successfully
2024.04.16 07:20:04.046 *I* [startup            ] Agent ID is 6334d94b-6ab3-46ba-8515-891f843f5907
2024.04.16 07:20:04.047 *I* [startup            ] Local host name is "d001-spooler01.local"
2024.04.16 07:20:04.047 *I* [startup            ] Using system name "d001-spooler01"
2024.04.16 07:20:04.047 *D* [comm               ] Maximum number of sessions set to 32
[New Thread 0x7ffff645a6c0 (LWP 33986)]
[New Thread 0x7ffff561e6c0 (LWP 33987)]
[New Thread 0x7ffff551d6c0 (LWP 33988)]
[New Thread 0x7ffff541c6c0 (LWP 33989)]
[New Thread 0x7ffff531b6c0 (LWP 33990)]
[New Thread 0x7ffff521a6c0 (LWP 33991)]
[New Thread 0x7ffff51196c0 (LWP 33992)]
[New Thread 0x7ffff50186c0 (LWP 33993)]
[New Thread 0x7ffff4f176c0 (LWP 33994)]
2024.04.16 07:20:04.049 *D* [threads.pool       ] Thread pool COMM initialized (min=8, max=512)
[New Thread 0x7ffff4e166c0 (LWP 33995)]
[New Thread 0x7ffff4dd56c0 (LWP 33996)]
[New Thread 0x7ffff4cd46c0 (LWP 33997)]
[New Thread 0x7ffff4bd36c0 (LWP 33998)]
[New Thread 0x7ffff4ad26c0 (LWP 33999)]
2024.04.16 07:20:04.049 *D* [threads.pool       ] Thread pool WEBSVC initialized (min=4, max=64)
2024.04.16 07:20:04.050 *D* [tunnel             ] Added server tunnel 172.16.xx.xx
2024.04.16 07:20:04.050 *I* [startup            ] Arbitrary command execution disabled
2024.04.16 07:20:04.064 *D* [smbios             ] BIOS read failed
[New Thread 0x7ffff49256c0 (LWP 34001)]
2024.04.16 07:20:05.084 *D* [linux              ] Using /sys/block to distinguish devices from partitions
2024.04.16 07:20:05.084 *D* [linux              ] CPU usage collector thread started
[New Thread 0x7ffff48246c0 (LWP 34002)]
[New Thread 0x7ffff47236c0 (LWP 34003)]
2024.04.16 07:20:05.089 *I* [subagents          ] Subagent "Linux" (/usr/lib/x86_64-linux-gnu/netxms/linux.nsm) loaded successfully (version 4.5.6)
2024.04.16 07:20:05.089 *I* [startup            ] System hardware ID is unknown
2024.04.16 07:20:05.090 *I* [linux              ] Unable to parse /proc/drbd, DRBD data collector will not start
2024.04.16 07:20:05.090 *D* [linux              ] ParseIoStat(): new device added (name=loop0 isRealDevice=1)
2024.04.16 07:20:05.090 *D* [linux              ] ParseIoStat(): new device added (name=loop1 isRealDevice=1)
2024.04.16 07:20:05.090 *D* [linux              ] ParseIoStat(): new device added (name=loop2 isRealDevice=1)
2024.04.16 07:20:05.091 *D* [linux              ] ParseIoStat(): new device added (name=loop3 isRealDevice=1)
2024.04.16 07:20:05.091 *D* [linux              ] ParseIoStat(): new device added (name=loop4 isRealDevice=1)
2024.04.16 07:20:05.091 *D* [linux              ] ParseIoStat(): new device added (name=loop5 isRealDevice=1)
2024.04.16 07:20:05.091 *D* [linux              ] ParseIoStat(): new device added (name=loop6 isRealDevice=1)
2024.04.16 07:20:05.092 *D* [linux              ] ParseIoStat(): new device added (name=loop7 isRealDevice=1)
2024.04.16 07:20:05.092 *D* [linux              ] ParseIoStat(): new device added (name=nvme0n1 isRealDevice=1)
2024.04.16 07:20:05.092 *D* [linux              ] ParseIoStat(): new device added (name=nvme0n1p1 isRealDevice=0)
2024.04.16 07:20:05.092 *D* [linux              ] ParseIoStat(): new device added (name=nvme0n1p2 isRealDevice=0)
2024.04.16 07:20:05.092 *D* [linux              ] ParseIoStat(): new device added (name=nvme0n1p3 isRealDevice=0)
2024.04.16 07:20:05.093 *D* [linux              ] ParseIoStat(): new device added (name=nvme0n1p4 isRealDevice=0)
2024.04.16 07:20:05.093 *D* [linux              ] ParseIoStat(): new device added (name=nvme0n1p128 isRealDevice=0)
2024.04.16 07:20:05.093 *D* [linux              ] ParseIoStat(): new device added (name=zd0 isRealDevice=1)
2024.04.16 07:20:05.093 *D* [linux              ] ParseIoStat(): new device added (name=zd16 isRealDevice=1)
2024.04.16 07:20:05.093 *D* [linux              ] ParseIoStat(): new device added (name=zd16p1 isRealDevice=0)
2024.04.16 07:20:05.094 *D* [linux              ] ParseIoStat(): new device added (name=zd16p14 isRealDevice=0)
2024.04.16 07:20:05.094 *D* [linux              ] ParseIoStat(): new device added (name=zd16p15 isRealDevice=0)
2024.04.16 07:20:05.094 *D* [linux              ] ParseIoStat(): new device added (name=zd32 isRealDevice=1)
2024.04.16 07:20:05.094 *D* [linux              ] ParseIoStat(): new device added (name=zd48 isRealDevice=1)
2024.04.16 07:20:05.095 *D* [linux              ] ParseIoStat(): new device added (name=zd64 isRealDevice=1)
2024.04.16 07:20:05.095 *D* [linux              ] ParseIoStat(): new device added (name=zd80 isRealDevice=1)
2024.04.16 07:20:05.095 *D* [linux              ] ParseIoStat(): new device added (name=zd96 isRealDevice=1)
2024.04.16 07:20:05.095 *D* [linux              ] ParseIoStat(): new device added (name=zd112 isRealDevice=1)
2024.04.16 07:20:05.096 *D* [linux              ] ParseIoStat(): new device added (name=zd112p1 isRealDevice=0)
2024.04.16 07:20:05.096 *D* [linux              ] ParseIoStat(): new device added (name=zd112p14 isRealDevice=0)
2024.04.16 07:20:05.096 *D* [linux              ] ParseIoStat(): new device added (name=zd112p15 isRealDevice=0)
2024.04.16 07:20:05.096 *D* [linux              ] ParseIoStat(): new device added (name=zd128 isRealDevice=1)
2024.04.16 07:20:05.096 *D* [linux              ] ParseIoStat(): new device added (name=zd144 isRealDevice=1)
2024.04.16 07:20:05.097 *D* [linux              ] ParseIoStat(): new device added (name=zd144p1 isRealDevice=0)
2024.04.16 07:20:05.097 *D* [linux              ] ParseIoStat(): new device added (name=zd144p14 isRealDevice=0)
2024.04.16 07:20:05.097 *D* [linux              ] ParseIoStat(): new device added (name=zd144p15 isRealDevice=0)
2024.04.16 07:20:05.097 *D* [linux              ] ParseIoStat(): new device added (name=zd160 isRealDevice=1)
2024.04.16 07:20:05.097 *D* [linux              ] ParseIoStat(): new device added (name=zd160p1 isRealDevice=0)
2024.04.16 07:20:05.097 *D* [linux              ] ParseIoStat(): new device added (name=zd160p2 isRealDevice=0)
2024.04.16 07:20:05.098 *D* [linux              ] ParseIoStat(): new device added (name=zd160p3 isRealDevice=0)
2024.04.16 07:20:05.098 *D* [linux              ] ParseIoStat(): new device added (name=zd160p4 isRealDevice=0)
2024.04.16 07:20:05.098 *D* [linux              ] ParseIoStat(): new device added (name=zd160p5 isRealDevice=0)
[Thread 0x7ffff47236c0 (LWP 34003) exited]
2024.04.16 07:20:05.099 *D* [linux              ] ParseIoStat(): new device added (name=zd160p6 isRealDevice=0)
2024.04.16 07:20:05.099 *D* [linux              ] ParseIoStat(): new device added (name=zd160p7 isRealDevice=0)
2024.04.16 07:20:05.099 *D* [linux              ] ParseIoStat(): new device added (name=zd160p8 isRealDevice=0)
[New Thread 0x7ffff39ff6c0 (LWP 34004)]
[New Thread 0x7ffff39be6c0 (LWP 34005)]
2024.04.16 07:20:06.091 *D* [threads.pool       ] Thread pool PROCEXEC initialized (min=1, max=16)
2024.04.16 07:20:06.091 *D* [notifications      ] StartNotificationProcessor: Loaded 1 servers
[New Thread 0x7ffff38bd6c0 (LWP 34006)]
2024.04.16 07:20:06.092 *D* [notifications      ] Notification processor started
2024.04.16 07:20:06.092 *D* [notifications      ] Notification housekeeper started
2024.04.16 07:20:06.092 *D* [notifications      ] Notification housekeeper execution completed
[New Thread 0x7ffff37bc6c0 (LWP 34007)]
[New Thread 0x7ffff377b6c0 (LWP 34008)]
[New Thread 0x7ffff31ff6c0 (LWP 34009)]
[New Thread 0x7ffff30fe6c0 (LWP 34010)]
[New Thread 0x7ffff2ffd6c0 (LWP 34011)]
2024.04.16 07:20:06.093 *D* [threads.pool       ] Thread pool DATACOLL initialized (min=4, max=64)
[New Thread 0x7ffff2efc6c0 (LWP 34012)]
2024.04.16 07:20:06.094 *D* [dc                 ] Data collection scheduler thread started
[New Thread 0x7ffff2dfb6c0 (LWP 34013)]
[New Thread 0x7ffff27ff6c0 (LWP 34014)]
[New Thread 0x7ffff26fe6c0 (LWP 34015)]
2024.04.16 07:20:06.094 *D* [dc                 ] Data sender thread started
[New Thread 0x7ffff25fd6c0 (LWP 34016)]
[New Thread 0x7ffff1dff6c0 (LWP 34017)]
[New Thread 0x7ffff1cfe6c0 (LWP 34018)]
2024.04.16 07:20:06.095 *D* [                   ] Session agent connector disabled
2024.04.16 07:20:06.096 *D* [comm.listener      ] SocketListener/ProxyHeartbeat: Trying to bind on 0.0.0.0:4700/udp
2024.04.16 07:20:06.096 *D* [comm.listener      ] SocketListener/ProxyHeartbeat: Trying to bind on [::]:4700/udp
2024.04.16 07:20:06.096 *D* [comm               ] Trying to bind on 0.0.0.0:4700
2024.04.16 07:20:06.096 *D* [                   ] Trying to bind on [::]:4700
2024.04.16 07:20:06.096 *I* [comm               ] Listening on socket 0.0.0.0:4700
2024.04.16 07:20:06.096 *I* [comm               ] Listening on socket :::4700
2024.04.16 07:20:06.096 *D* [dc                 ] Database writer thread started
2024.04.16 07:20:06.096 *D* [dc                 ] Data reconciliation thread started (block size 1024, timeout 60000 ms)
[New Thread 0x7ffff19fd6c0 (LWP 34019)]
[New Thread 0x7ffff18fc6c0 (LWP 34020)]
2024.04.16 07:20:06.107 *D* [                   ] NamedPipeListener(nxagentd.push): waiting for connection
2024.04.16 07:20:06.107 *D* [                   ] NamedPipeListener(nxagentd.events): waiting for connection
2024.04.16 07:20:06.107 *I* [comm               ] Control connector is disabled
[New Thread 0x7ffff17fb6c0 (LWP 34021)]
2024.04.16 07:20:06.108 *D* [tunnel             ] Tunnel manager started
2024.04.16 07:20:06.114 *D* [tunnel             ] 172.16.xx.xx: Certificate and private key loaded
2024.04.16 07:20:06.126 *D* [tunnel             ] 172.16.xx.xx: Server certificate subject is CN=d001-mgmt01
2024.04.16 07:20:06.127 *D* [tunnel             ] 172.16.xx.xx: Server certificate issuer is CN=d001
2024.04.16 07:20:06.127 *D* [tunnel             ] 172.16.xx.xx: Server certificate verification is disabled
2024.04.16 07:20:06.127 *D* [tunnel             ] 172.16.xx.xx: Server certificate pinning is disabled
[New Thread 0x7ffff16fa6c0 (LWP 34022)]
2024.04.16 07:20:06.171 *I* [                   ] Tunnel with 172.16.xx.xx established
2024.04.16 07:20:06.171 *D* [tunnel             ] 172.16.xx.xx: Tunnel is active
2024.04.16 07:20:07.109 *I* [filemon            ] Path list for file monitor is empty
2024.04.16 07:20:07.109 *I* [                   ] NetXMS Agent started
Agent running. Press Ctrl+C to shutdown.
2024.04.16 07:20:13.988 *D* [comm.cs.2          ] Session registered (control=false, master=false)
[New Thread 0x7ffff15f96c0 (LWP 34023)]
2024.04.16 07:20:13.989 *D* [comm.cs.2          ] Using protocol version 5
2024.04.16 07:20:13.994 *D* [comm.cs.2          ] Server capabilities: IPv6: yes; bulk reconciliation: yes; compression: yes
2024.04.16 07:20:13.994 *D* [comm.cs.2          ] Server ID set to 319612E4EC7036B6
2024.04.16 07:20:13.995 *D* [comm.cs.2          ] Server ID set to 319612E4EC7036B6
2024.04.16 07:20:14.005 *D* [dc                 ] 0 SNMP targets received from server 319612E4EC7036B6
2024.04.16 07:20:14.005 *D* [dc                 ] 0 proxies received from server 319612E4EC7036B6
2024.04.16 07:20:14.005 *D* [dc                 ] 0 data collection elements received from server 319612E4EC7036B6 (extended data: YES)
2024.04.16 07:20:14.005 *D* [dc                 ] Data collection for server 319612E4EC7036B6 reconfigured
2024.04.16 07:20:14.009 *D* [notifications      ] RegisterSessionForNotifications: starting background sync for server 319612E4EC7036B6
2024.04.16 07:20:14.009 *D* [notifications      ] Notification synchronization started
2024.04.16 07:20:14.009 *D* [notifications      ] Vacuum local database
2024.04.16 07:20:14.016 *D* [notifications      ] Notification synchronization finished
2024.04.16 07:48:09.800 *D* [comm.cs.3          ] Session registered (control=false, master=false)
[New Thread 0x7ffff11f86c0 (LWP 34029)]
2024.04.16 07:48:09.808 *D* [comm.cs.3          ] Using protocol version 5

Thread 42 "nxagentd" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffff11f86c0 (LWP 34029)]
0x00007ffff7a330a2 in ?? () from /lib/x86_64-linux-gnu/libcrypto.so.3
-------------------------------------------------
-------------------------------------------------
(gdb) bt
#0  0x00007ffff7a330a2 in ?? () from /lib/x86_64-linux-gnu/libcrypto.so.3
#1  0x00007ffff7a315af in OPENSSL_LH_doall_arg () from /lib/x86_64-linux-gnu/libcrypto.so.3
#2  0x00007ffff7a332ab in ?? () from /lib/x86_64-linux-gnu/libcrypto.so.3
#3  0x00007ffff7a24af6 in ?? () from /lib/x86_64-linux-gnu/libcrypto.so.3
#4  0x00007ffff7ec8fe0 in RSAPublicEncrypt (outlen=4096, padding=4, rsa=0x7ffff321e0e0, out=0x7ffff11ed990 "0\202\001\n\002\202\001\001", inlen=32, in=0x7ffff320b0a0) at ../../include/nxcrypto.h:211
#5  SetupEncryptionContext (msg=msg@entry=0x7ffff328b000, ppCtx=ppCtx@entry=0x7ffff11eea20, ppResponse=ppResponse@entry=0x7ffff11eea30, privateKey=privateKey@entry=0x0, nxcpVersion=<optimized out>) at ./src/libnetxms/crypto.cpp:474
#6  0x00005555555a8dfe in CommSession::readThread (this=0x7ffff1e61310) at ./src/agent/core/session.cpp:340
#7  0x00005555555aa9a2 in ThreadCreate_SharedPtr_Wrapper_0<CommSession> (arg=0x7ffff1e22080) at ../../../include/nms_threads.h:795
#8  0x00007ffff7357134 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#9  0x00007ffff73d77dc in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
(gdb)
-------------------------------------------------
thank you,

Br, Aleš

Hello,

Problem description:
- agent crashes immediately after trying to vacuum local sqlite database

Environment description:
- LXC container running (buster) on Proxmox running (bullseye) all up2date @04.04.2023
- kernel v6.2.6
- agent nxagent-4.3.3-linux-x86_64-static

Etc:
- sqlite database was created by nxagent-4.3.3-linux-x86_64-static
- filesystem zfs

---
# /opt/netxms/bin/nxagentd -v
NetXMS Core Agent Version 4.3.3 Build 4.3-416-g5911d89 (NON-UNICODE)

# sha256sum /opt/netxms/bin/nxagentd
2c208fb7eb7d8b4a99babad8bf2a11717d666652d8a0a5edb78c1e9ff62f1568  /opt/netxms/bin/nxagentd

# gdb /opt/netxms/bin/nxagentd

GNU gdb (Debian 8.2.1-2+b3) 8.2.1
...
(gdb) run -f -D 9
Starting program: /opt/netxms/bin/nxagentd -f -D 9
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
...
2023.04.04 08:59:26.011 *I* [config            ] Main configuration file: /etc/nxagentd.conf
2023.04.04 08:59:26.011 *I* [config            ] Configuration tree:
2023.04.04 08:59:26.011 *I* [config            ] config
2023.04.04 08:59:26.011 *I* [config            ]  +- CORE
2023.04.04 08:59:26.011 *I* [config            ]      +- ServerConnection
2023.04.04 08:59:26.011 *I* [config            ]      |    value: <removed>
2023.04.04 08:59:26.011 *I* [config            ]      +- Servers
2023.04.04 08:59:26.011 *I* [config            ]      |    value: <removed>
2023.04.04 08:59:26.011 *I* [config            ]      +- ControlServers
2023.04.04 08:59:26.011 *I* [config            ]      |    value: <removed>
2023.04.04 08:59:26.011 *I* [config            ]      +- MasterServers
2023.04.04 08:59:26.011 *I* [config            ]          value: <removed>
...
2023.04.04 08:59:26.014 *D* [crypto            ] Validating ciphers
2023.04.04 08:59:26.014 *D* [crypto            ]    AES-256 enabled
2023.04.04 08:59:26.014 *D* [crypto            ]    Blowfish-256 enabled
2023.04.04 08:59:26.014 *D* [crypto            ]    IDEA enabled
2023.04.04 08:59:26.014 *D* [crypto            ]    3DES enabled
2023.04.04 08:59:26.014 *D* [crypto            ]    AES-128 enabled
2023.04.04 08:59:26.014 *D* [crypto            ]    Blowfish-128 enabled
2023.04.04 08:59:26.014 *I* [crypto            ] Crypto library initialized (OpenSSL 1.1.1i  8 Dec 2020)
2023.04.04 08:59:26.014 *D* [dload              ] DLOpen: file="(null)", module=0x7ffff7ffe190
2023.04.04 08:59:26.014 *D* [dload              ] DLGetSymbolAddr: module=0x7ffff7ffe190, symbol=drvAPIVersion, address=0xbae490
2023.04.04 08:59:26.014 *D* [dload              ] DLGetSymbolAddr: module=0x7ffff7ffe190, symbol=drvName, address=0xbae488
2023.04.04 08:59:26.015 *D* [dload              ] DLGetSymbolAddr: module=0x7ffff7ffe190, symbol=drvCallTable, address=0xbae480
2023.04.04 08:59:26.015 *D* [db.drv.sqlite      ] SQLite version 3.36.0
2023.04.04 08:59:26.015 *I* [db.drv            ] Database driver ":self:" loaded and initialized successfully
2023.04.04 08:59:26.015 *D* [db.conn            ] DBConnect: server=(null) db=/opt/netxms/var/lib/netxms/nxagentd.db login=(null) schema=(null)
2023.04.04 08:59:26.015 *D* [db.conn            ] New DB connection opened: handle=0xc4afe0
2023.04.04 08:59:26.016 *D* [db.query          ] DB Library: global long running query threshold set to 250
2023.04.04 08:59:26.020 *I* [db.agent          ] Local database opened successfully
...
[New Thread 0x7ffff7856700 (LWP 1161774)]
.
2023.04.04 08:59:28.122 *D* [db.query          ] COMMIT TRANSACTION successful (level 0)
...
2023.04.04 08:59:33.123 *D* [db.query          ] BEGIN TRANSACTION successful (level 1)
2023.04.04 08:59:33.124 *D* [db.query          ] COMMIT TRANSACTION successful (level 0)
2023.04.04 08:59:33.124 *D* [dc                ] Database writer: 61 records inserted
...
2023.04.04 08:59:54.533 *D* [dc                ] 0 SNMP targets received from server <removed>
2023.04.04 08:59:54.533 *D* [dc                ] 0 proxies received from server <removed>
2023.04.04 08:59:54.533 *D* [dc                ] 62 data collection elements received from server <removed> (extended data: YES)
2023.04.04 08:59:54.534 *D* [dc                ] Data collection for server <removed> reconfigured
2023.04.04 08:59:54.534 *D* [comm.cs.64        ] Sending message CMD_REQUEST_COMPLETED (ID 6; size 32; uncompressed)
2023.04.04 08:59:54.534 *D* [comm.cs.64        ] Outgoing message dump:
  ** 000000 | 00 1D 50 00 00 00 00 20 00 00 00 06 00 00 00 01 | ..P.... ........
  ** 000010 | 00 00 00 1C 00 00 00 00 00 00 00 00 00 00 00 00 | ................
  ** code=0x001D (CMD_REQUEST_COMPLETED) version=5 flags=0x0000 id=6 size=32 numFields=1
  ** 000000: [    28] INT32      0
...
2023.04.04 08:59:54.564 *D* [comm.cs.64        ] Message dump:
  ** 000000 | 00 BA 50 00 00 00 00 10 00 00 00 07 00 00 00 00 | ..P.............
  ** code=0x00BA (CMD_ENABLE_AGENT_TRAPS) version=5 flags=0x0000 id=7 size=16 numFields=0

2023.04.04 08:59:54.564 *D* [comm.cs.64        ] Received message CMD_ENABLE_AGENT_TRAPS (7)
2023.04.04 08:59:54.565 *D* [notifications      ] RegisterSessionForNotifications: starting background sync for server <removed>
2023.04.04 08:59:54.565 *D* [comm.cs.64        ] Sending message CMD_REQUEST_COMPLETED (ID 7; size 32; uncompressed)
2023.04.04 08:59:54.565 *D* [comm.cs.64        ] Outgoing message dump:
  ** 000000 | 00 1D 50 00 00 00 00 20 00 00 00 07 00 00 00 01 | ..P.... ........
  ** 000010 | 00 00 00 1C 00 00 00 00 00 00 00 00 00 00 00 00 | ................
  ** code=0x001D (CMD_REQUEST_COMPLETED) version=5 flags=0x0000 id=7 size=32 numFields=1
  ** 000000: [    28] INT32      0

2023.04.04 08:59:54.565 *D* [notifications      ] Notification synchronization started
2023.04.04 08:59:54.565 *D* [notifications      ] Vacuum local database
free(): invalid pointer

Thread 5 "$COMM/WRK" received signal SIGABRT, Aborted.
[Switching to Thread 0x7ffff7856700 (LWP 1161774)]
__GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
50      ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) thread
[Current thread is 5 (Thread 0x7ffff7856700 (LWP 1161774))]
(gdb) backtrace
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1  0x00007ffff7ad6535 in __GI_abort () at abort.c:79
#2  0x00007ffff7b2d648 in __libc_message (action=action@entry=do_abort, fmt=fmt@entry=0x7ffff7c372a0 "%s\n") at ../sysdeps/posix/libc_fatal.c:181
#3  0x00007ffff7b33d6a in malloc_printerr (str=str@entry=0x7ffff7c3544e "free(): invalid pointer") at malloc.c:5359
#4  0x00007ffff7b3559c in _int_free (av=<optimized out>, p=<optimized out>, have_lock=<optimized out>) at malloc.c:4180
#5  0x0000000000547068 in sqlite3_free (p=0x2) at sqlite3.c:28180
#6  0x000000000056dc48 in sqlite3DbFree (p=<optimized out>, db=0x0) at sqlite3.c:28239
#7  sqlite3DeleteColumnNames (db=0x0, pTable=<optimized out>, pTable=<optimized out>) at sqlite3.c:47566
#8  0x000000000056f9b2 in deleteTable (db=0x0, pTable=0xc90cc8) at sqlite3.c:113163
#9  0x000000000056fb19 in sqlite3DeleteTable (db=<optimized out>, pTable=<optimized out>) at sqlite3.c:113180
#10 0x00000000005701ab in sqlite3SchemaClear (p=0xc3d618) at sqlite3.c:118313
#11 0x00000000005702bb in sqlite3ResetAllSchemasOfConnection (db=0xc3cd18) at sqlite3.c:113070
#12 0x00000000005c95c3 in sqlite3RunVacuum (pzErrMsg=pzErrMsg@entry=0xc47390, db=db@entry=0xc3cd18, iDb=<optimized out>, pOut=0x0) at sqlite3.c:143149
#13 0x00000000005a644b in sqlite3VdbeExec (p=<optimized out>) at sqlite3.c:93837
#14 0x00000000005ae200 in sqlite3Step (p=<optimized out>) at sqlite3.c:84869
#15 sqlite3_step (pStmt=0xc472e8) at sqlite3.c:19390
#16 0x00000000005aef85 in sqlite3_exec (db=0xc3cd18, zSql=<optimized out>, xCallback=0x0, pArg=0x0, pzErrMsg=0x0) at sqlite3.c:126198
#17 0x000000000052ea20 in QueryInternal (conn=conn@entry=0xc3d1b0, query=query@entry=0x7fffd4002af0 "VACUUM", errorText=errorText@entry=0x7ffff7854890 L"") at sqlite/sqlite.cpp:261
#18 0x000000000052eab3 in Query (connection=0xc3d1b0, query=<optimized out>, errorText=0x7ffff7854890 L"") at sqlite/sqlite.cpp:287
#19 0x00000000004b0412 in DBQueryEx (hConn=hConn@entry=0xc4afe0, query=query@entry=0x7e189e "VACUUM", errorText=errorText@entry=0x7ffff78558d0 "") at session.cpp:245
#20 0x00000000004b067f in DBQuery (hConn=hConn@entry=0xc4afe0, query=query@entry=0x7e189e "VACUUM") at session.cpp:291
#21 0x0000000000495098 in NotificationSync (session=std::shared_ptr<class CommSession> (use count 3, weak count 1) = {...}) at nproc.cpp:256
#22 0x0000000000495a0c in __ThreadPoolExecute_SharedPtr_Wrapper<CommSession> (arg=0x7fffc8015050) at ../../../include/nms_threads.h:993
#23 0x00000000004e4c4d in ProcessSerializedRequests (data=0x7fffc8079400) at tp.cpp:487
#24 0x00000000004e4eb9 in WorkerThread (threadInfo=0xc63de0) at tp.cpp:199
#25 0x00000000004e6b2a in ThreadCreate_Wrapper_1<WorkerThreadInfo*> (context=0xc63e00) at ../../include/nms_threads.h:539
#26 0x00007ffff7fa1fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#27 0x00007ffff7bad06f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
(gdb)

thank you,

Br, Aleš

If running nxagent as non root user within LXC container src/agent/subagents/linux/hypervisor.cpp will fail to detect container=lxc as it can not read /proc/1/environ.

Would it be possible to:

OR

1) extend src/agent/subagents/linux/hypervisor.cpp to implement methods to detect hypervisor and node type
    example: https://github.com/systemd/systemd/blob/main/src/basic/virt.c#L657
 
2) use systemd-detect-virt executable directly

3) try to read /run/systemd/container

4) enable/explain option to manually set:
    Node type: Container
    Hypervisor type: LXC

thank you,

Ales

Hello,

We are using ldap server that provides group name in cn attribute vs user name in uid attribute.
Would it be possible to split configuration parameter LdapMappingName to LdapUserMappingName
and LdapGroupMappingName and fix logic in src/server/core/ldap.cpp?

thank you,

Br, Ales

Environment:

-----------------------------------
os: CentOS release 6.7 (Final)
arch: x86_64
netxms: netxms-2.0-RC2
openldap: 2.4.40-6.el6_7
-----------------------------------

Ldap server:
-----------------------------------
MS AD
ldap policy: Show Values

Policy                          Current(New)

MaxPoolThreads        4
MaxDatagramRecv     4096
MaxReceiveBuffer      10485760
InitRecvTimeout        120
MaxConnections         5000
MaxConnIdleTime      900
MaxPageSize             20000
MaxQueryDuration     120
MaxTempTableSize    10000
MaxResultSetSize      262144
MinResultSets                0
MaxResultSetsPerConn    0
MaxNotificationPerConn   5
MaxValRange                 25000
ThreadMemoryLimit        0
SystemMemoryLimitPercent 0
-----------------------------------

-----------------------------------
netxmsd: debug 4
netxmsd: ldapsync

returns:
[DEBUG] LDAPConnection::initLDAP(): Connecting to LDAP server
[DEBUG] LDAPConnection::syncUsers(): LDAP could not get search results. Error code: Operations error
[DEBUG] LDAPConnection::closeLDAPConnection(): Disconnect form LDAP server
-----------------------------------

Query using command line ldapsearch works on server running netxmsd.

TcpDump shows:
-----------------------------------
1. bind was successful
2. search filter works and was applied correctly
3. search results were returned
-----------------------------------

tried:
- TLS (on/off) same results
- removal of all special characters on all places (password/bind user/ldap path)
- filter on/off
- ...


thank you for your help,

Br, Ales

Environment:

CentOS v5.11, x64

/etc/nxagentd.conf:
----
Servers = <removed>
ControlServers = <removed>
MasterServers = <removed>
ListenPort = 4700
LogFile = /var/log/netxms/nxagentd.log
FileStore = /var/nxagentd
EnableProxy = yes
EnableSNMPProxy = yes
----

Existing agent:
----
v1.2.2
install location: /opt/netxms
----

Fail (CentOS-5.11-x64_1.2.2-1.2.17.log):
----
sysinfo.cpp:370: warning: use of 'h' length modifier with 's' type character
  CXX    tools.o
  CXX    trap.o
  CXX    upgrade.o
  CXX    watchdog.o
  CXXLD  nxagentd
../../../src/libnetxms/.libs/libnetxms.so: undefined reference to `tre_regwexec'
../../../src/libnetxms/.libs/libnetxms.so: undefined reference to `tre_regwcomp'
collect2: ld returned 1 exit status
gmake[4]: *** [nxagentd] Error 1
gmake[4]: Leaving directory `/tmp/nxinst.z19411/inst/netxms-1.2.17/src/agent/core'
gmake[3]: *** [all-recursive] Error 1
gmake[3]: Leaving directory `/tmp/nxinst.z19411/inst/netxms-1.2.17/src/agent'
gmake[2]: *** [all-recursive] Error 1
gmake[2]: Leaving directory `/tmp/nxinst.z19411/inst/netxms-1.2.17/src'
gmake[1]: *** [all-recursive] Error 1
gmake[1]: Leaving directory `/tmp/nxinst.z19411/inst/netxms-1.2.17'
gmake: *** [all] Error 2
build failed, restarting old agent
----

Win update using procedure (CentOS-5.11-x64_1.2.17.log):
----
# service nxagentd stop
mv /opt/netxms /opt/_netxms
# cd /var/nxagentd
./nxagent-1.2.17.apkg
----

Thank you for your support

Br, Ales

- we are using configuration with zones
- when server receives trap following lines are logged:

[DD-MMM-YYYY HH:MM:SS] AgentConnectionEx::onTrap(): Received trap message from agent at a.b.c.d
[DD-MMM-YYYY HH:MM:SS] Cannot find node for IP address a.b.c.d

* this is the only trace on server that it received agent trap

- is there any known workaround for server in zone A to receive trap from agent in zone B

thank you,

Ales

What would be advised way to handle server to agent connection in case that server address
is source ip network address translated and translated IP is chosen dynamically from pool of
ip addresses. (static translation is not possible, not technical reason)

Practically:
SERVER (ip a.a.a.a/24) <-> SNAT (pool b.b.b.b/26) <- VPN -> agent (ip c.c.c.c/24)

Server translated address at time of connection could be b.b.b.1-b.b.b.127.

Solutions  (parameters Servers =, ControlServers =, MasterServers =)
* define all possible ip address in Servers = ...
* create dns A record with all possible ip addressees and use dns name as parameter value
* create feature request for parameters to accept CIDR notation

Possible solution:
* create feature request to add logic to client:
   if none of  Servers =, ControlServers =, MasterServers =  are defined
   and required authentication+encryption is configured
   allow configuration value 0.0.0.0
   as Servers =, ControlServers =, MasterServers = parameter value.
 
thank you,

Ales

Hello,

Is there any option to collect windows performance counters with
"#" in path name. I tried enclosing # in single/double quotes,
prefixing with "/", "\", defining path in agent config *WinPerf section
vs using PDH parameter.

In trace mode it looks like parsing stops at "#" character.

Agent used was: 0.2.26.1 (0.2.26).

Thank you,

Ales

Hello,

----------------------------
Current: .\src\agent\subagents\logwatch\eventlog.cpp is unable
to handle any other event log beside *System.

parser definition <file>*Security</file> for instance will try
to get EventMessageFile value from registry on path:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Eventlog\System\Security
----------------------------

----------------------------
<file>*Application</file>

should handle:
HKLM\SYSTEM\CurrentControlSet\Services\Eventlog\Application\Application
HKLM\SYSTEM\CurrentControlSet\Services\Eventlog\Application\<XY Application>
----------------------------

----------------------------
Could someone pretty please fix that and attach compiled logwatch.nsm
for v0.2.26.1 until next release.

Curent fix (for impatient grateful users) is to copy:
HKLM\SYSTEM\CurrentControlSet\Services\Eventlog\Security\Security
to
HKLM\SYSTEM\CurrentControlSet\Services\Eventlog\System\Security

for *Application hover you must copy relavant subkey of
HKLM\SYSTEM\CurrentControlSet\Services\Eventlog\Application\<xyz>
to
HKLM\SYSTEM\CurrentControlSet\Services\Eventlog\System\<xyz>
----------------------------

thank you,

Ales

Use this topic to post WinPerf counters.

Beside sensors readout from BCM agent could also provide:

- power on/off
- log collecting
- power state
- right click on node: Open Console (if KVM is provided)

Ales

Useful for:
- monitoring number of connections to specific TCP port
- checking if service is listening on specific port
- statistics of TCP connections table
- checking load on specific service as number of connections
- detecting TCP connection state based attacks and abnormalities

Requirements:
- agent on *nix system with net-snmp tools
- bash,grep,cat,snmpnetstat

Links:
http://net-snmp.sourceforge.net/docs/man/snmpnetstat.html
snmp_stat.sh attached

How-to:
1.
configure agent on node that has snmp connection to nodes you want to monitor:
ExternalParameter = TCP.Connections(*):/opt/netxms/scripts/snmp_stat.sh $1 $2 $3 $4

2.
add nodes you want to monitor to trusted nodes on that node

3.
parameters:
# $1 = hostname, monitored node ip
# $2 = community, monitored node snmp community string
# $3 = TCP state {ESTABLISHED|LISTEN|TIMEWAIT|TIMECLOSE|FINWAIT1|FINWAIT2|SYNSENT|SYNRECV}
# $4 = TCP port for detailed {ESTABLISHED|LISTEN}

4.
configure DCI's on node you want to monitor and specify node from 1.
as proxy node when creating DCI

5. examples DCI's:

# all established connections
TCP.Connections(192.168.1.1,public,ESTABLISHED)

# all established connections to TCP port 80
TCP.Connections(192.168.1.1,public,ESTABLISHED,80)

# will return 1 if service is listening on TCP port 25
TCP.Connections(192.168.1.1,public,LISTEN,25)


Feel free to use. Suggestions, spelling and bug fixes welcome.

Ales

When someone configures template, now has no practical way
to share it with others. By enabling exporting DCI(s),
configured in template, rightclick on Template, export,
a comunity sharing templates could be established.

This could also be useful:
- when establishing multiple netxms servers
- hand edit of exported file (xml ?)

Import would be done by creating empty Templat and Importing
exported file to it.

thank you,

A

When you right click on DCI  with configured thresholds there could be options:

- copy thresholds
- copy thresholds to other DCI

thank you,

A

hello,

It would be very functional if click on template object in object
browser would have beside Overview, Subordinates additional tab
"Last Values" that would show last values of DCI-s on Subordinates
the template is applyed to.

Most of my templates include redundant work to create custom graphs
for every template I apply to object.

In souch way you could quickly select three DCI-s from different
nodes on template object and graph data without need to: create graph,
add DCI-s from nodes, save graph, modify it when you apply template to
additional nodes.

thank you,

Ales

Ability to use jokers in File.Count(*)

It would be nice to have ability to use jokers
in File.Count parameter. For instance if you have

x:\folderA\subfolder\a\file1.txt
x:\folderA\subfolder\b\file2.txt
x:\folderA\subfolder\c\file3.txt
x:\folderA\subfolder\c\file4.txt


File.Count(x:\folderA\subfolder\*\*)
or
File.Count(x:\folderA\*\*\*)

Would return: 4


Or add additional DCI: File.CountEx(*)

File.CountEx(<root_path>,<depth>)

File.CountEx(x:\folderA,2)
Would return: 4


thank you,

Ales

hello,

I would like to get windows performance counter in agent.
As I checked in /src/agent/subagents/winperf/collect.cpp
additional windows performance counters can be pulled defining
them in config as:

<parameter name>:<counter path>:<number of samples>:<class>:<data type>:<description>

I tried test DCI as:

System.CPU.LoadAvg.test:"\\System\\Processor Queue Length":60:0:DCI_DT_FLOAT:"Average CPU load for last minute test"

I cannot figure out how to define this in: nxagentd.conf

thank you for your help,

Ales

- view: hide unmanaged objects
- user property: do not show unmanaged objects

- built in scheduler as you would have posibility to trigger action on agent not only
  using event procesing rules but define scheduled execution of on agent defined
  action