Cannot connect to NetXMS agent (Internal error)

Started by eliassal, October 29, 2021, 12:26:30 PM

Previous topic - Next topic

eliassal

I am following the video "NetXMS client and basic server settings" at https://wiki.netxms.org/wiki/NetXMS_Video_Tutorial_Series which are a little bit out-of-date, when I try to do poll > configuration
I always get


Quote[29.10.2021 11:14:26] Starting configuration poll for node WIN10-NETXMS
[29.10.2021 11:14:26] Checking node's capabilities...
[29.10.2021 11:14:26]    Checking NetXMS agent...
[29.10.2021 11:14:26]    Cannot connect to NetXMS agent (Internal error)
in spite of the fact that the NetXMS agent service is up and running. NetXMS is running on windows 10

Filipp Sudanov

Most probably this means, that the address of your netxms server is not in MasterServers line of agent configuration.

If the agent is on the same machine as the server, the default configuration for the agent includes MasterServers=127.0.0.1, but the node that is automatically created refers to external address of the machine.

eliassal

Hi Filipp, not sure I understnad what I should do but here ios the config for nxagentd.conf

QuoteMasterServers = 127.0.0.1,::1
ConfigIncludeDir = C:\NetXMS\etc\nxagentd.conf.d
LogFile = C:\NetXMS\log\nxagentd.log
FileStore = C:\NetXMS\var
SubAgent = portcheck.nsm
SubAgent = ssh.nsm
SubAgent = winperf.nsm
SubAgent = wmi.nsm
should I change this to something else? for example change 127.0.0.1 to the ip of the NetXMS server?

Filipp Sudanov

In the properties of node, which you are trying to poll under Communications there is "Primary host name". This is the address of agent to which netxms server is connecting. Take this address and add it to MasterServer line of agent config after comma.



kopecvi2

Hello I have the same issue. Does anybody know how to solve it?
The Agent is connected to NetXMS server via wireguard.

[Interface]
PrivateKey = XXXXXX=
Address = 10.220.59.8/24,fd11:5ee:bad:c0de::adc:3b08/64
#DNS = 9.9.9.9, 149.112.112.112
DNS = 46.36.40.255

[Peer]
PublicKey = YYYYYY=
PresharedKey = YYYYY=
Endpoint = 176.102.X.X:51820
AllowedIPs = 10.220.59.1/32, 10.220.59.2, fd11:5ee:bad:c0de::adc:3b01/64

There is how I have allowed ports in UFW
ufw status numbered
Status: active

     To                         Action      From
     --                         ------      ----
[ 1] 22/tcp                     LIMIT IN    Anywhere                
[ 2] 53                         ALLOW IN    Anywhere                
[ 3] 25/tcp                     ALLOW IN    Anywhere                
[ 4] 465/tcp                    ALLOW IN    Anywhere                
[ 5] 587/tcp                    ALLOW IN    Anywhere                
[ 6] 993/tcp                    ALLOW IN    Anywhere                
[ 7] 995/tcp                    ALLOW IN    Anywhere                
[ 8] 4190/tcp                   ALLOW IN    Anywhere                
[ 9] 80/tcp                     ALLOW IN    Anywhere                
[10] 443                        ALLOW IN    Anywhere                
[11] 4700                       ALLOW IN    10.220.59.1              
[12] 10.220.59.1 4700           ALLOW IN    Anywhere                
[13] 10.220.59.1 5161           ALLOW IN    Anywhere                
[14] 5161                       ALLOW IN    10.220.59.1              
[15] 22/tcp (v6)                LIMIT IN    Anywhere (v6)            
[16] 53 (v6)                    ALLOW IN    Anywhere (v6)            
[17] 25/tcp (v6)                ALLOW IN    Anywhere (v6)            
[18] 465/tcp (v6)               ALLOW IN    Anywhere (v6)            
[19] 587/tcp (v6)               ALLOW IN    Anywhere (v6)            
[20] 993/tcp (v6)               ALLOW IN    Anywhere (v6)            
[21] 995/tcp (v6)               ALLOW IN    Anywhere (v6)            
[22] 4190/tcp (v6)              ALLOW IN    Anywhere (v6)            
[23] 80/tcp (v6)                ALLOW IN    Anywhere (v6)            
[24] 443 (v6)                   ALLOW IN    Anywhere (v6)  

And there is conf for Agent:
# Log File
LogFile=/var/log/nxagentd.log
#[CORE]
# IP white list, can contain multiple records separated by comma.
# CIDR notation supported for subnets.
MasterServers= 127.0.0.1/32, 10.220.59.1:4700, 10.220.59.8


SubAgent=ssh.nsm
SubAgent = portcheck.nsm
SubAgent = filemgr.nsm

[filemgr]
RootFolder = /var/log

systemctl status nxagentd
● netxms-agent.service - NetXMS agent
     Loaded: loaded (/lib/systemd/system/netxms-agent.service; enabled; vendor preset: enabled)
     Active: active (running) since Sun 2025-11-23 11:46:05 CET; 28min ago
   Main PID: 941898 (nxagentd)
      Tasks: 38 (limit: 4558)
     Memory: 9.3M
        CPU: 1.079s
     CGroup: /system.slice/netxms-agent.service
             └─941898 /usr/bin/nxagentd -S

Nov 23 11:46:05 box.littlehill.xyz systemd[1]: Started NetXMS agent.

Filipp Sudanov

Add
DebugLevel = 6
in agent's config and restart the agent. They try running configuration poll for that node. Agent's log should have information from which IP it is getting the connection - it's probably some address that you don't have in MasterServers line.

The other option is to configure tunnel connection - this requires some effort to create certificates for the server, but then connection will be established from agent to the server (https://netxms.org/documentation/adminguide/server-management.html#server-configuration-for-agent-to-server-connection-tunnel-connection)

kopecvi2

#6
Thank you for your reply I add this line to /etc/nxagentd.conf:
cat /etc/nxagentd.conf
# Log File
LogFile=/var/log/nxagentd.log
#[CORE]
# IP white list, can contain multiple records separated by comma.
# CIDR notation supported for subnets.
MasterServers= 127.0.0.1/32, 10.220.59.1:4700, 10.220.59.8
DebugLevel = 6

SubAgent=ssh.nsm
SubAgent = portcheck.nsm
SubAgent = filemgr.nsm

[filemgr]
RootFolder = /var/log

But there was'nt nothing special in the output.
root@box:/home/kopecek# systemctl status nxagentd
● netxms-agent.service - NetXMS agent
     Loaded: loaded (/lib/systemd/system/netxms-agent.service; enabled; vendor preset: enabled)
     Active: active (running) since Mon 2025-11-24 15:22:29 CET; 2s ago
   Main PID: 1117491 (nxagentd)
      Tasks: 38 (limit: 4558)
     Memory: 9.1M
        CPU: 29ms
     CGroup: /system.slice/netxms-agent.service
             └─1117491 /usr/bin/nxagentd -S

Nov 24 15:22:29 box.kopecek.org systemd[1]: Started NetXMS agent.

And here is an ouput from NetXMS client of Poll scan:
[24.11.2025 15:18:37] **** Poll request sent to server ****
[24.11.2025 15:18:37] Poll request accepted, waiting for outstanding polling requests to complete...
[24.11.2025 15:18:37] Starting configuration of for node box.kopecek.org
[24.11.2025 15:18:37] Checking node's capabilities...
[24.11.2025 15:18:37]    Checking NetXMS agent...
[24.11.2025 15:18:47]    Cannot connect to NetXMS agent (Connect failed)
[24.11.2025 15:18:47]    Checking SNMP...
[24.11.2025 15:19:00]    No response from SNMP agent
[24.11.2025 15:19:00]    Checking SSH connectivity...
[24.11.2025 15:19:00]    Cannot connect to SSH
[24.11.2025 15:19:00]    Checking direct VNC connectivity...
[24.11.2025 15:19:02]    Cannot connect to VNC
[24.11.2025 15:19:02]    Checking EtherNet/IP...
[24.11.2025 15:19:07]    Cannot establish EtherNet/IP connection or get device identity (CONNECT FAILED)
[24.11.2025 15:19:07]    Checking Modbus TCP...
[24.11.2025 15:19:09]    Device does not respond to Modbus TCP request (Communication failure)
[24.11.2025 15:19:09] Capability check finished
[24.11.2025 15:19:09] Checking interface configuration...
[24.11.2025 15:19:09] Unable to get interface list from node
[24.11.2025 15:19:14] Interface configuration check finished
[24.11.2025 15:19:14] Checking node name
[24.11.2025 15:19:14] Node name cannot be interpreted as valid IP address, no need to resolve to host name
[24.11.2025 15:19:14] Updating general system hardware information
[24.11.2025 15:19:14] Processing template automatic apply rules
[24.11.2025 15:19:14] Processing container autobind rules
[24.11.2025 15:19:14] Processing cluster autobind rules
[24.11.2025 15:19:14] Finished configuration poll of node box.kopecek.org
[24.11.2025 15:19:14] Node configuration was not changed after poll
[24.11.2025 15:19:14] **** Poll completed successfully ****
But there is problems with connectivity now.

edit: i lost wireguard connection due to absence keep alive. But problem is still same (No connection to agent from server - [24.11.2025 16:05:27]    Cannot connect to NetXMS agent (Internal error)).

kopecvi2

And here is actual ouput from log file on the agent side..
2025.11.24 15:22:29.251 *I* [logger             ] Log file opened (rotation policy 2, max size 4194304)
2025.11.24 15:22:29.251 *I* [logger             ] Log rotation policy set to 2 (size=16777216, count=4)
2025.11.24 15:22:29.251 *I* [startup            ] Core agent version 5.2.7 (build tag 5.2-484-g5df9cb5911)
2025.11.24 15:22:29.251 *I* [startup            ] System time zone is CET+01CEST
2025.11.24 15:22:29.251 *I* [config             ] Additional configuration files was loaded from /etc/nxagentd.conf.d
2025.11.24 15:22:29.251 *I* [logger             ] Debug level set to 6
2025.11.24 15:22:29.251 *I* [config             ] Main configuration file: /etc/nxagentd.conf
2025.11.24 15:22:29.251 *I* [config             ] Configuration tree:
2025.11.24 15:22:29.251 *I* [config             ] config
2025.11.24 15:22:29.252 *I* [config             ]  +- CORE
2025.11.24 15:22:29.252 *I* [config             ]  |   +- LogFile
2025.11.24 15:22:29.252 *I* [config             ]  |   |    value: /var/log/nxagentd.log
2025.11.24 15:22:29.252 *I* [config             ]  |   +- MasterServers
2025.11.24 15:22:29.252 *I* [config             ]  |   |    value: 127.0.0.1/32, 10.220.59.1:4700, 10.220.59.8
2025.11.24 15:22:29.252 *I* [config             ]  |   +- DebugLevel
2025.11.24 15:22:29.252 *I* [config             ]  |   |    value: 6
2025.11.24 15:22:29.252 *I* [config             ]  |   +- SubAgent
2025.11.24 15:22:29.252 *I* [config             ]  |        value: ssh.nsm
2025.11.24 15:22:29.252 *I* [config             ]  |        value: portcheck.nsm
2025.11.24 15:22:29.252 *I* [config             ]  |        value: filemgr.nsm
2025.11.24 15:22:29.252 *I* [config             ]  +- filemgr
2025.11.24 15:22:29.252 *I* [config             ]      +- RootFolder
2025.11.24 15:22:29.252 *I* [config             ]           value: /var/log
2025.11.24 15:22:29.252 *I* [startup            ] PATH = /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin
2025.11.24 15:22:29.252 *I* [startup            ] LD_LIBRARY_PATH =
2025.11.24 15:22:29.252 *I* [startup            ] Data directory: /var/lib/netxms
2025.11.24 15:22:29.252 *I* [startup            ] File store: /tmp
2025.11.24 15:22:29.252 *D* [startup            ] Effective user ID 0
2025.11.24 15:22:29.252 *D* [startup            ] Effective group ID 0
2025.11.24 15:22:29.252 *D* [startup            ] Configuration policy directory: /var/lib/netxms/config_ap/
2025.11.24 15:22:29.252 *D* [startup            ] Log parser policy directory: /var/lib/netxms/logparser_ap/
2025.11.24 15:22:29.252 *D* [startup            ] User agent policy directory: /var/lib/netxms/useragent_ap/
2025.11.24 15:22:29.252 *D* [startup            ] Certificate directory: /var/lib/netxms/certificates/
2025.11.24 15:22:29.252 *D* [startup            ] CRL directory: /var/lib/netxms/crl/
2025.11.24 15:22:29.253 *D* [crypto             ] Validating ciphers
2025.11.24 15:22:29.253 *D* [crypto             ]    AES-256 enabled
2025.11.24 15:22:29.253 *D* [crypto             ]    Blowfish-256 disabled (validation failed)
2025.11.24 15:22:29.253 *D* [crypto             ]    IDEA disabled (config)
2025.11.24 15:22:29.253 *D* [crypto             ]    3DES enabled
2025.11.24 15:22:29.253 *D* [crypto             ]    AES-128 enabled
2025.11.24 15:22:29.253 *D* [crypto             ]    Blowfish-128 disabled (validation failed)
2025.11.24 15:22:29.253 *I* [crypto             ] Crypto library initialized (OpenSSL 3.0.2 15 Mar 2022)
2025.11.24 15:22:29.253 *D* [threads.pool       ] Thread pool TIMER initialized (min=2, max=16)
2025.11.24 15:22:29.253 *D* [startup            ] Subagent API initialized
2025.11.24 15:22:29.254 *D* [db.drv.sqlite      ] SQLite version 3.37.2
2025.11.24 15:22:29.254 *I* [db.drv             ] Database driver "sqlite.ddr" loaded and initialized successfully
2025.11.24 15:22:29.255 *D* [db.conn            ] New DB connection opened: handle=0x7eff9df629a0
2025.11.24 15:22:29.255 *D* [db.query           ] DB Library: global long running query threshold set to 250
2025.11.24 15:22:29.259 *I* [db.agent           ] Local database opened successfully
2025.11.24 15:22:29.260 *I* [startup            ] Agent ID is 576b4d1e-fa19-42a1-bf32-5080f654d46b
2025.11.24 15:22:29.260 *I* [startup            ] Local host name is "box"
2025.11.24 15:22:29.260 *I* [startup            ] Using system name "box"
2025.11.24 15:22:29.260 *D* [comm               ] Maximum number of sessions set to 32
2025.11.24 15:22:29.260 *D* [threads.pool       ] Thread pool COMM initialized (min=8, max=512)
2025.11.24 15:22:29.260 *D* [threads.pool       ] Thread pool WEBSVC initialized (min=4, max=64)
2025.11.24 15:22:29.260 *D* [comm               ] Added server access record 127.0.0.1/32 (control=true, master=true)
2025.11.24 15:22:29.260 *D* [comm               ] Added server access record 10.220.59.1:4700 (control=true, master=true)
2025.11.24 15:22:29.260 *D* [comm               ] Added server access record 10.220.59.8 (control=true, master=true)
2025.11.24 15:22:29.260 *I* [startup            ] Arbitrary command execution disabled
2025.11.24 15:22:29.261 *D* [smbios             ] System manufacturer: QEMU
2025.11.24 15:22:29.261 *D* [smbios             ] System product name: Standard PC (i440FX + PIIX, 1996)
2025.11.24 15:22:29.261 *D* [smbios             ] System serial number:
2025.11.24 15:22:29.261 *D* [smbios             ] System UUID: 5b784744-9231-4504-8aaa-5c5cfd09c04a
2025.11.24 15:22:29.261 *D* [smbios             ] Baseboard manufacturer:
2025.11.24 15:22:29.261 *D* [smbios             ] Baseboard product name:
2025.11.24 15:22:29.261 *D* [smbios             ] Baseboard serial number:
2025.11.24 15:22:29.261 *D* [smbios             ] Baseboard type:
2025.11.24 15:22:29.261 *D* [smbios             ] Baseboard version:
2025.11.24 15:22:29.261 *D* [smbios             ] BIOS vendor: SeaBIOS
2025.11.24 15:22:29.261 *D* [smbios             ] BIOS version: rel-1.16.3-0-ga6ed6b701f0a-prebuilt.qemu.org
2025.11.24 15:22:29.261 *D* [smbios             ] BIOS address: E800
2025.11.24 15:22:29.261 *W* [linux              ] Cannot open Intel RAPL sysfs root (No such file or directory)
2025.11.24 15:22:29.261 *D* [linux              ] Using /sys/block to distinguish devices from partitions
2025.11.24 15:22:29.262 *D* [linux              ] CPU usage collector thread started
2025.11.24 15:22:29.262 *D* [linux              ] ParseIoStat(): new device added (name=loop0 isRealDevice=1)
2025.11.24 15:22:29.262 *D* [linux              ] ParseIoStat(): new device added (name=loop1 isRealDevice=1)
2025.11.24 15:22:29.262 *D* [linux              ] ParseIoStat(): new device added (name=loop2 isRealDevice=1)
2025.11.24 15:22:29.262 *D* [linux              ] ParseIoStat(): new device added (name=loop3 isRealDevice=1)
2025.11.24 15:22:29.262 *D* [linux              ] ParseIoStat(): new device added (name=loop4 isRealDevice=1)
2025.11.24 15:22:29.262 *I* [linux              ] Unable to parse /proc/drbd, DRBD data collector will not start
2025.11.24 15:22:29.262 *D* [linux              ] ParseIoStat(): new device added (name=loop5 isRealDevice=1)
2025.11.24 15:22:29.262 *D* [linux              ] ParseIoStat(): new device added (name=loop6 isRealDevice=1)
2025.11.24 15:22:29.262 *D* [linux              ] ParseIoStat(): new device added (name=loop7 isRealDevice=1)
2025.11.24 15:22:29.262 *D* [linux              ] ParseIoStat(): new device added (name=sr0 isRealDevice=1)
2025.11.24 15:22:29.262 *D* [linux              ] ParseIoStat(): new device added (name=vda isRealDevice=1)
2025.11.24 15:22:29.262 *D* [linux              ] ParseIoStat(): new device added (name=vda1 isRealDevice=0)
2025.11.24 15:22:29.265 *I* [subagents          ] Subagent "Linux" (/usr/lib/x86_64-linux-gnu/netxms/linux.nsm) loaded successfully (version 5.2.7)
2025.11.24 15:22:29.265 *D* [ssh                ] Using libssh version 0.9.6/openssl/zlib
2025.11.24 15:22:29.265 *D* [ssh                ] Workaround for ssh_channel_read bug is disabled
2025.11.24 15:22:29.265 *D* [ssh                ] InitializeSessionPool: connection pool initialized
2025.11.24 15:22:29.265 *I* [subagents          ] Subagent "SSH" (/usr/lib/x86_64-linux-gnu/netxms/ssh.nsm) loaded successfully (version 5.2.7)
2025.11.24 15:22:29.265 *W* [subagents          ] Subagent module portcheck.nsm was substituted with netsvc.nsm (PORTCHECK subagent is superseded by NETSVC subagent)
2025.11.24 15:22:29.266 *D* [init.curl          ] cURL initialized (version: libcurl/7.81.0 OpenSSL/3.0.2 zlib/1.2.11 brotli/1.0.9 zstd/1.4.8 libidn2/2.3.2 libpsl/0.21.0 (+libidn2/2.3.2) libssh/0.9.6/openssl/zlib nghttp2/1.43.0 librtmp/2.3 OpenLDAP/2.5.19)
2025.11.24 15:22:29.266 *D* [init.curl          ] OpenSSL 3 is used as cURL SSL backend
2025.11.24 15:22:29.266 *D* [init.curl          ] cURL supported protocols: dict file ftp ftps gopher gophers http https imap imaps ldap ldaps mqtt pop3 pop3s rtmp rtsp scp sftp smb smbs smtp smtps telnet tftp
2025.11.24 15:22:29.267 *I* [subagents          ] Subagent "NETSVC" (/usr/lib/x86_64-linux-gnu/netxms/netsvc.nsm) loaded successfully (version 5.2.7)
2025.11.24 15:22:29.267 *I* [filemgr            ] Added file manager root directory "/var/log" (R/W)
2025.11.24 15:22:29.267 *I* [filemgr            ] File manager subagent initialized
2025.11.24 15:22:29.267 *I* [subagents          ] Subagent "FILEMGR" (/usr/lib/x86_64-linux-gnu/netxms/filemgr.nsm) loaded successfully (version 5.2.7)
2025.11.24 15:22:29.267 *I* [startup            ] System hardware ID is 25DF3B493FC58A4722956B98827A6ED1A07439B7
2025.11.24 15:22:30.269 *D* [threads.pool       ] Thread pool PROCEXEC initialized (min=1, max=16)
2025.11.24 15:22:30.269 *D* [notifications      ] StartNotificationProcessor: Loaded 0 servers
2025.11.24 15:22:30.269 *D* [notifications      ] Notification housekeeper started
2025.11.24 15:22:30.269 *D* [notifications      ] Notification processor started
2025.11.24 15:22:30.269 *D* [notifications      ] Notification housekeeper execution completed
2025.11.24 15:22:30.269 *D* [threads.pool       ] Thread pool DATACOLL initialized (min=4, max=64)
2025.11.24 15:22:30.270 *D* [dc                 ] Data collection scheduler thread started
2025.11.24 15:22:30.270 *D* [dc                 ] Data sender thread started
2025.11.24 15:22:30.270 *D* [dc                 ] Database writer thread started
2025.11.24 15:22:30.270 *D* [dc                 ] Data reconciliation thread started (block size 1024, timeout 60000 ms)
2025.11.24 15:22:30.270 *I* [sa                 ] Session agent connector disabled
2025.11.24 15:22:30.270 *D* [comm.listener      ] SocketListener/ProxyHeartbeat: Trying to bind on 0.0.0.0:4700/udp
2025.11.24 15:22:30.270 *D* [comm.listener      ] SocketListener/ProxyHeartbeat: Trying to bind on [::]:4700/udp
2025.11.24 15:22:30.270 *D* [comm               ] Trying to bind on 0.0.0.0:4700
2025.11.24 15:22:30.270 *D* [                   ] Trying to bind on [::]:4700
2025.11.24 15:22:30.270 *I* [comm               ] Listening on socket 0.0.0.0:4700
2025.11.24 15:22:30.270 *I* [comm               ] Listening on socket :::4700
2025.11.24 15:22:30.270 *D* [                   ] NamedPipeListener(nxagentd.push): waiting for connection
2025.11.24 15:22:30.270 *I* [comm               ] Control connector is disabled
2025.11.24 15:22:30.270 *D* [                   ] NamedPipeListener(nxagentd.events): waiting for connection
2025.11.24 15:22:30.270 *D* [tunnel             ] No tunnels configured, tunnel manager will not start
2025.11.24 15:22:31.271 *I* [filemon            ] Path list for file monitor is empty
2025.11.24 15:22:31.271 *I* [                   ] NetXMS Agent started
2025.11.24 15:27:31.271 *D* [websvc             ] WebServiceHousekeeper(): running cache entry check
2025.11.24 15:32:31.272 *D* [websvc             ] WebServiceHousekeeper(): running cache entry check
2025.11.24 15:34:51.846 *D* [comm               ] Incoming connection from 10.220.59.1
2025.11.24 15:34:51.846 *D* [comm               ] Connection from 10.220.59.1 rejected
2025.11.24 15:37:31.272 *D* [websvc             ] WebServiceHousekeeper(): running cache entry check
2025.11.24 15:37:47.359 *D* [comm               ] Incoming connection from 10.220.59.1
2025.11.24 15:37:47.359 *D* [comm               ] Connection from 10.220.59.1 rejected

Filipp Sudanov

Here's the relevant part in agent log:

2025.11.24 15:34:51.846 *D* [comm               ] Incoming connection from 10.220.59.1
2025.11.24 15:34:51.846 *D* [comm               ] Connection from 10.220.59.1 rejected

Ah, looks like
MasterServers= 127.0.0.1/32, 10.220.59.1:4700, 10.220.59.8
in agent config is wrong, there should be no port number in it:
MasterServers= 127.0.0.1/32, 10.220.59.1, 10.220.59.8

kopecvi2

Thank You very much, now it is working. Sorry for my stupid mistake. :-)