NetXMS Support Forum

Hi,

my setup involves several servers, all of which work fine.
However, recently, one of the agents has become unreachable.
After restarting the agent, the server could collect data, but the connection closed after around five minutes.
The agent logs show that the request from the server is accepted by the agent. What could be the issue here?

serverVersion: 6.1.1
agentVersion: 6.2.1
agentOS: Linux

The output from configuration poll:
[16.07.2026 10:31:44] **** Poll request sent to server ****
[16.07.2026 10:31:44] Poll request accepted, waiting for outstanding polling requests to complete...
[16.07.2026 10:31:44] Starting configuration of for node <nodename>
[16.07.2026 10:31:44] Checking node's capabilities...
[16.07.2026 10:31:44]    Checking NetXMS agent...
[16.07.2026 10:32:14]    Cannot connect to NetXMS agent (Request timeout)
[16.07.2026 10:32:14]    SNMP polling is disabled
[16.07.2026 10:32:14]    EtherNet/IP polling is disabled
[16.07.2026 10:32:14]    Modbus TCP polling is disabled
[16.07.2026 10:32:14] Capability check finished
[16.07.2026 10:32:14] Checking interface configuration...
[16.07.2026 10:32:14] Unable to get interface list from node
[16.07.2026 10:32:14] Interface configuration check finished
[16.07.2026 10:32:14] Checking node name
[16.07.2026 10:32:14] Node name cannot be interpreted as valid IP address, no need to resolve to host name
[16.07.2026 10:32:14] Updating general system hardware information
[16.07.2026 10:32:14] Processing template automatic apply rules
[16.07.2026 10:32:14] Processing container autobind rules
[16.07.2026 10:32:14] Processing cluster autobind rules
[16.07.2026 10:32:14] Finished configuration poll of node <nodename>
[16.07.2026 10:32:14] Node configuration was not changed after poll
[16.07.2026 10:32:14] **** Poll completed successfully ****
nxagentlog DebugLevel 8:
2026.07.16 08:20:21.884 *D* [comm               ] Incoming connection from <server-ip>
2026.07.16 08:20:21.884 *D* [comm               ] Connection from <server-ip> accepted
2026.07.16 08:20:21.884 *D* [comm.cs.4          ] Session registered (control=true, master=true, upgrade=true)
2026.07.16 08:20:21.884 *D* [comm.cs.4          ] Message dump:
  ** 000000 | 00 B6 50 20 00 00 00 10 00 00 00 00 00 00 00 00 | ..P ............
  ** code=0x00B6 (CMD_GET_NXCP_CAPS) version=5 flags=0x0020 id=0 size=16 numFields=0
  ** control message

2026.07.16 08:20:21.884 *D* [comm.cs.4          ] Received control message CMD_GET_NXCP_CAPS
2026.07.16 08:20:21.884 *D* [comm.cs.4          ] Using protocol version 5
2026.07.16 08:20:21.884 *D* [comm.cs.4          ] Sending message CMD_NXCP_CAPS (ID 0; size 16; uncompressed)
2026.07.16 08:20:21.884 *D* [comm.cs.4          ] Outgoing message dump:
  ** 000000 | 00 B7 50 20 00 00 00 10 00 00 00 00 05 00 00 00 | ..P ............
  ** code=0x00B7 (CMD_NXCP_CAPS) version=5 flags=0x0020 id=0 size=16 numFields=83886080
  ** control message

2026.07.16 08:20:21.884 *D* [comm.cs.4          ] Message dump:
  ** 000000 | 01 34 50 00 00 00 00 38 00 00 00 01 00 00 00 05 | .4P....8........
  ** 000010 | 00 00 00 97 03 01 00 01 00 00 02 17 03 01 00 01 | ................
  ** 000020 | 00 00 02 18 03 01 00 01 00 00 02 3A 03 01 00 01 | ...........:....
  ** 000030 | 00 00 03 12 03 01 00 01                         | ........
  ** code=0x0134 (CMD_SET_SERVER_CAPABILITIES) version=5 flags=0x0000 id=1 size=56 numFields=5
  ** 000000: [   151] INT16       1
  ** 000008: [   535] INT16       1
  ** 000010: [   536] INT16       1
  ** 000018: [   570] INT16       1
  ** 000020: [   786] INT16       1

2026.07.16 08:20:21.884 *D* [comm.cs.4          ] Received message CMD_SET_SERVER_CAPABILITIES (1)
2026.07.16 08:20:22.224 *D* [dc                 ] DataCollector: sleeping for 60 seconds
2026.07.16 08:20:47.229 *D* [comm.cs.1          ] Session disconnected by watchdog (timestamp = 110222679040, now = 110222799452)
2026.07.16 08:20:47.229 *D* [comm.cs.1          ] CommSession::disconnect()
2026.07.16 08:20:51.885 *D* [comm.cs.4          ] Communication channel closed by peer
2026.07.16 08:20:51.921 *D* [comm               ] Incoming connection from <server-ip>
2026.07.16 08:20:51.921 *D* [comm               ] Connection from <server-ip> accepted
2026.07.16 08:20:51.921 *D* [comm.cs.5          ] Session registered (control=true, master=true, upgrade=true)
2026.07.16 08:20:51.921 *D* [comm.cs.5          ] Message dump:
  ** 000000 | 00 B6 50 20 00 00 00 10 00 00 00 00 00 00 00 00 | ..P ............
  ** code=0x00B6 (CMD_GET_NXCP_CAPS) version=5 flags=0x0020 id=0 size=16 numFields=0
  ** control message

2026.07.16 08:20:51.921 *D* [comm.cs.5          ] Received control message CMD_GET_NXCP_CAPS
2026.07.16 08:20:51.921 *D* [comm.cs.5          ] Using protocol version 5
2026.07.16 08:20:51.921 *D* [comm.cs.5          ] Sending message CMD_NXCP_CAPS (ID 0; size 16; uncompressed)
2026.07.16 08:20:51.922 *D* [comm.cs.5          ] Outgoing message dump:
  ** 000000 | 00 B7 50 20 00 00 00 10 00 00 00 00 05 00 00 00 | ..P ............
  ** code=0x00B7 (CMD_NXCP_CAPS) version=5 flags=0x0020 id=0 size=16 numFields=83886080
  ** control message

2026.07.16 08:20:51.922 *D* [comm.cs.5          ] Message dump:
  ** 000000 | 01 34 50 00 00 00 00 38 00 00 00 01 00 00 00 05 | .4P....8........
  ** 000010 | 00 00 00 97 03 01 00 01 00 00 02 17 03 01 00 01 | ................
  ** 000020 | 00 00 02 18 03 01 00 01 00 00 02 3A 03 01 00 01 | ...........:....
  ** 000030 | 00 00 03 12 03 01 00 01                         | ........
  ** code=0x0134 (CMD_SET_SERVER_CAPABILITIES) version=5 flags=0x0000 id=1 size=56 numFields=5
  ** 000000: [   151] INT16       1
  ** 000008: [   535] INT16       1
  ** 000010: [   536] INT16       1
  ** 000018: [   570] INT16       1
  ** 000020: [   786] INT16       1

2026.07.16 08:20:51.922 *D* [comm.cs.5          ] Received message CMD_SET_SERVER_CAPABILITIES (1)
2026.07.16 08:20:52.229 *D* [comm.cs.2          ] Session disconnected by watchdog (timestamp = 110222679791, now = 110222804453)
2026.07.16 08:20:52.229 *D* [comm.cs.2          ] CommSession::disconnect()
2026.07.16 08:20:57.229 *D* [comm.cs.3          ] Session disconnected by watchdog (timestamp = 110222687458, now = 110222809453)
2026.07.16 08:20:57.230 *D* [comm.cs.3          ] CommSession::disconnect()
2026.07.16 08:21:21.922 *D* [comm.cs.5          ] Communication channel closed by peer
2026.07.16 08:21:22.224 *D* [dc                 ] DataCollector: sleeping for 60 seconds
2026.07.16 08:21:56.957 *D* [comm               ] Incoming connection from <server-ip>
2026.07.16 08:21:56.957 *D* [comm               ] Connection from <server-ip> accepted
2026.07.16 08:21:56.957 *D* [comm.cs.6          ] Session registered (control=true, master=true, upgrade=true)
Kind regards

Hi Tucson,

The agent is not actually unreachable — it accepts the connection, and the version mismatch is not the problem either. Your log shows what is going on:

- Each new connection completes the initial handshake (CMD_GET_NXCP_CAPS / CMD_NXCP_CAPS) — these control messages are processed directly by the session's network thread.
- The first real request, CMD_SET_SERVER_CAPABILITIES, is received but never answered. It is handed off to the agent's communication thread pool for processing, and that task never runs. The server waits 30 seconds for the response, reports "Request timeout", and closes the connection — you can see "Communication channel closed by peer" exactly 30 seconds after each connect.
- The "Session disconnected by watchdog" lines are not the cause: those are older sessions from previous failed attempts being cleaned up after the 120-second idle timeout (the timestamp deltas in those lines are all ~120 s).

This means all worker threads of the communication thread pool are stuck in requests that never complete. Typical causes are an ExternalParameter / ExternalMetric script that hangs, or a FileSystem.* metric blocking on a dead network mount (stale NFS/CIFS). Once all workers are blocked, the agent still accepts connections but cannot process any command — which matches your "works after restart, dies a few minutes later" pattern: after restart the pool is fresh, and it dies once the server's polls have handed it enough hanging requests. There is also an agent-side bug that prevents the pool from recovering in this situation — I have filed it as https://github.com/netxms/netxms/issues/3436.

To find what is blocking the threads on your side, please run on the agent host:

1. gdb -p $(pgrep nxagentd) --batch -ex 'thread apply all bt' > nxagentd-threads.txt and attach the output (install gdb if needed; eu-stack -p <pid> works too). The stacks of the threads named $COMM/WRK will show exactly where they are stuck.
2. nxget 127.0.0.1 Agent.Uptime — if this hangs, it confirms the thread pool is stuck.
3. df — if it hangs, you have a stale network mount, which is the common cause.

Also check whether your agent config has ExternalParameter / ExternalMetric entries, and whether those scripts can block (waiting on network, locks, prompts). Restarting the agent is a temporary workaround, but it will wedge again until the underlying hang is fixed.