Too many communication sessions open - unable to accept new connection

Started by dtk33d, August 30, 2023, 11:40:45 AM

Previous topic - Next topic

dtk33d

Hello,

i noticed in agentd log:


Too many communication sessions open - unable to accept new connection


and i noticed that my ssh dci's dont update ...

i read the manual and forum but could not find a solution.


Filipp Sudanov

Try adding
MaxSessions = 100
to agent configuration file (default for this setting is 32 - https://netxms.org/documentation/adminguide/appendix.html#agent-configuration-file-nxagentd-conf)

Are you collecting a lot of ssh metrics, do these metrics take long time to return value? Or is this agent a proxy or loaded with some other tasks? I just wonder why many sessions are used at the same time.

dtk33d

Hello,
yes i have a lot of ssh metrics configured.
i tried to set MaxSessions to 100 and 500 but the log is full of error message.

it is no agent proxy, agent is netxms server itself.


Filipp Sudanov

Have you restarted the agent after adding MaxSessions parameter?

dtk33d

yes

agent conf:

# Log File
LogFile=/var/log/nxagentd

# IP white list, can contain multiple records separated by comma.
# CIDR notation supported for subnets.
MasterServers=127.0.0.1
SubAgent=ssh.nsm
SubAgent=ping.nsm
[SSH]
ConfigFile = /etc/nxagentd-ssh-config
MaxSessions=500



even tried adding MaxSessions=500 in /etc/nxagentd-ssh-config....

no luck


Filipp Sudanov

How much time does it take since agent restart to consume all the sessions?

In out-of-the-box template for the agent there are metrics
Agent.ActiveConnections
Agent.AcceptedConnections

please show graphs for these for the time since agent start till the sessions run out

dtk33d

2023.10.02 18:32:48.183 *I* [linux              ] Unable to parse /proc/drbd, DRBD data collector will not start
2023.10.02 18:32:48.184 *I* [subagents          ] Subagent "Linux" (/usr/lib/x86_64-linux-gnu/netxms/linux.nsm) loaded successfully (version 4.4.2)
2023.10.02 18:32:48.187 *I* [subagents          ] Subagent "SSH" (/usr/lib/x86_64-linux-gnu/netxms/ssh.nsm) loaded successfully (version 4.4.2)
2023.10.02 18:32:48.187 *I* [subagents          ] Subagent "PING" (/usr/lib/x86_64-linux-gnu/netxms/ping.nsm) loaded successfully (version 4.4.2)
2023.10.02 18:32:48.188 *I* [startup            ] System hardware IDxxxxxxxxxxxxxx
2023.10.02 18:32:49.193 *I* [comm              ] Listening on socket 0.0.0.0:4700
2023.10.02 18:32:49.193 *I* [comm              ] Listening on socket :::4700
2023.10.02 18:32:49.193 *I* [comm              ] Control connector is disabled
2023.10.02 18:32:50.194 *I* [filemon            ] Path list for file monitor is empty
2023.10.02 18:32:50.194 *I* [                  ] NetXMS Agent started
2023.10.02 18:33:19.588 *W* [comm              ] Too many communication sessions open - unable to accept new connection
2023.10.02 18:33:19.588 *W* [comm              ] Too many communication sessions open - unable to accept new connection
2023.10.02 18:33:21.027 *W* [comm              ] Too many communication sessions open - unable to accept new connection
2023.10.02 18:33:24.592 *W* [comm              ] Too many communication sessions open - unable to accept new connection
2023.10.02 18:33:24.592 *W* [comm              ] Too many communication sessions open - unable to accept new connection
2023.10.02 18:33:24.592 *W* [comm              ] Too many communication sessions open - unable to accept new connection
2023.10.02 18:33:24.592 *W* [comm              ] Too many communication sessions open - unable to accept new connection
2023.10.02 18:33:25.038 *W* [comm              ] Too many communication sessions open - unable to accept new connection
2023.10.02 18:33:25.042 *W* [comm              ] Too many communication sessions open - unable to accept new connection
2023.10.02 18:33:25.044 *W* [comm              ] Too many communication sessions open - unable to accept new connection
2023.10.02 18:33:25.074 *W* [comm              ] Too many communication sessions open - unable to accept new connection
2023.10.02 18:33:35.065 *W* [comm              ] Too many communication sessions open - unable to accept new connection
2023.10.02 18:33:35.067 *W* [comm              ] Too many communication sessions open - unable to accept new connection
2023.10.02 18:33:35.069 *W* [comm              ] Too many communication sessions open - unable to accept new connection
2023.10.02 18:33:36.086 *W* [comm              ] Too many communication sessions open - unable to accept new connection
2023.10.02 18:33:39.568 *W* [comm              ] Too many communication sessions open - unable to accept new connection

Filipp Sudanov

Please add
DebugLevel = 6
to agent config file and share full agent log since it's start till sessions run out. You can share the log in private message if needed.