Nxagentd binding problem

Started by IT-Reszleg, June 19, 2018, 12:34:27 PM

Previous topic - Next topic

IT-Reszleg

Hi guys,

We use Netxms for monitoring mikrotik, synology devices and servers, and it works perfectly. We love it! :) But in this week i installed the agent to a Windows server 2012R2 x64 and a Windows 7 Pro x64 machine. In booth case the agent installed without errors, and run on background, but cannot accessible.

"nxagentd -D 1" says:

....
2018.06.19 11:20:29.040 Trying to bind on 0.0.0.0:4700
2018.06.19 11:20:29.040 Unable to bind socket: Only one usage of each socket address (protocol/network address/port) is normally permitted.
2018.06.19 11:20:29.040 Trying to bind on 127.0.0.1:28180
2018.06.19 11:20:29.040 Trying to bind on [::]:4700
2018.06.19 11:20:29.040 Unable to bind socket: Only one usage of each socket address (protocol/network address/port) is normally permitted.
2018.06.19 11:20:29.040 Unable to bind socket: Only one usage of each socket address (protocol/network address/port) is normally permitted.
2018.06.19 11:20:29.040 Session agent connector terminated (socket error)
.....


Windows system event log says:
Event ID: 9
Event source: NetXMS Win32 Agent
Error: "Unable to bind socket: Only one usage of each socket address (protocol/network address/port) is normally permitted."


"Netstat -an" says:

Active Connections

  Proto  Local Address          Foreign Address        State
  TCP    0.0.0.0:135            0.0.0.0:0              LISTENING
  TCP    0.0.0.0:445            0.0.0.0:0              LISTENING
  TCP    0.0.0.0:2179           0.0.0.0:0              LISTENING
  TCP    0.0.0.0:2301           0.0.0.0:0              LISTENING
  TCP    0.0.0.0:2381           0.0.0.0:0              LISTENING
  TCP    0.0.0.0:3389           0.0.0.0:0              LISTENING
  TCP    0.0.0.0:4700           0.0.0.0:0              LISTENING
  TCP    0.0.0.0:5985           0.0.0.0:0              LISTENING
  TCP    0.0.0.0:47001          0.0.0.0:0              LISTENING
  TCP    0.0.0.0:49152          0.0.0.0:0              LISTENING
  TCP    0.0.0.0:49153          0.0.0.0:0              LISTENING
  TCP    0.0.0.0:49154          0.0.0.0:0              LISTENING
  TCP    0.0.0.0:49155          0.0.0.0:0              LISTENING
  TCP    0.0.0.0:49157          0.0.0.0:0              LISTENING
  TCP    0.0.0.0:49158          0.0.0.0:0              LISTENING
  TCP    10.10.0.1:139          0.0.0.0:0              LISTENING
  TCP    10.10.0.1:3389         10.10.0.254:50669      ESTABLISHED
  TCP    10.10.0.1:49161        178.255.154.14:5938    ESTABLISHED
  TCP    127.0.0.1:5939         0.0.0.0:0              LISTENING
  TCP    127.0.0.1:5939         127.0.0.1:49162        ESTABLISHED
  TCP    127.0.0.1:9656         0.0.0.0:0              LISTENING
  TCP    127.0.0.1:28180        0.0.0.0:0              LISTENING
  TCP    127.0.0.1:49162        127.0.0.1:5939         ESTABLISHED
  TCP    127.0.0.1:49164        127.0.0.1:49165        ESTABLISHED
  TCP    127.0.0.1:49165        127.0.0.1:49164        ESTABLISHED
  TCP    [::]:135               [::]:0                 LISTENING
  TCP    [::]:445               [::]:0                 LISTENING
  TCP    [::]:2179              [::]:0                 LISTENING
  TCP    [::]:2301              [::]:0                 LISTENING
  TCP    [::]:2381              [::]:0                 LISTENING
  TCP    [::]:3389              [::]:0                 LISTENING
  TCP    [::]:4700              [::]:0                 LISTENING
  TCP    [::]:5985              [::]:0                 LISTENING
  TCP    [::]:47001             [::]:0                 LISTENING
  TCP    [::]:49152             [::]:0                 LISTENING
  TCP    [::]:49153             [::]:0                 LISTENING
  TCP    [::]:49154             [::]:0                 LISTENING
  TCP    [::]:49155             [::]:0                 LISTENING
  TCP    [::]:49157             [::]:0                 LISTENING
  TCP    [::]:49158             [::]:0                 LISTENING
  TCP    [fe80::a049:cfb7:f7c5:3418%15]:2179  [fe80::a049:cfb7:f7c5:3418%15]:49565  ESTABLISHED
  TCP    [fe80::a049:cfb7:f7c5:3418%15]:49565  [fe80::a049:cfb7:f7c5:3418%15]:2179  ESTABLISHED
  UDP    0.0.0.0:161            *:*
  UDP    0.0.0.0:500            *:*
  UDP    0.0.0.0:3389           *:*
  UDP    0.0.0.0:4500           *:*
  UDP    0.0.0.0:5355           *:*
  UDP    0.0.0.0:50992          *:*
  UDP    10.10.0.1:137          *:*
  UDP    10.10.0.1:138          *:*
  UDP    10.10.0.1:5353         *:*
  UDP    [::]:161               *:*
  UDP    [::]:500               *:*
  UDP    [::]:3389              *:*
  UDP    [::]:4500              *:*
  UDP    [::]:5355              *:*
  UDP    [::]:50993             *:*
  UDP    [::1]:5353             *:*


We tried agent version 2.23 and 2.26 but same result. Booth machine clean installed, no other service or software. I have no idea. Maybe you have a solution. :)

Thank you advance and have a nice day.  :)

voron

Quote2018.06.19 11:20:29.040 Trying to bind on 0.0.0.0:4700
2018.06.19 11:20:29.040 Unable to bind socket: Only one usage of each socket address (protocol/network address/port) is normally permitted.
2018.06.19 11:20:29.040 Trying to bind on 127.0.0.1:28180
2018.06.19 11:20:29.040 Trying to bind on [::]:4700
2018.06.19 11:20:29.040 Unable to bind socket: Only one usage of each socket address (protocol/network address/port) is normally permitted.
2018.06.19 11:20:29.040 Unable to bind socket: Only one usage of each socket address (protocol/network address/port) is normally permitted.
2018.06.19 11:20:29.040 Session agent connector terminated (socket error)
Because service Netxms Agent already running.
QuoteTCP    0.0.0.0:4700           0.0.0.0:0              LISTENING
Check firewall rules and MasterServers parameter in config file nxagentd.conf
Use nmap to check port

IT-Reszleg

Thanks for the quick reply, i will check these!  ;)

IT-Reszleg

I checked, bindig is ok (i was stupid  :)), but the communication between server and agent is not working. The agent is behind a Mikrotik router, the port 4700 is dst-nat-ed to this machine (and rdp port also, and that is worked). If i try to poll agent, then nothing happened. If i try register manually agent to server, then:

c:\NetXMS\bin>nxagentd.exe -r ***.***.***.***
2018.06.19 19:15:09.276 Core agent version 2.2.6
2018.06.19 19:15:09.276 Additional configs was loaded from C:\NetXMS\etc\nxagentd.conf.d
2018.06.19 19:15:09.276 Debug level set to 0
2018.06.19 19:15:09.292 Data directory: C:\Users\admin\AppData\Local\nxagentd
2018.06.19 19:15:09.292 File store: C:\NetXMS\var
2018.06.19 19:15:09.308 DB Library: Database driver "sqlite.ddr" loaded and initialized successfully
2018.06.19 19:15:09.308 Agent ID is 30fe9b11-3e3f-f14c-818c-0d447229976f
2018.06.19 19:15:09.308 Local host name is "WIndows7-VM"
2018.06.19 19:15:09.308 Using system name "WIndows7-VM"
2018.06.19 19:15:09.323 Subagent "WinNT" (WINNT.NSM) loaded successfully (version 2.2.6)
2018.06.19 19:15:09.323 Subagent "ECS" (ecs.nsm) loaded successfully (version 2.2.6)
2018.06.19 19:15:09.323 Subagent "FILEMGR" (filemgr.nsm) loaded successfully (version 2.2.6)
2018.06.19 19:15:09.323 Subagent "PING" (ping.nsm) loaded successfully (version 2.2.6)
2018.06.19 19:15:09.323 Subagent "LOGWATCH" (logwatch.nsm) loaded successfully (version 2.2.6)
2018.06.19 19:15:09.323 Subagent "PORTCHECK" (portcheck.nsm) loaded successfully (version 2.2.6)
2018.06.19 19:15:11.745 Subagent "WinPerf" (winperf.nsm) loaded successfully (version 2.2.6)
2018.06.19 19:15:11.745 Subagent "WMI" (wmi.nsm) loaded successfully (version 2.2.6)
2018.06.19 19:15:11.745 Subagent "UPS" (ups.nsm) loaded successfully (version 2.2.6)
2018.06.19 19:15:12.745 Listening on socket 0.0.0.0:4700
2018.06.19 19:15:12.745 Listening on socket [::]:4700
2018.06.19 19:15:33.776 Registration on management server failed: Unable to connect to management server
2018.06.19 19:15:34.776 NetXMS Agent started
Agent running. Press ESC to shutdown.


The management server is on remote place, and have fix public IP address. All of my other agents can communicate with this management server remotely, only this can't.

I have no idea, how to test the communication betwen server and agent. It seems the NAT is ok, and i can ping the management server from agent machine.

Tursiops

If you telnet to the public IP on port 4700 from your NetXMS server, does it connect or time out?
If it connects, check your agent configuration file and node setup in NetXMS. The agent might simply not be configured to allow the server to connect.
If it doesn't connect, there's some firewall in the way blocking the connection (could be the Windows firewall).

Victor Kirhenshtein

Hi,

do you want server to connect to agent, or vise versa? Try to make agent running without registration option. If connection from your NetXMS server is NAT'ed, make sure that you have listed correct address in MasterServers (it should be address that agent sees). You can set logging to file and run agent with debug level 6 or higher - then you'll see if there are any incoming connections and if they are rejected.

Best regards,
Victor

IT-Reszleg

Yes, i want the "server connect to agent" scenario. Tomorrow i will deal with it in whole day, and i will feed back. Thank you guys!!!:)

Victor Kirhenshtein

Then don't use -r option, just run agent as a service. Make sure server can connect on port 4700 (agent installer should add firewall exception). You can use nxget on NetXMS server to test connection. If it returns error 408 (timeout), then it's likely firewall issue. If it is error 500 (or other 5xx) then most likely source address of incoming connection is not listed in MasterServers.

Best regards,
Victor

IT-Reszleg

Problem solved. Solution and background: Agent has local lan IP address, and was behind a mikrotik router. I dst-nat-ed the TCP port 4700. In agent config file i set only the public IP address of server for "masterservers". When i run the agent with -D9 (debug mode), i saw the incoming connection rejected from "mikrotik's lan IP address"!

It is correct, because the agent saw the incoming connection come from his gateway (Mikrotik router) not from the public IP address of the server. So, i added the Mikrotik's lan address to Masterservers (in the agent's config file), and restarted the agent. And woila! Works perfectly!

So guys, i would like to big-big thank you for your help!!!  :D