NetXMS Support Forum

English Support => General Support => Topic started by: MHdby on March 01, 2019, 08:26:09 PM

Title: agent tunnel not connecting
Post by: MHdby on March 01, 2019, 08:26:09 PM
I have an agent tunnel on a remote site that, from what I can see in the agent log seems to be connecting. I see the node in the agent tunnel manager, but when i create and bind, it remains unbound and has a ? on the icon in infrastructure services. I do have other nodes from other sites that are working. I suspect this is related to something with the network the remote site has but I dont know what to request from them to allow through.
I redacted some info with x's
2019.03.01 17:48:26.326 *D* Trying to bind on 0.0.0.0:4700                                                                                                                                                                                                                 
2019.03.01 17:48:26.326 *D* Trying to bind on [::]:4700                                                                                                                                                                                                                     
2019.03.01 17:48:26.326 *D* Session agent connector disabled                                                                                                                                                                                                               
2019.03.01 17:48:26.326 *I* Listening on socket 0.0.0.0:4700                                                                                                                                                                                                               
2019.03.01 17:48:26.326 *D* Database writer thread started                                                                                                                                                                                                                 
2019.03.01 17:48:26.327 *I* Listening on socket [::]:4700                                                                                                                                                                                                                   
2019.03.01 17:48:26.327 *D* NamedPipeListener(nxagentd.push): waiting for connection                                                                                                                                                                                       
2019.03.01 17:48:26.327 *D* [tunnel             ] Tunnel manager started                                                                                                                                                                                                   
2019.03.01 17:48:26.330 *D* [tunnel             ] xxxxxx: Certificate and private key loaded                                                                                                                                                                       
2019.03.01 17:48:26.554 *D* [tunnel             ] xxxxxx: Server certificate subject is /C=xx/ST=xx/O=xxxxx/OU=xxxx/CN=xxxx/[email protected]                                                                                                       
2019.03.01 17:48:26.554 *D* [tunnel             ] xxxxxx: Server certificate issuer is /C=xx/ST=xx/O=xxxxx/OU=xxxx/CN=xxxx/[email protected]                                                                                                         
2019.03.01 17:48:26.554 *D* [VCS-1] Requesting parameter "System.PlatformName"                                                                                                                                                                                             
2019.03.01 17:48:26.554 *D* [VCS-1] Requesting parameter "System.UName"                                                                                                                                                                                                     
2019.03.01 17:48:26.554 *D* [tunnel             ] xxxxxx: Sending message CMD_SETUP_AGENT_TUNNEL                                                                                                                                                                   
2019.03.01 17:48:26.616 *D* [tunnel             ] xxxxxx: Received message CMD_REQUEST_COMPLETED                                                                                                                                                                   
2019.03.01 17:48:26.616 *I* Tunnel with xxxxxxx established                                                                                                                                                                                                         
2019.03.01 17:48:26.616 *D* [tunnel             ] xxxxxx: Tunnel is active                                                                                                                                                                                         
2019.03.01 17:48:27.328 *I* NetXMS Agent started 
2019.03.01 17:48:56.616 *D* [tunnel             ] 24.172.241.174: Sending message CMD_KEEPALIVE                                                                                                                                                                             
2019.03.01 17:48:56.677 *D* [tunnel             ] 24.172.241.174: Received message CMD_KEEPALIVE                                                                                                                                                                           
2019.03.01 17:49:26.677 *D* [tunnel             ] 24.172.241.174: Sending message CMD_KEEPALIVE                                                                                                                                                                             
2019.03.01 17:49:26.738 *D* [tunnel             ] 24.172.241.174: Received message CMD_KEEPALIVE                                                                                                                                                                           
2019.03.01 17:49:56.738 *D* [tunnel             ] 24.172.241.174: Sending message CMD_KEEPALIVE                                                                                                                                                                             
2019.03.01 17:49:56.799 *D* [tunnel             ] 24.172.241.174: Received message CMD_KEEPALIVE

From the server log im getting
2019.03.01 13:12:15.847 *D* [poll.manager       ] Node 451 "xxxx" queued for status poll
2019.03.01 13:12:15.847 *D* [poll.status        ] Starting status poll for node xxxx (ID: 451)
2019.03.01 13:12:15.847 *D* StatusPoll(xxxx [451]): unable to get system uptime
2019.03.01 13:12:15.847 *D* Finished status poll for node xxxx (ID: 451)

force full config check
2019.03.01 13:16:05.682 *D* [client.session.1   ] Sending message CMD_POLLING_INFO (96 bytes)
2019.03.01 13:16:05.682 *D* [client.session.1   ] Sending compressed message CMD_POLLING_INFO (112 bytes)
2019.03.01 13:16:05.682 *D* [poll.conf          ] Starting configuration poll for node xxxx (ID: 451)
2019.03.01 13:16:05.682 *D* [client.session.1   ] Sending message CMD_POLLING_INFO (88 bytes)
2019.03.01 13:16:05.686 *D* [client.session.1   ] Sending message CMD_POLLING_INFO (112 bytes)
2019.03.01 13:16:05.687 *D* [poll.conf          ] ConfPoll(xxxx): checking for NetXMS agent Flags={01000000} DynamicFlags={00040400}
2019.03.01 13:16:05.687 *D* [client.session.1   ] Sending message CMD_POLLING_INFO (104 bytes)
2019.03.01 13:16:05.687 *D* [client.session.1   ] Sending compressed message CMD_POLLING_INFO (128 bytes)
2019.03.01 13:16:05.687 *D* [poll.conf          ] ConfPoll(xxxx): node primary IP is invalid and there are no active tunnels
2019.03.01 13:16:05.687 *D* [client.session.1   ] Sending message CMD_POLLING_INFO (104 bytes)
2019.03.01 13:16:05.687 *D* [client.session.1   ] Sending message CMD_POLLING_INFO (120 bytes)
2019.03.01 13:16:05.687 *D* [client.session.1   ] Sending message CMD_POLLING_INFO (128 bytes)
2019.03.01 13:16:05.687 *D* [poll.conf          ] Node::updateInterfaceConfiguration(xxxx [451]): Unable to get interface list from node
2019.03.01 13:16:05.687 *D* [poll.conf          ] Node::updateInterfaceConfiguration(xxxx [451]): pIfList == NULL, dwCount = 0
2019.03.01 13:16:05.687 *D* Checking subnet bindings for node xxxx [451]
2019.03.01 13:16:05.687 *D* [client.session.1   ] Sending message CMD_POLLING_INFO (128 bytes)
2019.03.01 13:16:05.687 *D* [client.session.1   ] Sending message CMD_POLLING_INFO (88 bytes)
2019.03.01 13:16:05.687 *D* [client.session.1   ] Sending message CMD_POLLING_INFO (80 bytes)
2019.03.01 13:16:05.688 *D* [poll.conf          ] ConfPoll(xxxx): detected node type: 0 (Unknown)
2019.03.01 13:16:05.688 *D* [client.session.1   ] Sending compressed message CMD_POLLING_INFO (112 bytes)
2019.03.01 13:16:05.688 *D* [client.session.1   ] Sending compressed message CMD_POLLING_INFO (112 bytes)
2019.03.01 13:16:05.688 *D* Finished configuration poll for node xxxx (ID: 451)
2019.03.01 13:16:05.688 *D* [client.session.1   ] Sending message CMD_POLLING_INFO (32 bytes)
Title: Re: agent tunnel not connecting
Post by: Victor Kirhenshtein on March 02, 2019, 09:40:10 AM
Hi,

please make sure that you have debug level 6 for tag agent.tunnel on server side and provide debug output for tunnel establishing.

Best regards,
Victor
Title: Re: agent tunnel not connecting
Post by: MHdby on March 06, 2019, 04:51:18 PM
debug level of 6 in the server config was on. If i grep the log on agent.tunnel I am seeing some of these but not sure how to interpret it

2019.03.06 09:41:56.638 *D* [agent.tunnel.99697 ] Communication error (MSGRECV_COMM_FAILURE)
2019.03.06 09:41:56.638 *D* [agent.tunnel.99697 ] Tunnel unregistered
2019.03.06 09:41:56.638 *D* [agent.tunnel.99697 ] Receiver thread stopped
2019.03.06 09:41:56.638 *D* [agent.tunnel.99697 ] Tunnel shutdown
2019.03.06 09:41:56.638 *D* [agent.tunnel.99697 ] Tunnel destroyed
2019.03.06 09:41:56.664 *D* [agent.tunnel.24253 ] Received message CMD_CHANNEL_DATA
2019.03.06 09:41:56.672 *D* [agent.tunnel.4921  ] Received message CMD_CHANNEL_DATA
2019.03.06 09:41:56.673 *D* [agent.tunnel.4063  ] Received message CMD_CHANNEL_DATA
2019.03.06 09:41:56.674 *D* [agent.tunnel.4841  ] Received message CMD_CHANNEL_DATA
2019.03.06 09:41:56.675 *D* [agent.tunnel.14940 ] Received message CMD_CHANNEL_DATA
2019.03.06 09:41:56.675 *D* [agent.tunnel.4096  ] Received message CMD_CHANNEL_DATA
2019.03.06 09:41:56.675 *D* [agent.tunnel.4883  ] Received message CMD_CHANNEL_DATA
2019.03.06 09:41:56.676 *D* [agent.tunnel.4885  ] Received message CMD_CHANNEL_DATA
2019.03.06 09:41:56.677 *D* [agent.tunnel.4884  ] Received message CMD_CHANNEL_DATA
2019.03.06 09:41:56.680 *D* [agent.tunnel.934   ] Received message CMD_CHANNEL_DATA
2019.03.06 09:41:56.682 *D* [agent.tunnel       ] SetupTunnel(<public IP here>): Agent certificate not provided
2019.03.06 09:41:56.682 *D* [agent.tunnel.99698 ] Tunnel started
2019.03.06 09:41:56.682 *D* [agent.tunnel.99698 ] Tunnel closed by peer
2019.03.06 09:41:56.682 *D* [agent.tunnel.99698 ] Tunnel unregistered
2019.03.06 09:41:56.682 *D* [agent.tunnel.99698 ] Receiver thread stopped
2019.03.06 09:41:56.682 *D* [agent.tunnel.99698 ] Tunnel shutdown
2019.03.06 09:41:56.682 *D* [agent.tunnel.99698 ] Tunnel destroyed

I'm also seeing
2019.03.06 09:42:05.713 *D* [agent.tunnel.6     ] Received message CMD_CHANNEL_DATA
2019.03.06 09:42:05.714 *D* [agent.tunnel.4920  ] Received message CMD_CHANNEL_DATA
2019.03.06 09:42:05.757 *D* [agent.tunnel       ] SetupTunnel(<public ip here>): Agent certificate not provided
2019.03.06 09:42:05.757 *D* [agent.tunnel.99699 ] Tunnel started
2019.03.06 09:42:05.757 *D* [agent.tunnel.99699 ] Tunnel closed by peer
2019.03.06 09:42:05.757 *D* [agent.tunnel.99699 ] Tunnel unregistered
2019.03.06 09:42:05.757 *D* [agent.tunnel.99699 ] Receiver thread stopped
2019.03.06 09:42:05.758 *D* [agent.tunnel.99699 ] Tunnel shutdown
2019.03.06 09:42:05.758 *D* [agent.tunnel.99699 ] Tunnel destroyed
2019.03.06 09:42:05.767 *D* [agent.tunnel.18996 ] Received message CMD_KEEPALIVE
2019.03.06 09:42:05.768 *D* [agent.tunnel.18996 ] Sending message CMD_KEEPALIVE
2019.03.06 09:42:05.779 *D* [agent.tunnel.6     ] Received message CMD_CHANNEL_DATA
2019.03.06 09:42:05.844 *D* [agent.tunnel.6     ] Received message CMD_CHANNEL_DATA
2019.03.06 09:42:05.912 *D* [agent.tunnel.6     ] Received message CMD_CHANNEL_DATA
Title: Re: agent tunnel not connecting
Post by: Victor Kirhenshtein on March 20, 2019, 08:04:11 PM
From the server side it looks like agent did not provide certificate:

SetupTunnel(<public ip here>): Agent certificate not provided

Can you check that agent certificate actually exist on machine with agent? What operating system agent is running on? What versions you are using?

Best regards,
Victor