NetXMS Support Forum

Hi,

I can't get nxapush to work. I have a DCI called "LOCAL_TEST" with "push"
origin, the node is the server itself, and on the server I'm running this:


$ nxapush -vv LOCAL_TEST=7
1 data pair to send
Connected to NetXMS agent
Disconnected from NetXMS agent


nxapush doesn't complain, but the value doesn't show up in the "Last Values" table.

I've also tried to push to other nodes using "nxapush -vv -o 299 LOCAL_TEST=7" where 299 is the nodeId to which the DCI belongs. Again, the value doesn't make it to the node. CheckTrustedNodes is disabled in the server configuration.

What am I missing?

Thanks a lot,

   Hendrik

DCI matching is performed by parameter, so the parameter of your DCI should be "LOCAL_TEST".

Try setting debug level for the agent and server to 6, in there anything in the logs when you execute nxapush?

Yes, the DCI parameter is "LOCAL_TEST", origin is "push".

With DebugLevel=6 in netxmsd and nxagent the only output I get is in the agent log file. Nothing shows up in the server log. Here is the agent log:


2021.03.31 13:39:08.836 *D* [                   ] NamedPipeListener(nxagentd.push): accepted connection by user ats
2021.03.31 13:39:08.836 *D* [                   ] ProcessPushRequest: connection established
2021.03.31 13:39:08.836 *D* [                   ] ProcessPushRequest: received message CMD_PUSH_DCI_DATA
2021.03.31 13:39:08.836 *D* [                   ] PushData: "LOCAL_TEST" = "7"
2021.03.31 13:39:08.836 *D* [                   ] ProcessPushRequest: connection by user ats closed


Is there a way to tell the agent which server it should connect to? Is this localhost by default?

What's exactly netxms agent and server versions?
What kind of connection is between server and agent - is it normal or tunnel?

These are the versions we run at the moment:

$ netxmsd --version
NetXMS Server Version 3.8.250 Build 3.8-250-g21bbde29bb (UNICODE)
NXCP: 5.53.1.30 (AES-256, Blowfish-256, 3DES, AES-128, Blowfish-128)
Built with: g++ (Debian 8.3.0-6) 8.3.0


$ nxagentd -v
NetXMS Core Agent Version 3.8.250 Build 3.8-250-g21bbde29bb (UNICODE)


We have attempted to set up a tunnel connection following:
https://www.netxms.org/documentation/adminguide/server-management.html#self-signed-certificate-sample
placing certificates and keys in "/etc/netxmsd.conf.d/self/"

Server Config:

# Log file name
LogFile=/var/log/netxmsd

# Increase logging verbosity, 0 (only errors) to 9 (verbose debug)
DebugLevel=6

## PostgreSQL
DBDriver=pgsql.ddr
DBServer=127.0.0.1
DBName=netxms
DBLogin=netxms
DBPassword=our_db_password


TrustedCertificate = /etc/netxmsd.conf.d/self/rootCA.crt
ServerCertificate = /etc/netxmsd.conf.d/self/server.crt
ServerCertificateKey = /etc/netxmsd.conf.d/self/server.key


Agent Config:

# Log File
LogFile=/var/log/nxagentd

DebugLevel=8

# IP white list, can contain multiple records separated by comma.
# CIDR notation supported for subnets.
MasterServers=SERVER_IP

ServerConnection=SERVER_FQHN


######
The following logs are `tail -F /var/log/netxmsd /var/log/nxagentd | grep -i -e "crypto" -e "tunnel" -e "^==>"`'s output.
I have taken the liberty to:
  1) prefix the filenames to each line ( awk )
  2) substitute the first occurrence of "[" and "]" to "(" and ")" respectively in lines starting with "/var/log/netxmsd"
  3) abstract some detailed information
######

The first connection then succeeds,

/var/log/nxagentd 2021.04.08 08:20:43.023 *D* [tunnel           ] Tunnel manager started
/var/log/netxmsd  2021.04.08 08:20:43.026 *D* (                 ) SocketListener/AgentTunnels: Incoming connection from CLIENT_IP
/var/log/netxmsd  2021.04.08 08:20:43.026 *D* (                 ) SocketListener/AgentTunnels: Connection from CLIENT_IP accepted
/var/log/nxagentd 2021.04.08 08:20:43.026 *D* [tunnel           ] SERVER_FQHN: Loading certificate from file
/var/log/nxagentd 2021.04.08 08:20:43.026 *D* [tunnel           ] SERVER_FQHN: Cannot open file "/var/lib/netxms/certificates/53CBA2C8E6D236B46D68AB5D430E9E1BB742786B.crt" (No such file or directory)
/var/log/nxagentd 2021.04.08 08:20:43.026 *D* [tunnel           ] SERVER_FQHN: Cannot open file "/var/lib/netxms/certificates/120AFD110002000400020000000000000001.crt" (No such file or directory)
/var/log/netxmsd  2021.04.08 08:20:43.028 *D* (agent.tunnel     ) SetupTunnel(CLIENT_IP): minimal TLS version set to 1.2
/var/log/netxmsd  2021.04.08 08:20:43.028 *D* (crypto.cert      ) CreateTrustedCertificatesStore: trusted certificate "/etc/netxmsd.conf.d/self/rootCA.crt" added
/var/log/netxmsd  2021.04.08 08:20:43.028 *D* (crypto.cert      ) CreateTrustedCertificatesStore: added system certificate store at "/etc/ssl/certs"
/var/log/netxmsd  2021.04.08 08:20:43.029 *D* (crypto.cert      ) Certificate "/C=CC/ST=ST/L=LLLLLL/O=OOOOOO/CN=SERVER_FQHN" verification error 18 (self signed certificate) at depth 0
/var/log/nxagentd 2021.04.08 08:20:43.034 *D* [tunnel           ] SERVER_FQHN: Server certificate subject is /C=CC/ST=ST/L=LLLLLL/O=OOOOOO/CN=SERVER_FQHN
/var/log/nxagentd 2021.04.08 08:20:43.034 *D* [tunnel           ] SERVER_FQHN: Server certificate issuer is /C=CC/ST=ST/L=LLLLLL/O=OOOOOO/CN=SERVER_FQHN
/var/log/nxagentd 2021.04.08 08:20:43.034 *D* [tunnel           ] SERVER_FQHN: Server certificate verification is disabled
/var/log/nxagentd 2021.04.08 08:20:43.034 *D* [tunnel           ] SERVER_FQHN: Sending message CMD_SETUP_AGENT_TUNNEL
/var/log/netxmsd  2021.04.08 08:20:43.034 *D* (agent.tunnel     ) SetupTunnel(CLIENT_IP): Agent certificate not provided
/var/log/netxmsd  2021.04.08 08:20:43.035 *D* (agent.tunnel     ) SetupTunnel(CLIENT_IP): assigned to poller #0
/var/log/netxmsd  2021.04.08 08:20:43.035 *D* (agent.tunnel.1   ) Tunnel started
/var/log/netxmsd  2021.04.08 08:20:43.035 *D* (agent.tunnel.1   ) Received message CMD_SETUP_AGENT_TUNNEL
/var/log/netxmsd  2021.04.08 08:20:43.035 *D* (agent.tunnel.1   ) Unbound tunnel initialized
/var/log/netxmsd  2021.04.08 08:20:43.035 *D* (agent.tunnel.1   ) System name..............: SERVER_FQHN
/var/log/netxmsd  2021.04.08 08:20:43.035 *D* (agent.tunnel.1   ) Hostname.................: SERVER_FQHN
/var/log/netxmsd  2021.04.08 08:20:43.035 *D* (agent.tunnel.1   ) System information.......: Linux SERVER_FQHN 4.19.0-14-amd64 #1 SMP Debian 4.19.171-2 (2021-01-30) x86_64
/var/log/netxmsd  2021.04.08 08:20:43.036 *D* (agent.tunnel.1   ) Platform name............: Linux-x86_64
/var/log/netxmsd  2021.04.08 08:20:43.036 *D* (agent.tunnel.1   ) Hardware ID..............: 384518613C4CA702A5CFB61248FA459DC0E33B96
/var/log/netxmsd  2021.04.08 08:20:43.036 *D* (agent.tunnel.1   ) Agent ID.................: 2529de7e-dc33-4595-b2b7-b3420ac838fe
/var/log/netxmsd  2021.04.08 08:20:43.036 *D* (agent.tunnel.1   ) Agent version............: 3.8.250
/var/log/netxmsd  2021.04.08 08:20:43.036 *D* (agent.tunnel.1   ) Zone UIN.................: 0
/var/log/netxmsd  2021.04.08 08:20:43.036 *D* (agent.tunnel.1   ) Agent proxy..............: NO
/var/log/netxmsd  2021.04.08 08:20:43.036 *D* (agent.tunnel.1   ) SNMP proxy...............: NO
/var/log/netxmsd  2021.04.08 08:20:43.036 *D* (agent.tunnel.1   ) SNMP trap proxy..........: NO
/var/log/netxmsd  2021.04.08 08:20:43.036 *D* (agent.tunnel.1   ) Syslog proxy.............: NO
/var/log/netxmsd  2021.04.08 08:20:43.036 *D* (agent.tunnel.1   ) User agent...............: NO
/var/log/netxmsd  2021.04.08 08:20:43.036 *D* (agent.tunnel.1   ) Hook script not found
/var/log/netxmsd  2021.04.08 08:20:43.036 *D* (agent.tunnel.1   ) Sending message CMD_REQUEST_COMPLETED
/var/log/nxagentd 2021.04.08 08:20:43.077 *D* [tunnel           ] SERVER_FQHN: Received message CMD_REQUEST_COMPLETED
/var/log/nxagentd 2021.04.08 08:20:43.077 *I* [                 ] Tunnel with SERVER_FQHN established
/var/log/nxagentd 2021.04.08 08:20:43.077 *D* [tunnel           ] SERVER_FQHN: Tunnel is active


resulting in the tunnel showing up as unbound in:

# nxadm -c "show tunnels"

BOUND TUNNELS
ID  | Node ID | EP  | Chan. | Peer IP Address          | System Name              | Hostname                 | Platform Name    | Agent Version | Agent Build Tag
-----+---------+-----+-------+--------------------------+--------------------------+--------------------------+------------------+---------------+--------------------------

UNBOUND TUNNELS
ID  | EP  | Peer IP Address          | System Name              | Hostname                 | Platform Name    | Agent Version | Agent Build Tag
-----+-----+--------------------------+--------------------------+--------------------------+------------------+---------------+------------------------------------
1   | NO  | CLIENT_IP                | SERVER_FQHN              | SERVER_FQHN              | Linux-x86_64     | 3.8.250       | 3.8-250-g21bbde29bb


Issuing an agent tunnel bind command to the server's node has netxmsd generate a key/cert pair in /var/lib/netxms/certificates/ and send it to the agent, which acknowledges the files and saves them:

/var/log/netxmsd  2021.04.08 08:24:25.841 *D* (client.session.0 ) Received message CMD_BIND_AGENT_TUNNEL
/var/log/netxmsd  2021.04.08 08:24:25.841 *D* (agent.tunnel     ) BindAgentTunnel: processing bind request 1 -> 1727 by user admin
/var/log/netxmsd  2021.04.08 08:24:25.841 *D* (agent.tunnel.1   ) Node agent ID (00000000-0000-0000-0000-000000000000) do not match tunnel agent ID (2529de7e-dc33-4595-b2b7-b3420ac838fe) on bind
/var/log/netxmsd  2021.04.08 08:24:25.841 *D* (agent.tunnel.1   ) Sending message CMD_BIND_AGENT_TUNNEL
/var/log/netxmsd  2021.04.08 08:24:25.841 *D* (event.corr       ) CorrelateEvent: event SYS_TUNNEL_AGENT_ID_MISMATCH id 332805 source SERVER_FQHN [1727]
/var/log/netxmsd  2021.04.08 08:24:25.841 *D* (event.proc       ) EVENT SYS_TUNNEL_AGENT_ID_MISMATCH [96] at {0} (ID:332805 F:0x0001 S:1 TAGS:"") FROM SERVER_FQHN: Agent ID 2529de7e-dc33-4595-b2b7-b3420ac838fe on node do not match agent ID 00000000-0000-0000-0000-000000000000 on tunnel from SERVER_FQHN (CLIENT_IP) at bind
/var/log/nxagentd 2021.04.08 08:24:25.842 *D* [tunnel           ] SERVER_FQHN: Received message CMD_BIND_AGENT_TUNNEL
/var/log/nxagentd 2021.04.08 08:24:25.893 *D* [tunnel           ] SERVER_FQHN: Sending message CMD_REQUEST_CERTIFICATE
/var/log/netxmsd  2021.04.08 08:24:25.893 *D* (agent.tunnel.1   ) Received message CMD_REQUEST_CERTIFICATE
/var/log/netxmsd  2021.04.08 08:24:25.893 *D* (crypto.cert      ) IssueCertificate: new certificate request (CN override: 63c86ba9-08e4-4b4a-8e4a-6a09e831ddac, OU override: 84bff5a5-d832-45fe-a18b-e2ddee7519a3)
/var/log/netxmsd  2021.04.08 08:24:25.895 *D* (crypto.cert      ) IssueCertificate: new certificate with subject "/C=CC/O=OOOOOO/OU=84bff5a5-d832-45fe-a18b-e2ddee7519a3/CN=63c86ba9-08e4-4b4a-8e4a-6a09e831ddac" issued successfully
/var/log/netxmsd  2021.04.08 08:24:25.896 *D* (agent.tunnel.1   ) New certificate issued
/var/log/netxmsd  2021.04.08 08:24:25.896 *D* (node.agent       ) Tunnel ID for node SERVER_FQHN [1727] set to 63c86ba9-08e4-4b4a-8e4a-6a09e831ddac
/var/log/netxmsd  2021.04.08 08:24:25.896 *D* (agent.tunnel.1   ) Sending message CMD_NEW_CERTIFICATE
/var/log/nxagentd 2021.04.08 08:24:25.897 *D* [tunnel           ] SERVER_FQHN: Received message CMD_NEW_CERTIFICATE
/var/log/nxagentd 2021.04.08 08:24:25.898 *D* [tunnel           ] SERVER_FQHN: Certificate and private key saved
/var/log/nxagentd 2021.04.08 08:24:25.898 *D* [tunnel           ] SERVER_FQHN: Sending message CMD_REQUEST_COMPLETED
/var/log/netxmsd  2021.04.08 08:24:25.898 *D* (agent.tunnel.1   ) Received message CMD_REQUEST_COMPLETED
/var/log/netxmsd  2021.04.08 08:24:25.898 *D* (agent.tunnel.1   ) Certificate successfully issued and transferred to agent
/var/log/netxmsd  2021.04.08 08:24:25.898 *D* (agent.tunnel.1   ) Bind successful, resetting tunnel
/var/log/netxmsd  2021.04.08 08:24:25.898 *D* (agent.tunnel.1   ) Sending message CMD_RESET_TUNNEL
/var/log/nxagentd 2021.04.08 08:24:25.898 *D* [tunnel           ] SERVER_FQHN: Received message CMD_RESET_TUNNEL
/var/log/nxagentd 2021.04.08 08:24:25.898 *D* [tunnel           ] SERVER_FQHN: Receiver thread stopped (tunnel reset)
/var/log/nxagentd 2021.04.08 08:24:25.898 *W* [                 ] Tunnel with SERVER_FQHN closed
/var/log/netxmsd  2021.04.08 08:24:25.964 *D* (client.session.0 ) Received message CMD_GET_AGENT_TUNNELS
/var/log/nxagentd 2021.04.08 08:24:43.084 *D* [tunnel           ] SERVER_FQHN: Resetting tunnel
/var/log/netxmsd  2021.04.08 08:24:43.084 *D* (agent.tunnel.1   ) Tunnel closed by peer
/var/log/netxmsd  2021.04.08 08:24:43.085 *D* (agent.tunnel.1   ) Tunnel unregistered
/var/log/netxmsd  2021.04.08 08:24:43.085 *D* (agent.tunnel.1   ) Tunnel closure completed
/var/log/netxmsd  2021.04.08 08:24:43.085 *D* (agent.tunnel.1   ) Tunnel shutdown
/var/log/netxmsd  2021.04.08 08:24:43.085 *D* (agent.tunnel.1   ) Tunnel destroyed


The tunnel is reset automatically and all following connection attempts show up in the logs as follows:

/var/log/netxmsd  2021.04.08 08:24:43.085 *D* (                 ) SocketListener/AgentTunnels: Incoming connection from CLIENT_IP
/var/log/netxmsd  2021.04.08 08:24:43.085 *D* (                 ) SocketListener/AgentTunnels: Connection from CLIENT_IP accepted
/var/log/netxmsd  2021.04.08 08:24:43.085 *D* (agent.tunnel     ) SetupTunnel(CLIENT_IP): minimal TLS version set to 1.2
/var/log/netxmsd  2021.04.08 08:24:43.086 *D* (crypto.cert      ) CreateTrustedCertificatesStore: trusted certificate "/etc/netxmsd.conf.d/self/rootCA.crt" added
/var/log/netxmsd  2021.04.08 08:24:43.086 *D* (crypto.cert      ) CreateTrustedCertificatesStore: added system certificate store at "/etc/ssl/certs"
/var/log/nxagentd 2021.04.08 08:24:43.085 *D* [tunnel           ] SERVER_FQHN: Loading certificate from file
/var/log/nxagentd 2021.04.08 08:24:43.089 *D* [tunnel           ] SERVER_FQHN: Certificate and private key loaded
/var/log/netxmsd  2021.04.08 08:24:43.089 *D* (crypto.cert      ) Certificate "/C=CC/ST=ST/L=LLLLLL/O=OOOOOO/CN=SERVER_FQHN" verification error 18 (self signed certificate) at depth 0
/var/log/nxagentd 2021.04.08 08:24:43.092 *D* [tunnel           ] SERVER_FQHN: Server certificate subject is /C=CC/ST=ST/L=LLLLLL/O=OOOOOO/CN=SERVER_FQHN
/var/log/nxagentd 2021.04.08 08:24:43.092 *D* [tunnel           ] SERVER_FQHN: Server certificate issuer is /C=CC/ST=ST/L=LLLLLL/O=OOOOOO/CN=SERVER_FQHN
/var/log/nxagentd 2021.04.08 08:24:43.092 *D* [tunnel           ] SERVER_FQHN: Server certificate verification is disabled
/var/log/netxmsd  2021.04.08 08:24:43.093 *D* (crypto.cert      ) Certificate "/C=CC/O=OOOOOO/OU=84bff5a5-d832-45fe-a18b-e2ddee7519a3/CN=63c86ba9-08e4-4b4a-8e4a-6a09e831ddac" verification error 7 (certificate signature failure) at depth 0
/var/log/netxmsd  2021.04.08 08:24:43.093 *D* (agent.tunnel     ) SetupTunnel(CLIENT_IP): TLS handshake failed (error:00000001:lib(0):func(0):reason(1))
/var/log/nxagentd 2021.04.08 08:24:43.093 *D* [tunnel           ] SERVER_FQHN: Sending message CMD_SETUP_AGENT_TUNNEL
/var/log/nxagentd 2021.04.08 08:24:43.093 *D* [crypto           ] OpenSSL error stack:
/var/log/nxagentd 2021.04.08 08:24:43.093 *D* [crypto           ] error:1409441B:SSL routines:ssl3_read_bytes:tlsv1 alert decrypt error
/var/log/nxagentd 2021.04.08 08:24:43.093 *D* [tunnel           ] SERVER_FQHN: SSL_write error (bytes=-1 ssl_err=6 socket_err=0)
/var/log/nxagentd 2021.04.08 08:24:43.093 *D* [tunnel           ] SERVER_FQHN: Receiver thread stopped (MSGRECV_CLOSED)
/var/log/nxagentd 2021.04.08 08:24:43.093 *W* [                 ] Tunnel with SERVER_FQHN closed
/var/log/nxagentd 2021.04.08 08:24:53.093 *D* [tunnel           ] SERVER_FQHN: Cannot configure tunnel (request timeout)
/var/log/nxagentd 2021.04.08 08:25:23.094 *D* [tunnel           ] SERVER_FQHN: Resetting tunnel


How come the server is building a new key/cert pair for the client, based on data that later cannot be found for verification?


Best regards,

  Florian

Do you have any DPI system between agent and server that could intercept ssl connections?

Quote from: Filipp Sudanov on April 12, 2021, 06:57:15 PM
Do you have any DPI system between agent and server that could intercept ssl connections?

No. Server and agent are living on the same physical machine.

Using agent tunnel connection on the same machine does not have any benefits (but should work anyways).
Did you try tunnel connection from agent on another machine? Did it work?
What's exact OS version? If it's debian/ubuntu, what's the output of   dpkg -l | grep libssl

Well, okay then: How do I connect the agent with the server when living on the same machine? We don't need the agent for anything else but nxapush, so we haven't tried it from another machine.

We are running Debian Buster. libssl:

ii  libssl1.1:amd64                      1.1.1d-0+deb10u6             amd64        Secure


1) install netxms-agent package - you probably did that
2) in /etc/nxagentd.conf there should be
MasterServers=127.0.0.0/8
3) there should be a node in netxms that corresponds to your netxms server. This node is automatically created with external IP of your machine. In properties of that node -> Communications change primary host name to 127.0.0.1
4) perform Poll -> Configuration (full) on that node. Poll output should say that there is communication with the agent.
5) try nxapush

Also, if you need to push values from server there's nxpush utility that connects directly to netxms server.

Thanks! nxapush works perfectly now.

Out of curiosity - would you expect your solution to work without the full configuration poll, given a netxmsd and nxagent restart? Why?
We feel like having tried something very similar without success.

---

We had some misconceptions regarding the respective feature sets of nxapush vs nxpush, resulting in us not considering the latter.
After some more fiddling with it, triggered by your suggestion, we could clear those up and find some significant advantages of using nxpush over nxapush in our use case.
Namely the more flexible reference methods for nodes.
Thank you again for that.


Kind regards,

  Florian

Yes, right, nxapush is intended to run on remote nodes, not on the server.

If the system is correctly configured initially - agent configuration file allows connection from server - the configuration poll will happen automatically and server will properly operate with the agent. If the agent is not accessible when server is launched for the first time, the server will remember that agent is not available on that node.
That's standard procedure - server remembers, if agent or SNMP is available on each node. This is seen in Object Details - Capabiliest (isAgent, etc). Capabilities are changen on Full configuration poll (and also automatically, but with very big time interval).

Ah! Makes a lot of sense put in context of capability memory.

Thanks for clarifying!