Added a subnet to NetXMS yesterday, ran a discovery on it and then disabled the discovery. This is monitored via a NetXMS agent as a proxy.
It has picked up 45 devices, a mixture of switches, APs and firewalls. I am not really sure exactly how many metrics are being collected in total across the whole network, but I have not specifically started collecting [for example] interface throughput stats or created any custom templates. NetXMS will just be collecting whatever it does by default, things like ICMP reachability, FDB, interface and instance discovery.
In Server Configuration, every parameter with word "poll" or "interval" is at the default except NetworkDiscovery.ActiveDiscovery.Interval and NetworkDiscovery.PassiveDiscovery.Interval which are set to 43000s.
So the reason I am asking this, is that since this network was added, the firewall is showing an additional 2000-4000 connections in to this network from the NetXMS agent. One switch has 64 SNMP connections to it. 19 were opened in the same second. Each connection has <100 TX/RX bytes. So it seems to me that NetXMS is being a bit profligate with how it opens connections.
Seems to be too much requests, but could be legit, as NetXMS requests a lot of information via snmp.
If you are using unencrypted snmp communications, please capture network exchange dumps. Then we can see specific OIDs that are being requested.
If your server is on Linux, use this command to capture to file in wireshark format:
tcpdump -s 0 -w snmp-exchange.pcap host = DEVICE_IP
On Windows you can use Wireshark for that.
This is a bit tricky, the host in question also runs a Zabbix proxy. But it's definitely NetXMS generating this traffic. I will see what I can do.
Lightning maint window, stopped Zabbix proxy service for a few minutes.
This is the cause:
17:32:56.994760 IP 192.168.1.1.54475 > 192.168.10.18.161: C="public" GetRequest(28) .1.3.6.1.2.1.1.2.0
17:32:57.029465 IP 192.168.10.18.161 > 192.168.1.1.54475: C="public" GetResponse(39) .1.3.6.1.2.1.1.2.0=.1.3.6.1.4.1.9.6.1.81.28.2
17:32:57.032267 IP 192.168.1.1.49414 > 192.168.10.18.161: C="public" GetRequest(30) .1.3.6.1.2.1.2.2.1.7.49
....
17:33:04.334785 IP 192.168.10.18.161 > 192.168.1.1.36845: C="public" GetResponse(33) .1.3.6.1.2.1.2.2.1.5.100409=0
17:33:04.346389 IP 192.168.1.1.51426 > 192.168.10.18.161: C="public" GetRequest(28) .1.3.6.1.2.1.1.3.0
17:33:04.381043 IP 192.168.10.18.161 > 192.168.1.1.51426: C="public" GetResponse(33) .1.3.6.1.2.1.1.3.0=2487595200
The source port is different every time, so this is a new "connection" [UDP is connectionless] every time from the firewall's POV. In this case, 172 connections in 8 seconds. The real age of the connection is only a few tens of ms but the firewall doesn't know this and only ages it out after 60s of no traffic.
We are simply collecting more data with NetXMS than Zabbix, so the apparent effect is amplified.
This is firewall design problem in my opinion, as this is perfectly normal traffic pattern. However, I think we can implement configurable option in our SNMP stack to keep pool of sockets and reuse them when possible.
Added feature request to the tracker: https://track.radensolutions.com/issue/NX-2614 (https://track.radensolutions.com/issue/NX-2614)
Best regards,
Victor
Quote from: Victor Kirhenshtein on October 08, 2024, 03:18:33 PMThis is firewall design problem in my opinion, as this is perfectly normal traffic pattern.
Sure, understood. Because firewall keeps connection state table entry hanging around for 60s this exacerbates the apparent "issue".