NetXMS Support Forum

Hi,

I see that database writer queue is quite high (5721598 in log file you've sent). Try to monitor it for some time (you can use internal DCIs on server node or debug console command "show queues"). If it is constantly growing it may indicate database performance issue and can cause excessive memory consumption (because more and more data kept in memory).

Best regards,
Victor

Hi,

definitely looks like memory leak. What was previous version that didn't have this issue? What are you monitoring, are you using scripts, proxies?
If possible please run diagnostic collection script nx-collect-server-diag.cmd and provide created zip file.

Best regards,
Victor

Quote from: bquiggle on January 05, 2021, 06:26:09 PM
I currently have the parser and actions setup to monitor the logs for a specific string, and if it finds it, it emails me the line from the log containing the string. If the string occurs twice in immediate succession, it will send me two separate emails. Is there a good way to group event notification emails when the event occurs multiple times in quick succession?

You can do that with scripts in event processing policy. When event is processed, update some key in persistent storage with current timestamp, and add additional script filter which will compare that key to current timestamp, and only allow further event processing if previous event is far enough in the past. See screenshot for example (it will send notification for SNMP_EGP_NEIGHBOUR_LOSS only if it comes more than 10 minutes after previous alert).

Quote from: bquiggle on January 05, 2021, 06:26:09 PM
How would I set up the parser to trigger an event if the log hasn't been written to after a set amount of time? Is this possible?

Using parser it can be done only in very complex way - you can match each line and generate event, setting timestamp from taht event, and have scheduled task or DCI to check how old that timestamp is.
Alternatively you can use DCI to read file log file modification time and check how old it is (assuming that modification time updated after write which may not always be the case if application keeps log file open).

Best regards,
Victor

This happens when you have Java 8. Try to upgrade to Java 11.

Best regards,
Victor

Quote from: Zebble on January 05, 2021, 05:54:10 AM
FYI, the server console from the Tools menu seems to have disappeared in 3.7.95.  I'm using nxcmd instead.

Make sure you are using Java 11 - it seems that this plugin is incompatible with Java 8.

Best regards,
Victor

Hello all!

NetXMS 3.7 is released (build 3.7.95). It is mostly about bug fixex and small improvements.

Please also note that web and desktop UI now requires Java 11.

Changes since version 3.6:

- UI framework updated to Eclipse 4.17 / RAP 3.14
- Improved clipboard support in web UI
- Improved access control logic for enter/leave/schedule maintenance
- Improved generic Cisco driver
- Improved support for Cisco Nexus switches
- Added command line tool nxaevent for sending events to server via local agent
- New agent parameter Agent.IsRestartPending
- Default DCI instance retention time changed to 7 days
- Container auto bind and template auto apply for sensors, mobile devices, and access points is configurable and off by default
- Explicit boolean data type in NXSL
- NXSL regular expression matching operator returns boolean "false" instead of null value if there is no match
- New method setIfXTableUsageMode in NXSL class "Node"
- New methods enableAgentStatusPolling, enableICMPStatusPolling, and enableSNMPStatusPolling in NXSL class "Interface"
- SQL query trace in server can be enabled independently from global debug level and changed at runtime
- Fixed unrecoverable "objects out of sync" state on client
- Fixed issues with file following
- Fixed issues:
        NX-770 (Instance discovery poll a node when an instance DCI is added)
        NX-790 (Ability to disable 802.1x checking)
        NX-1058 (Repeat event in DCI threshold may not be generated if repeat interval is the same as polling interval)
        NX-1173 (Add bulk DCI update functionality)
        NX-1469 (Show warning to user on attempt to force poll node in unmanaged state)
        NX-1602 (Option to disable ICMP poll for individual child Interface objects)
        NX-1623 (DCI > "Clear collected data" should refresh Last Values)
        NX-1948 (Create nxaevent utility similar to nxapush)
        NX-1956 (Allow applying templates to Access Points)
        NX-1975 (Per DCI multiplier settings)
        NX-1983 (Add automatic bind rules for clusters)
        NX-1984 (File delivery policy incorrectly handles Russian letters)
        NX-1985 (File name is cleaned on save in GUI log parser policy editor)

Best regards,
Victor

Actually I didn't immediately notice that those addresses are in zone 0. Lines like those:

2020.12.30 12:46:06.029 *D* [snmp.discovery     ] SnmpCheckCommSettings(192.168.1.5): failed
2020.12.30 12:46:06.030 *D* [obj.poll.node      ] AcceptNewNode(192.168.1.5): host is not reachable

I'll double check log tomorrow.

Best regards,
Victor

Hi,

from server log it looks like server considers all nodes discovered from ARP cache as unreachable. Just in case - are you running proxy agent as root? Also, are you sure that devices in remote zone accept SNMP requests from proxy agent IP address and that community strings are correct?

Best regards,
Victor

Hi,

I just created clean system, added new zone, single node as proxy in it, and passive discovery works as expected. Below are screenshots from my configuration (and you can see object tree in zone "Test" being populated with devices).
Could you please make clean system again, set debug level to 6 on both agent and server, add only proxy node, enable passive discovery, and send me server an agent logs?

Best regards,
Victor

Hello!

Currently log parser cannot expand single <file> entry into multiple files. You can use timestamp macros or external scripts to monitor files with changing names, but not changing set of files simultaneously.
You can safely ignore <logName> tag. It is purely cosmetic issue - instead of omitting it altogether if not set (as with other tags) code that generates XML always put it. This tag can be used to specify Windows event log name to filter by specific log (useful for Windows event synchronization).
Macros section moving to the bottom is a bug - we will fix it.

Best regards,
Victor

Hi!

I will try to reproduce this on my test system tomorrow - but it definitely looks wrong. Passive discovery should go in a same way as for local zone. Can proxy agent read local ARP cache and is there anything to be used for discovery? You can check that by selecting "Execute server script" on proxy node in UI and running the following script:

for(line : $node->readAgentList("Net.ArpCache"))
   println(line);

It should print some lines with ARP entries.
Am I understood correctly that proxy node is in zone 1, not in default zone (it is how it should be)?

Best regards,
Victor

Yes, that's actually a problem - executeSSHCommand uses SSH settings from node it is called on, and it is actually not possible to set SSH port for node.
In NXSL workaround is to use agentReadList - all that executeSSHCommand does is reading agent list with necessary arguments (properly escaped). But it has to be fixed anyway - SSH port for node definitely should be configurable, and maybe optional port number, login, and password arguments can be added to executeSSHCommand.

Best regards,
Victor

Hi,

port can be specified as part of host name after : character. This is relevant piece of code from SSH subagent: https://github.com/netxms/netxms/blob/819a9418773ce2fe4b5589397a70716d736c652e/src/agent/subagents/ssh/handlers.cpp#L39

Best regards,
Victor

Quote from: Zebble on December 28, 2020, 08:14:28 PM
Before I forget to ask, are passive discoveries possible through a proxy as well?

Yes, but you have to enable SNMP proxy in agent (by adding EnableSNMPProxy = yes to core section) and you must use zoning, with this agent set as zone proxy.

Best regards,
Victor

Hi,

try to check with lsof how many file descriptors actually open. It could be descriptor leak. Also, are you sure that your changes in open file limit actually reflected on nxagentd process? If it is being started with systemd you may need to make changes to unit file.

Best regards,
Victor