NetXMS Support Forum

Thank you for the help, I discovered in the script a section that had the potential to infinitely loop if the webservice it was accessing was offline. I fixed this oversight in the script, and moved the script to be the Event Processing action as per your first suggestion, and we haven't had the issue since.
Thanks again!

I am not sure if this is relevant, but our database is located on a seperate VM, netxms-dbg installed several new packages to the system, 'sudo apt-get install netxms-dbg' broke our installation, I have updated everything and brought it back online now, and will send the new files momentarily.

Our running version was 5.1.3, it is now 5.1.4. The issue continues.
Events.Processor.PoolSize is 1200.

Setting debug event.* 8 helped immensely for the logs, it locked up processing yesterday while I was out but I was still able to view what I needed as the logs hadn't overwritten yet. 
The last event that shows up in the Events Log tool in NXMC is SYS_SSH_OK for CN-RKr-BH at 17:58:21, in the screenshot I can see what I believe to be the issue beginning right after that, the SQL Query Failed logs begin after this and pretty much fill up everything since that time.
Logs at time of lockup


I have files generated from running the script but I am unsure of the best way to share them on the forum, how would you like me to share those?

Just recently we've started having an issue where NetXMS will completely stop event processing. Checking the events log through nxmc will show the last processed event and will never update, and the event processing queue will begin to pile up. Restarting netxmsd.service clears the queue and processing will work again for a while.
The VM has plenty of resources, and it doesn't seem like memory/CPU/io is limiting.
I set debug level to 6 but was unable to see anything in the logs that looked relevant, however debug level 6 does spit out a ton of logging so I may have missed something . Is there a higher debug level I should be using ? My biggest concern is that the logs at higher levels put out so much information that I would need to catch the issue within minutes at our current log retention , can I set the logs to be specific topics or even just for longer retention ?
Any other troubleshooting advice would be welcome as well
Thanks

Thank you, I was able to use this to filter through all my devices and clear out the DCIs that were duplicated.

I have somehow wound up with a large number of nodes (>1500) that have a DCI that was created by a template dci with instance discovery, but the instance discovery DCI is no longer part of the template. Because of this, if I want to remove these DCI (which I do), I have to go through every node and individually delete each one.
If I go to edit the DCI, I get the popup that says 'This DCI was created through instance discovery' , but there is no instance discovery DCI left on the node.
I am at a loss, it seems there is no way to automate deleting a DCI through API or scripting, and I really don't want to pay someone to go through all of these nodes and sort out which DCI need to be deleted.
Is there a way to bulk delete DCI that I am missing?