I have been seeing some strange behavior for a few days. Database performance seems fine, running on flash. Attempting to handle/terminate/resolve more than even a few alarm entries at once results in a pegged CPU core with netxmsd.
netxmsd: show dbstats
SQL query counters:
Total .......... 2061537
SELECT ......... 861140
Non-SELECT ..... 1200397
Long running ... 0
Failed ......... 0
Background writer requests:
DCI data ....... 20263
DCI raw data ... 20262
Others ......... 49
netxmsd: show msgwq
0 active queues
Housekeeper thread state is RUNNING
Show pollers shows about half and half in cleanup and awaiting execution.
netxmsd: show queues
Data collector : 0
DCI cache loader : 0
Template updates : 0
Database writer : 0
Database writer (IData) : 0
Database writer (raw DCI values) : 0
Event processor : 0
Event log writer : 0
Poller : 0
Node discovery poller : 0
Syslog processing : 0
Syslog writer : 0
Scheduler : 0
Show stats will time out while the CPU core is pegged.
netxmsd: show watchdog
Thread Interval Status
----------------------------------------------------------------------------
Item Poller 10 Running
Syncer Thread 30 Sleeping
Poll Manager 5 Sleeping
Ad hoc scheduler 5 Sleeping
Recurrent scheduler 5 Sleeping
Stopping the netxmsd process and repairing the DB will resolve the stuck CPU temporarily.
Viewing the logs, I get a lot of "Poll Manager" does not respond to watchdog thread.
Anything further to check?