Odd hanging upon alarm clear, polling stuck

Started by jermudgeon, May 13, 2019, 06:10:02 PM

Previous topic - Next topic

jermudgeon

I have been seeing some strange behavior for a few days. Database performance seems fine, running on flash. Attempting to handle/terminate/resolve more than even a few alarm entries at once results in a pegged CPU core with netxmsd.

netxmsd: show dbstats
SQL query counters:
   Total .......... 2061537
   SELECT ......... 861140
   Non-SELECT ..... 1200397
   Long running ... 0
   Failed ......... 0
Background writer requests:
   DCI data ....... 20263
   DCI raw data ... 20262
   Others ......... 49


netxmsd: show msgwq
0 active queues
Housekeeper thread state is RUNNING


Show pollers shows about half and half in cleanup and awaiting execution.

netxmsd: show queues
Data collector                   : 0
DCI cache loader                 : 0
Template updates                 : 0
Database writer                  : 0
Database writer (IData)          : 0
Database writer (raw DCI values) : 0
Event processor                  : 0
Event log writer                 : 0
Poller                           : 0
Node discovery poller            : 0
Syslog processing                : 0
Syslog writer                    : 0
Scheduler                        : 0



Show stats will time out while the CPU core is pegged.

netxmsd: show watchdog
Thread                                           Interval Status
----------------------------------------------------------------------------
Item Poller                                      10       Running
Syncer Thread                                    30       Sleeping
Poll Manager                                     5        Sleeping
Ad hoc scheduler                                 5        Sleeping
Recurrent scheduler                              5        Sleeping



Stopping the netxmsd process and repairing the DB will resolve the stuck CPU temporarily.

Viewing the logs, I get a lot of "Poll Manager" does not respond to watchdog thread.
Anything further to check?