DDCC client: Thread Item "Item poller" not responding

Started by isoftcom, January 31, 2019, 03:00:27 PM

Previous topic - Next topic

isoftcom

Hello Devs

I have been seeing this popping almost every after 30 seconds

DDCC client: Thread Item "Item poller" not responding

This is where Netxms is installed and other servers and its currently having a red x.

How do i sort this problem out?

Victor Kirhenshtein

Hi,

this points to some internal locks being held for prolonged period. What version you are using, on what OS? Can you show event log for NetXMS server node?

Best regards,
Victor

isoftcom

#2
OS version - AIX 7.2
the version of netxms is 2.0.4
i cleared the logs to have fresh logs so i detect the problem. by event logs you mean netxmsd logs right?

some logs include the following

-----------------------------------------------------------------
[21-Jan-2019 07:24:50.357] [ERROR] SQL query failed (Query = "INSERT INTO tdata_155 (item_id,tdata_timestamp,record_id) VALUES (?,?,?)"): ORA-00001: unique constraint (NETXMS.SYS_C0012623) violated
[21-Jan-2019 07:24:50.909] [ERROR] SQL query failed (Query = "INSERT INTO tdata_155 (item_id,tdata_timestamp,record_id) VALUES (?,?,?)"): ORA-00001: unique constraint (NETXMS.SYS_C0012623) violated
[21-Jan-2019 07:24:56.374] [ERROR] SQL query failed (Query = "INSERT INTO tdata_151 (item_id,tdata_timestamp,record_id) VALUES (?,?,?)"): ORA-00001: unique constraint (NETXMS.SYS_C0012621) violated
[21-Jan-2019 07:30:01.469] [ERROR] Thread "Syncer Thread" does not respond to watchdog thread
[21-Jan-2019 07:30:01.473] [ERROR] Thread "Poll Manager" does not respond to watchdog thread
[21-Jan-2019 07:34:50.389] [ERROR] SQL query failed (Query = "INSERT INTO tdata_155 (item_id,tdata_timestamp,record_id) VALUES (?,?,?)"): ORA-00001: unique constraint (NETXMS.SYS_C0012623) violated
[21-Jan-2019 07:34:50.885] [ERROR] SQL query failed (Query = "INSERT INTO tdata_151 (item_id,tdata_timestamp,record_id) VALUES (?,?,?)"): ORA-00001: unique constraint (NETXMS.SYS_C0012621) violated
[21-Jan-2019 07:34:52.000] [ERROR] SQL query failed (Query = "INSERT INTO tdata_155 (item_id,tdata_timestamp,record_id) VALUES (?,?,?)"): ORA-00001: unique constraint (NETXMS.SYS_C0012623) violated
[21-Jan-2019 07:36:41.570] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 07:37:21.593] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 07:38:01.618] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 07:38:41.632] [ERROR] Thread "Syncer Thread" does not respond to watchdog thread
[21-Jan-2019 07:39:01.649] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 07:40:01.675] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 07:41:21.702] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 07:44:01.767] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 07:47:32.390] [ERROR] SQL query failed (Query = "INSERT INTO tdata_151 (item_id,tdata_timestamp,record_id) VALUES (?,?,?)"): ORA-00001: unique constraint (NETXMS.SYS_C0012621) violated
[21-Jan-2019 07:49:01.828] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 07:50:01.831] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 07:50:06.598] [ERROR] SQL query failed (Query = "INSERT INTO tdata_151 (item_id,tdata_timestamp,record_id) VALUES (?,?,?)"): ORA-00001: unique constraint (NETXMS.SYS_C0012621) violated
[21-Jan-2019 07:51:01.849] [ERROR] Thread "Syncer Thread" does not respond to watchdog thread
[21-Jan-2019 07:54:01.918] [ERROR] Thread "Syncer Thread" does not respond to watchdog thread
[21-Jan-2019 07:54:41.924] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 07:55:21.936] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 07:58:01.971] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 07:59:21.995] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 08:00:01.078] [DEBUG] IST clean syslg history called
[21-Jan-2019 08:00:02.000] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 08:01:02.018] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 08:02:42.077] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 08:03:42.096] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 08:04:22.101] [ERROR] Thread "Syncer Thread" does not respond to watchdog thread
[21-Jan-2019 08:05:22.109] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 08:07:02.148] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 08:14:02.234] [ERROR] Thread "Syncer Thread" does not respond to watchdog thread
[21-Jan-2019 08:14:42.260] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 08:15:42.292] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 08:16:22.307] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 08:18:02.327] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 08:19:02.345] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 08:20:22.362] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 08:20:54.132] [ERROR] SQL query failed (Query = "INSERT INTO tdata_155 (item_id,tdata_timestamp,record_id) VALUES (?,?,?)"): ORA-00001: unique constraint (NETXMS.SYS_C0012623) violated
[21-Jan-2019 08:20:54.137] [ERROR] SQL query failed (Query = "INSERT INTO tdata_155 (item_id,tdata_timestamp,record_id) VALUES (?,?,?)"): ORA-00001: unique constraint (NETXMS.SYS_C0012623) violated
[21-Jan-2019 08:22:42.407] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 08:24:22.453] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 08:26:42.511] [ERROR] Thread "Poll Manager" does not respond to watchdog thread
[21-Jan-2019 08:28:22.531] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 08:29:22.548] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 08:29:59.587] [ERROR] SQL query failed (Query = "INSERT INTO tdata_151 (item_id,tdata_timestamp,record_id) VALUES (?,?,?)"): ORA-00001: unique constraint (NETXMS.SYS_C0012621) violated
[21-Jan-2019 08:30:42.565] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 08:33:02.599] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 08:33:42.610] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 08:34:42.624] [ERROR] Thread "Syncer Thread" does not respond to watchdog thread
[21-Jan-2019 08:35:02.631] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 08:36:02.654] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 08:37:22.683] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 08:39:42.724] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 08:40:22.747] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 08:41:42.765] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 08:41:42.766] [ERROR] Thread "Syncer Thread" does not respond to watchdog thread
[21-Jan-2019 08:42:42.826] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 08:44:22.867] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 08:46:22.920] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 08:47:42.952] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 08:49:02.972] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 08:49:36.626] [ERROR] SQL query failed (Query = "INSERT INTO tdata_155 (item_id,tdata_timestamp,record_id) VALUES (?,?,?)"): ORA-00001: unique constraint (NETXMS.SYS_C0012623) violated
[21-Jan-2019 08:49:36.627] [ERROR] SQL query failed (Query = "INSERT INTO tdata_155 (item_id,tdata_timestamp,record_id) VALUES (?,?,?)"): ORA-00001: unique constraint (NETXMS.SYS_C0012623) violated
[21-Jan-2019 08:50:02.994] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 08:52:57.180] [DEBUG] Database Connection Pool: connection 1106bc590 reconnected
[21-Jan-2019 08:52:57.847] [DEBUG] Database Connection Pool: connection 11033b730 reconnected
[21-Jan-2019 08:52:58.788] [DEBUG] Database Connection Pool: connection 1103301b0 reconnected
[21-Jan-2019 08:52:59.681] [DEBUG] Database Connection Pool: connection 1100c8d10 reconnected
[21-Jan-2019 08:52:59.970] [DEBUG] Database Connection Pool: connection 1100eedd0 reconnected
[21-Jan-2019 08:53:23.051] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 08:54:23.062] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 08:54:37.828] [ERROR] SQL query failed (Query = "INSERT INTO tdata_151 (item_id,tdata_timestamp,record_id) VALUES (?,?,?)"): ORA-00001: unique constraint (NETXMS.SYS_C0012621) violated
[21-Jan-2019 08:54:40.867] [ERROR] SQL query failed (Query = "INSERT INTO tdata_151 (item_id,tdata_timestamp,record_id) VALUES (?,?,?)"): ORA-00001: unique constraint (NETXMS.SYS_C0012621) violated
[21-Jan-2019 09:00:23.134] [ERROR] Thread "Syncer Thread" does not respond to watchdog thread
[21-Jan-2019 09:03:03.190] [ERROR] Thread "Poll Manager" does not respond to watchdog thread
[21-Jan-2019 09:05:39.028] [DEBUG] IST clean syslg history called
[21-Jan-2019 09:05:40.744] [ERROR] SQL query failed (Query = "INSERT INTO tdata_151 (item_id,tdata_timestamp,record_id) VALUES (?,?,?)"): ORA-00001: unique constraint (NETXMS.SYS_C0012621) violated
[21-Jan-2019 09:05:41.990] [ERROR] SQL query failed (Query = "UPDATE raw_dci_values SET raw_value=?,transformed_value=?,last_poll_time=? WHERE item_id=?"): ORA-00060: deadlock detected while waiting for resource
[21-Jan-2019 09:05:42.309] [ERROR] SQL query failed (Query = "INSERT INTO tdata_151 (item_id,tdata_timestamp,record_id) VALUES (?,?,?)"): ORA-00001: unique constraint (NETXMS.SYS_C0012621) violated
[21-Jan-2019 09:10:43.309] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 09:11:52.536] [ERROR] SQL query failed (Query = "INSERT INTO tdata_151 (item_id,tdata_timestamp,record_id) VALUES (?,?,?)"): ORA-00001: unique constraint (NETXMS.SYS_C0012621) violated
[21-Jan-2019 09:11:52.650] [ERROR] SQL query failed (Query = "INSERT INTO tdata_155 (item_id,tdata_timestamp,record_id) VALUES (?,?,?)"): ORA-00001: unique constraint (NETXMS.SYS_C0012623) violated
[21-Jan-2019 09:11:52.786] [ERROR] SQL query failed (Query = "INSERT INTO tdata_151 (item_id,tdata_timestamp,record_id) VALUES (?,?,?)"): ORA-00001: unique constraint (NETXMS.SYS_C0012621) violated
[21-Jan-2019 09:12:23.353] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 09:13:43.374] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 09:14:43.393] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 09:18:43.447] [ERROR] Thread "Syncer Thread" does not respond to watchdog thread
[21-Jan-2019 09:19:43.459] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 09:22:03.488] [ERROR] Thread "Poll Manager" does not respond to watchdog thread
[21-Jan-2019 09:25:23.555] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 09:26:23.576] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 09:27:03.589] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 09:28:43.676] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 09:31:43.746] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 09:33:03.763] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 09:33:03.767] [ERROR] Thread "Syncer Thread" does not respond to watchdog thread
[21-Jan-2019 09:34:23.795] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 09:36:23.823] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 09:41:43.884] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 09:43:03.928] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 09:43:11.050] [ERROR] SQL query failed (Query = "INSERT INTO tdata_151 (item_id,tdata_timestamp,record_id) VALUES (?,?,?)"): ORA-00001: unique constraint (NETXMS.SYS_C0012621) violated
[21-Jan-2019 09:43:43.955] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 09:44:43.975] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 09:47:44.027] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 09:49:04
----------------------------------------------

Victor Kirhenshtein

This version is very old. I definitely recommend upgrade to 2.2.12 first. We have fixed lot of issues, including internal locks and performance issues.

Best regards,
Victor