NetXMS Support Forum

English Support => General Support => Topic started by: isoftcom on January 31, 2019, 03:00:27 pm

Title: DDCC client: Thread Item "Item poller" not responding
Post by: isoftcom on January 31, 2019, 03:00:27 pm
Hello Devs

I have been seeing this popping almost every after 30 seconds

DDCC client: Thread Item "Item poller" not responding

This is where Netxms is installed and other servers and its currently having a red x.

How do i sort this problem out?
Title: Re: DDCC client: Thread Item "Item poller" not responding
Post by: Victor Kirhenshtein on January 31, 2019, 03:13:59 pm
Hi,

this points to some internal locks being held for prolonged period. What version you are using, on what OS? Can you show event log for NetXMS server node?

Best regards,
Victor
 
Title: Re: DDCC client: Thread Item "Item poller" not responding
Post by: isoftcom on January 31, 2019, 04:13:49 pm
OS version - AIX 7.2
the version of netxms is 2.0.4
i cleared the logs to have fresh logs so i detect the problem. by event logs you mean netxmsd logs right?

some logs include the following

-----------------------------------------------------------------
[21-Jan-2019 07:24:50.357] [ERROR] SQL query failed (Query = "INSERT INTO tdata_155 (item_id,tdata_timestamp,record_id) VALUES (?,?,?)"): ORA-00001: unique constraint (NETXMS.SYS_C0012623) violated
[21-Jan-2019 07:24:50.909] [ERROR] SQL query failed (Query = "INSERT INTO tdata_155 (item_id,tdata_timestamp,record_id) VALUES (?,?,?)"): ORA-00001: unique constraint (NETXMS.SYS_C0012623) violated
[21-Jan-2019 07:24:56.374] [ERROR] SQL query failed (Query = "INSERT INTO tdata_151 (item_id,tdata_timestamp,record_id) VALUES (?,?,?)"): ORA-00001: unique constraint (NETXMS.SYS_C0012621) violated
[21-Jan-2019 07:30:01.469] [ERROR] Thread "Syncer Thread" does not respond to watchdog thread
[21-Jan-2019 07:30:01.473] [ERROR] Thread "Poll Manager" does not respond to watchdog thread
[21-Jan-2019 07:34:50.389] [ERROR] SQL query failed (Query = "INSERT INTO tdata_155 (item_id,tdata_timestamp,record_id) VALUES (?,?,?)"): ORA-00001: unique constraint (NETXMS.SYS_C0012623) violated
[21-Jan-2019 07:34:50.885] [ERROR] SQL query failed (Query = "INSERT INTO tdata_151 (item_id,tdata_timestamp,record_id) VALUES (?,?,?)"): ORA-00001: unique constraint (NETXMS.SYS_C0012621) violated
[21-Jan-2019 07:34:52.000] [ERROR] SQL query failed (Query = "INSERT INTO tdata_155 (item_id,tdata_timestamp,record_id) VALUES (?,?,?)"): ORA-00001: unique constraint (NETXMS.SYS_C0012623) violated
[21-Jan-2019 07:36:41.570] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 07:37:21.593] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 07:38:01.618] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 07:38:41.632] [ERROR] Thread "Syncer Thread" does not respond to watchdog thread
[21-Jan-2019 07:39:01.649] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 07:40:01.675] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 07:41:21.702] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 07:44:01.767] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 07:47:32.390] [ERROR] SQL query failed (Query = "INSERT INTO tdata_151 (item_id,tdata_timestamp,record_id) VALUES (?,?,?)"): ORA-00001: unique constraint (NETXMS.SYS_C0012621) violated
[21-Jan-2019 07:49:01.828] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 07:50:01.831] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 07:50:06.598] [ERROR] SQL query failed (Query = "INSERT INTO tdata_151 (item_id,tdata_timestamp,record_id) VALUES (?,?,?)"): ORA-00001: unique constraint (NETXMS.SYS_C0012621) violated
[21-Jan-2019 07:51:01.849] [ERROR] Thread "Syncer Thread" does not respond to watchdog thread
[21-Jan-2019 07:54:01.918] [ERROR] Thread "Syncer Thread" does not respond to watchdog thread
[21-Jan-2019 07:54:41.924] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 07:55:21.936] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 07:58:01.971] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 07:59:21.995] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 08:00:01.078] [DEBUG] IST clean syslg history called
[21-Jan-2019 08:00:02.000] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 08:01:02.018] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 08:02:42.077] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 08:03:42.096] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 08:04:22.101] [ERROR] Thread "Syncer Thread" does not respond to watchdog thread
[21-Jan-2019 08:05:22.109] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 08:07:02.148] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 08:14:02.234] [ERROR] Thread "Syncer Thread" does not respond to watchdog thread
[21-Jan-2019 08:14:42.260] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 08:15:42.292] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 08:16:22.307] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 08:18:02.327] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 08:19:02.345] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 08:20:22.362] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 08:20:54.132] [ERROR] SQL query failed (Query = "INSERT INTO tdata_155 (item_id,tdata_timestamp,record_id) VALUES (?,?,?)"): ORA-00001: unique constraint (NETXMS.SYS_C0012623) violated
[21-Jan-2019 08:20:54.137] [ERROR] SQL query failed (Query = "INSERT INTO tdata_155 (item_id,tdata_timestamp,record_id) VALUES (?,?,?)"): ORA-00001: unique constraint (NETXMS.SYS_C0012623) violated
[21-Jan-2019 08:22:42.407] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 08:24:22.453] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 08:26:42.511] [ERROR] Thread "Poll Manager" does not respond to watchdog thread
[21-Jan-2019 08:28:22.531] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 08:29:22.548] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 08:29:59.587] [ERROR] SQL query failed (Query = "INSERT INTO tdata_151 (item_id,tdata_timestamp,record_id) VALUES (?,?,?)"): ORA-00001: unique constraint (NETXMS.SYS_C0012621) violated
[21-Jan-2019 08:30:42.565] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 08:33:02.599] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 08:33:42.610] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 08:34:42.624] [ERROR] Thread "Syncer Thread" does not respond to watchdog thread
[21-Jan-2019 08:35:02.631] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 08:36:02.654] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 08:37:22.683] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 08:39:42.724] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 08:40:22.747] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 08:41:42.765] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 08:41:42.766] [ERROR] Thread "Syncer Thread" does not respond to watchdog thread
[21-Jan-2019 08:42:42.826] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 08:44:22.867] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 08:46:22.920] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 08:47:42.952] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 08:49:02.972] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 08:49:36.626] [ERROR] SQL query failed (Query = "INSERT INTO tdata_155 (item_id,tdata_timestamp,record_id) VALUES (?,?,?)"): ORA-00001: unique constraint (NETXMS.SYS_C0012623) violated
[21-Jan-2019 08:49:36.627] [ERROR] SQL query failed (Query = "INSERT INTO tdata_155 (item_id,tdata_timestamp,record_id) VALUES (?,?,?)"): ORA-00001: unique constraint (NETXMS.SYS_C0012623) violated
[21-Jan-2019 08:50:02.994] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 08:52:57.180] [DEBUG] Database Connection Pool: connection 1106bc590 reconnected
[21-Jan-2019 08:52:57.847] [DEBUG] Database Connection Pool: connection 11033b730 reconnected
[21-Jan-2019 08:52:58.788] [DEBUG] Database Connection Pool: connection 1103301b0 reconnected
[21-Jan-2019 08:52:59.681] [DEBUG] Database Connection Pool: connection 1100c8d10 reconnected
[21-Jan-2019 08:52:59.970] [DEBUG] Database Connection Pool: connection 1100eedd0 reconnected
[21-Jan-2019 08:53:23.051] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 08:54:23.062] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 08:54:37.828] [ERROR] SQL query failed (Query = "INSERT INTO tdata_151 (item_id,tdata_timestamp,record_id) VALUES (?,?,?)"): ORA-00001: unique constraint (NETXMS.SYS_C0012621) violated
[21-Jan-2019 08:54:40.867] [ERROR] SQL query failed (Query = "INSERT INTO tdata_151 (item_id,tdata_timestamp,record_id) VALUES (?,?,?)"): ORA-00001: unique constraint (NETXMS.SYS_C0012621) violated
[21-Jan-2019 09:00:23.134] [ERROR] Thread "Syncer Thread" does not respond to watchdog thread
[21-Jan-2019 09:03:03.190] [ERROR] Thread "Poll Manager" does not respond to watchdog thread
[21-Jan-2019 09:05:39.028] [DEBUG] IST clean syslg history called
[21-Jan-2019 09:05:40.744] [ERROR] SQL query failed (Query = "INSERT INTO tdata_151 (item_id,tdata_timestamp,record_id) VALUES (?,?,?)"): ORA-00001: unique constraint (NETXMS.SYS_C0012621) violated
[21-Jan-2019 09:05:41.990] [ERROR] SQL query failed (Query = "UPDATE raw_dci_values SET raw_value=?,transformed_value=?,last_poll_time=? WHERE item_id=?"): ORA-00060: deadlock detected while waiting for resource
[21-Jan-2019 09:05:42.309] [ERROR] SQL query failed (Query = "INSERT INTO tdata_151 (item_id,tdata_timestamp,record_id) VALUES (?,?,?)"): ORA-00001: unique constraint (NETXMS.SYS_C0012621) violated
[21-Jan-2019 09:10:43.309] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 09:11:52.536] [ERROR] SQL query failed (Query = "INSERT INTO tdata_151 (item_id,tdata_timestamp,record_id) VALUES (?,?,?)"): ORA-00001: unique constraint (NETXMS.SYS_C0012621) violated
[21-Jan-2019 09:11:52.650] [ERROR] SQL query failed (Query = "INSERT INTO tdata_155 (item_id,tdata_timestamp,record_id) VALUES (?,?,?)"): ORA-00001: unique constraint (NETXMS.SYS_C0012623) violated
[21-Jan-2019 09:11:52.786] [ERROR] SQL query failed (Query = "INSERT INTO tdata_151 (item_id,tdata_timestamp,record_id) VALUES (?,?,?)"): ORA-00001: unique constraint (NETXMS.SYS_C0012621) violated
[21-Jan-2019 09:12:23.353] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 09:13:43.374] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 09:14:43.393] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 09:18:43.447] [ERROR] Thread "Syncer Thread" does not respond to watchdog thread
[21-Jan-2019 09:19:43.459] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 09:22:03.488] [ERROR] Thread "Poll Manager" does not respond to watchdog thread
[21-Jan-2019 09:25:23.555] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 09:26:23.576] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 09:27:03.589] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 09:28:43.676] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 09:31:43.746] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 09:33:03.763] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 09:33:03.767] [ERROR] Thread "Syncer Thread" does not respond to watchdog thread
[21-Jan-2019 09:34:23.795] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 09:36:23.823] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 09:41:43.884] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 09:43:03.928] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 09:43:11.050] [ERROR] SQL query failed (Query = "INSERT INTO tdata_151 (item_id,tdata_timestamp,record_id) VALUES (?,?,?)"): ORA-00001: unique constraint (NETXMS.SYS_C0012621) violated
[21-Jan-2019 09:43:43.955] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 09:44:43.975] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 09:47:44.027] [ERROR] Thread "Item Poller" does not respond to watchdog thread
[21-Jan-2019 09:49:04
----------------------------------------------
Title: Re: DDCC client: Thread Item "Item poller" not responding
Post by: Victor Kirhenshtein on January 31, 2019, 05:25:05 pm
This version is very old. I definitely recommend upgrade to 2.2.12 first. We have fixed lot of issues, including internal locks and performance issues.

Best regards,
Victor