Show Posts

This section allows you to view all posts made by this member. Note that you can only see posts made in areas you currently have access to.


Messages - pzandvoort

Pages: [1] 2 3
1
General Support / Re: Housekeeper / SQL Server Deadlock
« on: July 07, 2020, 08:54:43 pm »
Filipp,

Thanks for the research.
I look forward to the next version!

Peter

2
General Support / Re: Housekeeper / SQL Server Deadlock
« on: June 23, 2020, 04:12:11 pm »
Here we go! 4 deadlocks last night....

FYI: our logging goes to the Windows event log, so "eventlog 2020-06-23.csv" is an export of that. Note the reverse order! I have replace devicenames with *** to protect the innocent.
Also attached: the 4 deadlock events from SQL Server.

I don't see anything specific in here, but hope this helps!
Thanks,

Peter

3
General Support / Re: Housekeeper / SQL Server Deadlock
« on: June 18, 2020, 04:52:43 pm »
I've set the debug level to 7 for the housekeeper.
Of course, no deadlocks last night. It's the proverbial watched pot that never boils :)

4
General Support / Re: Housekeeper / SQL Server Deadlock
« on: June 11, 2020, 05:50:06 pm »
None happened last night, but here's the one from yesterday.



Code: [Select]
<deadlock>
 <victim-list>
  <victimProcess id="process14ebe971468" />
 </victim-list>
 <process-list>
  <process id="process14ebe971468" taskpriority="0" logused="40088" waitresource="PAGE: 5:1:1192460 " waittime="5506" ownerId="16049071" transactionname="user_transaction" lasttranstarted="2020-06-10T02:02:30.297" XDES="0x14eb61df590" lockMode="IX" schedulerid="24" kpid="19196" status="suspended" spid="60" sbid="0" ecid="0" priority="0" trancount="2" lastbatchstarted="2020-06-10T02:02:31.320" lastbatchcompleted="2020-06-10T02:02:31.320" lastattention="1900-01-01T00:00:00.320" clientapp="NetXMS" hostname="MONITOR1" hostpid="8632" loginname="CAMPUSDOOR\monitor" isolationlevel="read committed (2)" xactid="16049071" currentdb="5" currentdbname="netxms" lockTimeout="4294967295" clientoption1="671088672" clientoption2="128056">
   <executionStack>
    <frame procname="adhoc" line="1" stmtstart="98" stmtend="294" sqlhandle="0x020000006b0a903581f03fcc658c71f5b8537b2dde9aac390000000000000000000000000000000000000000">
unknown    </frame>
    <frame procname="adhoc" line="1" stmtend="236" sqlhandle="0x0200000065e8a32aad77be1795d7ff93c21bc8140ac3c6b70000000000000000000000000000000000000000">
unknown    </frame>
   </executionStack>
   <inputbuf>
INSERT INTO idata_2349 (item_id,idata_timestamp,idata_value,raw_value) VALUES (1109,1591768951,'91.331634','91.331634')   </inputbuf>
  </process>
  <process id="process14ebe036ca8" taskpriority="0" logused="3221500" waitresource="PAGE: 5:1:1193485 " waittime="4987" ownerId="16049218" transactionname="DELETE" lasttranstarted="2020-06-10T02:02:30.450" XDES="0x14ead5fb650" lockMode="U" schedulerid="4" kpid="12052" status="suspended" spid="61" sbid="0" ecid="0" priority="0" trancount="2" lastbatchstarted="2020-06-10T02:02:29.383" lastbatchcompleted="2020-06-10T02:02:29.383" lastattention="1900-01-01T00:00:00.383" clientapp="NetXMS" hostname="MONITOR1" hostpid="8632" loginname="CAMPUSDOOR\monitor" isolationlevel="read committed (2)" xactid="16049218" currentdb="5" currentdbname="netxms" lockTimeout="4294967295" clientoption1="671088672" clientoption2="128056">
   <executionStack>
    <frame procname="adhoc" line="1" stmtend="4878" sqlhandle="0x0200000083c4a620a2fa781ce6e93abf014fae0f6f1089850000000000000000000000000000000000000000">
unknown    </frame>
   </executionStack>
   <inputbuf>
DELETE FROM idata_2349 WHERE (item_id=1101 AND idata_timestamp&lt;1583992949) OR (item_id=1102 AND idata_timestamp&lt;1583992949) OR (item_id=1103 AND idata_timestamp&lt;1583992949) OR (item_id=1104 AND idata_timestamp&lt;1583992949) OR (item_id=1105 AND idata_timestamp&lt;1583992949) OR (item_id=1106 AND idata_timestamp&lt;1583992949) OR (item_id=1107 AND idata_timestamp&lt;1583992949) OR (item_id=1109 AND idata_timestamp&lt;1583992949) OR (item_id=1111 AND idata_timestamp&lt;1583992949) OR (item_id=1173 AND idata_timestamp&lt;1583992949) OR (item_id=1174 AND idata_timestamp&lt;1583992949) OR (item_id=1175 AND idata_timestamp&lt;1583992949) OR (item_id=1178 AND idata_timestamp&lt;1583992949) OR (item_id=1179 AND idata_timestamp&lt;1583992949) OR (item_id=1180 AND idata_timestamp&lt;1583992949) OR (item_id=1181 AND idata_timestamp&lt;1583992949) OR (item_id=1184 AND idata_timestamp&lt;1583992949) OR (item_id=1185 AND idata_timestamp&lt;1583992949) OR (item_id=1186 AND idata_timestamp&lt;1583992949) OR (item_id=1187 AND idata_timestamp&lt;1583992949) OR (item_id=14205   </inputbuf>
  </process>
 </process-list>
 <resource-list>
  <pagelock fileid="1" pageid="1192460" dbid="5" subresource="FULL" objectname="netxms.dbo.idata_2349" id="lock14dc42cf500" mode="X" associatedObjectId="72057594066829312">
   <owner-list>
    <owner id="process14ebe036ca8" mode="X" />
   </owner-list>
   <waiter-list>
    <waiter id="process14ebe971468" mode="IX" requestType="wait" />
   </waiter-list>
  </pagelock>
  <pagelock fileid="1" pageid="1193485" dbid="5" subresource="FULL" objectname="netxms.dbo.idata_2349" id="lock14e39817400" mode="IX" associatedObjectId="72057594066829312">
   <owner-list>
    <owner id="process14ebe971468" mode="IX" />
   </owner-list>
   <waiter-list>
    <waiter id="process14ebe036ca8" mode="U" requestType="wait" />
   </waiter-list>
  </pagelock>
 </resource-list>
</deadlock>

There were quite a few on 6/9.... Screenshot attached.
Below is the last one from 6/9.
Code: [Select]
<deadlock>
 <victim-list>
  <victimProcess id="process14ebe047468" />
 </victim-list>
 <process-list>
  <process id="process14ebe047468" taskpriority="0" logused="0" waitresource="PAGE: 5:1:2303392 " waittime="10425" ownerId="14398297" transactionname="DELETE" lasttranstarted="2020-06-09T02:20:42.220" XDES="0x14ebde89020" lockMode="U" schedulerid="6" kpid="16336" status="suspended" spid="63" sbid="0" ecid="8" priority="0" trancount="0" lastbatchstarted="2020-06-09T02:20:34.203" lastbatchcompleted="2020-06-09T02:20:34.203" lastattention="1900-01-01T00:00:00.203" clientapp="NetXMS" hostname="MONITOR1" hostpid="8632" isolationlevel="read committed (2)" xactid="14398297" currentdb="5" currentdbname="netxms" lockTimeout="4294967295" clientoption1="671088672" clientoption2="128056">
   <executionStack>
    <frame procname="adhoc" line="1" stmtend="7324" sqlhandle="0x020000009844b73552aecc44670076b3495534916ebe69be0000000000000000000000000000000000000000">
unknown    </frame>
   </executionStack>
   <inputbuf>
DELETE FROM idata_2637 WHERE (item_id=1817 AND idata_timestamp&lt;1583907634) OR (item_id=1818 AND idata_timestamp&lt;1583907634) OR (item_id=1819 AND idata_timestamp&lt;1583907634) OR (item_id=1822 AND idata_timestamp&lt;1583907634) OR (item_id=1823 AND idata_timestamp&lt;1583907634) OR (item_id=1824 AND idata_timestamp&lt;1583907634) OR (item_id=1825 AND idata_timestamp&lt;1583907634) OR (item_id=1828 AND idata_timestamp&lt;1583907634) OR (item_id=1829 AND idata_timestamp&lt;1583907634) OR (item_id=1830 AND idata_timestamp&lt;1583907634) OR (item_id=1831 AND idata_timestamp&lt;1583907634) OR (item_id=1970 AND idata_timestamp&lt;1583907634) OR (item_id=24756 AND idata_timestamp&lt;1583907634) OR (item_id=24757 AND idata_timestamp&lt;1583907634) OR (item_id=24758 AND idata_timestamp&lt;1583907634) OR (item_id=24759 AND idata_timestamp&lt;1583907634) OR (item_id=24760 AND idata_timestamp&lt;1583907634) OR (item_id=24761 AND idata_timestamp&lt;1583907634) OR (item_id=24762 AND idata_timestamp&lt;1583907634) OR (item_id=24763 AND idata_timestamp&lt;1583907634) OR (item_   </inputbuf>
  </process>
  <process id="process14ebe037c28" taskpriority="0" logused="9584" waitresource="PAGE: 5:1:2303257 " waittime="9865" ownerId="14398160" transactionname="user_transaction" lasttranstarted="2020-06-09T02:20:41.787" XDES="0x14eb6185390" lockMode="IX" schedulerid="4" kpid="16576" status="suspended" spid="59" sbid="0" ecid="0" priority="0" trancount="2" lastbatchstarted="2020-06-09T02:20:43.150" lastbatchcompleted="2020-06-09T02:20:43.150" lastattention="1900-01-01T00:00:00.150" clientapp="NetXMS" hostname="MONITOR1" hostpid="8632" loginname="CAMPUSDOOR\monitor" isolationlevel="read committed (2)" xactid="14398160" currentdb="5" currentdbname="netxms" lockTimeout="4294967295" clientoption1="671088672" clientoption2="128056">
   <executionStack>
    <frame procname="adhoc" line="1" stmtstart="98" stmtend="294" sqlhandle="0x020000003e4d2b1ad6d052f85d4ee2831814cedcb3d912610000000000000000000000000000000000000000">
unknown    </frame>
    <frame procname="adhoc" line="1" stmtend="218" sqlhandle="0x0200000059d8b81b923ecd02bb46bd15cb5d570c8e585c430000000000000000000000000000000000000000">
unknown    </frame>
   </executionStack>
   <inputbuf>
INSERT INTO idata_2637 (item_id,idata_timestamp,idata_value,raw_value) VALUES (24770,1591683643,'Running','0')   </inputbuf>
  </process>
  <process id="process14ebe06f848" taskpriority="0" logused="10000" waittime="4455" schedulerid="11" kpid="15704" status="suspended" spid="63" sbid="0" ecid="7" priority="0" trancount="0" lastbatchstarted="2020-06-09T02:20:34.203" lastbatchcompleted="2020-06-09T02:20:34.203" lastattention="1900-01-01T00:00:00.203" clientapp="NetXMS" hostname="MONITOR1" hostpid="8632" isolationlevel="read committed (2)" xactid="14398297" currentdb="5" currentdbname="netxms" lockTimeout="4294967295" clientoption1="671088672" clientoption2="128056">
   <executionStack>
    <frame procname="adhoc" line="1" stmtend="7324" sqlhandle="0x020000009844b73552aecc44670076b3495534916ebe69be0000000000000000000000000000000000000000">
unknown    </frame>
   </executionStack>
   <inputbuf>
DELETE FROM idata_2637 WHERE (item_id=1817 AND idata_timestamp&lt;1583907634) OR (item_id=1818 AND idata_timestamp&lt;1583907634) OR (item_id=1819 AND idata_timestamp&lt;1583907634) OR (item_id=1822 AND idata_timestamp&lt;1583907634) OR (item_id=1823 AND idata_timestamp&lt;1583907634) OR (item_id=1824 AND idata_timestamp&lt;1583907634) OR (item_id=1825 AND idata_timestamp&lt;1583907634) OR (item_id=1828 AND idata_timestamp&lt;1583907634) OR (item_id=1829 AND idata_timestamp&lt;1583907634) OR (item_id=1830 AND idata_timestamp&lt;1583907634) OR (item_id=1831 AND idata_timestamp&lt;1583907634) OR (item_id=1970 AND idata_timestamp&lt;1583907634) OR (item_id=24756 AND idata_timestamp&lt;1583907634) OR (item_id=24757 AND idata_timestamp&lt;1583907634) OR (item_id=24758 AND idata_timestamp&lt;1583907634) OR (item_id=24759 AND idata_timestamp&lt;1583907634) OR (item_id=24760 AND idata_timestamp&lt;1583907634) OR (item_id=24761 AND idata_timestamp&lt;1583907634) OR (item_id=24762 AND idata_timestamp&lt;1583907634) OR (item_id=24763 AND idata_timestamp&lt;1583907634) OR (item_   </inputbuf>
  </process>
  <process id="process14ebe01d848" taskpriority="0" logused="10000" waittime="9768" schedulerid="1" kpid="13264" status="suspended" spid="63" sbid="0" ecid="0" priority="0" trancount="2" lastbatchstarted="2020-06-09T02:20:34.203" lastbatchcompleted="2020-06-09T02:20:34.203" lastattention="1900-01-01T00:00:00.203" clientapp="NetXMS" hostname="MONITOR1" hostpid="8632" loginname="CAMPUSDOOR\monitor" isolationlevel="read committed (2)" xactid="14398297" currentdb="5" currentdbname="netxms" lockTimeout="4294967295" clientoption1="671088672" clientoption2="128056">
   <executionStack>
    <frame procname="adhoc" line="1" stmtend="7324" sqlhandle="0x020000009844b73552aecc44670076b3495534916ebe69be0000000000000000000000000000000000000000">
unknown    </frame>
   </executionStack>
   <inputbuf>
DELETE FROM idata_2637 WHERE (item_id=1817 AND idata_timestamp&lt;1583907634) OR (item_id=1818 AND idata_timestamp&lt;1583907634) OR (item_id=1819 AND idata_timestamp&lt;1583907634) OR (item_id=1822 AND idata_timestamp&lt;1583907634) OR (item_id=1823 AND idata_timestamp&lt;1583907634) OR (item_id=1824 AND idata_timestamp&lt;1583907634) OR (item_id=1825 AND idata_timestamp&lt;1583907634) OR (item_id=1828 AND idata_timestamp&lt;1583907634) OR (item_id=1829 AND idata_timestamp&lt;1583907634) OR (item_id=1830 AND idata_timestamp&lt;1583907634) OR (item_id=1831 AND idata_timestamp&lt;1583907634) OR (item_id=1970 AND idata_timestamp&lt;1583907634) OR (item_id=24756 AND idata_timestamp&lt;1583907634) OR (item_id=24757 AND idata_timestamp&lt;1583907634) OR (item_id=24758 AND idata_timestamp&lt;1583907634) OR (item_id=24759 AND idata_timestamp&lt;1583907634) OR (item_id=24760 AND idata_timestamp&lt;1583907634) OR (item_id=24761 AND idata_timestamp&lt;1583907634) OR (item_id=24762 AND idata_timestamp&lt;1583907634) OR (item_id=24763 AND idata_timestamp&lt;1583907634) OR (item_   </inputbuf>
  </process>
 </process-list>
 <resource-list>
  <pagelock fileid="1" pageid="2303392" dbid="5" subresource="FULL" objectname="netxms.dbo.idata_2637" id="lock14ddf8c6080" mode="IX" associatedObjectId="72057594076790784">
   <owner-list>
    <owner id="process14ebe037c28" mode="IX" />
   </owner-list>
   <waiter-list>
    <waiter id="process14ebe047468" mode="U" requestType="wait" />
   </waiter-list>
  </pagelock>
  <pagelock fileid="1" pageid="2303257" dbid="5" subresource="FULL" objectname="netxms.dbo.idata_2637" id="lock14e0026c980" mode="U" associatedObjectId="72057594076790784">
   <owner-list>
    <owner id="process14ebe06f848" mode="U" />
   </owner-list>
   <waiter-list>
    <waiter id="process14ebe037c28" mode="IX" requestType="wait" />
   </waiter-list>
  </pagelock>
  <exchangeEvent id="Pipe15935351630" WaitType="e_waitPipeNewRow" waiterType="Producer" nodeId="2" tid="11" ownerActivity="receivedData" waiterActivity="tryToSendData" merging="true" spilling="true" waitingToClose="true">
   <owner-list>
    <owner id="process14ebe01d848" />
   </owner-list>
   <waiter-list>
    <waiter id="process14ebe06f848" />
   </waiter-list>
  </exchangeEvent>
  <exchangeEvent id="Pipe14d50d18a30" WaitType="e_waitPipeGetRow" waiterType="Coordinator" nodeId="2" tid="0" ownerActivity="sentData" waiterActivity="needMoreData" merging="true" spilling="false" waitingToClose="false">
   <owner-list>
    <owner id="process14ebe047468" />
   </owner-list>
   <waiter-list>
    <waiter id="process14ebe01d848" />
   </waiter-list>
  </exchangeEvent>
 </resource-list>
</deadlock>

5
General Support / Re: Housekeeper / SQL Server Deadlock
« on: June 11, 2020, 07:35:46 am »
Hi Philipp,

Thanks for your quick reply!
The system isn't anything special, but not completely bad: Dell R420, Dual E5-2430 @ 2.2GHz, 32GB Ram, 2x 1TB drive in RAID1. It's normally pretty bored. It's not doing anything other than NetXMS and SQL. The database is 19GB.

sh stat:
- netxmsd: Objects 2473
- Monitored Nodes: 141
- Collectible DCIs: 3689

SELECT var_name, var_value FROM config WHERE var_name like 'DefaultDCI%';
DefaultDCIPollingInterval   60
DefaultDCIRetentionTime   90

SELECT polling_interval, retention_time, count(polling_interval) FROM items GROUP BY polling_interval, retention_time ORDER BY retention_time, polling_interval;
0   0   3830
300   0   24
3600   0   14
60   90   1

So most things just poll every minute and retain for 90 days.

Isn't a deadlock different from a regular lock? If all NetXMS is doing is just deletes and some inserts, I would maybe expect some regular locks/waits, but they resolve themselves given enough time. From what I understand, a deadlock is normally something special where 2 things wait for each other in a circle with no way out. (https://docs.microsoft.com/en-us/previous-versions/sql/sql-server-2005/ms177433%28v%3dsql.90%29)



6
General Support / Housekeeper / SQL Server Deadlock
« on: June 09, 2020, 05:07:00 pm »
Many nights at 2am, when the housekeeper runs, we get a few deadlocks on SQL Server.

SQL query failed (Query = "INSERT INTO idata_2301 (item_id,idata_timestamp,idata_value,raw_value) VALUES (1061,1591683195,'Ok','3')"): [Microsoft][SQL Server Native Client 11.0][SQL Server]Transaction (Process ID 57) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.

We also see some of the DELETE statements from the housekeeper failing for similar reasons: "SQL query failed (Query = "DELETE FROM idata_2637 WHERE (item_id=1817 AND idata_timestamp<1583907634) OR …" [Microsoft][SQL Server Native Client 11.0][SQL Server]Transaction (Process ID 63) was deadlocked on lock | communication buffer resources with another process and has been chosen as the deadlock victim. Rerun the transaction.

This doesn't happen every night, and the number of deadlocks isn't the same every time, but it's often enough that it's waking people up at 2am.

Anyone have any thoughts on where to go to troubleshoot?

netxms 3.3-323
SQL Server 2016 SP2 CU12 (13.0.5698.0)

Thanks!
Peter

7
General Support / ExternalParameters - Sequential?
« on: February 14, 2020, 12:01:33 am »
I can't find any documentation on it, but it looks like the agent polls ExternalParameters sequentially. Is this correct? If so, is there a way to run them simultaneously (say with some maximum degree of parallelism)?

I have a few external parameters that usually complete pretty quickly, but may sometimes take pretty long. It seems like they're pushing out some of the others....

8
Victor,

That makes perfect sense and matches exactly what we're seeing. For now, we've disabled the generation of the alarm on SYS_DCI_UNSUPPORTED to suppress the result, but if you can make it work like you describe that'd be awesome! Looking forward to the next release.
Thanks!

Peter

9
We have the exact same issue.

Every time the housekeeper runs (2:00am by default) it seems to re-apply the templates. Since not every DCI in the template is supported by all nodes, "SYS_DCI_UNSUPPORTED" fires and we get the "Status of DCI changed to UNSUPPORTED" alarm. This makes perfect sense the first time the template gets applied, since it's a new DCI for that node and it just figured out that the DCI isn't supported. But it shouldn't happen if the node already has that DCI and that DCI is already known to be unsupported or disabled.

2.2.10 did this correctly. The logic seems broken in 2.2.11 and up.

Did you figure out a workaround to this? We can obviously suppress the alarm by changing the response to SYS_DCI_UNSUPPORTED, but that seems wrong.

Peter

10
Announcements / Re: NetXMS 2.2.8 released
« on: September 04, 2018, 10:11:44 pm »

11
General Support / Re: Parameters in Event Monitor?
« on: September 04, 2018, 03:51:15 pm »
Thank you, Victor!

12
General Support / Parameters in Event Monitor?
« on: August 31, 2018, 11:07:47 pm »
Maybe I'm just dense and not seeing it, but is there a way to display the event parameters in the Event Monitor? I guess the same question goes for a node's EventLog.
Many thanks,

Peter

13
General Support / Re: Max Parameter Length?
« on: July 10, 2018, 09:38:48 pm »
On further investigation, this doesn't seem to be limited to just Service.Check(*). Unless I'm looking at this wrong, all checks and DCIs seem to break after the parameters reach a certain length.

14
General Support / Max Parameter Length?
« on: July 10, 2018, 09:09:55 pm »
I'm trying to use Service.Check(*) to check a URL. Unfortunately, the URL I need to check is really long and looks like this:

https://hostA.mydomain.com/file1.aspx?KaB6rTg6%2fcIKlmbmN%40ecOi%2fjkr%407oXTOfLonrtxEolozX7HjHuvdXzfSZLXGSyM%2fz%2f88GXwVAXPhKIP893OYWC1N4zkKcFEoGnXlORpSV0GyA%40ljEZT3xDK72S46Y6luNEnz0lBYEqb8bzPEJ%40V28pJbIRpMLDp3uKPUnqit19t%40I5uY%40bZtcGI7NXZ5O81c8HDPmmxYci%40xpL7oAqLR9A%3d%3d

The URL above doesn't work and results in a "Cannot get current parameter value: Requested data collection item is not supported by agent" error.
If I shorten the URL slightly, the error goes away and it actually makes the call (but then obviously I have the wrong URL!)

https://hostA.mydomain.com/file1.aspx?KaB6rTg6%2fcIKlmbmN%40ecOi%2fjkr%407oXTOfLonrtxEolozX7HjHuvdXzfSZLXGSyM%2fz%2f88GXwVAXPhKIP893OYWC1N4zkKcFEoGnXlORpSV0GyA%40ljEZT3xDK72S46Y6luNEnz0lBYEqb8bzPEJ%40V28pJbIRpMLDp3uKPUnqit19t%40I5uY%40bZtcG

... works. If I add one more character, it stops working.

Is there a length limit to what's passed to Service.Check? If so, what's the limit?
Thanks!

Peter


15
You can use /CENTRALCONFIG or /LOCALCONFIG to set this (defaults to /LOCALCONFIG).
We use "/VERYSILENT /DIR="C:\Program Files\NetXMS" /CENTRALCONFIG /SERVER=hostname_of_netxms_server /NOSUBAGENT=winperf" to do a completely blank install, and then serve the config from the server.

Peter

Pages: [1] 2 3