NetXMS Support Forum

Victor,

First, thank you very much for all this support. I would be happy to find a way to donate, or, is Raden Solutions a direct connection to you folks? I am completely open to changing my OS from Win2008 R2 64bit to anything you suggest for better outcomes. Although I am also happy to continue helping us both resolve this issue.

Just let me know.

Regards,

Thanks Victor,

I tried all of this, I'm now running on 1.07 with the ProcessAffinityMask = 1.

I received the following crash dump. The system ran for about 4 hours before the application faulted. Here is the contents of the dump file.

NETXMSD CRASH DUMP
Tue Nov 09 11:52:15 2010

EXCEPTION: C0000005 (Access violation) at 776AC120

NetXMS Version: 1.0.7
OS Version: Windows Server 2008 R2 Build 7600
Processor architecture: Intel x86

Register information:
  eax=00163F00  ebx=00059000  ecx=00006394  edx=00000000
  esi=00163EFC  edi=00163FFC  ebp=04F1ECE0  esp=04F1ECD8
  cs=0023  ds=002B  es=002B  ss=002B  fs=0053  gs=002B  flags=00010612

Call stack:
  [msvcrt:776AC120]: (function-name not available)
  [libnetxms:C:\Source\NetXMS-1.0.x\src\libnetxms\queue.cpp:95]: Queue::Put
  [nxcore:C:\Source\NetXMS-1.0.x\src\server\core\dbwrite.cpp:55]: QueueSQLRequest
  [nxcore:C:\Source\NetXMS-1.0.x\src\server\core\dcitem.cpp:1017]: DCItem::processNewValue
  [nxcore:C:\Source\NetXMS-1.0.x\src\server\core\node.cpp:3514]: Node::processNewDciValue
  [nxcore:C:\Source\NetXMS-1.0.x\src\server\core\datacoll.cpp:147]: DataCollector
  [(module-name not available):04F1FF48]: (function-name not available)
  [(module-name not available):03808570]: (function-name not available)

Seems to be crashing every 7-9 hours, same debug info in the dump file each time. Tomorrow I'll try to run those nxadm debug statements every hour or so in order to track the situation.

Thanks

Here is the dump capture, the mdmp file was empty.


NETXMSD CRASH DUMP
Sat Nov 06 12:33:35 2010

EXCEPTION: C0000005 (Access violation) at 7571C120

NetXMS Version: 1.0.5
OS Version: Windows Server 2008 R2 Build 7600
Processor architecture: Intel x86

Register information:
 eax=0015DFC0  ebx=00057800  ecx=0002F9F3  edx=00000000
 esi=0015DFBC  edi=0015DFFC  ebp=054DECE0  esp=054DECD8
 cs=0023  ds=002B  es=002B  ss=002B  fs=0053  gs=002B  flags=00010616

Call stack:
 [msvcrt:7571C120]: (function-name not available)
 [libnetxms:0022A532]: (function-name not available)
 [nxcore:1000F10B]: (function-name not available)
 [nxcore:100114E4]: (function-name not available)
 [(module-name not available):054DEE74]: (function-name not available)
 [(module-name not available):4B8AF098]: (function-name not available)
 [libnxsrv:002564BA]: (function-name not available)

I enabled the crash dumps, next time it happens I'll post.

These commands were run about 2-3 minutes after a reboot to get the system operational again. I'll run them again in a few hours to see if something sticks out later on.

show pollers:



show queues:

netxmsd: show queues
Condition poller                 : 0
Configuration poller             : 3590
Data collector                   : 7096
Database writer                  : 0
Event processor                  : 0
Network discovery poller         : 0
Node poller                      : 1
Routing table poller             : 2307
Status poller                    : 1940

show flags:

netxmsd: show flags
Flags: 0x43300257
  AF_DAEMON                        = 1
  AF_USE_SYSLOG                    = 1
  AF_ENABLE_NETWORK_DISCOVERY      = 1
  AF_ACTIVE_NETWORK_DISCOVERY      = 0
  AF_LOG_SQL_ERRORS                = 1
  AF_DELETE_EMPTY_SUBNETS          = 0
  AF_ENABLE_SNMP_TRAPD             = 1
  AF_ENABLE_ZONING                 = 0
  AF_SYNC_NODE_NAMES_WITH_DNS      = 0
  AF_CHECK_TRUSTED_NODES           = 1
  AF_WRITE_FULL_DUMP               = 0
  AF_RESOLVE_NODE_NAMES            = 1
  AF_CATCH_EXCEPTIONS              = 1
  AF_INTERNAL_CA                   = 0
  AF_DB_LOCKED                     = 1
  AF_ENABLE_MULTIPLE_DB_CONN       = 1
  AF_DB_CONNECTION_LOST            = 0
  AF_NO_NETWORK_CONNECTIVITY       = 0
  AF_EVENT_STORM_DETECTED          = 0
  AF_SERVER_INITIALIZED            = 1
  AF_SHUTDOWN                      = 0

show mutex:

netxmsd: show mutex
Mutex status:
  g_hMutexIdIndex: unlocked
  g_hMutexNodeIndex: unlocked
  g_hMutexSubnetIndex: unlocked
  g_hMutexInterfaceIndex: unlocked

Thanks

Server is, Windows 2008 R2 64 Bit with SQL Server 2008 db on a separate machine. Running 1.0.5 server with 1.0.6 clients.

The following error is happening,



We have about 450 agents and maybe a total of 1500 nodes. Only about 4-6 DCI per node, vast majority are just monitoring windows event logs for errors and criticals.

I am noticing that we are getting a fair amount of agents dropping connectivity and then restoring in blocks of about 10-20 every 10-20 minutes. The crashes only started after we went from about 300 agents installed to 450 agents installed. I put the numberofstatuspollers up to 50 at first and that didn't seem to help, now I am putting the number back down to 30. That seems related but I'm not sure. The netowrk utilization is steady at about 2-3 percent. The processor is steady at about 10-20 percent. Disk IO is nothing...

Any suggestions based on this information would be helpful.

I have configured DCI's for checking drive space that are working on the 1.06 32 bit client but the x64 clients are never executing, even though the items are defined and active on the client. I noticed by default the x64 client was showing them as disabled, but I set them active and they still never update.

As I wrote this post I then realized the FileSystem.FreePerc is not working on the x64 agent but the Disk.FreePerc is working on the x64 client.

I was going to cancel the post but I thought it would be good information to share.

Thanks

Thanks, I just removed the filtering script since I think I wasn't using it for the purpose intended anyhow.

Windows 2008 Enterprise R2 64 bit
SQL Server DB backend
netXMS Server 1.04

QuoteScript (Agentcfg: :1) execution error: Error in line 9: syntax error, unexpected $end

Any ideas on what the source of this would be or how to correct?

Thanks,

Victor,

Yes, thank you that seemed to correct the processor pegging issue.

Regards,

Clark

Windows 2003 R2
32bit systems running the 1.04 client

The netXMS client is monitoring system, security, and application windows event logs and 1 file log from a jboss/tomcat server using the following configuration. Whenever I restart the jboss/tomcat portal service the NXAGENTD peg's the processor at 100% utilization and my server takes 30 minutes to restart the portal. Even when the portal finally completes and is started the NXAGENTD service continues to hammer the processor until I kill the process and restart the service.

I have this exact scenario happening on two different systems. Both are the same environment.

Any input on how I can keep the process from going crazy and/or pinpoint why it is happening? The only thing I have played around with is the processAll option, I have turned that on and off a couple times.

<parser processAll="0">
   <file>C:\portal\server\default\log\server.log</file>
   <macros>
      <macro name="timestamp">[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2},[0-9]{3}</macro>
   </macros>
   <rules>
      <rule break="1">
         <match>@{timestamp}.ERROR (.*)</match>
         <event params="1">100002</event>
      </rule>
        </rules>
</parser>

Thanks,

In the meantime here is the solution I implemented based on your suggestion. I just call this from a batch file and run it on reboot and on a daily schedule then restart the service. If you redefine the filter and/or the path it should work for most people who would need it (on windows systems).

Also the log parser def in this sample was used to parse errors from a JBOSS-TOMCAT 4.2 bundle stdout log.

This code should generate this sort of output,

<parser processAll="1">
<file>C:\Program Files\ApplicationName\logs\stdout_20091207.log</file>
<macros>
<macro name="timestamp">[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2},[0-9]{3}</macro>
</macros>
<rules>
<rule break="1">
<match>@{timestamp}.ERROR (.*)</match>
<event params="1">100002</event>
</rule>
</rules>
</parser>


LogConfig.vbs

Dim defaultLogFileName
Dim netXMSAgentCFGFileName
Dim quoteChar
Dim Filter

Dim dynamicLogLocation

dynamicLogLocation = "C:\Program Files\ApplicationName\logs\"

' Define the filter used (if desired) to find a specific log file in a folder
Filter = "out*.log"

' Used if unable to find a log file in the folder
defaultLogFileName = "out.log"

' Location and name of log file definition
netXMSAgentCFGFileName = "C:\NetXMS\var\dynamic_log.xml"

quoteChar = CHR(34)


Const OverwriteFile = 2
Set objFSO = CreateObject("Scripting.FileSystemObject")

logFileName = GetNewestFile(dynamicLogLocation)
If logFileName = "" Then
  logFileName = defaultLogFileName
End If

Set objTextFile = objFSO.OpenTextFile(netXMSAgentCFGFileName, OverwriteFile, True)

objTextFile.WriteLine("<parser processAll=" + quoteChar + "1" + quoteChar + ">")
objTextFile.WriteLine(vbTab + "<file>" + logFileName + "</file>")
objTextFile.WriteLine(vbTab + "<macros>")
objTextFile.WriteLine(vbTab + vbTab + "<macro name=" + quoteChar + "timestamp" + quoteChar + ">[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2},[0-9]{3}</macro>")
objTextFile.WriteLine(vbTab + "</macros>")
objTextFile.WriteLine(vbTab + "<rules>")
objTextFile.WriteLine(vbTab + vbTab + "<rule break=" + quoteChar + "1" + quoteChar + ">")
objTextFile.WriteLine(vbTab + vbTab + vbTab + "<match>@{timestamp}.ERROR (.*)</match>")
objTextFile.WriteLine(vbTab + vbTab + vbTab + "<event params=" + quoteChar + "1" + quoteChar + ">100002</event>")
objTextFile.WriteLine(vbTab + vbTab + "</rule>")
objTextFile.WriteLine(vbTab + "</rules>")
objTextFile.WriteLine("</parser>")
objTextFile.Close

'Then ...  net stop NetXMSAgentdW32 and net start NetXMSAgentdW32

Function GetNewestFile(ByVal sPath)

  sNewestFile = Null   ' init value


  Set oFSO = CreateObject("Scripting.FileSystemObject")
  Set oFolder = oFSO.GetFolder(sPath)
  Set oFiles = oFolder.Files

  ' enumerate the files in the folder, finding the newest file
  For Each oFile In oFiles
    On Error Resume Next
    If CompareFileName(oFile.Name,Filter) Then
       If IsNull(sNewestFile) Then
         sNewestFile = oFile.Path
         dPrevDate = oFile.DateLastModified
       Elseif dPrevDate < oFile.DateLastModified Then
         sNewestFile = oFile.Path
       End If
    End If
    On Error Goto 0
  Next

  If IsNull(sNewestFile) Then sNewestFile = ""

  GetNewestFile = sNewestFile

End Function


Private Function CompareFileName (ByVal Name, ByVal Filter) ' (recursive)
  CompareFileName = False
  Dim np, fp: np = 1: fp = 1
  Do
     If fp > Len(Filter) Then CompareFileName = np > len(name): Exit Function
     If Mid(Filter,fp) = ".*" Then    ' special case: ".*" at end of filter
        If np > Len(Name) Then CompareFileName = True: Exit Function
        End If
     If Mid(Filter,fp) = "." Then     ' special case: "." at end of filter
        CompareFileName = np > Len(Name): Exit Function
        End If
     Dim fc: fc = Mid(Filter,fp,1): fp = fp + 1
     Select Case fc
        Case "*"
           CompareFileName = CompareFileName2(name,np,filter,fp)
           Exit Function
        Case "?"
           If np <= Len(Name) And Mid(Name,np,1) <> "." Then np = np + 1
        Case Else
           If np > Len(Name) Then Exit Function
           Dim nc: nc = Mid(Name,np,1): np = np + 1
           If Strcomp(fc,nc,vbTextCompare)<>0 Then Exit Function
        End Select
  Loop
End Function

Private Function CompareFileName2 (ByVal Name, ByVal np0, ByVal Filter, ByVal fp0)
  Dim fp: fp = fp0
  Dim fc2
  Do                                  ' skip over "*" and "?" characters in filter
     If fp > Len(Filter) Then CompareFileName2 = True: Exit Function
     fc2 = Mid(Filter,fp,1): fp = fp + 1
     If fc2 <> "*" And fc2 <> "?" Then Exit Do
     Loop
  If fc2 = "." Then
     If Mid(Filter,fp) = "*" Then     ' special case: ".*" at end of filter
        CompareFileName2 = True: Exit Function
        End If
     If fp > Len(Filter) Then         ' special case: "." at end of filter
        CompareFileName2 = InStr(np0,Name,".") = 0: Exit Function
        End If
     End If
  Dim np
  For np = np0 To Len(Name)
     Dim nc: nc = Mid(Name,np,1)
     If StrComp(fc2,nc,vbTextCompare)=0 Then
        If CompareFileName(Mid(Name,np+1),Mid(Filter,fp)) Then
           CompareFileName2 = True: Exit Function
           End If
        End If
  Next
  CompareFileName2 = False
End Function


How can I setup a logmonitor parser for a variable filename?

log_20100920.txt

The filename is not reset daily it is reset each time the server is restarted, so what I want is to monitor the most recent created filename in the folder starting with log_

Is this accomplished using a macro? If so what the best way to approach this?

Thanks for any input