NetXMS Support Forum

Hello @all,

today i updated NetXMS Server (Windows 10 64bit, MySQL 5.7) from V3.7.130 to V3.8.226...

The NetXMS Server seems to run properly but the NetXMS WebUI + Reporting Server (both same Version as NetXMS Server) doesen't start.

The NetXMS Agent (at the NetXMS Server) drops thousands and thousand of errors per second in Windows Eventlog (Message: "CRT Error"). Open the NetXMS Console seems to hang in cause of the thousands of Windows Eventlog Alarms from Agent. We have high CPU Usage on NetXMS Server for the windows eventlog service in cause of the many Errors generated by Agent.

Due to this behaviour I have restored the complete NetXMS Server to the previous Version (V3.7.130).

Are there any prerequesits for V3.8.226 i have to do or install on NetXMS Server that i can upgrade to V3.8.226 ??


Greetings
Marco

Running on Windows server 2008 R2 here and after upgrade to 3.8.226 (from 3.7.95) the WebUI will not start, which is our only method of access.

Further info:
Windows event ID: 7024
Description:
  The NetXMS WebUI service terminated with service-specific error The system cannot open the file..

Unfortunately it doesn't tell me which file :-(

Hi Marco,

for the WebUI look whether ther Service will be run as "LocalService" and set it to "LocalSystem".

Cheers Frank

Thanks for the reply, I had seen your post and tried that but it did not work in this instance. I've checked the permissions on the NextXMS folder structure and the account running the service should have access so I don't think the file it can't find is in there. I've re-installed the WebUI a couple of times now, still no joy, the server appears to be running OK, using the java console appears to work and no info is missing, currently at a loss as to what the problem is.

DH

Hey Frank,

thanks for the hint with the Service User...  :-)   I check it at my next try to install V3.8.

Have anyone any idea whats the "CRT Error" of the Agent means?  So i can take a look at my installation to find any errors or prerequesits that i need for the update to 3.8.226?

Greetings
Marco

I am having the exact problem with running the 3.8.226 x64 and 3.8.250 x64 agent running on windows 2012R2, 2016, and 2019 against netxms core 3.4.313 on windows server 2016.  The agent will run for a few minutes and then start filling up the system event log with the following error.
Log Name:      System
Source:        NetXMS Win32 Agent
Date:          4/4/2021 1:14:18 PM
Event ID:      1000
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      mycomputer.mydomain.com
Description:
CRT error detected
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="NetXMS Win32 Agent" />
    <EventID Qualifiers="0">1000</EventID>
    <Level>2</Level>
    <Task>0</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2021-04-04T17:14:18.692060400Z" />
    <EventRecordID>12345</EventRecordID>
    <Channel>System</Channel>
    <Computer>mycomputer.mydomain.com</Computer>
    <Security />
  </System>
  <EventData>
    <Data>CRT error detected</Data>
  </EventData>
</Event>

Once the error storm starts, agent computer has high CPU usuage and netxms core starts consuming high memory.

I think I have narrowed it down to having SubAgent = logwatch.nsm enabled.  When a logwatch event is raised I receive the CRT error detected.
<parser trace="0" name="LogParser-EL-Application">
   <file encoding="UTF-8">*Application</file>
   <rules>
      <rule name="ApplicationEventLogError" break="true">
         <match repeatCount="0" repeatInterval="60">(.*)</match>
         <event>EL_APPLICATION_ERROR</event>
         <level>1</level>
      </rule>
      <rule name="ApplicationEventLogWarning" break="true">
         <match repeatCount="0" repeatInterval="60">(.*)</match>
         <event>EL_APPLICATION_WARNING</event>
         <level>2</level>
      </rule>
      <rule name="ApplicationWindowsErrorReporting" break="true">
         <match repeatCount="0" repeatInterval="1">(.*)</match>
         <event>EL_APPLICATION_WINDOWS_ERROR_REPORTING</event>
         <level>4</level>
         <id>1001</id>
         <source>Windows Error Reporting</source>
      </rule>
   </rules>
   <macros/>
</parser>

Hi suphu,

many thanks for the infos... We have logwatch.nsm enabled at every NetXMS Agent installation. So, i would wait with the upgrade to 3.8, until a new version is available...

Are there any infos from the NetXMS developers for this (maybe) bug?

Greetings
Marco

Hi,

we likely found the issue. New patch release will be available in a next few days, please re-test this issue then.

Best regards,
Victor

Hi Victor,

thanks for the fast response. I try the new version at release and will post a feedback.

Greetings
Marco

Thanks!
I installed and tested nxagent-3.8.262-x64.exe on Windows Server 2012R2, 2016, and 2019.  All appears to be running normally, with no CRT error detected.

Hello @all,

today i installed NetXMS Server V3.8.267: Server runs, no CRT Error from Agent... :-) ... Thanks for the Patch Viktor... :-)

I installed the WebUI Server V3.8.262 too, but the service doesn't start... I tried to run the service as "Lokales System" (as Frank suggested) but the service still doesn't start.

I attached the Log File from the WebUI service.. Maybe one of the Devs can take a look into it...


Greetings
Marco

Hi,

log file just states that service cannot be started. Is there any other log files?

Best regards,
Victor

Hi Victor,

there are three files in the  C:\NetXMS\WebUI\logs  Directory:

commons-daemon.2021-04-13.log    (attached in the last post)
nxwebui-stderr.2021-04-13.log          (only one line of content  "Failed to start service.")
nxwebui-stdout.2021-04-13.log         (empty file)

Are there any log at other locations or can i activate verbose logging for the WebUI Service?

Greetings
Marco