Hello @all,
today i updated NetXMS Server (Windows 10 64bit, MySQL 5.7) from V3.7.130 to V3.8.226...
The NetXMS Server seems to run properly but the NetXMS WebUI + Reporting Server (both same Version as NetXMS Server) doesen't start.
The NetXMS Agent (at the NetXMS Server) drops thousands and thousand of errors per second in Windows Eventlog (Message: "CRT Error"). Open the NetXMS Console seems to hang in cause of the thousands of Windows Eventlog Alarms from Agent. We have high CPU Usage on NetXMS Server for the windows eventlog service in cause of the many Errors generated by Agent.
Due to this behaviour I have restored the complete NetXMS Server to the previous Version (V3.7.130).
Are there any prerequesits for V3.8.226 i have to do or install on NetXMS Server that i can upgrade to V3.8.226 ??
Greetings
Marco
Running on Windows server 2008 R2 here and after upgrade to 3.8.226 (from 3.7.95) the WebUI will not start, which is our only method of access.
Further info:
Windows event ID: 7024
Description:
The NetXMS WebUI service terminated with service-specific error The system cannot open the file..
Unfortunately it doesn't tell me which file :-(
Hi Marco,
for the WebUI look whether ther Service will be run as "LocalService" and set it to "LocalSystem".
Cheers Frank
Thanks for the reply, I had seen your post and tried that but it did not work in this instance. I've checked the permissions on the NextXMS folder structure and the account running the service should have access so I don't think the file it can't find is in there. I've re-installed the WebUI a couple of times now, still no joy, the server appears to be running OK, using the java console appears to work and no info is missing, currently at a loss as to what the problem is.
DH
Hey Frank,
thanks for the hint with the Service User... :-) I check it at my next try to install V3.8.
Have anyone any idea whats the "CRT Error" of the Agent means? So i can take a look at my installation to find any errors or prerequesits that i need for the update to 3.8.226?
Greetings
Marco
I am having the exact problem with running the 3.8.226 x64 and 3.8.250 x64 agent running on windows 2012R2, 2016, and 2019 against netxms core 3.4.313 on windows server 2016. The agent will run for a few minutes and then start filling up the system event log with the following error.
Log Name: System
Source: NetXMS Win32 Agent
Date: 4/4/2021 1:14:18 PM
Event ID: 1000
Task Category: None
Level: Error
Keywords: Classic
User: N/A
Computer: mycomputer.mydomain.com
Description:
CRT error detected
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<System>
<Provider Name="NetXMS Win32 Agent" />
<EventID Qualifiers="0">1000</EventID>
<Level>2</Level>
<Task>0</Task>
<Keywords>0x80000000000000</Keywords>
<TimeCreated SystemTime="2021-04-04T17:14:18.692060400Z" />
<EventRecordID>12345</EventRecordID>
<Channel>System</Channel>
<Computer>mycomputer.mydomain.com</Computer>
<Security />
</System>
<EventData>
<Data>CRT error detected</Data>
</EventData>
</Event>
Once the error storm starts, agent computer has high CPU usuage and netxms core starts consuming high memory.
I think I have narrowed it down to having SubAgent = logwatch.nsm enabled. When a logwatch event is raised I receive the CRT error detected.
<parser trace="0" name="LogParser-EL-Application">
<file encoding="UTF-8">*Application</file>
<rules>
<rule name="ApplicationEventLogError" break="true">
<match repeatCount="0" repeatInterval="60">(.*)</match>
<event>EL_APPLICATION_ERROR</event>
<level>1</level>
</rule>
<rule name="ApplicationEventLogWarning" break="true">
<match repeatCount="0" repeatInterval="60">(.*)</match>
<event>EL_APPLICATION_WARNING</event>
<level>2</level>
</rule>
<rule name="ApplicationWindowsErrorReporting" break="true">
<match repeatCount="0" repeatInterval="1">(.*)</match>
<event>EL_APPLICATION_WINDOWS_ERROR_REPORTING</event>
<level>4</level>
<id>1001</id>
<source>Windows Error Reporting</source>
</rule>
</rules>
<macros/>
</parser>
Hi suphu,
many thanks for the infos... We have logwatch.nsm enabled at every NetXMS Agent installation. So, i would wait with the upgrade to 3.8, until a new version is available...
Are there any infos from the NetXMS developers for this (maybe) bug?
Greetings
Marco
Hi,
we likely found the issue. New patch release will be available in a next few days, please re-test this issue then.
Best regards,
Victor
Hi Victor,
thanks for the fast response. I try the new version at release and will post a feedback.
Greetings
Marco
Thanks!
I installed and tested nxagent-3.8.262-x64.exe on Windows Server 2012R2, 2016, and 2019. All appears to be running normally, with no CRT error detected.
Hello @all,
today i installed NetXMS Server V3.8.267: Server runs, no CRT Error from Agent... :-) ... Thanks for the Patch Viktor... :-)
I installed the WebUI Server V3.8.262 too, but the service doesn't start... I tried to run the service as "Lokales System" (as Frank suggested) but the service still doesn't start.
I attached the Log File from the WebUI service.. Maybe one of the Devs can take a look into it...
Greetings
Marco
Hi,
log file just states that service cannot be started. Is there any other log files?
Best regards,
Victor
Hi Victor,
there are three files in the C:\NetXMS\WebUI\logs Directory:
commons-daemon.2021-04-13.log (attached in the last post)
nxwebui-stderr.2021-04-13.log (only one line of content "Failed to start service.")
nxwebui-stdout.2021-04-13.log (empty file)
Are there any log at other locations or can i activate verbose logging for the WebUI Service?
Greetings
Marco
Did you install it on top of old version? If yes, try to remove it and do clean install (make sure to manually delete all remaining work directories after uninstall). Select bundled JVM when installing.
Best regards,
Victor
Hi Victor,
first i installed it over the old version > not worked.
Then i deinstalled it, deleted the c:\netxms\webui manualy and reinstall (with jre) > not worked.
What for directorys i had to clean after doing a "normal" deinstall? Is the "c:\netxms\webui" the only one?
By the way: In the c:\netxms directory is also the NetXMS Server installed...
At WebUI install i let the suggested installation path ("c:\NetXMS") unchanged. NetXMS Web UI is installed in c:\NetXMS\WebUI. The JRE is installed in c:\NetXMS\bin\jre.
Is this correct?
Greetings
Marco
Did you change service to run under "Local System" after clean install?
Best regards,
Victor
Hi Victor,
i have gone thru following steps:
- deinstalled NetXMS WebUI (over Windows Program deinstallation)
- manually deleted c:\NetXMS\WebUI
- installed NetXMS WebUI with netxms-webui-3.8.262-x64.exe
- Service not comes up after installation
- change service user to System Account
- start service > service start fails (please see log files, now there are more informations in log)
Greetings
Marco
Very strange, I just did the same and service started correctly, so it somehow related to environment. Try to install Java 11 from AdoptOpenJDK separately - maybe we miss something when pack embedded JRE. Those are log lines that are most suspicious:
[2021-04-14 09:41:19] [error] ( javajni.c:828 ) [10492] CreateJavaVM Failed with error [-1]
[2021-04-14 09:41:19] [error] ( javajni.c:828 ) [10492] Das System kann die angegebene Datei nicht finden.
Best regards,
Victor
Also found this thread: https://stackoverflow.com/questions/63012015/jni-createjavavm-fails-with-adopt-openjdk-11-0-8 (https://stackoverflow.com/questions/63012015/jni-createjavavm-fails-with-adopt-openjdk-11-0-8). Do you have JAVA_HOME environment set? If yes, try to remove it completely (preferred) or point to C:\NetXMS\bin\jre.
Best regards,
Victor
Hi Victor,
JAVA_HOME was not set. AdoptOpenJDK is installed in V8 + V11, i deinstalled both (needed these only for the NextXMS Client). Then i reinstalled NetXMS WebUI as described in my last post.... Same errors in log..
After that, i deinstalled NetXMS WebUI an reinstalled it at another setup path (new patch: c:\NetXMS_WebUI) an voila: Service is starting and i can access NetXMS with the WebUI...
I would now leave the setup as it is.... Can i delete the c:\netxms\bin\jre path? or is this jre needed for/from the NetXMS Server (i have the NetXMS Server installed at c:\NetXMS)
Greetings
Marco
Hi,
yes, I think you can delete c:\netxms\bin\jre. I have no idea why it was not working when installed into same directory. But we are going to merge server ad web UI installers into one anyway in version 4, so it should not be an issue after that.
Best regards,
Victor