NetXMS Support Forum

English Support => General Support => Topic started by: xelander on July 29, 2020, 11:51:47 AM

Title: DCI last values don't store history data
Post by: xelander on July 29, 2020, 11:51:47 AM
Hello,
I'm new to NetXMS, I just installed it on a CentOS 8 VM with MariaDB 10.3.17 and it's working fine, but there is one thing I cannot understand.

I configured an internal DCI (ICMP.ResponseTime.Max) and an SNMP DCI (.1.3.6.1.2.1.2.2.1.11.1 - ifInUcastPkts) and they seem to work fine, since I can see the last values.
But when I try to see the History or a Chart for a value they are both empty (see attachments).

Can you help me? Am I doing something wrong?

Thanks,
Alessandro
Title: Re: DCI last values don't store history data
Post by: Victor Kirhenshtein on July 29, 2020, 02:30:17 PM
Hi,

could be database issue or time zone issue. In database you should have table idata_nnn, where nnn is actual node object ID. Check if you have any records in that table. If yes, check that time on your server is correct and that time zone is correct as well. Internally server and client uses UTC timestamps, so even when time looks identical it may in fact be incorrect on one side when concerted to UTC. If there are no records in idata table then likely there is issue with storing data to database - in that case check server log for SQL errors.

Best regards,
Victor
Title: Re: DCI last values don't store history data
Post by: xelander on July 29, 2020, 05:54:57 PM
Hi,
thank you very much for your answer, I checked the DB and the data seems to be there (see screeshots).

Timestamps seem correct too: in the table they're UTC and on the server and client the time is CEST (UTC+1, +2 with daylight savings).
Even if they were wrong, shouldn't I at least see the values in the "History" view, albeit with wrong timestamps?

Thanks,
Alessandro
Title: Re: DCI last values don't store history data
Post by: Filipp Sudanov on July 31, 2020, 05:19:05 PM
This looks to be very strange. I would suggest to turn on debugging from the database side to see what exactly queries are generated when you request history / graph from the front-end. And then checking if such a query really returns any data.
Also, if it's fine for you to recreate the database you can try to move to Postgres - it's generally more tested because more netxms users are using it.
Title: Re: DCI last values don't store history data
Post by: Filipp Sudanov on August 03, 2020, 12:42:50 PM
Also - 3.4.284 is now available, it has a fix for one error that was also related to not displaying historic data, could be that it's the same error here. Please try this new version.
Title: Re: DCI last values don't store history data
Post by: xelander on August 03, 2020, 03:29:45 PM
Hello,
I enabled query logging on the DB and the query seems to be correct, and it returns valid data (see attachements):

SELECT idata_timestamp,idata_value,raw_value FROM idata_2069 WHERE item_id=682 ORDER BY idata_timestamp DESC LIMIT 4096

I'll try the new version as soon as possible, and I'll let you know.

Thanks,
Alessandro

Title: Re: DCI last values don't store history data
Post by: xelander on August 04, 2020, 10:43:10 AM
Hi,
I upgraded to version 3.4.284, both the server and the management console, but unfortunately nothing changed.

The query is the same:
SELECT idata_timestamp,idata_value,raw_value FROM idata_2069 WHERE item_id=682 ORDER BY idata_timestamp DESC LIMIT 4096
but still no data is displayed in the History view or in the Charts view, and I cannot find any error in MariaDB nor Netxms logs.

Is there anything alse I can try to debug the problem?

Thanks,
Alessandro

Title: Re: DCI last values don't store history data
Post by: Victor Kirhenshtein on August 04, 2020, 11:01:27 AM
Try to change DCI data type to "String" and request history - will it show something?

Best regards,
Victor
Title: Re: DCI last values don't store history data
Post by: xelander on August 04, 2020, 11:12:10 AM
Nothing changed, sorry  :'(



Title: Re: DCI last values don't store history data
Post by: Filipp Sudanov on August 05, 2020, 05:24:47 PM
How did you compile netxms - can you show config.log file? Can you show the output of yum list installed
Title: Re: DCI last values don't store history data
Post by: xelander on August 06, 2020, 09:18:36 AM
Hi,I followed the directions on the "NetXMS Administrator Guide, Release 3.4.178", chapter "3.6  Installing from sources", that is:
# ./configure --with-server --with-mysql --with-agent
# make
# make install

I'm attaching the config.log and the output of yum list installed.If you need anything else, please let me know.

Thank you,Alessandro

Title: Re: DCI last values don't store history data
Post by: Filipp Sudanov on August 06, 2020, 03:00:32 PM
As an experiment - can you try to a different configure command:

./configure prefix=/opt/netxms-master --with-server --with-mariadb --with-mariadb-compat-headers --with-agent
Title: Re: DCI last values don't store history data
Post by: xelander on August 06, 2020, 05:04:37 PM
I tried (see attached config.log), but the issue is still present.
The query is the same as before:200806 15:54:06    266 Prepare  SELECT idata_timestamp,idata_value,raw_value FROM idata_2069 WHERE item_id=? ORDER BY idata_timestamp DESC LIMIT 4096
                   266 Execute  SELECT idata_timestamp,idata_value,raw_value FROM idata_2069 WHERE item_id=683 ORDER BY idata_timestamp DESC LIMIT 4096
                   266 Close stmt

I was wondering if it could be a client issue, instead of a server one.
After all, if the server had a problem SELECT'ing data from the DB, almost nothing would work...

Title: Re: DCI last values don't store history data
Post by: Filipp Sudanov on August 06, 2020, 05:46:12 PM
What's the operating system where the client is running? What's client version? Is it downloaded from netxms web site?
For testing - may be you could try the client on another machine, preferrably on a different OS.
Title: Re: DCI last values don't store history data
Post by: xelander on August 06, 2020, 06:03:55 PM
I tried the Windows client (nxmc-3.4.284-win32-x64.zip (https://www.netxms.org/download/releases/3.4/nxmc-3.4.284-win32-x64.zip)) both on Windows 7 and on Windows 10, but I also tried the web interface (nxmc-3.4.284.war (https://www.netxms.org/download/releases/3.4/nxmc-3.4.284.war)), everything downloaded from the netxms website, so it seems the issue doesn't depend on the client OS.
I still have to try on Linux...
Title: Re: DCI last values don't store history data
Post by: xelander on August 07, 2020, 01:42:27 PM
Today I also tried on Linux CentOS 8, (nxmc-3.4.284-linux-gtk-x64.tar.gz (https://www.netxms.org/download/releases/3.4/nxmc-3.4.284-linux-gtk-x64.tar.gz)) but nothing changed: no historical data.This is so strange that now I wonder if I'm doing something wrong...

Title: Re: DCI last values don't store history data
Post by: Filipp Sudanov on August 07, 2020, 02:22:48 PM
Does it happen that there are no historical data for all DCIs in system, or just for some DCIs?
Title: Re: DCI last values don't store history data
Post by: xelander on August 07, 2020, 03:01:27 PM
I've not configured many DCIs yet, but it seems no DCI shows historical data.
I've added a couple of other DCI on a different switch (3Com vs CISCO), but still no historical data in the GUI.

Even default DCIs for the NetXMS server itself show no historical data (see attachments).
The only strange thing I'm noticing now, is the <<ERROR>> value for the only DCI coming from the NetXMS agent on the server.
Title: Re: DCI last values don't store history data
Post by: Filipp Sudanov on August 07, 2020, 06:38:09 PM
One more thing to have a look at - console writes it's own log file in user's home folder in \.nxmc\data\.metadata
Can you share what you have there?
Title: Re: DCI last values don't store history data
Post by: xelander on August 07, 2020, 07:14:01 PM
Here they are.
Title: Re: DCI last values don't store history data
Post by: Filipp Sudanov on August 08, 2020, 02:04:18 AM
Did you try to open the dci graph / history during the duration of that log?
Title: Re: DCI last values don't store history data
Post by: xelander on August 08, 2020, 08:01:43 PM
Yes, of course.
Title: Re: DCI last values don't store history data
Post by: Filipp Sudanov on August 10, 2020, 04:57:36 PM
Let's try to get historical data from java API using nxshell - this way we could see if the issue is with the server or the GUI.

You will need to recompile to get nxshell, for that please run
export JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64
add then run your ./configure command with the following keys added:
--with-jdk=/usr/lib/jvm/java-11-openjdk-amd64 --with-client

make & make install should produce nxshell executable in your bin folder.

Then make a file, say collected_data.py
from java.util import Date
data = s.getCollectedData(200, 2000, Date(Date().time - 60 * 60 * 1000), Date(), 100, HistoricalDataType.valueOf('PROCESSED'))
print data
for v in data.values:
    print v.timestamp.toString() + " = " + str(v.value)

replace 200 with a node ID and 2000 with a DCI ID.

Then run the following (-u username, same as in management console and -P is the password):
nxshell -H 127.0.0.1 -u admin -P ""  collected_data.py
and share the output. This script script should return datapoints for a give DCI for last hour.

Title: Re: DCI last values don't store history data
Post by: xelander on August 11, 2020, 11:23:25 AM
Hi, here they are.
Node 2069, DCI 682
nxshell -H 127.0.0.1 -u admin -P XXXX collected_data.py
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.python.core.PySystemState (file:/usr/local/lib/netxms/nxshell.jar) to method java.io.Console.encoding()
WARNING: Please consider reporting this to the maintainers of org.python.core.PySystemState
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
DciData [nodeId=2069, dciId=682, dataType=INT32, valuesSize=0]

Node 2069, DCI 683nxshell -H 127.0.0.1 -u admin -P XXXX collected_data.py
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.python.core.PySystemState (file:/usr/local/lib/netxms/nxshell.jar) to method java.io.Console.encoding()
WARNING: Please consider reporting this to the maintainers of org.python.core.PySystemState
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
DciData [nodeId=2069, dciId=683, dataType=INT32, valuesSize=0]


It seems nxshell doesn't get any data too  :(
Title: Re: DCI last values don't store history data
Post by: Filipp Sudanov on August 11, 2020, 12:07:31 PM
Can we possible get remote access to your netxms server? If yes, please send a message to [email protected]

If no, please collect server log at debuglevel = 8. Start console during the log and open the graph and history of a DCI.
You can change debut level on the fly:
nxadm -c "debug 8"
will change it

nxadm -c "debug"
will print current debug level
Title: Re: DCI last values don't store history data
Post by: xelander on August 11, 2020, 12:41:29 PM
Yes, I can set up a remote access for you, thank you very much.
I have to setup some port forwarding on our firewall: do you only need SSH or also port 4701 for the console?
Can you tell me the public IP address you will be using to connect?

In the meanwhile, I changed the debug level and opened the history and graph of DCI 682 and 683 of node 2069, see attached log.I think they are at line 5744, timestamp 2020.08.11 11:21:55.828

Title: Re: DCI last values don't store history data
Post by: Filipp Sudanov on August 11, 2020, 01:53:10 PM
Sent you a PM.

Meanwhile - it's recommended to check netxms log for *E* and *W* messages - there is one SQL query error, which is not related to issue discussed here, we will have a look. And there's also a warning about autobind script.
Title: Re: DCI last values don't store history data
Post by: xelander on August 11, 2020, 03:55:13 PM
The SQL errors are related to two new groups I was creating (see screenshot), perhaps I made something wrong.
About the autobind script, I don't know, I'm new to NetXMS and I'm still exploring all its features.
I remember opening the script library, perhaps I unvoluntarily tampered with the scripts.

Title: Re: DCI last values don't store history data
Post by: Filipp Sudanov on August 11, 2020, 05:20:23 PM
There was one thing we missed - after recompiling --with-mariadb --with-mariadb-compat-headers, database driver in netxmsd.conf should have been updated, but mysql.ddr was left there.
The correct configuration is:

DBDriver = mariadb.ddr

P.S. pls disable system user in netxms - it's currently enabled.

Title: Re: DCI last values don't store history data
Post by: xelander on August 11, 2020, 05:44:26 PM
Oh! I'm really sorry, I completely missed this step  :-[ .
Thank you very, very much for your support and for your time.
P.S. I disabled the system user, thank you.

Title: Re: DCI last values don't store history data
Post by: logic_sam on December 11, 2020, 04:48:48 PM
 :) :) :)
(https://sc.mogicons.com/c/179.jpg)

BIG HUG FOR Filipp Sudanov

- NetXMS on CentOS 8
- MariaDB

Data was not written in the history.
I was able to see "last values" but no history, no graphs.
netxms data collection was not working (i add this for google searches)

Thanks to Filipp :
- i learned to look for datas in "idata_xxx" tables
- i recompiled with mariadb options
- i changed netxmsd.conf to "DBDriver = mariadb.ddr"

God bless you Filipp !


for the records :

./configure --with-server --with-mariadb --with-mariadb-compat-headers --with-agent
/usr/local/etc/netxmsd.conf
DBDriver = mariadb.ddr