NetXMS Support Forum

Hi NetXMS team,

recently, I was investigating the usage of TimescaleDB for the underlying NetXMS database. The system works fine, however when taking a look at the implementation, I noticed that it only uses the feature of Hypertables and does not use Continuous Aggregation options.

I tried to set up Continuous Aggregation on my local instance of NetXMS by adding VIEWs to the DB schema, using the TimescaleDB tutorials as a guide (https://docs.timescale.com/latest/using-timescaledb/continuous-aggregates).

The SQL code for my tests is added in the file continuous_aggregation_for_netxms_tsdb.sql.

For testing, I only used the idata_sc_default table and left all DCIs in the default storage class. The SQL code basically creates five "layers" of continuous aggregation, each with decreasing granularity compared to the previous one:

• Older than 1 hour: Calc average over 3 minutes intervals (idata_continuous_3minutes_gt_1hour)
• Older than 1 day: Clac average over 5 minutes intervals (idata_continuous_5minutes_gt_1day)
• Older than 1 week: Clac average over 10 minutes intervals (idata_continuous_10minutes_gt_1week)
• Older than 1 month: Clac average over 30 minutes intervals (idata_continuous_30minutes_gt_1month)
• Older than 3 months: Clac average over 1 hour intervals (idata_continuous_1hour_gt_3months)

All these VIEWs are then combined in one single VIEW idata_continuous, including the original data for the most recent 60 minutes.

The VIEW itself works as expected, however, making NetXMS actually read the data through this VIEW it not possible without changing NetXMS sourcecode, as by default when using TSDB syntax, NetXMS reads the data directly from the storage class (idata_sc_XYZ) tables and not from the combined idata VIEW.

-----------------------------------------------

Now for my actual feature request (sorry btw. for the verbose introduction  )

Would it be possible to introduce this as an optional feature? As a first implementation, I thought about a globally definable policy for multiple levels of Continuous Aggregation (possibly even define individual aggregation functions?), which could be configured via the NXMC GUI. As this feature does only use VIEWs, which are mostly read-only, they do not alter any materialized data in the DB and thus could even be configured on an existing database without breaking the system.

The most obvious advantage of this feature would be a huge performance boost for read queries that request large amounts of data over long period of time (e.g. 1 year of 60s data points) where it is of greater interest to observe a certain trend of data instead of the actual detailed datapoints. This would especially be useful in combination with Grafana, where it is quite common to increase the size of the displayed timeframe to investigate certain data-trends instead of being interested in exact values of very old data.

One minor downside of this would be increased storage consumption, as those VIEWS are updated periodically by TimescaleDB and thus store additional data next to the original values.


As for the actual implementation, there are two problems which I would consider hard to overcome - How to handle string/text based data values and how to handle table DCIs?

Anyway, this is just meant as an idea and to provide some suggestions for potential ways to implement it.

Hoping for your feedback,
best regards
fbu

Dear NetXMS team,

I seem to have found a scenario related to DCI Data Recalculation, during which the server (Version 3.2-400) does not behave as expected. The scenario is as follows:

For some of my DCIs I use secondary DCIs, which feature a longer retention period and a bigger polling interval, to avoid storing data with fine granularity over a long period of time.

These secondary DCIs use different aggregation scripts, based on what kind of data the primary DCI returns. The most common script is a simple average calculation, like this:

return GetAvgDCIValue($node, FindDCIByDescription($node, "gateway.RTT"), time()-3600, time());

You can see that (with a polling interval of 1 hour) the secondary DCI simply returns the average value of the primary DCI for the last hour.

This method works perfectly fine, as long as the server is running.

And this is where the problem comes in:

If for some reason, the server was down and the secondary DCIs could not poll any data (the primary DCIs still gathers data, as it is coming from a NetXMS Agent, which stores the data on its local DB during server downtime), I want to use the DCI Data Recalculation feature, to recalculate the secondary DCI data, based on the data that the primary DCI could still gather during the downtime via the Agent.

However, it seems like the NXSL function time() always returns the actual current timestamp, instead of the timestamp of the value to recalculate.
Just to make it a little clearer: Imagine the current timestamp is X and the server should recalculate a DCI value at the timestamp Y (with Y < X). The transformation script shown above is supposed to calculate the average of the primary DCI for the period of one hour before timestamp Y and not X.

This behavior leads to the situation that for every recalculated value of the secondary DCI, the time() function returns timestamp X, meaning the recalculated average value will always be the one for the latest one hour period (relative to actual present), which is not correct.


One way I could see to resolve this problem is to have an option for the DCI Data Recalculation, which makes the server always take the timestamp of the value that is currently being recalculated and (temporarily) set this as the current timestamp, which will then be returned by the time() function, just for the process of data recalculation.

Additionally, I could also think of an option, which allows to "fill gaps" in the data (e.g. caused by server downtime). Maybe by checking the polling interval and generating the missing values. If no RAW values are present, the transformation script could still be executed. For a script like the one above, this would still return values to the final data.


Hopefully, these thoughts will be helpful to you.

Thank you and kind regards,
fbu

Dear NetXMS,

I am currently trying to receive data from our NetXMS Server through the REST API.

Logging in via the basic Authentication header works fine and GET requests to /objects.*, /alarms.*, /info or /predefinedgraphs endpoints returns proper data on valid and error messages on invalid requests.

However, using POST requests on the /sessions and /summaryTable/adHoc endpoints results in {"error":46} responses, when adding a JSON object to the request.

Omitting the JSON object results in an "Invalid argument" message.

Digging through the sourcecode of the latest version (3.0.2329) and cross-checking with the Tomcat9 logs, I could find out that when not adding a JSON object to the request, the logging message in line 100 of AbstractHandler.java ("No POST data in call") is being printed and the method returns the expected error object.

But: When a JSON object is added to the request, the debug message in line 105 ("POST: data = [...]") is not being printed, although it should be and the loglevel is set to "debug" (Other debug messages of the API are visible).

The tomcat logs then state that the internal error is caused by a NullPointerException, so my assumption would be that either new JsonRepresentation(entity) or .getJsonObject() in line 104 of AbstractHandler.java return NULL, thus causing this exception.

Is there anything special that I did not take into account or did I stumble upon a bug? (Stacktrace is attached as screenshot "tomcat9_stacktrace.png")

Just as additional information: I am using Postman (note the missing "description" field in the response, as seen in the attached screenshot "postman_error_46.png") for testing, but I also tried with cURL, giving the same error.

I would be glad, if you could help me with this 

Kind regards,
fbu

Hi NetXMS team,

I found a few instances of Dashboard Templates being mentioned on this forum and that they might be considered for development.

I think this would be a great addition to the tool, especially when having a network with numerous similar nodes which all should have individual dashboards.

Currently, we need to create the dashboards manually, which is very time consuming and dashboard templates would be really helpful with this.


Is there any update on this topic?


Kind regards,
fbu

Dear NetXMS team,

as requested by Victor in my previous post under 'General Support' (https://www.netxms.org/forum/general-support/multiple-values-assigned-to-same-timestamp-with-nxapush/), I now post this topic as a feature request.

To sum up my point: I try to use nxapush to update DCI values at given timestamps, which results in multiple values being assigned to the same timestamp, as seen in the two screenshots attached to this post.

My idea would be to have an option (maybe -u) indicating that any value currently assigned to the given timestamp should be deleted and replaced by the new value, allowing a more flexible use of both nxpush and nxapush.

Best regards,
fbu

Hi,

currently, if you want to display in a dashboard how the values of a certain DCI developed during a certain period of time, you use a line chart.

I think it would be cool, if it was possible to do something similar with bar- and tube-charts, as well.

Maybe even give the possibility to define intervals in which the period should be split up to and how the data in these intervals should be accumulated.

E.g.: Something like - "For the period of the last 8 months, display the sum of all data of each individual month", resulting in a chart similar to this:
https://preply.com/wp-content/uploads/2018/08/representation-bar-graph.png.

Kind regards,
fbu

Dear NetXMS team,

I am currently facing an issue with Line charts which use data from DCI, populated by the NetXMS Agent push:

The way, data collection works for these DCIs is that (e.g.) every five minutes, a script pushes 24 timestamp-value pairs (one for every one of the last full hours) to the DCI using nxapush.

This is done every five minutes, because the values might change over the course of an hour. The idea here is that it always overwrites the previously assigned values to those timestamps, and here comes the problem:

I noticed (at first in the line charts) that there are multiple different values assigned to equal timestamps, resulting in vertical steps within the graph, as seen in 'double_value_assign.png'.

Then, I checked the history data of the DCI and as seen in 'multiple_value_assign1.png', there are in fact multiple instances of timestamp-value pairs, which have the same timestamp.

(I marked the two entries, which refer to the right-most step in the line chart screenshot)

Is this expected behavior, or a bug?

Anyway, my assumption is that most users would expect nx(a)push to just update the value, if an entry with the given timestamp is already present on the DCI.


Kind regards,
fbu

Hi all,

maybe someone can help me with the following:

I am currently trying to set up a SSO (Single sign on) solution for NetXMS using Apereo CAS. After several days of research and digging, I stumbled across a few lines in the WebUI source code, indicating that it has native SSO support, which was then confirmed by the change log of version 1.2.8 (https://www.netxms.org/forum/announcements/netxms-1-2-8-released/).

However, I could not find a single line of documentation nor a single forum thread on this topic, which is why I started one, by myself.

Currently, the following are the main questions related to this:

1. How does the Webapp handle the SSO session?
-> Is there a special URL to which CAS needs to redirect upon successful login?
-> Which attributes (if any) are needed from and for CAS?
-> Is SLO (single log out) also implemented?
2. How is the user management handled?
-> Does the user still need to be present in the NetXMS database?
-> If not, how are access rights handled (maybe via a special group?)
3. Can the official documentation be extended with this topic? It would really be cool to have a proper guide on how to set this up

I would be really happy, if someone could find the time to help me with this topic 

Thank you and kind regards,
fbu

Dear NetXMS team,

here is another thing that came to my mind when creating line charts in NetXMS:

Currently, NetXMS provides the ability to create "stacked" line chart diagrams, however, it only allows to stack all or none of the data from the source DCIs.

I think it would be useful to manually choose, which data to stack on top of each other and to have a more user friendly way of defining the stacking order (right now it is defined by the order the DCI sources).

Imagine the following: We have the four metrics TCP upstream, UDP upstream, HTTP upstream and overall upstream (downstream equivalent). To combine all four within one diagram, it should be possible to only stack UDP on top of TCP, as HTTP is a subset of the TCP traffic and the overall upstream is the superset of all three (which would only be added as a reference line).

What do you think about this idea?

Thanks and kind regards,
fbu 

Hi NetXMS team,

here's another idea of a potentially useful feature addition:

A NXSL function which allows to manually enable and disable DCIs or set their states. The function could be defined such as "setDCIStatus(dci, status)", where dci would be the DCI-object and status an integer with 0=Active, 1=Disabled and 2=Not Supported.

Maybe "setDCIStatusByID(dciID, node, status)", with dciID being an integer representing the DCI ID and node being a node-object as optional parameter, would also be possible.


The idea came to my mind, because a fairly large set of the nodes within my current setup is receiving monitoring data via NXAgent extension scripts from a centralized database, and are initially created via a big NXSL script under "Infrastructure Services", based on this data, as it is impossible to have a direct monitoring communication with NetXMS - not even with the use of NXAgent and proxy agents.

Because these devices might change in their states and different metrics temporarily might not return data any more, it would be useful to have a NXSL function, to disable those metrics via a script, as the "unsupported"-detection is also not working with this setup.

I know, template automatic apply rules can be used for this, however, once a template is un-applied from a node, the DCIs on it are gone and there is no way to access their history data via the management console until the template is re-applied, again.


What do you think, could this be useful if so, would it be difficult to implement?

Kind regards,
fbu

Dear NetXMS team,

as this is my first post on this forum, I shortly want to thank you for creating this awesome software product.

Over the course of the last few months, I got to research through NetXMS and learn, how most of its components and features are working.

During this time, there where some situations, in which I felt like there would be something useful missing.

Now, I want to share my ideas with you in this and the upcoming feature request posts, hoping that I can help to improve the tool for all users. 

############################

In my current setup, there are about 100 nodes to monitor, which are all bound to a container hierarchy under "Infrastructure Services".

For some of them, I created dashboards, containing only "Status Indicator" elements, to show the states of certain sets of nodes.

As this hierarchy currently has a depth of six, it can be pretty annoying to manually navigating through the containers, in order to access the object details of special nodes, each time their states are turning to something non-normal.

(To clarify: On each node's performance tab, there is a number of charts for its applied metrics, which is the preferred display option, as creating around 100 individual dashboards would be to time-consuming -> Which refers to one of my other feature requests of "Dashboard templates"  )

My idea would be to have a connection from certain dashboard elements (e.g. Status indicators) to the object details of the related node.

Basically, I came up with two potential solutions:

1. Enable right-click dropdown menus on these dashboard elements and add the menu entry "Show object details" - The way I see it is, that this is the simpler solution, as this menu entry is already available, when right-clicking a node in the "Service Dependency" view of a node.

2. Allow object details to be added as "Drill down objects" - I would assume this to be harder to implement, but it would also be kind of useful, as double clicking a status indicator would directly lead the user to the node's object details.

Thank you for considering my idea. Within the upcoming days, a few more will appear on the forum. 

Kind regards,
fbu