NetXMS Support Forum

Hi,

Not fixed - I've reopened https://github.com/netxms/netxms/issues/3137. It was closed against a different fix (#3240) that only covered part of the problem.

Your combination is not the version mismatch Filipp mentioned - agent 6.2.3 and server 6.2.2 both contain everything from that work, so the "server must also be 6.2" note does not apply to you.

What is left: the agent can still change the reported user name, and sometimes the display name, for the same Store package between two polls, without signalling that the data is incomplete. Since 6.1 the server treats package name plus user as the package identity, so any such change comes out as a remove plus an install. Most likely triggers are a user profile whose registry hive is not on disk while the user is logged off (roaming profiles, FSLogix, UPD), and a failed SID-to-username lookup.

To tell which one you are hitting, please post:

• nxagentd log with debug level 6 for tag "winnt", covering a full logon and logoff
• System.InstalledProducts table taken twice - once with a user logged on, once without
• whether these are RDS/terminal servers, and whether profiles are roaming, FSLogix or UPD

Workaround is poor. There is no way to exclude Store packages from the inventory, and the inventory is collected on every configuration poll. Since 6.1 SYS_PACKAGE_INSTALLED and SYS_PACKAGE_REMOVED carry the user name as an event parameter, but filtering on it only removes half of each pair - the other half is reported as a system-wide package with empty user.

Hi,

"Driver Error" is as generic as it looks - it only means the driver loaded, your config parsed, and the POST to the webhook came back with something other than HTTP 200 and a body of exactly "ok". A transport failure, a non-200 code and an error string from Slack all end up as that same message. (If the driver itself were missing or unloadable you'd get "Driver not initialized" instead, so that part is fine.)

To get the actual error, in Tools -> Server Debug Console:

debug ncd.slack 7
debug nc 7

then send again and look in the server log (path is set by LogFile in netxmsd.conf, /var/log/netxmsd by default). It will be one of:

• Call to curl_easy_perform() failed - transport level: DNS, TLS, firewall. Note the Slack driver has no proxy option, so if this server reaches the internet through a proxy, that alone will stop it.
• Error response from webhook: HTTP response code is N
• Error response from webhook: <text> - Slack's own error, e.g. channel_not_found or invalid_payload.

Two things worth checking before you do that:

• url must be a classic incoming webhook - https://hooks.slack.com/services/T.../B.../... - which replies with the literal string "ok". A Workflow Builder trigger URL (https://hooks.slack.com/triggers/...) replies with JSON instead, and the driver will report Driver Error even though Slack accepted the message. This is the easiest one to get wrong right now, since Slack pushes you towards Workflow Builder.
• Recipient is mandatory for this driver and goes out as the Slack channel override. If it names a channel that doesn't exist you get channel_not_found back.

Your post shows url= with nothing after it - assuming that's just removed for the forum, but worth confirming the value actually saved.

Hi,

Not your script - 6.2 added a server configuration parameter Scripts.RestrictWriteAccess, enabled by default on new installations and on in-place upgrades. With it on, script-origin DCIs run under a read-only security context, and executeAgentCommandWithOutput() is one of the denied operations: https://netxms.org/documentation/adminguide/scripting.html#script-write-access-restrictions

Here's what's actually happening: the denied call does not raise an error. It returns null, the script keeps running, and it dies one line later where res.length hits that null - which is why the error points at line 8 and says "argument is not an object" instead of naming the restriction. In the Script Executor the same script runs under your user's rights, which include the control permission, so it works there and only there.

To confirm, enable debug tag nxsl.security at level 7 - the denial logs "Read-only script access denied" with the object name and id. Nothing appears at default levels.

Two ways out:

• Turn the restriction off: Server Configuration -> Scripts.RestrictWriteAccess -> 0. Takes effect immediately, no restart. It's a global switch though - it re-enables writes for every restricted script type, not just your DCI.
• Move the ping to the agent. Better fit here, since you're collecting a value rather than modifying anything. In nxagentd.conf, pointing at the same commands your ping-suli / ping-voda actions run:

ExternalMetric = PingSuli(*):["/path/to/ping-suli", "$1"]
ExternalMetric = PingVoda(*):["/path/to/ping-voda", "$1"]
Then two Agent-origin DCIs with metrics PingSuli(1.1.1.1) and PingVoda(1.1.1.1). No NXSL, so the restriction doesn't apply - and the if ($2) switch disappears, since you had two DCIs anyway.

I've opened https://github.com/netxms/netxms/issues/3619 on whether script-origin DCIs should be restricted at all - they're a data source, not an analysis script - and https://github.com/netxms/netxms/issues/3620 on the denial being invisible and surfacing as an unrelated error.

Your fresh install working is the useful data point - this is something about that specific server, not the pinboard code.

Here's what's actually happening: all web UI settings live in a single properties file on the machine running the web UI, under the servlet container's work directory, named nxmc.preferences.<uuid>. The uuid comes from a cookie called nxmcStoreId in your browser. If that cookie doesn't come back on the next login, the web UI generates a new uuid and opens an empty file - and the old one stays on disk with all your settings still in it. Pinboard, the "Show server clock" checkbox and column widths all live in that one file, which is why they go together.

Three things to check, and they split the problem cleanly:

• How many contexts are deployed. Look at /var/lib/tomcat10/work/Catalina/localhost/ - if more than one nxmc context is there (nxmc and nxmc-6.2.4, say), each has its own state directory, and the files you find may belong to a context you are no longer using. The web UI logs a line "State directory:" at every startup naming the one actually in use.
• Whether a new file appears per login. List the state directory, log in again, list it again. A new nxmc.preferences.<uuid> each time means the cookie is not coming back. The same file with a new timestamp means it is, and the problem is elsewhere.
• The cookie itself. Browser dev tools, Application - Cookies, look for nxmcStoreId. It is set with a 90 day lifetime, so it should survive a logout. If your nginx config caches or strips Set-Cookie on the proxied responses, it won't.

If you already have a file there containing pinboard$<number> entries, that settles most of it on its own: saving works, and the session that comes up empty is reading a different file.

The web UI log goes to catalina.out on a Tomcat install - grep it for "State directory", "local preferences" and "Invalid store ID". (-Dnxmc.logfile is only set in our Jetty instructions, so on Tomcat everything lands in the console log.)

Note: even once this works, that directory is Tomcat's per-context work directory and it is cleared on war redeploy. So web UI settings are lost on every web UI upgrade by design - that is likely part of what you saw across the three version upgrades.

Hi,

Not a setting — there's nothing to enable, pinboard is saved when the session ends and restored on the next login. Looks like a regression, but two different things can produce an empty pinboard in the web UI and they need different fixes.

So: 1) which exact version are you on? 2) does it stay empty after an explicit Logout, or only after closing the browser? 3) do other web UI settings survive — last active perspective, splitter positions, table column widths — or do those reset too?

That last one matters because web UI settings are not stored in the server database. They live in a file on the machine running the web UI, in the servlet container's temp directory, keyed by a cookie in your browser. If everything resets, you're losing that file — typically a web UI restart that recreates the temp dir — and that's a separate problem from the pinboard itself.

Добрый день,

Метрика указана неверно. System.Registry.Value(*) — это шаблон из каталога метрик агента, а не готовая метрика: у неё два аргумента, путь к ключу и имя value. Для вашего случая:

System.Registry.Value(HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Windows\Safer\CodeIdentifiers,{instance})
{instance} нужно добавить и в description, иначе все шесть DCI получатся с одинаковым именем.

С литеральной * агент не может разобрать первый аргумент как корневой ключ и возвращает "unknown metric", поэтому созданные DCI сразу уходят в статус unsupported — в event log ноды должно быть шесть SYS_DCI_UNSUPPORTED. В Data Collection в режиме просмотра проверьте, что включён показ unsupported DCI.

Instance discovery настроен правильно, external list здесь не нужен — System.Registry.Values(...) возвращает именно имена values.

Отдельно: сам DCI, на котором настроен instance discovery, в режиме просмотра не отображается, в списке видны только созданные по нему DCI. Это ожидаемое поведение.

Hi,

That error means the server is not counting your license as valid — it falls back to the built-in limit of 250 managed nodes and blocks creation above it. "Unlimited" in the key is not enough by itself: a key is ignored if it is expired, capped to a version below 6.2, or bound to a different machine, and none of that is logged.

Run nxlicmgr list from the server's bin directory. Check that the key is actually listed, and that Quantity, Hardware Id, System Id, Version and Expiration are all green. Red in any column is your answer — Version red means the key predates 6.2, Hardware Id or System Id red means it was issued for a different machine.

If the key is missing from the list, or anything is red, write to [email protected] with that output — those cases need the key re-issued.

update: another issue is fixed. this one is tracked here: https://github.com/netxms/netxms/issues/3593

it's fixed already, will be included in next release

Ключ ни на что не влияет в работе программы — собирать без него правильно.

Он добавляет в сборку только libipfix, библиотеку разбора NetFlow/IPFIX. Демон-коллектор nxflowd, для которого она предназначалась, в сборку не входит вообще, а сервер и консоль с flow-данными не работают: обработка так и не была реализована. В 7.0 ключ удалён вместе с кодом.

Вместо собственного коллектора в 7.0 планируется интеграция с внешними анализаторами трафика, первый коннектор — ntopng: https://github.com/netxms/netxms/issues/3429

Hi,

Confirmed, and thanks for tracing it - your reading is exactly right. When no multiplier applies the value stays a String, so the %s path skips the decimal trimming. Tracked as https://github.com/netxms/netxms/issues/3577.

Until that's fixed, setting an explicit display format on the DCI works around it: %{m,u}.2f instead of leaving it empty. The f conversion parses the string into a number first, so the trimming happens. Cost is that it's per-DCI and pins you to two decimals regardless of magnitude.

Да, это ровно тот случай из Note в ответе от 09.08: отложенное действие было заведено до входа в обслуживание, а при срабатывании режим обслуживания не проверяется — ни флаг "Accept correlated events", ни скрипт в фильтре правила на этом этапе не работают, они отрабатывают в момент постановки таймера. Настройками это не обходится, поэтому завёл тикет: https://github.com/netxms/netxms/issues/3566

Hi,

Yes, there is one - it sits on the notification channel, not on the EPP action, which is why you didn't find it. Added in 6.1 (https://github.com/netxms/netxms/issues/2987), and it's a token bucket, so "max N per time window" is exactly what it does.

Your reading of snoozeTime/blockingTimerKey is correct: the check is "does a scheduled task with this key exist", not a count against a threshold. N=1 only, never N.

Server configuration parameters:

NotificationChannels.RateLimit.ChannelBurst      10       # your N
NotificationChannels.RateLimit.ChannelRate       1        # refill rate
NotificationChannels.RateLimit.ChannelRateUnit   minute   # second | minute | hour

10 immediately, then one more per minute - 10 per 600s. Defaults are 0 = disabled, and they are re-read every 10 seconds, so no restart. There is a matching RecipientBurst/RecipientRate pair counted per recipient address. Anything over the limit is queued rather than dropped, and once the backlog passes DigestThreshold (50) it's folded into one digest mail every DigestInterval (300s).

The catch: it counts per channel and per recipient, and cannot key on the event message. That is the one thing your script does that this doesn't.

For that part there's a lighter option than persistent storage - the alarm already counts repeats per alarmKey, and a filter script can read it:

alarm = FindAlarmByKey($key);
return (alarm == null) || (alarm.repeatCount < 10);

The count disappears when the alarm terminates, so there is nothing to clean up. Note the filter runs before the rule updates the alarm, so have an earlier rule create it and a later one send the mail. If you stay with persistent storage instead, two things will bite you: keys are silently truncated at 127 characters (hash the message with Crypto::SHA256()), and nothing ever expires them - delete with WritePersistentStorage(key, null).

On the alarmTimeout chain, two corrections. Generating an alarm does not suppress the rule's actions - the alarmKey de-duplicates the alarm, not the notification, and the action list still runs for every matching event. So whatever gives you 1 mail per window there is your filter script, not the alarm. And alarmTimeout is measured from the alarm's last change, so every repeat event resets it: during a flood SYS_ALARM_TIMEOUT never fires, it fires only once the events stop for the full timeout period.

It's an inactivity timer, not a window timer. Terminate-on-timeout is the right way to build a self-resetting alarm - just don't expect a fixed 600s cycle out of it.

If you enable the channel limit, be on 6.1.3+ or any 6.2.x - there was a shutdown crash in the throttling thread, fixed in https://github.com/netxms/netxms/issues/3238. None of these parameters are in the Admin Guide yet - filed https://github.com/netxms/netxms-doc/issues/70 for that.

Hi,

No - Title and Group do not take scripts. Neither field goes through a macro engine on the server: the whole Performance View block is stored with the DCI as-is, and the client substitutes a small fixed set of placeholders when it draws the tab. That is the entire mechanism, so there is nothing to hook a script into.

Two separate things are off in the screenshot. %[...] is a real macro syntax, but it belongs to the text expansion used for event and alarm messages, notification text and object tools - and there the form is %[scriptName], without the script: prefix. The script: prefix belongs to %{script:name}, which is a template macro for DCI name, description and instance fields, expanded when the DCI is created from a template - once per node, not per discovered instance. You combined the two, and neither engine is applied to Performance View fields.

What the fields do accept, substituted by the client at render time:

• Title - {instance}, {instance-name}, {node-name}
• Group - {instance}, {instance-name} (no {node-name} here)
• Name in legend - {instance}, {instance-name}, {node-name}

Leave Title or Name in legend empty and the DCI description is used instead. Charts are grouped by the substituted Group value, so instances that produce identical text land on one graph.

For splitting instances into two or more groups there are two ways.

Simplest approach is one master DCI per group: copy the instance discovery DCI, give each copy a fixed Group name, and let each instance filter script accept only the instances belonging to that group. The group stays static text and the script only decides membership, so each group can also have its own title, colors and time range.

If you want to keep a single master DCI and let the script decide, use the instance display name as the carrier - the filter script sets it, and Group reads it back:

if ($1 ilike "*volt*")
   return [true, $1, "Voltage"];
return [true, $1, "Current"];
Second element of the array is the instance value ({instance}), third is the instance display name ({instance-name}). Set both Group and Title to {instance-name} and Name in legend to {instance} - instances the script labelled "Voltage" end up on one graph titled Voltage, "Current" on another.

Two caveats: {instance-name} then carries the group label everywhere it is shown, Last Values and DCI descriptions included, so use {instance} where you need the instance itself; and group matching is exact, case included.

Filling these fields from a script is not possible today, and the macros that do work are not in the documentation - the Performance view section only mentions grouping by identical Group text. I raised https://github.com/netxms/netxms-doc/issues/69 to get them written down. If script-driven titles are important for your setup, say so and I will open a feature request for it.

Hi,

No problems, and running pvesh from the agent works fine. On the second question the choice isn't quite what it looks like though, because with NetXMS the REST API route is itself an agent feature.

pvesh works from the agent for a reason worth knowing about: it invokes API functions directly without going through the REST/HTTPS server, and it requires root. The packaged agent runs as root - the systemd unit does not drop privileges, and the agent changes uid only if you pass -u/-g or set UserId/GroupId in nxagentd.conf. So it works with no extra setup, which also means anyone who can add a DCI can make the agent fork a root command on the hypervisor. Worth deciding that deliberately rather than finding it out later.

What will actually bite you is process spawning. ExternalParameter (ExternalMetric in current naming) forks a process per metric per poll, and pvesh is a Perl program that loads the PVE API modules on every invocation. One metric per guest on a 60 second interval turns into a continuous stream of interpreter startups on the host. The default execution timeout is 5 seconds (ExecTimeout, or ExternalMetricTimeout for these specifically), and a slow pvesh call will hit it and get killed.

Use ExternalDataProvider instead - added in 5.2.0. It runs the command once per polling interval, caches the output, and serves any number of metrics out of the cached document with jq queries, evaluated inside the agent so there is no jq binary to install:

[ExternalDataProvider/PVE]
Command = ["/usr/bin/pvesh", "get", "/cluster/resources", "--type", "vm", "--output-format", "json"]
PollingInterval = 60
Timeout = 30000

[ExternalDataProvider/PVE/Metrics]
PVE.Guest.Status(*) = .[] | select(.vmid == $1) | .status
PVE.Guest.CPU(*) = .[] | select(.vmid == $1) | .cpu
PVE.Guest.CPU(*).dataType = float
PVE.Guest.Memory(*) = .[] | select(.vmid == $1) | .mem
PVE.Guest.Memory(*).dataType = uint64

[ExternalDataProvider/PVE/Lists]
PVE.Guests = .[] | .vmid

That is one pvesh call a minute for the whole cluster no matter how many DCIs you build on top, and PVE.Guests gives you instance discovery over vmid. Two details: Timeout here is in milliseconds, and metrics with parameters ($1) need agent 6.0 or newer - on 5.2 define each metric with a fixed query instead.

On agent versus REST: there is no version of the REST route that doesn't involve an agent. Web service DCIs are executed by an agent acting as web service proxy - the server never makes the HTTP request itself. If the node has no web service proxy configured it falls back to the zone proxy, and finally to the management server node, so "just using the REST API" in practice means "the server's own agent makes the call". The choice is which agent and where it sits, not whether there is one.

Consequences:

• Responses are cached per URL for the retention time set on the web service definition, so every DCI reading out of the same document costs one HTTP request. Against /cluster/resources that is a single call per interval covering every guest in the cluster.
• The extraction is identical either way - web service DCIs and ExternalDataProvider share the same jq-based extractor, so the queries you write don't change with the route.
• Only an agent on the PVE host can reach 8006 on localhost, which matters if that port isn't open to the server. Poll from an agent elsewhere and you install nothing on the hypervisor at all.

So the real trade is narrow. pvesh saves you managing an API token, at the price of forking a root-privileged Perl process on the hypervisor. The web service route costs you a token and gives you a read-only credential in exchange (PVEAuditor is enough), one HTTP call per interval, and the freedom to poll from an agent that isn't on the PVE host. I'd take the REST API as the default and keep pvesh for anything the HTTP API doesn't expose.

One thing has changed since my post above. The OTLP route I said didn't work now does - the receiver accepted protobuf only, which is why PVE 9's OpenTelemetry metric server couldn't reach it, and that is fixed in 6.2.3 (https://github.com/netxms/netxms/issues/3482), released on 9 August. On 6.2.3 or later you can point Datacenter - Metric Server - OpenTelemetry straight at the NetXMS web API and have PVE push guest metrics on its own: no collector in between, no polling, and no agent on the host. It needs an auth token passed through the otel-headers option, and the receiver is loaded as a server module (Module = otlp in netxmsd.conf). None of the ingestion side is documented yet - https://github.com/netxms/netxms-doc/issues/67 covers that.