NetXMS Support Forum

We have a pfSense node with SNMP activated that gets recreated as a object every hr because its a GW/FW with public IP as VIP and nodes behind that public IP is NAT attached.
Node behind will have public IP as primary IP, that will be a duplicate of GW/FW.

I assume its the "NetworkDiscovery.MergeDuplicateNodes" that causes this to happen and it would be greate if this could have been disabled on selected Object only also

Hi

After upgrade to NetXMS 4.0.x a API call we have used stopped working.
Did anything change in way summaryTable/adHoc works?

http://_HOST_/netxms-websvc/summaryTable/adHoc with this payload gives Internal error 46.
In NetXMS 3.x it returned the summarytable.


{
      "baseObject":"Infrastructure Services",
      "multiInstance":False,
      "columns": [
          {
          "columnName":"IP",
          "dciName":"Dummy(PrimaryIP)"
          },
          {
          "columnName":"Type",
          "dciName":".1.0.8802.1.1.2.1.5.1.1.2.7.0"
          },
          {
          "columnName":"OS",
          "dciName":"Dummy(System.OS)"
          }
      ]
  }

Have been running a server with timescaledb on Ubuntu 20.04 for 2months now.
Its been abit unstable compared to have running default PostGresql over years on other setups.

Could i have a currupt database?

nxdbmgr will give this "in loop" when running "nxdbmgr check -d -f" as it fails to commit the fix:

Found collected data for non-existing DCI [6152]. Delete invalid records? (Yes/No/All/Skip) Y
* DCI history records                                                  [  66% ]

SQL query failed (25P02 ERROR:  current transaction is aborted, commands ignored until end of transaction block):
SELECT distinct(item_id) FROM idata_sc_30                                                                                                                                                                                                                                                       75SQL query failed (25P02 ERROR:  current transaction is aborted, commands ignored until end of transaction block):
SELECT distinct(item_id) FROM idata_sc_90                                                                                                                                                                                                                                                       83SQL query failed (25P02 ERROR:  current transaction is aborted, commands ignored until end of transaction block):
SELECT distinct(item_id) FROM idata_sc_180                                                                                                                                                                                                                                                      91SQL query failed (25P02 ERROR:  current transaction is aborted, commands ignored until end of transaction block):
SELECT distinct(item_id) FROM idata_sc_other                                                                                                                                                                                                                                                 [FIXED ]
* Table DCI history records                                            [   0SQL query failed (25P02 ERROR:  current transaction is aborted, commands ignored until end of transaction block):
SELECT count(*) FROM tdata_sc_default WHERE tdata_timestamp > to_timestamp(1617706540)                                                                                                                                                                                                           8SQL query failed (25P02 ERROR:  current transaction is aborted, commands ignored until end of transaction block):
SELECT count(*) FROM tdata_sc_7 WHERE tdata_timestamp > to_timestamp(1617706540)                                                                                                                                                                                                                16SQL query failed (25P02 ERROR:  current transaction is aborted, commands ignored until end of transaction block):
SELECT count(*) FROM tdata_sc_30 WHERE tdata_timestamp > to_timestamp(1617706540)                                                                                                                                                                                                               25SQL query failed (25P02 ERROR:  current transaction is aborted, commands ignored until end of transaction block):
SELECT count(*) FROM tdata_sc_90 WHERE tdata_timestamp > to_timestamp(1617706540)                                                                                                                                                                                                               33SQL query failed (25P02 ERROR:  current transaction is aborted, commands ignored until end of transaction block):
SELECT count(*) FROM tdata_sc_180 WHERE tdata_timestamp > to_timestamp(1617706540)                                                                                                                                                                                                              41SQL query failed (25P02 ERROR:  current transaction is aborted, commands ignored until end of transaction block):
SELECT count(*) FROM tdata_sc_other WHERE tdata_timestamp > to_timestamp(1617706540)                                                                                                                                                                                                            50SQL query failed (25P02 ERROR:  current transaction is aborted, commands ignored until end of transaction block):
SELECT distinct(item_id) FROM tdata_sc_default                                                                                                                                                                                                                                                  58SQL query failed (25P02 ERROR:  current transaction is aborted, commands ignored until end of transaction block):
SELECT distinct(item_id) FROM tdata_sc_7                                                                                                                                                                                                                                                        66SQL query failed (25P02 ERROR:  current transaction is aborted, commands ignored until end of transaction block):
SELECT distinct(item_id) FROM tdata_sc_30                                                                                                                                                                                                                                                       75SQL query failed (25P02 ERROR:  current transaction is aborted, commands ignored until end of transaction block):
SELECT distinct(item_id) FROM tdata_sc_90                                                                                                                                                                                                                                                       83SQL query failed (25P02 ERROR:  current transaction is aborted, commands ignored until end of transaction block):
SELECT distinct(item_id) FROM tdata_sc_180                                                                                                                                                                                                                                                      91SQL query failed (25P02 ERROR:  current transaction is aborted, commands ignored until end of transaction block):
SELECT distinct(item_id) FROM tdata_sc_other                                                                                                                                                                                                                                                 [PASSED]
106 errors was found, 106 errors was corrected
All errors in database was fixed
Commit changes? (Yes/No) Y


We are using Cluster object on small OpenVPN AS cluster with two nodes.
They have a floating IP switching between the nodes with UCARP detecting wich node should be primary.

Latly when doing apt upgrade for some reason primary detach the IP without secondary taking over so there is no master of the cluster. Probly some bug in UCARP setup, nothing todo with NetXMS..

But NetXMS does not detect that the floating IP is not active on any node, even not pingable..
First when old primary network is manually restarted alarm with "Cluster resource "X" moved from node A to node "
In Cluster log there is a event that floating/cluster IP is deleted too..
Some version back im pritty sure this situation caused this untouch netxms setup to trigger a "Cluster down" alarm.

Did anything change latly version of NetXMS for Cluster Objects?

In old NetXMS v2. days we didnt use this way of monitoring such cluster, but added a 3th node with the floating IP as PrimaryHostname.
But thats not possible anymore with the detection and merging of nodes with multi IPs to one Object..

Hi

Requesting Ubuntu 20.04 (focal fossa) deb in repository 

Possible to set chmod and user rights (chown) in Agent Policy File Delivery.
Atleast possible to set chmod +x for scripts to be used later.

We have a few nodes inside LXC on Proxmox.
They get stuck on "High CPU utilization (100.000000%)" when CPU load hit that, but does not leave 100% before shutdown LXC and start again.

Looking at load avg, it isnt having much work todo.
How does NetXMS agent measure the CPU load?

16 CPU cores 32 Threads
@proxmox:~$ w
07:39:43 up 15 days,  9:28,  1 user,  load average: 0.59, 0.64, 0.72

1 vCPU
@lxc01:~$ w
07:36:43 up 1 day, 12:02,  1 user,  load average: 0.64, 0.73, 0.77

1 vCPU
@lxc02:~$ w
07:38:35 up 1 day, 13:18,  2 users,  load average: 0.59, 0.64, 0.73

Hi

Im trying to understand Agent Policies - File Delivery.
Documentation is empty on the topic: https://www.netxms.org/documentation/adminguide/agent-management.html#file-delivery-policy

Im able to create a policy and add "dir".
But when adding a file, nothing seems to happen.

Is there anything required on agent or server to get this function work?

NetXMS Server 3.1-300
NXMC 3.1.343
nxagent 3.1-300

Have been using AgentReadList in different scenarios to dynamic apply template and dci instances
Could need some help to get it rewritten for NXLSv3


What worked in 2.2.x

Scenario 1:

DCI on agentless node, using a "source node" and "Instance discovery method: Script", called with

QuoteDW::DiscoverActiveMQ("DLQDetailsQueue")

DW::DiscoverActiveMQ script is:
if ($1 == "ActiveMQ") {
  //return AgentReadList($node, "DiscoverActiveMQ(" . $targetObject->ipAddr . ")");
  return $targetObject->readAgentList("DiscoverActiveMQ(" . $targetObject->ipAddr . ")");
}

ExternalList DiscoverActiveMQ do a do a call from source node to node and get a instance list to create DCI to poll info using a ExternalParameter (script on source node -> node) on each instance.




Scenario 2:
AutoApply template if user X is within /etc/passwd.

Using a script library with following code and "ExternalList = UserList:cut -d: -f1 /etc/passwd"
sub ServiceCheck(ServiceName) {
  if ($node->isAgent && $node->platformName like "Linux-*") {
    users = AgentReadList($node,"UserList");
    if ( users == null ) return null;
    foreach ( user : users ) {
      if ( user == ServiceName ) return true;
    }
  }
  return false;
}

Hi

After upgrading from 2.2.17 to 3.0.2292 runtimeFlags and state in NXLS does not show correct values.

Debugging with a Dummy dci on nodes and return $node->runtimeFlags + $node->state gives this results:

Node with active alarm, not responding to ICMP (dead) and only ICMP poll:
$node->runtimeFlags = 4
$node->state = 5

Node with no active alarm, responding to ICMP and only ICMP poll :
$node->runtimeFlags = 4
$node->state = 4

Node with no active alarm, responding to ICMP and nxagent+ICMP poll:
$node->runtimeFlags = 4
$node->state = 0

How can i stable check whetever Node is dead or not within NXLS?
We are using this on dummy/external+proxy DCI's to prevent them raising alarms when node is dead(not responding to ICMP or Agent)

We used this before:

Quoteif ($node->runtimeFlags & 0x0004) {
    return true;
}

Hi

Im having problem with Websvc API after upgrading to v3
This works in 2.2.17, but fails in v3 with response code 404 in catalina.out.

https://www.netxms.org/documentation/adminguide/advanced.html#get-object-by-id
curl -X GET --cookie "session_handle=$SESSION_HANDLE" "$API_HOME/objects/1579"

Did anything change in API?

This still works:
https://www.netxms.org/documentation/adminguide/advanced.html#get-multiple-objects-with-filters
curl -X GET --cookie "session_handle=$SESSION_HANDLE" "$API_HOME/objects?class=container,node"

Our server have been unstable since upgrading to 2.0.x branch.
We have been getting more nodes at same time, so problem could exist on older versions too..

Server Setup:
OS: Ubuntu 14.04.05-LTS (ESXi)
CPU: 4x E5-2690 @ 2.90GHz
Mem: 8GB

Server stats:
Total number of objects:     10490
Number of monitored nodes:   3737
Number of collectable DCIs:  33514

Server config:
PollerThreadPoolBaseSize: 300
PollerThreadPoolMaxSize: 800
NumberOfDataCollectors: 800


Been talking on Telegram about this, and this night one of our NOC had some time to do debug and found this:

-- Quote --
It seems that Netxms doesn't handle more than 1024 sockets very well and crashes if an attempt to retransmit data when the send buffers are full on a fd equal to or larger than 1024.

_opt_netxms206_bin_netxmsd.0.crash

(gdb) bt
#0  0x00007ff8ffe79c37 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1  0x00007ff8ffe7d028 in __GI_abort () at abort.c:89
#2  0x00007ff8ffeb62a4 in __libc_message (do_abort=do_abort@entry=2, fmt=fmt@entry=0x7ff8fffc2113 "*** %s ***: %s terminated\n") at ../sysdeps/posix/libc_fatal.c:175
#3  0x00007ff8fff4dbbc in __GI___fortify_fail (msg=<optimized out>, msg@entry=0x7ff8fffc20aa "buffer overflow detected") at fortify_fail.c:38
#4  0x00007ff8fff4ca90 in __GI___chk_fail () at chk_fail.c:28
#5  0x00007ff8fff4db07 in __fdelt_chk (d=<optimized out>) at fdelt_chk.c:25
#6  0x00007ff9004603bb in SendEx (hSocket=1149, data=data@entry=0x7ff8b226e580, len=1016, flags=flags@entry=0, mutex=0x7ff8b4172160) at tools.cpp:1084
#7  0x00007ff90097dd1f in ClientSession::sendMessage (this=0x7ff8b418a910, msg=<optimized out>) at session.cpp:1588
#8  0x00007ff900980060 in ClientSession::sendAllObjects (this=this@entry=0x7ff8b418a910, pRequest=pRequest@entry=0x7ff8b02cbcf0) at session.cpp:2294
#9  0x00007ff90099f08d in ClientSession::processingThread (this=0x7ff8b418a910) at session.cpp:798
#10 0x00007ff90099f219 in ClientSession::processingThreadStarter (pArg=<optimized out>) at session.cpp:215
#11 0x00007ff900210184 in start_thread (arg=0x7ff7c5359700) at pthread_create.c:312
#12 0x00007ff8fff3d37d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111


2016-11-12_22-34

(gdb) bt
#0  0x00007f47494aac37 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1  0x00007f47494ae028 in __GI_abort () at abort.c:89
#2  0x00007f47494e72a4 in __libc_message (do_abort=do_abort@entry=2, fmt=fmt@entry=0x7f47495f3113 "*** %s ***: %s terminated\n") at ../sysdeps/posix/libc_fatal.c:175
#3  0x00007f474957ebbc in __GI___fortify_fail (msg=<optimized out>, msg@entry=0x7f47495f30aa "buffer overflow detected") at fortify_fail.c:38
#4  0x00007f474957da90 in __GI___chk_fail () at chk_fail.c:28
#5  0x00007f474957eb07 in __fdelt_chk (d=<optimized out>) at fdelt_chk.c:25
#6  0x00007f4749a913bb in SendEx (hSocket=1180, data=data@entry=0x7f470036ab00, len=424, flags=flags@entry=0, mutex=0x7f470c262ff0) at tools.cpp:1084
#7  0x00007f4749faed1f in ClientSession::sendMessage (this=0x7f470c17dce0, msg=<optimized out>) at session.cpp:1588
#8  0x00007f4749faf0a5 in ClientSession::updateThread (this=0x7f470c17dce0) at session.cpp:658
#9  0x00007f4749faf2b9 in ClientSession::updateThreadStarter (pArg=<optimized out>) at session.cpp:224
#10 0x00007f4749841184 in start_thread (arg=0x7f4627cc1700) at pthread_create.c:312
#11 0x00007f474956e37d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111


2016-11-12_22-46.crash

(gdb) bt
#0  0x00007f77edd68c37 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1  0x00007f77edd6c028 in __GI_abort () at abort.c:89
#2  0x00007f77edda52a4 in __libc_message (do_abort=do_abort@entry=2, fmt=fmt@entry=0x7f77edeb1113 "*** %s ***: %s terminated\n") at ../sysdeps/posix/libc_fatal.c:175
#3  0x00007f77ede3cbbc in __GI___fortify_fail (msg=<optimized out>, msg@entry=0x7f77edeb10aa "buffer overflow detected") at fortify_fail.c:38
#4  0x00007f77ede3ba90 in __GI___chk_fail () at chk_fail.c:28
#5  0x00007f77ede3cb07 in __fdelt_chk (d=<optimized out>) at fdelt_chk.c:25
#6  0x00007f77ee34f3bb in SendEx (hSocket=1125, data=data@entry=0x7f77980f3380, len=424, flags=flags@entry=0, mutex=0x7f77a004f3e0) at tools.cpp:1084
#7  0x00007f77ee86cd1f in ClientSession::sendMessage (this=0x7f77a0239b70, msg=<optimized out>) at session.cpp:1588
#8  0x00007f77ee86d0a5 in ClientSession::updateThread (this=0x7f77a0239b70) at session.cpp:658
#9  0x00007f77ee86d2b9 in ClientSession::updateThreadStarter (pArg=<optimized out>) at session.cpp:224
#10 0x00007f77ee0ff184 in start_thread (arg=0x7f76c3891700) at pthread_create.c:312
#11 0x00007f77ede2c37d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111



excerpt of SendEx(SOCKET hSocket, const void *data, size_t len, int flags, MUTEX mutex) in tools.cpp:

do
{
retry:
#ifdef MSG_NOSIGNAL
nRet = send(hSocket, ((char *)data) + (len - nLeft), nLeft, flags | MSG_NOSIGNAL);
#else
nRet = send(hSocket, ((char *)data) + (len - nLeft), nLeft, flags);
#endif
if (nRet <= 0)
{
if ((WSAGetLastError() == WSAEWOULDBLOCK)
#ifndef _WIN32
    || (errno == EAGAIN)
#endif
   )
{
// Wait until socket becomes available for writing
struct timeval tv;
fd_set wfds;

tv.tv_sec = 60;
tv.tv_usec = 0;
FD_ZERO(&wfds);
FD_SET(hSocket, &wfds);
nRet = select(SELECT_NFDS(hSocket + 1), NULL, &wfds, NULL, &tv);
if ((nRet > 0) || ((nRet == -1) && (errno == EINTR)))
goto retry;
}
break;
}
nLeft -= nRet;
} while (nLeft > 0);


line 1084 is FD_SET(hSocket, &wfds);


To quote "man select":

Quote
       An  fd_set is a fixed size buffer.  Executing FD_CLR() or FD_SET() with
       a value of fd that is negative or is equal to or larger than FD_SETSIZE
       will result in undefined behavior.

hSocket is 1149, 1180 and 1125 in our crashdumps.


FD_SETSIZE on Linux is 1024:

Quote
    /usr/include/sys/select.h:#define   FD_SETSIZE      __FD_SETSIZE
    /usr/include/bits/typesizes.h:#define   __FD_SETSIZE        1024

Also consider the conditions for a crash. send() must fail with WSAEWOULDBLOCK, meaning that the send buffers are full. This can happen if the network is saturated or if the other side simply doesn't acknowledge the received data. Only then and iif the socket fd is equal to or larger than 1024 would lead to this crash. This would explain the inconsistent behaviour and perceived correlation with external factors.

Im running a proxy/source-node against multiply nodes that have a webpage to display values/statues (Inhouse applications/systems)

Each system is added to netxms as individual node, using a external DCI call on proxynode.
This is a python script that download webpage and parse out the values for each DCI.
There is a cache for this so node that i want more then 1 value from same page does not download it for each DCI as long age of cached download isnt more than X-time, making it goes faster on +2nd poll.

When AgentCache is off, this will slow down DCI poll because of serial(?) polling to the node used as proxy?
Turning on AgentCache the proxynode is going bananas on load and some DCI's get overpopulated with records every minute even if interval is 5min
Looking at Agent log, i can see its polling that many times..

It looks like Agent just spawns threads to it breaks.. Is there a config for MaxPollThread on agent?
And are there any

Im able to use Maintenance Enter/Leave/Schedule with all "System Rights" for a user, but do not manage to set this on a user with only specific rights.

I could find "Schedule object maintenance time interval" system rights, but this only allows a user to schedule a maintenanche and not put it direct into it or leave.

We just upgraded from 1.2.17 to 2.0.3 and discovered that Object Tools is no longer found in dropdown menu from AlarmBrowser (Rightclick alarm)

Do we need to enable a option or is it removed?

I still get Object Tools when rightclick node in Infrastructure.

we are using the JIRA integration to create issues on alarms.
But sometimes we allready have created issue via JIRA
Would it be possible to link alarm to JIRA ticket?

In some cases, we have more then one alarm on same JIRA ticket. In this cases it would be nice to link each alarm to same ticket..
This feature could also add option to update JIRA link in situation where ticket is moved and get another ID (Even if JIRA support forwarding of old ticket-id to new id..)

We are randomly getting problem with clients on different computers hang during login with NetXMS client.
This connection session do not timeout and we are not able to connect any new clients before server restart

1: Is it possible to boot/kick a client session with nxadm or other command?
2: Is there a limit/config for max active client sessions?
3: Would it be a problem connect nxjclient 1.2.16 to a server v1.2.17?

I have a nxshell script that takes linux formated hosts (/etc/hosts) list and adds it as nodes.
When after running it ok 2 times i get "Request timed out" on 3th run.
Then i need to wait xx minutes before i get it to work again.

Do the script open a connection and then not closing it proper so i need to wait for some timeout?

Script:
#!/usr/bin/python
#
# 2015-07-29 - EspenGB - Working 1.2.17
# Usage:
# $0 hostname
# * Ex: $0 node.netxms.tld

# $0 hostfile(/etc/hostname)
# * Ex: $0 /etc/hosts

# $0 hostname/hostfile containerId
# * Ex: $0 /etc/hosts 1234

import sys, socket, os, string, re

if len(sys.argv) < 2:
    print >>sys.stderr, 'Usage: ' + sys.argv[0] + ' <file/host> [<containerId>]'
    sys.exit(1)
   
name = sys.argv[1]

def addNode(name):
    try:
        test = socket.inet_aton(name)
        host_ip = name
    except socket.error:
        try:
            host_ip = socket.gethostbyname(name)
        except socket.error:
            host_ip = "0.0.0.0"

    if sys.argv[2]:
containerId = int(sys.argv[2])
    else:
        if sys.argv[0] == "nxshell_test":
            containerId = int(5636)
        else:
            containerId = int(5636)
    flags = NXCObjectCreationData.CF_DISABLE_NXCP | \
            NXCObjectCreationData.CF_DISABLE_SNMP
   
    cd = NXCObjectCreationData(objects.GenericObject.OBJECT_NODE, name, containerId);
    cd.setCreationFlags(flags);
    cd.setPrimaryName(host_ip) # Create node with IP address
    nodeId = session.createObject(cd)
    print " * Added {0} to containerID: {1}".format(name, containerId)
   
def is_valid_ip(ip):
    m = re.match(r"^(\d{1,3})\.(\d{1,3})\.(\d{1,3})\.(\d{1,3})$", ip)
    return bool(m) and all(map(lambda n: 0 <= int(n) <= 255, m.groups()))

if os.path.isfile(name):
    print "File found: "
    for names in open(name).readlines():
hostnames = re.split(r'\s+', names)
host_ip = hostnames[0].strip()
host_name = hostnames[1].strip()
if is_valid_ip(host_ip):
    print "Add: '" + host_ip + "' '" + host_name + "'"
            addNode(host_name.strip());
else:
    print "Add: " + name.strip()
    addNode(name.strip());

I'm using "Automatic Apply Rules" to apply templates to nodes depending on what value some DCIs have.
When i restart netxms, Apply rule rutines starts before DCI have any last values, making it goes bananas removing and adding templates randomly untill next scheduled "Apply Rule" runs and DCIs are having data.

I tried adding this to "Automatic Apply Rules", but abort doesnt work here:

$node2 = FindObject("netxms-server.tld", $node);
if($node2 != null) {
  $uptimeCheck = GetDCIValue($node2, FindDCIByDescription($node2, "System uptime (min)"));
  if ($uptimeCheck != null && $uptimeCheck < 360) {
    // Abort if netxms server just restarted
    abort;
  }
}

Are there any other way to delay Apply Rule routines?

Have been running netxms 1.2.9 stable for a long time now, but feel the urge to upgrade and get new functions.

From 1.2.15 we have been running test servers, but keep getting the same problem with "Node Down" alarm even if the node is not having problem.

Production server have not this problem.
Production and test server are on same VMWare
Test server is a copy of production and then upgraded.

Removed all custom templates, only using ICMP ping and still problem both with agent and non-agent nodes.
If i decrease the amount of nodes, problem goes away.

Do other have the same problem?
What could i do to debug?