NetXMS Support Forum

Hi Victor,

The 2.2.9 Upgrade solved all issues

everything works just great!!

Regards

Clifford Dsouza

Hi Victor

Great!!!, Eagerly waiting for v2.2.9

Thanks and Regards

Clifford

Hi,

Please find the output of script attached for your reference.

Thanks
Clifford

Hi,

We have just upgraded Netxms  Server to 2.2.8, now the response seems normal from Management console however CPU utilization is showing 300% and above, below are the stats after the upgrade. Just wanted to know if it is normal.


NetXMS Server Remote Console V2.2.8 Ready
Enter "help" for command list

netxmsd: show threads
MAIN
   Threads.............. 256 (8/256)
   Load average......... 7838118.55 7292039.09 5187107.27
   Current load......... 3084405%
   Usage................ 100%
   Active requests...... 7896078
   Scheduled requests... 0
   Total requests....... 12741590
   Thread starts........ 248
   Thread stops......... 0
   Average wait time.... 1641274 ms

POLLERS
   Threads.............. 250 (10/250)
   Load average......... 103.76 109.10 117.88
   Current load......... 83%
   Usage................ 100%
   Active requests...... 208
   Scheduled requests... 0
   Total requests....... 30786539
   Thread starts........ 240
   Thread stops......... 0
   Average wait time.... 221 ms

DATACOLL
   Threads.............. 96 (10/250)
   Load average......... 4.58 5.48 5.53
   Current load......... 1%
   Usage................ 38%
   Active requests...... 1
   Scheduled requests... 0
   Total requests....... 17575738
   Thread starts........ 382
   Thread stops......... 296
   Average wait time.... 0 ms

SCHEDULER
   Threads.............. 1 (1/64)
   Load average......... 0.00 0.00 0.00
   Current load......... 0%
   Usage................ 1%
   Active requests...... 0
   Scheduled requests... 0
   Total requests....... 1750
   Thread starts........ 0
   Thread stops......... 0
   Average wait time.... 0 ms

AGENT
   Threads.............. 4 (4/256)
   Load average......... 0.00 0.00 0.00
   Current load......... 0%
   Usage................ 1%
   Active requests...... 0
   Scheduled requests... 0
   Total requests....... 0
   Thread starts........ 0
   Thread stops......... 0
   Average wait time.... 0 ms

CLIENT
   Threads.............. 16 (16/512)
   Load average......... 0.00 0.00 0.00
   Current load......... 0%
   Usage................ 3%
   Active requests...... 0
   Scheduled requests... 0
   Total requests....... 4586
   Thread starts........ 0
   Thread stops......... 0
   Average wait time.... 0 ms


Thank you
Clifford.

Hi

Cool! i'll try the web console

was wondering will shifting to ubuntu help?

Regards

Clifford Dsouza

Hi

I didn't get the "server is unable to deliver all updates to the client" part, cause almost all objects are switches and routers, so all we expect from the NMS is to report when the node or link is down. so all of it is SNMP Queries to the clients to check for update status

also it is important for me to view the link status as in my below query

https://www.netxms.org/forum/configuration/unable-to-get-bandwidth-details-on-map/msg23166/#msg23166

hope this gets implemented, it would be the greatest thing for me,

My NMS would be super complete

Regards
Clifford

Hi Tatjana,

Both Servers are Virtual with same configuration and OS (Centos 7). Please see the node stats of both Server below.

Server with issue:

NetXMS Server Remote Console V2.2.7 Ready
Enter "help" for command list

netxmsd: show stats
Objects............ 35197
Monitored nodes.... 1122
Collectible DCIs... 2146
Active alarms...... 416

Server working fine:

NetXMS Server Remote Console V2.2.7 Ready
Enter "help" for command list

netxmsd: show stats
Objects............ 32410
Monitored nodes.... 853
Collectible DCIs... 288
Active alarms...... 232


Thanks
Clifford

Hello Victor,

Any update regarding the issue which we are facing?

Thanks
Clifford.

Hi Victor,

I recompiled after commenting out the 2 lines as suggested. However the problem remains the same.
Actually we have 2 different Servers for which we recently upgraded from 2.2.5 to 2.2.7 one of these server is running fine after upgrade. Issue is with only this Server, earlier this Server was working Good.

Thank you
Clifford

Hi Victor,

I check  in src/server/core/session.cpp line 1894 abd 1895 are already commented out. Please find the attached snapshot.

Hi Victor,

I rebuild the server with-zlib option but situation is same. after few minutes CPU goes high. I have captured the threads with script which you had shared earlier. Please find the attached screenshot with thread output files.

Thank you
Clifford.

Hi Victor,

Please find the attached scripts output for your reference.

Best Regards
Clifford

Hi Victor

thanks for ur reply, i need clarity on the 3rd point

3. Capture threads using attached script (you should have gdb and all relevant netxms-*-dbg packages installed
how do i verify i have gdb and dbg packages installed , i did the standard netxms install on centos 7.

beside that i dont see any attached scripts, and an example how to capture the threads

Regards

Clifford

Quote from: clifford on August 08, 2018, 09:17:25 AM

Quote from: clifford on August 08, 2018, 09:16:21 AM

Quote from: clifford on August 08, 2018, 08:40:18 AM

Quote from: clifford on August 07, 2018, 02:31:00 PM
Thanks,

i changed the values to 512, hope this helps..
however i could not find the ThreadPool.Syncer.MaxSize parameter in the server configuration., so i manually created one, when you say it should be greater than 1, what could be the recommended value based on the stats i posted.

netxmsd: show threads
MAIN
   Threads:            8 (8/512)
   Load average:       0.00 0.00 0.00
   Current load:       0%
   Usage:              1%
   Active requests:    0
   Scheduled requests: 0

POLLERS
   Threads:            509 (10/512)
   Load average:       51.59 54.84 62.20
   Current load:       8%
   Usage:              99%
   Active requests:    44
   Scheduled requests: 0

DATACOLL
   Threads:            250 (10/250)
   Load average:       4.68 4.33 2.69
   Current load:       0%
   Usage:              100%
   Active requests:    0
   Scheduled requests: 0

SCHEDULER
   Threads:            1 (1/64)
   Load average:       0.00 0.00 0.00
   Current load:       0%
   Usage:              1%
   Active requests:    0
   Scheduled requests: 0

AGENT
   Threads:            4 (4/256)
   Load average:       0.00 0.00 0.00
   Current load:       0%
   Usage:              1%
   Active requests:    0
   Scheduled requests: 0


Thanks

Clifford

When everything is fine and the CPU is at 3% to 25 % Usage

netxmsd:  show threads
MAIN
   Threads:            36 (8/512)
   Load average:       0.00 0.00 0.00
   Current load:       0%
   Usage:              7%
   Active requests:    0
   Scheduled requests: 0

POLLERS
   Threads:            510 (10/512)
   Load average:       41.75 112.32 76.88
   Current load:       44%
   Usage:              99%
   Active requests:    226
   Scheduled requests: 0

DATACOLL
   Threads:            249 (10/250)
   Load average:       0.72 6.04 4.81
   Current load:       0%
   Usage:              99%
   Active requests:    2
   Scheduled requests: 0

SCHEDULER
   Threads:            1 (1/64)
   Load average:       0.00 0.00 0.00
   Current load:       0%
   Usage:              1%
   Active requests:    0
   Scheduled requests: 0

AGENT
   Threads:            4 (4/256)
   Load average:       0.00 0.00 0.00
   Current load:       0%
   Usage:              1%
   Active requests:    0
   Scheduled requests: 0

SYNCER
   Threads:            10 (1/10)
   Load average:       17.84 29.54 14.71
   Current load:       0%
   Usage:              100%
   Active requests:    0
   Scheduled requests: 0


When things are bad



netxmsd: show threads
MAIN
   Threads:            2048 (8/2048)
   Load average:       2996493.60 2968151.86 2119779.01
   Current load:       145523%
   Usage:              100%
   Active requests:    2980327
   Scheduled requests: 0

POLLERS
   Threads:            1003 (10/1024)
   Load average:       51.45 48.24 45.01
   Current load:       5%
   Usage:              97%
   Active requests:    51
   Scheduled requests: 0

DATACOLL
   Threads:            512 (10/512)
   Load average:       5.44 2.90 1.57
   Current load:       0%
   Usage:              100%
   Active requests:    0
   Scheduled requests: 0

SCHEDULER
   Threads:            1 (1/64)
   Load average:       0.00 0.00 0.00
   Current load:       0%
   Usage:              1%
   Active requests:    0
   Scheduled requests: 0

AGENT
   Threads:            4 (4/256)
   Load average:       0.00 0.00 0.00
   Current load:       0%
   Usage:              1%
   Active requests:    0
   Scheduled requests: 0

SYNCER
   Threads:            7 (1/10)
   Load average:       0.00 1.44 5.04
   Current load:       0%
   Usage:              70%
   Active requests:    0
   Scheduled requests: 0

System stats

all this is just snmp traffic

things got normal when i quit the console and all is well since reconnecting the console again

Regards
Clifford

Sadly had to quite the management console again, tried with the 32 bit console also, after few minutes the system goes for a toss again

Regards
Clifford

Quote from: clifford on August 08, 2018, 09:16:21 AM

Quote from: clifford on August 08, 2018, 08:40:18 AM

Quote from: clifford on August 07, 2018, 02:31:00 PM
Thanks,

i changed the values to 512, hope this helps..
however i could not find the ThreadPool.Syncer.MaxSize parameter in the server configuration., so i manually created one, when you say it should be greater than 1, what could be the recommended value based on the stats i posted.

netxmsd: show threads
MAIN
   Threads:            8 (8/512)
   Load average:       0.00 0.00 0.00
   Current load:       0%
   Usage:              1%
   Active requests:    0
   Scheduled requests: 0

POLLERS
   Threads:            509 (10/512)
   Load average:       51.59 54.84 62.20
   Current load:       8%
   Usage:              99%
   Active requests:    44
   Scheduled requests: 0

DATACOLL
   Threads:            250 (10/250)
   Load average:       4.68 4.33 2.69
   Current load:       0%
   Usage:              100%
   Active requests:    0
   Scheduled requests: 0

SCHEDULER
   Threads:            1 (1/64)
   Load average:       0.00 0.00 0.00
   Current load:       0%
   Usage:              1%
   Active requests:    0
   Scheduled requests: 0

AGENT
   Threads:            4 (4/256)
   Load average:       0.00 0.00 0.00
   Current load:       0%
   Usage:              1%
   Active requests:    0
   Scheduled requests: 0


Thanks

Clifford

When everything is fine and the CPU is at 3% to 25 % Usage

netxmsd:  show threads
MAIN
   Threads:            36 (8/512)
   Load average:       0.00 0.00 0.00
   Current load:       0%
   Usage:              7%
   Active requests:    0
   Scheduled requests: 0

POLLERS
   Threads:            510 (10/512)
   Load average:       41.75 112.32 76.88
   Current load:       44%
   Usage:              99%
   Active requests:    226
   Scheduled requests: 0

DATACOLL
   Threads:            249 (10/250)
   Load average:       0.72 6.04 4.81
   Current load:       0%
   Usage:              99%
   Active requests:    2
   Scheduled requests: 0

SCHEDULER
   Threads:            1 (1/64)
   Load average:       0.00 0.00 0.00
   Current load:       0%
   Usage:              1%
   Active requests:    0
   Scheduled requests: 0

AGENT
   Threads:            4 (4/256)
   Load average:       0.00 0.00 0.00
   Current load:       0%
   Usage:              1%
   Active requests:    0
   Scheduled requests: 0

SYNCER
   Threads:            10 (1/10)
   Load average:       17.84 29.54 14.71
   Current load:       0%
   Usage:              100%
   Active requests:    0
   Scheduled requests: 0


When things are bad



netxmsd: show threads
MAIN
   Threads:            2048 (8/2048)
   Load average:       2996493.60 2968151.86 2119779.01
   Current load:       145523%
   Usage:              100%
   Active requests:    2980327
   Scheduled requests: 0

POLLERS
   Threads:            1003 (10/1024)
   Load average:       51.45 48.24 45.01
   Current load:       5%
   Usage:              97%
   Active requests:    51
   Scheduled requests: 0

DATACOLL
   Threads:            512 (10/512)
   Load average:       5.44 2.90 1.57
   Current load:       0%
   Usage:              100%
   Active requests:    0
   Scheduled requests: 0

SCHEDULER
   Threads:            1 (1/64)
   Load average:       0.00 0.00 0.00
   Current load:       0%
   Usage:              1%
   Active requests:    0
   Scheduled requests: 0

AGENT
   Threads:            4 (4/256)
   Load average:       0.00 0.00 0.00
   Current load:       0%
   Usage:              1%
   Active requests:    0
   Scheduled requests: 0

SYNCER
   Threads:            7 (1/10)
   Load average:       0.00 1.44 5.04
   Current load:       0%
   Usage:              70%
   Active requests:    0
   Scheduled requests: 0

System stats

all this is just snmp traffic

things got normal when i quit the console and all is well since reconnecting the console again

Regards
Clifford