Deleted nodes still throwing events

Started by bdefloo, January 31, 2013, 10:33:27 AM

Previous topic - Next topic

bdefloo

Hi,

I've been having a strange issue on my v1.2.5 Windows 2003 x86 setup (but I think it's since v1.2.4):
I have a DCI on all my server nodes which checks if the last 5 ICMP ping results are above a certain value.
When the NetXMS core service starts, and one of these nodes is down, it will be triggered and post an event after 5 polls, which then sends an e-mail.

However, I appear to be getting these events and the related e-mails even for nodes I already deleted weeks ago! The node is nowhere to be found in the console object browser, and the alarms I'd expect to see aren't there either, but when I look in the database I can still find the deleted node in the object_properties and nodes tables, the DCIs are still there in the items table, ...
The is_deleted field is also still set to 0 in the object_properties table.

Could something be going wrong with the node deletion process?

Thanks in advance for any help you can provide!

Kind regards,
bdefloo

Victor Kirhenshtein

Hi!

Looks like some problem with node deletion. Can you try to stop NetXMS server and run nxdbmgr check? If node is really node bound to any subnet or container, it should be detected and deleted.

Best regards,
Victor

bdefloo

Hi,

That appears to have fixed it:

NetXMS Database Manager Version 1.2.5

Checking database (excluding collected data):
* Checking zone objects...                                            [PASSED]
Unlinked node object 31895 ("x") can be linked to subnet 93 (x.x.x.x). Link? (Y/N) Y
Unlinked node object 31963 ("x") can be linked to subnet 93 (x.x.x.x). Link? (Y/N) Y
Unlinked node object 62336 ("x") can be linked to subnet 9262 (x.x.x.x). Link? (Y/N) Y
* Checking node objects...                                            [FIXED ]
Unlinked interface object 36721 ("x"). Delete it? (Y/N) Y
Unlinked interface object 36722 ("x"). Delete it? (Y/N) Y
* Checking interface objects...                                       [FIXED ]
* Checking network service objects...                                 [PASSED]
* Checking cluster objects...                                         [PASSED]
* Checking template to node mapping...                                [PASSED]
* Checking object properties...                                       [PASSED]
* Checking event processing policy...                                 [PASSED]
5 errors was found, 5 errors was corrected
All errors in database was fixed
Commit changes? (Y/N) Y
Committing changes...
Changes was successfully committed to database
Database check completed

(Note: I removed hostnames/IPs)

After starting the server again and removing those nodes again, they were gone from the database and a subsequent nxdbmgr check didn't find any issues.

Thanks!

testos

#3
Hi.

Is a good practice to check, repair and backup database every night. In linux you can make a bash script like this:
Quote# Check_db.sh
#
# Check, repair and backup netxms database in remote host
#
### Variables ###
date=`date +%d-%m-%Y`
DateTimeStamp="date +%d/%m/%Y-%H:%M:%S"
# Paths
logfile="/var/log/mysql_backup.log"
tempbackdir="/tmp"
remotebackdir="/datos/backup/mysql_netxms_pro"
sourcedir="/var/lib/mysql"

### Check and repair data base ###
### First stop netxms agent and server, and then repair netxms data base using nxdbmgr
/usr/bin/killall nxagentd >> $logfile
/usr/bin/killall netxmsd >> $logfile
/usr/local/bin/nxdbmgr -f check >> $logfile
sleep 60
### Repair all mysql database (-Avrs parametres can be different depending on mysql version)
echo "#################################################" >> $logfile
echo `$DateTimeStamp` - All processes have been stopped. Analysis and repair MySQL DB started: >> $logfile
/usr/bin/mysqlcheck -Avrs -uroot -pxxxxxxxxx >> $logfile

### Backup databases locally whit compression ###
### Stop mysqld deamon
/sbin/service mysqld stop
sleep 60
### Compress with tar command
echo `$DateTimeStamp` - MySQL databases backup started: >> $logfile
tar -pcjvf $tempbackdir/mysql_backup-$date.tar.bz2 $sourcedir/ ### >> $logfile
tar -tjvf $backdir/mysql_backup-$date.tar.bz2 $sourcedir/ ### >> $logfile
sleep 60
/sbin/service mysqld start >> $logfile
/usr/local/bin/netxmsd -d >> $logfile
/usr/local/bin/nxagentd -d >> $logfile
echo `$DateTimeStamp` - MySQL databases backup completed. All netxms processes have been started >> $logfile

### Move local backup to remote server # # #
### Note that the remote server must contain the server's public key (generated with the
###"ssh-keygen-t rsa" command) from which you send the backup in
###"$HOME/.ssh/authorized_keys" file
scp $tempbackdir/mysql_backup-$date.tar.bz2 [email protected]:$remotebackdir >> $logfile
echo `$DateTimeStamp` - Local backup has been moved to remote server >> $logfile

### Remove daily backups over 32 days old on the remote server ###
ssh [email protected] find $remotebackdir/ -name '*.tar.bz2' -type f -mtime +32 -exec "rm -f {} \;" >> $logfile
echo `$DateTimeStamp` - Remote backups over 32 days old have been removed >> $logfile
echo "###" >> $logfile
echo "###" >> $logfile

exit 0

Finally you must assign the appropriate permissions to the script and then schedule it in crontab:
Quote00 3 * * * /usr/local/scripts/Check_db.sh


Best regards.

millerpaint

Testos,

Thanks for posting this script - it will be very helpful !


-Kevin C.