NetXMS Support Forum

Hi,

Our NetXMS has been working fine for months and has just keeled over. The core service is refusing to start. I have run it from the command line and here is the last part of the output

[10-Sep-2013 10:06:21.591] Successful sync query: "SELECT id FROM containers WHE
RE object_class=28" [0 ms]
[10-Sep-2013 10:06:21.591] {0000000003A8F920} Successful prepare: "SELECT name,s
tatus,is_deleted,inherit_access_rights,last_modified,status_calc_alg,status_prop
_alg,status_fixed_val,status_shift,status_translation,status_single_threshold,st
atus_thresholds,comments,is_system,location_type,latitude,longitude,location_acc
uracy,location_timestamp,guid,image,submap_id FROM object_properties WHERE objec
t_id=?" [0 ms]
[10-Sep-2013 10:06:21.591] Successful async query: "SELECT idata_value,idata_tim
estamp FROM idata_171 WHERE item_id=1504 ORDER BY idata_timestamp DESC LIMIT 1"
[0 ms]
[10-Sep-2013 10:06:21.591] {0000000003A8F920} bind at pos 1: "886"
[10-Sep-2013 10:06:21.591] Successful prepared sync query: "SELECT name,status,i
s_deleted,inherit_access_rights,last_modified,status_calc_alg,status_prop_alg,st
atus_fixed_val,status_shift,status_translation,status_single_threshold,status_th
resholds,comments,is_system,location_type,latitude,longitude,location_accuracy,l
ocation_timestamp,guid,image,submap_id FROM object_properties WHERE object_id=?"
[0 ms]
[10-Sep-2013 10:06:21.607] {0000000003A8F920} Successful prepare: "SELECT attr_n
ame,attr_value FROM object_custom_attributes WHERE object_id=?" [0 ms]
[10-Sep-2013 10:06:21.607] {0000000003A8F920} bind at pos 1: "886"
[10-Sep-2013 10:06:21.607] Successful prepared sync query: "SELECT attr_name,att
r_value FROM object_custom_attributes WHERE object_id=?" [0 ms]
[10-Sep-2013 10:06:21.607] Successful sync query: "SELECT target_node_id FROM tr
usted_nodes WHERE source_object_id=886" [0 ms]
[10-Sep-2013 10:06:21.607] Successful sync query: "SELECT category,flags,auto_bi
nd_filter FROM containers WHERE id=886" [0 ms]
[10-Sep-2013 10:06:21.607] Successful async query: "SELECT idata_value,idata_tim
estamp FROM idata_171 WHERE item_id=1507 ORDER BY idata_timestamp DESC LIMIT 1"
[0 ms]
[10-Sep-2013 10:06:21.607] {0000000003A8F920} Successful prepare: "SELECT user_i
d,access_rights FROM acl WHERE object_id=?" [0 ms]
[10-Sep-2013 10:06:21.607] {0000000003A8F920} bind at pos 1: "886"
[10-Sep-2013 10:06:21.607] Successful prepared sync query: "SELECT user_id,acces
s_rights FROM acl WHERE object_id=?" [0 ms]
[10-Sep-2013 10:06:21.622] Successful sync query: "SELECT object_id FROM contain
er_members WHERE container_id=886" [0 ms]
[10-Sep-2013 10:06:21.622] {0000000003A8F920} Successful prepare: "SELECT change
_timestamp,new_status FROM slm_service_history WHERE service_id=? AND change_tim
estamp>?" [0 ms]
[10-Sep-2013 10:06:21.622] {0000000003A8F920} bind at pos 1: "886"
[10-Sep-2013 10:06:21.622] {0000000003A8F920} bind at pos 2: "1378767600"
[10-Sep-2013 10:06:21.622] Successful async query: "SELECT idata_value,idata_tim
estamp FROM idata_171 WHERE item_id=1509 ORDER BY idata_timestamp DESC LIMIT 1"
[0 ms]
[10-Sep-2013 10:06:21.622] Successful prepared sync query: "SELECT change_timest
amp,new_status FROM slm_service_history WHERE service_id=? AND change_timestamp>
?" [0 ms]
[10-Sep-2013 10:06:21.622] ++++ ServiceContainer::getUptimeFromDBFor(), downtime
0
[10-Sep-2013 10:06:21.622] {0000000003A8F920} Successful prepare: "SELECT change
_timestamp,new_status FROM slm_service_history WHERE service_id=? AND change_tim
estamp>?" [0 ms]
[10-Sep-2013 10:06:21.622] {0000000003A8F920} bind at pos 1: "886"
[10-Sep-2013 10:06:21.638] {0000000003A8F920} bind at pos 2: "1378681200"
[10-Sep-2013 10:06:21.638] Successful prepared sync query: "SELECT change_timest
amp,new_status FROM slm_service_history WHERE service_id=? AND change_timestamp>
?" [0 ms]
[10-Sep-2013 10:06:21.638] ++++ ServiceContainer::getUptimeFromDBFor(), downtime
0
[10-Sep-2013 10:06:21.638] Successful async query: "SELECT idata_value,idata_tim
estamp FROM idata_171 WHERE item_id=1520 ORDER BY idata_timestamp DESC LIMIT 1"
[0 ms]
[10-Sep-2013 10:06:21.638] {0000000003A8F920} Successful prepare: "SELECT change
_timestamp,new_status FROM slm_service_history WHERE service_id=? AND change_tim
estamp>?" [0 ms]
[10-Sep-2013 10:06:21.638] {0000000003A8F920} bind at pos 1: "886"
[10-Sep-2013 10:06:21.638] {0000000003A8F920} bind at pos 2: "1377990000"
[10-Sep-2013 10:06:21.638] Successful prepared sync query: "SELECT change_timest
amp,new_status FROM slm_service_history WHERE service_id=? AND change_timestamp>
?" [0 ms]
[10-Sep-2013 10:06:21.638] ++++ ServiceContainer::getUptimeFromDBFor(), downtime
0
[10-Sep-2013 10:06:21.638] ++++ ServiceContainer::initUptimeStats() id=886 100.0
00000 100.000000 100.000000
[10-Sep-2013 10:06:21.654] {0000000003A8F920} Successful prepare: "SELECT servic
e_id FROM business_services WHERE service_id=?" [16 ms]
[10-Sep-2013 10:06:21.654] {0000000003A8F920} bind at pos 1: "886"
[10-Sep-2013 10:06:21.654] Successful prepared sync query: "SELECT service_id FR
OM business_services WHERE service_id=?" [0 ms]
[10-Sep-2013 10:06:21.654] Successful async query: "SELECT idata_value,idata_tim
estamp FROM idata_171 WHERE item_id=1521 ORDER BY idata_timestamp DESC LIMIT 1"
[0 ms]
[10-Sep-2013 10:06:21.654] Loading node links...
[10-Sep-2013 10:06:21.654] Successful sync query: "SELECT id FROM containers WHE
RE object_class=29" [0 ms]
[10-Sep-2013 10:06:21.654] {0000000003A8FA40} Successful prepare: "SELECT name,s
tatus,is_deleted,inherit_access_rights,last_modified,status_calc_alg,status_prop
_alg,status_fixed_val,status_shift,status_translation,status_single_threshold,st
atus_thresholds,comments,is_system,location_type,latitude,longitude,location_acc
uracy,location_timestamp,guid,image,submap_id FROM object_properties WHERE objec
t_id=?" [0 ms]
[10-Sep-2013 10:06:21.654] {0000000003A8FA40} bind at pos 1: "887"
[10-Sep-2013 10:06:21.669] Successful prepared sync query: "SELECT name,status,i
s_deleted,inherit_access_rights,last_modified,status_calc_alg,status_prop_alg,st
atus_fixed_val,status_shift,status_translation,status_single_threshold,status_th
resholds,comments,is_system,location_type,latitude,longitude,location_accuracy,l
ocation_timestamp,guid,image,submap_id FROM object_properties WHERE object_id=?"
[0 ms]
[10-Sep-2013 10:06:21.669] Successful async query: "SELECT idata_value,idata_tim
estamp FROM idata_171 WHERE item_id=1523 ORDER BY idata_timestamp DESC LIMIT 1"
[0 ms]
[10-Sep-2013 10:06:21.669] {0000000003A8FA40} Successful prepare: "SELECT attr_n
ame,attr_value FROM object_custom_attributes WHERE object_id=?" [0 ms]
[10-Sep-2013 10:06:21.669] {0000000003A8FA40} bind at pos 1: "887"
[10-Sep-2013 10:06:21.669] Successful prepared sync query: "SELECT attr_name,att
r_value FROM object_custom_attributes WHERE object_id=?" [0 ms]
[10-Sep-2013 10:06:21.669] Successful sync query: "SELECT target_node_id FROM tr
usted_nodes WHERE source_object_id=887" [0 ms]
[10-Sep-2013 10:06:21.669] Successful sync query: "SELECT category,flags,auto_bi
nd_filter FROM containers WHERE id=887" [0 ms]
[10-Sep-2013 10:06:21.669] {0000000003A8FA40} Successful prepare: "SELECT user_i
d,access_rights FROM acl WHERE object_id=?" [0 ms]
[10-Sep-2013 10:06:21.669] {0000000003A8FA40} bind at pos 1: "887"
[10-Sep-2013 10:06:21.685] Successful prepared sync query: "SELECT user_id,acces
s_rights FROM acl WHERE object_id=?" [16 ms]
[10-Sep-2013 10:06:21.685] Successful async query: "SELECT idata_value,idata_tim
estamp FROM idata_182 WHERE item_id=46 ORDER BY idata_timestamp DESC LIMIT 1" [1
6 ms]
[10-Sep-2013 10:06:21.685] Successful sync query: "SELECT object_id FROM contain
er_members WHERE container_id=887" [0 ms]
[10-Sep-2013 10:06:21.685] {0000000003A8FA40} Successful prepare: "SELECT change
_timestamp,new_status FROM slm_service_history WHERE service_id=? AND change_tim
estamp>?" [0 ms]
[10-Sep-2013 10:06:21.685] {0000000003A8FA40} bind at pos 1: "887"
[10-Sep-2013 10:06:21.685] {0000000003A8FA40} bind at pos 2: "1378767600"
[10-Sep-2013 10:06:21.685] Successful prepared sync query: "SELECT change_timest
amp,new_status FROM slm_service_history WHERE service_id=? AND change_timestamp>
?" [0 ms]
[10-Sep-2013 10:06:21.685] ++++ ServiceContainer::getUptimeFromDBFor(), downtime
0
[10-Sep-2013 10:06:21.685] {0000000003A8FA40} Successful prepare: "SELECT change
_timestamp,new_status FROM slm_service_history WHERE service_id=? AND change_tim
estamp>?" [0 ms]
[10-Sep-2013 10:06:21.685] {0000000003A8FA40} bind at pos 1: "887"
[10-Sep-2013 10:06:21.685] {0000000003A8FA40} bind at pos 2: "1378681200"
[10-Sep-2013 10:06:21.685] Successful prepared sync query: "SELECT change_timest
amp,new_status FROM slm_service_history WHERE service_id=? AND change_timestamp>
?" [0 ms]
[10-Sep-2013 10:06:21.685] ++++ ServiceContainer::getUptimeFromDBFor(), downtime
0
[10-Sep-2013 10:06:21.685] {0000000003A8FA40} Successful prepare: "SELECT change
_timestamp,new_status FROM slm_service_history WHERE service_id=? AND change_tim
estamp>?" [0 ms]
[10-Sep-2013 10:06:21.685] {0000000003A8FA40} bind at pos 1: "887"
[10-Sep-2013 10:06:21.700] {0000000003A8FA40} bind at pos 2: "1377990000"
[10-Sep-2013 10:06:21.700] Successful prepared sync query: "SELECT change_timest
amp,new_status FROM slm_service_history WHERE service_id=? AND change_timestamp>
?" [0 ms]
[10-Sep-2013 10:06:21.700] ++++ ServiceContainer::getUptimeFromDBFor(), downtime
0
[10-Sep-2013 10:06:21.700] ++++ ServiceContainer::initUptimeStats() id=887 100.0
00000 100.000000 100.000000
[10-Sep-2013 10:06:21.700] {0000000003A8FA40} Successful prepare: "SELECT node_i
d FROM node_links WHERE nodelink_id=?" [0 ms]
[10-Sep-2013 10:06:21.700] {0000000003A8FA40} bind at pos 1: "887"
[10-Sep-2013 10:06:21.700] Successful prepared sync query: "SELECT node_id FROM
node_links WHERE nodelink_id=?" [0 ms]
[10-Sep-2013 10:06:21.700] {0000000003A8FB60} Successful prepare: "SELECT name,s
tatus,is_deleted,inherit_access_rights,last_modified,status_calc_alg,status_prop
_alg,status_fixed_val,status_shift,status_translation,status_single_threshold,st
atus_thresholds,comments,is_system,location_type,latitude,longitude,location_acc
uracy,location_timestamp,guid,image,submap_id FROM object_properties WHERE objec
t_id=?" [0 ms]
[10-Sep-2013 10:06:21.700] {0000000003A8FB60} bind at pos 1: "888"
[10-Sep-2013 10:06:21.700] Successful prepared sync query: "SELECT name,status,i
s_deleted,inherit_access_rights,last_modified,status_calc_alg,status_prop_alg,st
atus_fixed_val,status_shift,status_translation,status_single_threshold,status_th
resholds,comments,is_system,location_type,latitude,longitude,location_accuracy,l
ocation_timestamp,guid,image,submap_id FROM object_properties WHERE object_id=?"
[0 ms]
[10-Sep-2013 10:06:21.700] {0000000003A8FB60} Successful prepare: "SELECT attr_n
ame,attr_value FROM object_custom_attributes WHERE object_id=?" [0 ms]
[10-Sep-2013 10:06:21.700] {0000000003A8FB60} bind at pos 1: "888"
[10-Sep-2013 10:06:21.700] Successful prepared sync query: "SELECT attr_name,att
r_value FROM object_custom_attributes WHERE object_id=?" [0 ms]
[10-Sep-2013 10:06:21.716] Successful sync query: "SELECT target_node_id FROM tr
usted_nodes WHERE source_object_id=888" [0 ms]
[10-Sep-2013 10:06:21.716] Successful sync query: "SELECT category,flags,auto_bi
nd_filter FROM containers WHERE id=888" [0 ms]
[10-Sep-2013 10:06:21.716] {0000000003A8FB60} Successful prepare: "SELECT user_i
d,access_rights FROM acl WHERE object_id=?" [0 ms]
[10-Sep-2013 10:06:21.716] {0000000003A8FB60} bind at pos 1: "888"
[10-Sep-2013 10:06:21.716] Successful prepared sync query: "SELECT user_id,acces
s_rights FROM acl WHERE object_id=?" [0 ms]
[10-Sep-2013 10:06:21.716] Successful sync query: "SELECT object_id FROM contain
er_members WHERE container_id=888" [0 ms]
[10-Sep-2013 10:06:21.716] {0000000003A8FB60} Successful prepare: "SELECT change
_timestamp,new_status FROM slm_service_history WHERE service_id=? AND change_tim
estamp>?" [0 ms]
[10-Sep-2013 10:06:21.716] {0000000003A8FB60} bind at pos 1: "888"
[10-Sep-2013 10:06:21.716] {0000000003A8FB60} bind at pos 2: "1378767600"
[10-Sep-2013 10:06:21.716] Successful prepared sync query: "SELECT change_timest
amp,new_status FROM slm_service_history WHERE service_id=? AND change_timestamp>
?" [0 ms]
[10-Sep-2013 10:06:21.716] ++++ ServiceContainer::getUptimeFromDBFor(), downtime
0
[10-Sep-2013 10:06:21.716] {0000000003A8FB60} Successful prepare: "SELECT change
_timestamp,new_status FROM slm_service_history WHERE service_id=? AND change_tim
estamp>?" [0 ms]
[10-Sep-2013 10:06:21.716] {0000000003A8FB60} bind at pos 1: "888"
[10-Sep-2013 10:06:21.716] {0000000003A8FB60} bind at pos 2: "1378681200"
[10-Sep-2013 10:06:21.732] Successful prepared sync query: "SELECT change_timest
amp,new_status FROM slm_service_history WHERE service_id=? AND change_timestamp>
?" [16 ms]
[10-Sep-2013 10:06:21.732] ++++ ServiceContainer::getUptimeFromDBFor(), downtime
0
[10-Sep-2013 10:06:21.732] {0000000003A8FB60} Successful prepare: "SELECT change
_timestamp,new_status FROM slm_service_history WHERE service_id=? AND change_tim
estamp>?" [0 ms]
[10-Sep-2013 10:06:21.732] {0000000003A8FB60} bind at pos 1: "888"
[10-Sep-2013 10:06:21.732] {0000000003A8FB60} bind at pos 2: "1377990000"
[10-Sep-2013 10:06:21.732] Successful prepared sync query: "SELECT change_timest
amp,new_status FROM slm_service_history WHERE service_id=? AND change_timestamp>
?" [0 ms]
[10-Sep-2013 10:06:21.732] ++++ ServiceContainer::getUptimeFromDBFor(), downtime
0
[10-Sep-2013 10:06:21.732] ++++ ServiceContainer::initUptimeStats() id=888 100.0
00000 100.000000 100.000000
[10-Sep-2013 10:06:21.732] {0000000003A8FB60} Successful prepare: "SELECT node_i
d FROM node_links WHERE nodelink_id=?" [0 ms]
[10-Sep-2013 10:06:21.732] {0000000003A8FB60} bind at pos 1: "888"
[10-Sep-2013 10:06:21.732] Successful prepared sync query: "SELECT node_id FROM
node_links WHERE nodelink_id=?" [0 ms]
[10-Sep-2013 10:06:21.732] Loading service checks...
[10-Sep-2013 10:06:21.732] Successful sync query: "SELECT id FROM slm_checks" [0
ms]
[10-Sep-2013 10:06:21.732] Linking objects...
[10-Sep-2013 10:06:21.747] Successful sync query: "SELECT object_id FROM contain
er_members WHERE container_id=2" [15 ms]
[10-Sep-2013 10:06:21.747] Successful async query: "SELECT idata_value,idata_tim
estamp FROM idata_186 WHERE item_id=48 ORDER BY idata_timestamp DESC LIMIT 1" [1
5 ms]
[10-Sep-2013 10:06:21.747] Successful sync query: "SELECT object_id FROM contain
er_members WHERE container_id=3" [0 ms]
[10-Sep-2013 10:06:21.747] Successful sync query: "SELECT object_id FROM contain
er_members WHERE container_id=5" [0 ms]
[10-Sep-2013 10:06:21.747] Successful sync query: "SELECT object_id FROM contain
er_members WHERE container_id=6" [0 ms]
[10-Sep-2013 10:06:21.747] Successful sync query: "SELECT object_id FROM contain
er_members WHERE container_id=7" [0 ms]
[10-Sep-2013 10:06:21.747] Successful sync query: "SELECT object_id FROM contain
er_members WHERE container_id=8" [0 ms]
[10-Sep-2013 10:06:21.747] Successful sync query: "SELECT object_id FROM contain
er_members WHERE container_id=9" [0 ms]
[10-Sep-2013 10:06:21.747] #### CalculateCompoundStatus for id 9
[10-Sep-2013 10:06:21.747] ServiceContainer::calculateCompoundStatus(Business Se
rvices [9]): old_status=5 new_status=5
[10-Sep-2013 10:06:21.747] Objects loaded and initialized
[10-Sep-2013 10:06:21.747] Map update thread started
[10-Sep-2013 10:06:21.763] Successful sync query: "SELECT id,name,comments FROM
situations ORDER BY id" [16 ms]
[10-Sep-2013 10:06:21.763] Situations loaded and initialized
[10-Sep-2013 10:06:21.763] Successful sync query: "SELECT action_id,action_name,
action_type,is_disabled,rcpt_addr,email_subject,action_data FROM actions ORDER B
Y action_id" [0 ms]
[10-Sep-2013 10:06:21.763] Failed sync query: "SELECT event_code,severity,flags,
message,description,event_name FROM event_cfg ORDER BY event_code" [0 ms]
[10-Sep-2013 10:06:21.763] SQL query failed (Query = "SELECT event_code,severity
,flags,message,description,event_name FROM event_cfg ORDER BY event_code"): XX00
0 ERROR:  unexpected chunk number 0 (expected 1) for toast value 225167 in pg_to
ast_2619
[10-Sep-2013 10:06:21.763] Unable to load events from database
NetXMS Core initialization failed
[10-Sep-2013 10:06:21.763] {0000000003A8FC80} Successful prepare: "SELECT var_va
lue FROM config WHERE var_name=?" [0 ms]
[10-Sep-2013 10:06:21.763] {0000000003A8FC80} bind at pos 1: "DBLockStatus"
[10-Sep-2013 10:06:21.763] Successful prepared sync query: "SELECT var_value FRO
M config WHERE var_name=?" [0 ms]
[10-Sep-2013 10:06:21.763] {0000000003A8FC80} Successful prepare: "UPDATE config
SET var_value=? WHERE var_name=?" [0 ms]
[10-Sep-2013 10:06:21.763] {0000000003A8FC80} bind at pos 1: "UNLOCKED"
[10-Sep-2013 10:06:21.763] {0000000003A8FC80} bind at pos 2: "DBLockStatus"
[10-Sep-2013 10:06:21.778] Successful prepared sync query: "UPDATE config SET va
r_value=? WHERE var_name=?" [0 ms]
[10-Sep-2013 10:06:21.778] {0000000003A8FC80} Successful prepare: "SELECT var_va
lue FROM config WHERE var_name=?" [0 ms]
[10-Sep-2013 10:06:21.778] {0000000003A8FC80} bind at pos 1: "DBLockInfo"
[10-Sep-2013 10:06:21.778] Successful prepared sync query: "SELECT var_value FRO
M config WHERE var_name=?" [0 ms]
[10-Sep-2013 10:06:21.778] {0000000003A8FC80} Successful prepare: "UPDATE config
SET var_value=? WHERE var_name=?" [0 ms]
[10-Sep-2013 10:06:21.778] {0000000003A8FC80} bind at pos 1: ""
[10-Sep-2013 10:06:21.778] {0000000003A8FC80} bind at pos 2: "DBLockInfo"
[10-Sep-2013 10:06:21.810] Successful prepared sync query: "UPDATE config SET va
r_value=? WHERE var_name=?" [32 ms]
[10-Sep-2013 10:06:21.810] {0000000003A8FC80} Successful prepare: "SELECT var_va
lue FROM config WHERE var_name=?" [0 ms]
[10-Sep-2013 10:06:21.810] {0000000003A8FC80} bind at pos 1: "DBLockPID"
[10-Sep-2013 10:06:21.810] Successful prepared sync query: "SELECT var_value FRO
M config WHERE var_name=?" [0 ms]
[10-Sep-2013 10:06:21.810] {0000000003A8FC80} Successful prepare: "UPDATE config
SET var_value=? WHERE var_name=?" [0 ms]
[10-Sep-2013 10:06:21.810] {0000000003A8FC80} bind at pos 1: "0"
[10-Sep-2013 10:06:21.810] {0000000003A8FC80} bind at pos 2: "DBLockPID"
[10-Sep-2013 10:06:21.825] Successful prepared sync query: "UPDATE config SET va
r_value=? WHERE var_name=?" [15 ms]
[10-Sep-2013 10:06:21.825] DB connection 000000000089FF60 closed


It seems to be seeing the Database fine but not running. Any pointers would be gratefully received.

Don't worry - a vacuum of the Database seemed to fix it so must have been a corruption somewhere.