Event generated but could not see it in event_log

Started by Nagav, July 29, 2015, 03:15:09 PM

Previous topic - Next topic

Nagav

Hi

I am trying to generate event using logwatch.nsm subagent

I have parser as below
<parser>
  <file>/apps/uspsqa1/switch/site/log/sys/SunOSsyslg.log</file>
  <rules>
    <rule>
      <match>F-(.*)</match>
      <event params="1">IST_SYSLG1_RECORD</event>
    </rule>
  </rules>
</parser>


and I can see following lines in nxagentd.log when match take place

nxagentd.log:[29-Jul-2015 06:57:24.022] [DEBUG] SendTrap(): event_code=0, event_name=IST_SYSLG1_RECORD, num_args=1, arg[0]="RTTSRF-0017: read equ record failed" arg[1]="(null)" arg[2]="(null)"

However, I could not see event being logged into event_log table, as a result I could not see it event log monitor and I could not use it to generate alarts

In event IST_SYSLG1_RECORD configuration,  I have also enabled the option to Write to even log

But still I could not see event being logged in event_log table

Please help me

Regards
Naga

Victor Kirhenshtein

Hi,

configuration on agent side seems to be ok, and SendTrap record in log confirms that event is being generated. Check that you run agent with debug level 6 or higher and try to find record similar to "sending message CMD_TRAP" after SendTrap() record. If you'll find it, it mean that message was actually sent to server and problem is on server side. If not, problem is on agent side.

Best regards,
Victor

Nagav

Hi Victor

I have made debug level of agent to 9 and restated the agent.
But I could not see message ""sending message CMD_TRAP" after SendTrap() record" in agent log file

In agent log I could see only following two line
[05-Aug-2015 03:16:15.483] [DEBUG] LogParser: new data avialable in file "/apps/uspsqa1/switch/site/log/sys/SunOSsyslg.log"
[05-Aug-2015 03:16:15.484] [DEBUG] SendTrap(): event_code=0, event_name=IST_SYSLG1_RECORD, num_args=1, arg[0]="Fatal error " arg[1]="(null)" arg[2]="(null)"

Should I do any change to get "sending message CMD_TRAP" after SendTrap() record
Please help

Regards
Naga

Victor Kirhenshtein

Hi,

most likely it indicated that there are no server connection accepting traps. Can you share agent configuration file (nxagentd.conf)? Also, how this node is configured in NetXMS server and what is configuration poll output for it?

Best regards,
Victor

Nagav

Hi Victor

Attached nxagentd.conf file and word doc that has screenshot of node property and node configuration poll details

Regards
Naga

Victor Kirhenshtein

Hi,

your agent version (2.0-M2) has a bug that can cause incorrect server access if server address listed in multiple categories (Servers, ControlServers, MasterServers). Try to comment out Servers and ControlServers options in nxagentd.conf leaving only MasterServers.

Best regards,
Victor

Nagav

Hi Victor

Tried your suggestion, still I could see only two line in agent log file
[06-Aug-2015 05:00:21.367] [DEBUG] LogParser: new data avialable in file "/apps/uspsqa1/switch/site/log/sys/SunOSsyslg.log"
[06-Aug-2015 05:00:21.367] [DEBUG] SendTrap(): event_code=0, event_name=IST_SYSLG1_RECORD, num_args=1, arg[0]="fatal error" arg[1]="(null)" arg[2]="(null)"

Regards
Naga

Victor Kirhenshtein

Can you post agent config for verification please?

Best regards,
Victor

Nagav

Hi

Attached updated agent configuration

Thanks
Naga

Nagav

Hi

Just now I noticed that

if I comment out server name in server and control list
I see following logs in agent and DCI's did not collect data

[06-Aug-2015 05:20:36.789] [DEBUG] Connection from 127.0.0.1 rejected
[06-Aug-2015 05:20:36.789] [DEBUG] Incoming connection from 127.0.0.1
[06-Aug-2015 05:20:36.789] [DEBUG] Connection from 127.0.0.1 rejected
[06-Aug-2015 05:20:36.789] [DEBUG] Incoming connection from 127.0.0.1
[06-Aug-2015 05:20:36.789] [DEBUG] Connection from 127.0.0.1 rejected
[06-Aug-2015 05:20:36.790] [DEBUG] Incoming connection from 127.0.0.1
[06-Aug-2015 05:20:36.790] [DEBUG] Connection from 127.0.0.1 rejected
[06-Aug-2015 05:20:36.790] [DEBUG] Incoming connection from 127.0.0.1
[06-Aug-2015 05:20:36.790] [DEBUG] Connection from 127.0.0.1 rejected
[06-Aug-2015 05:20:36.791] [DEBUG] Incoming connection from 127.0.0.1
[06-Aug-2015 05:20:36.791] [DEBUG] Connection from 127.0.0.1 rejected
[06-Aug-2015 05:20:36.791] [DEBUG] Incoming connection from 127.0.0.1
[06-Aug-2015 05:20:36.791] [DEBUG] Connection from 127.0.0.1 rejected
[06-Aug-2015 05:20:36.792] [DEBUG] Incoming connection from 127.0.0.1
[06-Aug-2015 05:20:36.792] [DEBUG] Connection from 127.0.0.1 rejected
[06-Aug-2015 05:20:36.792] [DEBUG] Incoming connection from 127.0.0.1
[06-Aug-2015 05:20:36.792] [DEBUG] Connection from 127.0.0.1 rejected

Thanks
Naga

Victor Kirhenshtein

So, there are two problems. First, you should add 127.0.0.1 to list of MasterServers for agent running on NetXMS server. Second, you should check that primary host name for node is not set to 127.0.0.1, but to real IP address.

Best regards,
Victor

Nagav

Excellent Victor, thanks for your support
After making the changes suggested by you, I could see approriate log messages in both agent log and server log


Agent log:
[06-Aug-2015 23:59:32.636] [DEBUG] [session:0] GetTableValue(): result is 0 (SUCCESS)
[06-Aug-2015 23:59:32.637] [DEBUG] [session:0] Sending message CMD_REQUEST_COMPLETED (size 2880)
[06-Aug-2015 23:59:33.055] [DEBUG] LogParser: new data avialable in file "/apps/uspsqa1/switch/site/log/sys/SunOSsyslg.log"
[06-Aug-2015 23:59:33.055] [DEBUG] SendTrap(): event_code=0, event_name=IST_SYSLG1_RECORD, num_args=1, arg[0]="fatal error" arg[1]="(null)" arg[2]="(null)"
[06-Aug-2015 23:59:33.055] [DEBUG] [session:0] Sending message CMD_TRAP (size 144)

Server log:
g-2015 23:59:06.692] [DEBUG] StatusPoll(s1): bAllDown=false, dynFlags=0x00000001
[06-Aug-2015 23:59:06.694] [DEBUG] StatusPoll(s1 [112]): boot time set to 1438269511 from agent
[06-Aug-2015 23:59:06.695] [DEBUG] Finished status poll for node s1 (ID: 112)
[06-Aug-2015 23:59:33.056] [DEBUG] AgentConnectionEx::onTrap(): Received trap message from agent at 10.80.226.33, node ID 112
[06-Aug-2015 23:59:33.056] [DEBUG] AgentConnectionEx::onTrap(): trapID is valid
[06-Aug-2015 23:59:33.056] [DEBUG] Event from trap: 100066
[06-Aug-2015 23:59:33.056] [DEBUG] CorrelateEvent: event IST_SYSLG1_RECORD id 44 source s1 [112]
[06-Aug-2015 23:59:33.056] [DEBUG] CorrelateEvent: finished, rootId=0
[06-Aug-2015 23:59:33.056] [DEBUG] EVENT 100066 (ID:44 F:0x0001 S:0 TAG:"") FROM s1: fatal error
[06-Aug-2015 23:59:33.057] [DEBUG] Event 44 match EPP rule 27
[06-Aug-2015 23:59:46.486] [DEBUG] Updating maps...

Also could see event being logged in event_log table

Regards
Naga