Event processing

Event Processing Policy

Actions taken by event processor for any specific event determined by set of rules called Event Processing Policy.

_images/event_processing_policy.png

Event Processing Policy Screen

Every rule has two parts - a matching part (called Condition in the rule configuration dialog), which determines if the rule is appropriate for the current event, and an action part, which determines actions to be taken for matched events.

Each event passes through all rules in the policy, so if it matches to more than one rule, actions specified in all matched rules will be executed. You can change this behavior by setting Stop Processing flag for the rule. If this flag is set and rule matched, processing of current event will be stopped.

You can create and modify Event Processing Policy using Event Processing Policy Editor. To access the Event Processing Policy Editor window, press F4 or select Tools ‣ Event Processing Policy menu.

To create event policy right click on entry before or after witch new Event Processing Policy should appear and select Insert before or Insert after. Drag and drop can be used for rule reorganization.

_images/epp_entity_menue.png

Edit buttons

To edit Event Processing Policy, filter or action click on icon in right corner of an entry, it will open general properties of Event Processing Policy.

_images/epp_edit_button.png

Edit buttons

In EPP properties there are following sections:

Section Description
Comments There can be added comment about this policy that will be displayed like EPP name.
Condition This part determines if the rule is appropriate for the current event. There can be disabled check of this EPP.
Condition –> Source Objects One or more event’s source objects. This list can be left empty, which matches any node, or contain nodes, subnets, or containers. If you specify subnet or container, any node within it will be matched.
Condition –> Events Event code. This field can be left empty, which matches any event, or list of event codes.
Condition –> Severity Filter Event’s severity. This field contains selection of event severities to be matched.
Condition –> Filtering Script Optional matching script written in NXSL. If this field is empty, no additional checks performed. Otherwise, the event will be considered as matched only if the script returns non-zero (TRUE) return code. For more information about NetXMS scripting language please refer to the chapter Scripting in this manual.
Action In action part you can set what action should be done if event meets condition. There also can be set flag Stop event processing, that will stop check on this rule.
Action –> Alarm There can be set rules connected with alarm generation. Alarm can be created, resolved or terminated ore done nothing.
Action –> Persistent Storage NXLS Persistent Storage action like add/update or delete can be made.
Action –> Server Actions There is defined list of actions to be executed if condition is met. For action configuration refer to Actions chapter.
Action –> Timer Cancellations There is defined list of timmers that should be canceled if condition is met.
_images/epp_properties.png

Event Processing Policy properties

_images/epp_toolbar.png

Description of EPP toolbar form left to right: save changes, expand all, collapse all, horizontal layout, vertical layout, cut EPP, copy EPP, paset EPP, delete EPP

*After all manipulations are done - save changes by pressing save icon.*

Examples

This rule defines that for every major or critical event originated from a node named “IPSO” two e-mail actions must be executed.

_images/EPP_rule_config_example_1.png

Example 1

Alarms

Alarms Overview

As a result of event processing some events can be shown up as alarms. Usually alarm represents something that needs attention of network administrators or network control center operators, for example low free disk space on a server.

All alarm events are logged to alarm log. A number of days the server keeps an alarm history can be configured by “AlarmHistoryRetentionTime” server configuration parameter. Alarm log can be viewied in “Alarm Log View”(Alt+F8). This view give option to query in alarm log required information.

Every alarm has the following attributes:

Attribute Description
Creation time Time when alarm was created.
Last change time Time when alarm was last changed (for example, acknowledged).
State Current state of the alarm, see table bellow
Message Message text (usually derived from originating event’s message text).
Severity Alarm’s severity - Normal, Warning, Minor, Major, or Critical.
Source Source node (derived from originating event).
Key Text string used to identify duplicate alarms and for automatic alarm termination.

Possible alarm states:

Outstanding New alarm.
Acknowledged When network administrator sees an alarm, he may acknowledge it to indicate that somebody already aware of that problem and working on it. A new event with the same alarm ID will reset the alarm state back to outstanding
Sticky Acknowledged for time Alarm will remain acknowledged for given time interval even after new matching events, after time will pass alarm will be moved to outstanding state. This option can be used like snooze. When you know that there will be new matching events, but it will not change the situation. But after some time someone should check this problem. For example, if you have problem that cannot be solved until next week, so this alarm can be sticky acknowledged for 7 days. After 7 days this problem again will be in outstanding state. This type of acknowledge can be disabled by parameter “EnableTimedAlarmAck” in server configuration view.
Sticky Acknowledged Alarm will remain acknowledged event after new matching events. This can be useful when you know that there will be new matching events, but it will not change the situation. For example, if you have network device which will send new SNMP trap every minute until problem solved, sticky acknowledge will help to eliminate unnecessary outstanding alarms.
Resolved Network administrator sets this state when the problem is solved.
Terminated Inactive alarm. When problem is solved, network administrator can terminate alarm. This will remove alarm from active alarms list and it will not be seen in console, but alarm record will remain in database.

There are 2 types of alarm state flows: strict and not strict. This option can be configured in Preference page of Alarms or on server configuration page, parameter “StrictAlarmStatusFlow”. The difference between them is that in strict mode Terminate can be done only after Resolve state.

_images/AlarmStatesTransitionsInvokedByUser-NOTstrict.png

Not strict(default)

_images/AlarmStatesTransitionsInvokedByUser-strict.png

Strict

Alarm Melodies

On each severity of alarm can be set melody to play. This melody will be played when new alarm in state outstanding will occur. Melody that should be played should exist on server in wav format. See instruction there: Upload file on server. By default there are no sounds on alarms.

To set sound open preferences, there select Alarms ‣ Alarm Sounds tab. There in drop-down will be seen all possible options. If sound will not be chosen, alarm with this severity will come silently.

_images/Alarm_Sound_Preferences.png

Alarm Browser

When an alarm is generated it will appear in the Alarm Browser where information about currently active alarms can be viewed.

_images/alarm_browser.png

Alarm Comments

For each alarm can be created comments in “Alarm Details”

_images/alarm_details_comments.png

or “Alarm Comments” views.

_images/alarm_comments.png

Comment can be created, edited or deleted. All comments will be deleted after alarm termination.

Alarm Summary Emails

It is possible to schedule emails which contain a summary of all currently active alarms, similar to what can be seen in the Alarm Browser.

To enable Alarm Summary Emails it is required to configure the following server parameters:

Name
SMTPFromAddr
SMTPFromName
SMTPPort
SMTPRetryCount
SMTPServer
EnableAlarmSummaryEmails
AlarmSummaryEmailSchedule
AlarmSummaryEmailRecipients

Further information on server configuration parameters can be found in Server configuration parameters.

Generating Alarms

To generate alarms from events, you should edit Alarm field in appropriate rule of Event Processing Policy. Alarm configuration dialog will look like this:

_images/Alarm_config.png

You should select Generate new alarm radio button to enable alarm generation from current rule. In the Message field enter alarm’s text, and in the alarm key enter value which will be used for repeated alarms detection and automatic alarm termination. In both fields you can use macros described in the Macros for Event Processing section.

You can also configure sending of additional event if alarm will stay in Outstanding state for given period of time. To enable this, enter desired number of seconds in Seconds field, and select event to be sent. Entering value of 0 for seconds will disable additional event sending.

Alarms generated by rules can by categorised to limit what alarms can be seen by what users. This can be done by applying a category in the Alarm Category field, which can be created and configured in the Alarm Caregory Configurator.

Alarm Caregory Configurator

Alarm categories can be created and configured in the Alarm Category Configurator which can be found in Configuration ‣ Alarm Category Configurator menu:

_images/Alarm_category_config.png

Alarm Category Configurator

Alarm categories provide the possibility to configure access rights for viewing generated alarms on a per user or per group basis. When creating an alarm category, it is possible to set the Category name, Description.

_images/Alarm_category_properties.png

Alarm Category properties

Alarm category access rights can be configured by adding users or groups to the access list of the category in the Access Control property page.

_images/Alarm_category_access.png

Alarm Category Access Control

By default, all alarms can be viewed by all users due to the View all alarms system right being set as default to the Everyone user group. In order to limit the viewing of alarms, this system right should be removed and the access rights configured in the categories themselves. When the categories have been configured, they can be applied to the necessary Event Processing Policy rules.

If an alarm category has been applied to an Event Processing Policy rule, it will appear in the Event Processing Policy Editor when a rule is expanded under the Action section.

_images/EPP_rule_expanded.png

Event Processing Policy expanded

Automatic Alarm Termination/Resolve

You can terminate or resolve all active alarms with given key as a reaction for the event. To do this, select Terminate alarm radio button or Resolve alarm radio button in alarm configuration dialog and enter value for alarm key. For that field you can use macros described in the Macros for Event Processing chapter.

Escalation

As it was described in Generating Alarms chapter there is possibility to generate new event if alarm stay in Outstanding state for too long. Escalation is built on this option. When alarm was generated, but no action was done from operator in predefined time, new event can be generated and this time email or SMS can be sent to operator or to it’s manager. This escalation process can have as many steps as it is required.

Actions

In addition to alarm generation server can perform various types of actions as a reaction to an event. Action types available in NetXMS are described in the following sections. Each action can be separately disabled in action configuration.

After the action is added, it can be edited to add delay time and timer key. This option can be used to prevent notification send in case if problem solved quickly enough. Key is a free form string that support macro and delay is the delay time in seconds before action is executed.

The next example shows the configuration for the situation when there is no need to notify anyone if node went down and back up in just a minute.

_images/delayed_action.png

Escalation

One EPP rule can contain multiple actions with different delays. Delay timers are canceled by other rule in case of problem resolution.

The next example shows that if node went down, then
  1. after 1 minute responsible person will be notified if the problem still persists
  2. after 30 minutes the support manager will be notified if the problem still persists
  3. after 1 hour the IT manager will be notified if the problem still persists
_images/delayed_action_escalation.png

Action types

Execute command on management server

Executes provided command on server node. Check that user under witch netxmsd process run has permission to run this command.

Execute command on remote node

Executes provided command name defined in this nodes agent configuration file. To this command can be given parameters in format: commandName param1 param2 param3... Check that user under witch nxagentd process run has permission to run this command.

As the Remote Host can be used hostname or object name(int format: @objectName). Second option allows action execution on node behind proxy.

Send e-mail

Send email to one or more recipients. Multiple recipients can be separated by semicolons. Required server configuration parameters to send emails: SMTPFromAddr, SMTPFromName, SMTPRetryCount, SMTPServer. For detailed description of parameters check Server configuration parameters.

In message text can be used Macros for Event Processing.

Send SMS

Send SMS to one or more recipients. Multiple recipients can be separated by semicolons. Server will use SMS driver for actual message sending.

In message text can be used Macros for Event Processing.

Send XMPP message

Sends XMPP/Jabber message to one or more recipients. Multiple recipients can be separated by semicolons. equired server configuration parameters to send XMPP message: XMPPLogin, XMPPPassword, XMPPPort, XMPPServer, EnableXMPPConnector. For detailed description of parameters check Server configuration parameters.

In message text can be used Macros for Event Processing.

Execute NXSL script

This action executes script form scrip library. In action configuration should be defined name of script. Information about scripting and library can be found there.

Forward event

NetXMS does not support configuration synchronization between two NetXMS servers(Distributed Monitoring). But it is possible to forward events from one server to another. This option allow synchronize events between servers but there are some limitation.

Configuration
Source server configuration:
  1. Create new action of type “forward event” - it will have destination server address property.
  2. Create a rule in event processing policy with filter for events you want to forward and add forwarding action as action.
Destination server configuration:
  1. Enable EnableISCListener and ReceiveForwardedEvents in server configuration.
  2. Open port 4702.
  3. Check that receiving server have all events as on a sending server
Limitation
Limitations of event forwarding:
  1. Event template with same event code or event name must exist on recipient server
  2. Node object with same IP address as event’s source node’s address must exist on recipient server
  3. Does not work with zones

Events not met these conditions are discarded. It is possible to check if and why incoming events are discarded if turn on level 5 debug on receiving server.

There can be used one of two options if it is required to disable polling of sender server nodes on recipient server: disable all polling protocols or unmanage nodes. Chose depends on how you wish to see node’s status. For unmanaged node, it always be “unmanaged”, regardless of active alarms. If you disable polling, node’s status will be “unknown” unless there will be active alarms for that node - in that case node’s status will change to severity of most critical alarm.

NXLS Persistent Storage

NXSL

There are 2 functions:
  • ReadPersistentStorage(“key”) - read value by key
  • WritePersistentStorage(“key”, “value”) - insert or update value by key. If value will be empty - variable will be deleted.

View

Persistent Storage view (Configuration ‣ Persistent Storage) provide information about current state of Persistent Storage variables.

_images/pstorage.png

Note

Situations functionality is depricated. Persistent storage should be used instead.

Macros for Event Processing

On various stages of event processing you may need to use macros to include information like event source, severity, or parameter in your event texts, alarms, or actions. You may use the following macros to accomplish this:

Macro Description
%a IP address of event source object.
%A Alarm’s text (can be used only in actions to put text of alarm from the same event processing policy rule).
%c Event’s code.
%g Globally unique identifier (GUID) of event source object.
%i Unique ID of event source object in hexadecimal form. Always prefixed with 0x and contains exactly 8 digits (for example 0x000029AC).
%I Unique ID of event source object in decimal form.
%K Alarm’s key (can be used only in actions to put text of alarm from the same event processing policy rule).
%m Event’s message text (meaningless in event template).
%M Custom message text. Can be set in filtering script by setting CUSTOM_MESSAGE variable.
%n Name of event source object.
%N Event’s name.
%s
Event’s severity code as number. Possible values are:
  • 0 - Normal
  • 1 - Warning
  • 2 - Minor
  • 3 - Major
  • 4 - Critical
%S Event’s severity code as text.
%t Event’s timestamp is a form day-month-year hour:minute:second.
%T Event’s timestamp as a number of seconds since epoch (as returned by time() function).
%u User tag associated with the event.
%v NetXMS server’s version.
%[name] Value returned by script. You should specify name of the script from script library.
%{name} Value of custom attribute.
%<name> Event’s parameter with given name.
%1 - %99 Event’s parameter number 1 .. 99.
%% Insert % character.

If you need to insert special characters (like carriage return) you can use the following notations:

Char Description
\t Tab Character (0x09)
\n New line, CR/LF character pair
\\ Backslash character

Event’s parameter with given name

Threshold reached/rearmed named parameters:
  • %<dciId>
  • %<dciName>
  • %<dciDescription>
  • %<thresholdValue>
  • %<currentValue>
  • %<instance>
  • %<isRepeatedEvent> - set only for DCI reached events
  • %<dciValue>