Hi all,
We're monitoring log-based error events(Event Log Entry, Error in Log File, ASP .NET Error) and sending email notifications via an EPP rule. The issue we have is that when something goes wrong we can be spammed with hundreds of emails.
We have two different methods to resolve this but wanted input on which is better and if there are any other options.
- A custom NXSL filter script on the EPP rule that builds a key and tracks via WritePersistentStorage(). This allows 10 emails per a rolling 600 second window per key (based on the message).
- Generate an alarm with an alarmKey based on the message, using a custom script, and then using alarmTimeout and a companion rule that terminates the alarm. This is obviously a native solution but only allows 1 email per window.
Questions for the community:
- Is there a native mechanism we've overlooked for "max N notifications per time window"? (We've looked at action snoozeTime/blockingTimerKey, but those suppress rather than count.)
- For email throttling specifically, is the persistent-storage script the accepted pattern, or is there a cleaner way?
- Is the alarmTimeout → SYS_ALARM_TIMEOUT → terminate-rule chain the right way to get a self-resetting alarm?
Thanks in advance.
Hi,
Yes, there is one - it sits on the notification channel, not on the EPP action, which is why you didn't find it. Added in 6.1 (https://github.com/netxms/netxms/issues/2987), and it's a token bucket, so "max N per time window" is exactly what it does.
Your reading of snoozeTime/blockingTimerKey is correct: the check is "does a scheduled task with this key exist", not a count against a threshold. N=1 only, never N.
Server configuration parameters:
NotificationChannels.RateLimit.ChannelBurst 10 # your N
NotificationChannels.RateLimit.ChannelRate 1 # refill rate
NotificationChannels.RateLimit.ChannelRateUnit minute # second | minute | hour
10 immediately, then one more per minute - 10 per 600s. Defaults are 0 = disabled, and they are re-read every 10 seconds, so no restart. There is a matching RecipientBurst/RecipientRate pair counted per recipient address. Anything over the limit is queued rather than dropped, and once the backlog passes DigestThreshold (50) it's folded into one digest mail every DigestInterval (300s).
The catch: it counts per channel and per recipient, and cannot key on the event message. That is the one thing your script does that this doesn't.
For that part there's a lighter option than persistent storage - the alarm already counts repeats per alarmKey, and a filter script can read it:
alarm = FindAlarmByKey($key);
return (alarm == null) || (alarm.repeatCount < 10);
The count disappears when the alarm terminates, so there is nothing to clean up. Note the filter runs before the rule updates the alarm, so have an earlier rule create it and a later one send the mail. If you stay with persistent storage instead, two things will bite you: keys are silently truncated at 127 characters (hash the message with Crypto::SHA256()), and nothing ever expires them - delete with WritePersistentStorage(key, null).
On the alarmTimeout chain, two corrections. Generating an alarm does not suppress the rule's actions - the alarmKey de-duplicates the alarm, not the notification, and the action list still runs for every matching event. So whatever gives you 1 mail per window there is your filter script, not the alarm. And alarmTimeout is measured from the alarm's last change, so every repeat event resets it: during a flood SYS_ALARM_TIMEOUT never fires, it fires only once the events stop for the full timeout period.
It's an inactivity timer, not a window timer. Terminate-on-timeout is the right way to build a self-resetting alarm - just don't expect a fixed 600s cycle out of it.
If you enable the channel limit, be on 6.1.3+ or any 6.2.x - there was a shutdown crash in the throttling thread, fixed in https://github.com/netxms/netxms/issues/3238. None of these parameters are in the Admin Guide yet - filed https://github.com/netxms/netxms-doc/issues/70 for that.