Re: Log monitoring

Hello,

I have thought about log monitoring a while ago, while looking at some
other solutions for such monitoring. However, I have been disappointed
by all the monitoring packages that I have looked for the same reason
I dislike most IDS and antivirus systems: you need to write rules what
normal behavior _doesn't_ look like, instead of describing what normal
behavior is and then forgetting about the system, knowing that it will
alert you when something doesn't go as planned.

I have been doing some thinking and I think a possible way of solving
this would be something like this:

* event occurs: "check pass; user unknown"
* alert is fired
* administrator decides that any event that matches a regexp created
from this rule isn't worthy of an alert
* the next time such an event occurs, it's ignored

another example:

* event occurs: "foo is 100"
* alert is fired
* administrator creates a rule that any event matching "foo is $1" for
0 < foo < 150 is normal, creates a rule for critical alert when foo >
300...
* the next time foo is over 150, an event occurs, as well as when foo
is -1 (due to a bug perhaps?) or over 300 (a critical condition)

after a while, the system would "learn" what normal behavior is and
alert only when the log deviates from this behavior.

I can give more thought to how this could be implemented if this idea
is found acceptable.

Bostjan

On 9/13/05, Victor Kirhenshtein <victor_at_opticom.lv> wrote:
>
> Hi all!
>
> Below is a proposed scheme for log monitoring.
>
> I. Key points
>
> 1. Logs should be processed by agents.
> 2. Agent should check each log record against log processing policy, and
> geberate appropriate event if match detected.
> 3. Agent can have more than one log processing policy installed.
> 4. Log processing policies should be created and distributed from
> administrator's console, allowing centralized management of log
> processing.
>
>
> II. Log processing policy
>
> Log processing policy is a set of matching rules with the following
> attributes:
>
> Matching criterias:
> * Event id range (integer range)
> * Event source (regexp)
> * Log message (regexp)
>
> Action data:
> * NetXMS event id
>
> Event ID range and event source matching used only for Windows Event
> Log. All text log files should be matched only by message text.
> Also, log processing policy should have version attribute, so agent
> after restart could determine if he has latest policy installed and
> request latest one from server if needed.
>
>
> III. Additional questions to discuss
>
> 1. Should we implement log processing as a separate subagent or as part
> of core agent?
> 2. How we can guarantee event delivery from agent to server?
> 3. What to do on agent startup - parse already existing log records or
> just wait for new records?
>
> Any comments?
>
> Best regards,
> Victor
>
>
Received on Tue Sep 13 2005 - 21:04:51 EEST