SV: Request for comments

Hi,

Im still a bit fuzzy on the concepts. To get this clear:

a. A Policy. This is something that does a general match against a log
file and if it matches applies one or several rules and/or rulesets?

b. A ruleset is mere a set of rules.

c. A rule is a match to an explicit rule, and an event as a result of
this match?

Is the above correct?

In regards to imports/exports; it seems resonable that doing an export
one should be asked if one wants to include the required dependencies.
Or even better at the time of selecting what to export, one can choose
to export custom events and possibly other exportable data.

About applying policies to containers/folders/groups one should have the
opportunity to set these to fail gracefully meaning that if say you try
to apply *nix filters to a MS server, these inapplicable rules should
just silently fail. This means in effect that one can make a general
ruleset or policy for servers and then apply this policy to a mixed
enviroment. I can see that this would make somewhat sense, separating
between server and router/switch policies etc. Im not sure though if
that is the best approach.

Another approach would be to simply have all rules apply to all
collected logdata. Consider this;

As long as you have a syslog server running in netxms it seems sensible
that logs are stored aswell as parsed on the server, this could though
pose several issues in regards to scalability, however;

First one defines various classes then tags a node with a
classification, say a cisco router is tagget as a router.

Secondly we collect data from several nodes, windows, linux and cisco
nodes through the syslog interface. However we do not separate on
logfile names, that is an irrelevant information because we store all
data in an sql database table (this might put a strain on servers, I
don't know) but we store what node it came from and other syslogish data
like severity etc.

Then we create rules for say, bind log entries and events to happen when
rules are matched or not matched and so forth. Just in the good
oldfashion rules kind of way.

Then we make policies that binds rules or rulesets to a certain class.
And then just parse all logentries according to the rules valid for this
class.

That is one approach to the whole issue.

A different one would be to "export" rulesets from netxms to the agent
and then have the agent apply these rulesets to local logfiles then
"syslog" them over to the netxms server.

I do believe though that one should try to accomplis the following goals
though;

Rules/policies should be easy to apply directly to any single node. This
should be sets and not standalone rules, however one should also be able
to make standalone rules for any single host, just to cover special
needs.

Also one should, through the existing network tree, be able to apply
sets/policies on a higher level, like a folder/container (Im not sure
what the difference is). This would allow a very dynamic setup. One
should also be able to go and look on any node below this
container/folder and see currently applied policies/rules and see that
these are inherited from objects higher up in the tree.

Also, importing/exporting is important. I would personally hate having
to set up 500 rules and not beeing able to share them with anyone. Or
have someone else share with me.

There should also be some sort of rule wizard or rule constructor.

Im just playing around with various ideas right now so they are probably
a bit incoherent. I have to reread what you wrote and have it sink in
before replying to it.

Best regards,
Rune Nilssen
rune_at_directconnect.no

-----Opprinnelig melding-----
Fra: Victor Kirhenshtein [mailto:victor_at_opticom.lv]
Sendt: 21. oktober 2005 16:27
Til: NetXMS Users
Emne: Re: [netxms-users] Request for comments

Hi,

it was very interesting comment. As I can imagine it now:

1. Policy is a set of rules, where each rule is either a direct matching
rule (i.e. if message text matches to regexp, send event, as described
in first post), or a link to predefined set of rules (ruleset). This
will allow administartor to create some common sets of rules an reuse
them in different policies. Policy may have default file name set
(probably also allowing usage of environment variables), which can be
overrided when policy being applied to host. Import/export of rulesets
and policies should be easy, with one exception - if one uses custom
events in the ruleset, how they should be exported and imported? Should
event in the rule be replaced with standard event (not very good
solution) or exported with the ruleset?

2. Policy applies to host/file pair, so one policy can be applied to
more than one file on a target system. Probably it also will be good to
allow applying policies to host groups via container or subnet obects.

3. Policies and rulesets can be organized into hierarchical structure
with folders.

For the user this can be represented as a tree with three subtrees:

1. Policies - tree structure with optional folders and policies as
leafs.
2. Rulesets - same as policies (or policies and rulesets may be in one
tree?)
3. Current bindings - Nodes on first level and associated policies on
second.

Like this:

* Policies
        [Folder 1]
                policy 1
                policy 2
        [Folder 2]
                policy 3
* Rulesets
        [UNIX]
                bind logs
                squid logs
        [Windows]
                Exchange
                SQL Server
        some_common_rules
* Current bindings
        Node 1
                policy 1: /var/log/squid
                policy 1: /var/log/postfix
                policy 3: /var/log/messages
        Node 2
                policy 2: {EventLog:System}
                policy 3: C:\squid\squid.log

Policy may look like this:

Policy 1
Default file: /var/log/messages
Report unmatched records: no
Rules:
        1. if match "^sshd.*" send event 1
        2. call ruleset "/UNIX/bind logs"

What do you think?

Best regards,
Victor

-----Original Message-----
From: Rune Nilssen [mailto:rune_at_directconnect.no]
Sent: Thursday, October 20, 2005 8:11 PM
To: NetXMS Users
Subject: Re: [netxms-users] Request for comments

Hi,

1. Today I gather my logs on a common log server for all syslog enabled
units. Have you considered how agents would deal with such enviroments?
I guess theres several approaches that could be done by the user in such
a situation. The advantage however is that I would be able to parse
syslogs from Cisco routers etc.

2. Sounds resonable enough, I would suggest however that one is able to
group rules/policies togeather.. It would make it easier to apply
certain sets of predefined rules, say one special set for handling
kernel messages, one set of rules for handling bind log entries etc. It
would sure be handy not having to set up everything from scratch each
time, also if one takes importing/exporting into account it would help
people get up and running faster if say, someone made a few default
rulesets or similar. This might be what you had in mind in the first
place considering your opening line. :)

3 & 4. I would suggest a multi-multi approach. Meaning that one ruleset
can apply to anything from one to several or all logfiles a client
monitors aswell as across clients. Also it would make sense if a ruleset
has default files it applies to, and this beeing a default value that
can be overridden on a host or host-group basis.

As far as the interface goes, I can only imagine some sort of
tree-structure. Maybe one could have a set of default rules in a
"rulebook" that's set up as a tree with rulesets as folders. One would
doubleclick any rule to edit its default values and restrictions about
what kind of units it can be applied to(?). Further one would have a
different tree for applied rules/rulesets or maybe this belong in the
nodetree? Anyway, one could apply a rule/ruleset to a folder in the tree
and all units this is applicable for would automagically adapt these
rules, and if needed tweak this on a folder or unit level. Im not sure,
this is really just loose thoughts.

What do you think?

Sorry about the delayed response, work is taxing, as it should. :)

Best regards,
Rune Nilssen
rune_at_directconnect.no

-----Opprinnelig melding-----
Fra: Victor Kirhenshtein [mailto:victor_at_opticom.lv]
Sendt: 8. oktober 2005 22:35
Til: NetXMS Users
Emne: [netxms-users] Request for comments

Hi all!

Currently we are working on a log processing functionality. It's based
on the following key concepts:

1. All log processing should be done by agents.
2. Agent processes log files according to "log processing policy", which
is a set of matching rules for log records. If log record match to the
rule, agent either generate an event specified in the rule, or ignores
the record. It is also possible to generate special event for unmatched
records. 3. Agent can have multiple log processing policies installed,
usually one per file. 4. One log processing policy can be installed on
multiple agents, allowing to configure monitoring of similar files on
different hosts only once. 5. All log processing configuration should be
done from the management console in centralized way.

Does anybody have any comments on that scheme? Also, I'll be very happy
to see suggestions, how user interface for log processing configuration
can looks like.

Best regards,
Victor
Received on Fri Oct 21 2005 - 20:02:08 EEST