sec.pl

[-conf=<file pattern> ...] [-input=<file pattern>[=<context>] ...] [-input_timeout=<input timeout>] [-timeout_script=<timeout script>] [-reopen_timeout=<reopen timeout>] [-check_timeout=<check timeout>] [-poll_timeout=<poll timeout>] [-blocksize=<io block size>] [-bufsize=<input buffer size>] [-evstoresize=<event store size>] [-cleantime=<clean time>] [-log=<logfile>] [-syslog=<facility>] [-debug=<debuglevel>] [-pid=<pidfile>] [-dump=<dumpfile>] [-quoting | -noquoting] [-tail | -notail] [-fromstart | -nofromstart] [-detach | -nodetach] [-intevents | -nointevents] [-intcontexts | -nointcontexts] [-testonly | -notestonly] [-help] [-?] [-version]

After startup SEC reads lines from files, named pipes, or standard input, matches the lines with patterns (regular expressions, Perl subroutines, etc.) for recognizing input events, and correlates events according to the rules in its configuration file(s). In order to detect a composite event, rules are used sequentially in the same order as they are given in the configuration file. If there are two or more configuration files, rule sequences from different files are used virtually in parallel. SEC can be configured to produce its output by executing user-specified shell commands, and by using utilities like snmptrap(1) or snmpnotify(1), SEC can generate network management events as output. Other options for producing output events are described further in this man page.

Following rule types are currently implemented in SEC for detecting composite events:

Single - match input event and execute an action immediately.

SingleWithScript - match input event and depending on the exit value of an external script, execute an action.

SingleWithSuppress - match input event and execute an action immediately, but ignore following matching events for the next t seconds.

Pair - match input event, execute an action immediately, and ignore following matching events until some other input event arrives. On the arrival of the second event execute another action.

PairWithWindow - match input event and wait for t seconds for other input event to arrive. If that event is not observed within a given time window, execute an action. If the event arrives on time, execute another action.

SingleWithThreshold - count matching input events during t seconds and if a given threshold is exceeded, execute an action and ignore all matching events during the rest of the time window.

SingleWith2Thresholds - count matching input events during t1 seconds and if a given threshold is exceeded, execute an action. Then start the counting of matching events again and if their number per t2 seconds drops below the second threshold, execute another action.

Suppress - suppress matching input events (used to keep the events from being matched by later rules).

Calendar - execute an action at specific times.

If a rule defines the correlation of several events in a certain time window, the rule will start an event correlation operation when the first matching event occurs. In order to distinguish between different event correlation operations that have been started by the rules, every operation is assigned a unique key that is generated from the rule ID, the name of the configuration file the rule belongs to, and the textual description of the composite event that the correlation operation is trying to detect (see RULES AND EVENT CORRELATION OPERATIONS section for more information).

Rules support not only the execution of shell commands as actions, but also the creation and deletion of contexts that decide whether a particular rule can be applied at a given moment, the generation of synthetic events that will act as input for other rules, the clearing of correlation operations that have been started by other rules, etc. This makes it possible to combine several rules and form more complex event correlation schemes. Contexts can not only be used for activating or deactivating a particular rule, but they can also serve as event stores (in logsurfer(1) the contexts are used in a similar way). There are actions for adding events to a context, reporting all events associated with a context, etc. By using the features of SEC in an appropriate way, one can accomplish a wide range of event correlation and event consolidation tasks.

-conf=<file pattern>

expand <file pattern> to filenames (with the Perl glob() function) and read event correlation rules from every file. Multiple -conf options can be specified at command line. Each time SEC receives a signal that forces a configuration reload, <file pattern> is re-evaluated. See also TIMING AND IPC section for a discussion about the rule processing order when multiple configuration files are involved.

-input=<file pattern>[=<context>]

expand <file pattern> to filenames (with the Perl glob() function) and use the files as event sources. An input file can be a regular file, named pipe, or standard input if - was specified. Multiple -input options can be specified at command line. Each time SEC receives a signal that forces it to reopen its input files, <file pattern> is re-evaluated. If SEC experiences a system error when reading from an input file, it will close the file (use the -reopen_timeout option for reopening the file). If <context> is given, SEC will set up the context <context> each time it reads a line from input files that correspond to <file pattern>. This will help the user to write rules that match data from particular input source(s) only. When there is an -input option with <context> specified, it will automatically enable the -intcontexts option. See TIMING AND IPC section for more information.

-input_timeout=<input_timeout>, -timeout_script=<timeout_script>

if SEC has not observed new data in an input file during <input_timeout> seconds (or the file was closed <input_timeout> seconds ago), <timeout_script> will be executed with command line parameters 1 and <the name of the input file>. If fresh data become available again, <timeout_script> will be executed with command line parameters 0 and <the name of the input file>.

-reopen_timeout=<reopen_timeout>

if an input file is in closed state (e.g., because SEC has failed to open it at startup), SEC will attempt to reopen this input file after every <reopen_timeout> seconds until open succeeds.

-check_timeout=<check_timeout>

if SEC has not observed new data in an input file, SEC will not poll the file (both for status and data) during the next <check_timeout> seconds.

-poll_timeout=<poll_timeout>

a real number that specifies how many seconds SEC will sleep when no new data were read from input files. Default is 0.1 seconds.

-blocksize=<io_block_size>

size of the block in bytes that SEC will attempt to read from input files at once. Default is 1024 (i.e., read from input files by 1KB blocks).

-bufsize=<input_buffer_size>

set input buffer to hold last <input_buffer_size> lines that have been read from input files. The content of input buffer will be compared with patterns that are part of rule definitions (i.e., no more than <input_buffer_size> lines can be matched by a pattern at a time). Default size is 10 lines.

-evstoresize=<event_store_size>

set an upper limit to the size of context event stores - if an event store already contains <event_store_size> events, SEC will add no more events to that context, regardless of its configuration file directives.

-cleantime=<clean_time>

time interval in seconds that specifies how often internal event correlation and context lists are processed, in order to accomplish time-related tasks and to remove obsolete elements. Default is 1 second.

-log=<logfile>

use <logfile> for logging SEC activities. Note that if the SEC standard error is connected to a terminal, messages will be logged there, in order to facilitate debugging. If the -detach option has also been specified, no logging to standard error will take place.

-syslog=<facility>

use syslog for logging SEC activities. All messages will be logged with the facility <facility>, e.g., local0 (see (3) for possible facility values). Warning: be careful with this option if you use SEC for monitoring syslog logfiles, because it might create message loops (SEC log messages are written to SEC input files that trigger new log messages).

-debug=<debuglevel>

set logging verbosity for the SEC logfile. Setting debuglevel to <debuglevel> means that all messages from <debuglevel> and lower levels are logged (e.g., if <debuglevel> is 3, messages from levels 1, 2, and 3 are logged). The following levels are recognized by SEC: 1 - critical messages (severe faults that cause SEC to terminate, e.g., a failed system call) 2 - error messages (faults that need attention but don't cause SEC to terminate, e.g., an incorrect rule definition in a configuration file) 3 - warning messages (possible faults, e.g., a command forked from SEC terminated with a non-zero exit code) 4 - notification messages (normal system level events and interrupts, e.g., the reception of a signal) 5 - informative messages (information about external commands forked from SEC) 6 - debug messages (detailed information about all SEC activities) Default <debuglevel> is 6 (i.e., log everything).

-pid=<pidfile>

SEC will store its process ID to <pidfile> at startup.

-dump=<dumpfile>

SEC will use <dumpfile> as its dumpfile. See SIGNALS section for more information. Default is /tmp/sec.dump.

-quoting, -noquoting

if the -quoting option is specified, event description strings that are supplied to external shell commands of shellcmd and spawn actions will be put inside apostrophes. All apostrophes ' that strings originally contain will be masked. This option prevents the shell from interpreting special symbols that event description strings might contain. Default is -noquoting.

-tail, -notail

if the -notail option is specified, SEC will process all data that are currently available in input files and exit after reaching all EOFs. Default is -tail - SEC will jump to the end of input files and wait for new lines to arrive. With the -tail option, SEC follows an input file by its name (and not merely by its i-node). If the input file is recreated or truncated, SEC will reopen it and process its content from the start. If the input file is removed (i.e., there is just an i-node left without name), SEC will keep the i-node open and wait for the input file to be recreated.

-fromstart, -nofromstart

these flags have no meaning when the -notail option is also specified. When used in combination with -tail (or alone, since -tail is enabled by default), -fromstart will force SEC to read and process input files from the beginning to the end, before the 'tail' mode is entered at SEC startup. Default is -nofromstart.

-detach, -nodetach

if the -detach option is specified, SEC will disassociate itself from the controlling terminal and become a daemon at startup (note that SEC will close its standard input, standard output, and standard error, and change its working directory to the root directory). Default is -nodetach.

-intevents, -nointevents

SEC will generate internal events when it starts up, when it receives certain signals, and when it terminates normally. Specific rules can be written to match those internal events, in order to take some action at SEC startup, restart, and shutdown. See TIMING AND IPC section for more information. Default is -nointevents.

-intcontexts, -nointcontexts

SEC will create an internal context when it reads a line from an input file. This will help the user to write rules that match data from particular input source only. See TIMING AND IPC section for more information. Default is -nointcontexts.

-testonly, -notestonly

if the -testonly option is specified, SEC will exit immediately after parsing the configuration file(s). If the configuration file(s) contained no faulty rules, SEC will exit with 0, otherwise with 1. Default is -notestonly.

-help, -?

SEC will output usage information and exit.

-version

SEC will output version information and exit.

Note that one can introduce options both with a single dash (-) and double dash (--), and also use both an equal sign (=) and whitespace as a separator between the option name and the option value, e.g., -conf=<file pattern> and --conf <file pattern> are equivalent.

Before describing each rule type in detail, patterns and pattern types, context expressions, and action lists are discussed, since they are important parts of the rule definition.

SubStr[<number>]

pattern is assumed to be a substring that will be searched in last <number> input lines L1, L2, ..., L<number>. The input lines are joined into a single string with the newline character acting as a separator, and the resulting string "L1\nL2\n...\nL<number>" is searched for the substring. (Note that if <number> is 1, the last input line without a terminating newline is searched.) If the substring is found, the pattern matches. The backslash constructs \t, \n, \r, \s, and \0 can be used in the pattern to denote tabulation, newline, carriage return, space character, and empty string, while \\ denotes backslash itself. As an example, consider the following pattern definition:

ptype=substr pattern=Backup done:\tsuccess

The pattern matches lines containing "Backup done:<TAB>success".

RegExp[<number>]

pattern is assumed to be a regular expression that last <number> input lines L1, L2, ..., L<number> are compared with. The input lines are joined into a single string with the newline character acting as a separator, and the resulting string "L1\nL2\n...\nL<number>" is compared with the regular expression pattern. (Note that if <number> is 1, the last input line without a terminating newline is compared with the pattern.) If the pattern matches, backreference values will be assigned to the special variables $1, $2, ..., and the special variable $0 will be set to "L1\nL2\n...\nL<number>" (i.e., to the matching input line(s)). These special variables can be used in some other parts of the rule definition. All regular expression constructs that Perl allows are allowed in the pattern (see perlre(1) for more information). As an example, consider the following pattern definition:

ptype=regexp pattern=(?i)(\S+\.mydomain).*printer: toner\/ink low

The pattern matches "printer: toner/ink low" messages in a case insensitive manner from printers belonging to .mydomain. Note that the printer hostname is assigned to $1, while the whole message line is assigned to $0. As another example, the following pattern definition

ptype=regexp2 pattern=^AAA\nBBB$

produces a match if the last two input lines are AAA and BBB, and sets $0 to "AAA\nBBB".

PerlFunc[<number>]

pattern is assumed to be a Perl function that last <number> input lines L1, L2, ..., L<number> are submitted to. The Perl function is compiled at SEC startup by calling the Perl eval() function, and eval() must return a code reference for the pattern to be valid (also see VARIABLES AND EVAL section for more information). In order to check whether the pattern matches, SEC will call the function in list context and pass lines L1, L2, ..., L<number> and the names of corresponding input sources S1, S2, ..., S<number> to the function as parameters:

function(L1, L2, ..., L<number>, S1, S2, ..., S<number>)

(if the input line has been generated with the event action, its input source name will be set to 'undef'). If the function returns several values or a single value that is TRUE in boolean context, the pattern matches. If the pattern matches, return values will be assigned to the special variables $1, $2, ..., and the special variable $0 will be set to "L1\nL2\n...\nL<number>" (i.e., to the matching input line(s)). These special variables can be used in some other parts of the rule definition. As an example, consider the following pattern definition:

ptype=perlfunc2 pattern=sub { return ($_[0] cmp $_[1]); }

The pattern compares last two input lines in a stringwise manner ($_[1] holds the last line and $_[0] the preceding one), and matches if the lines are different. Note that the result of the comparison is assigned to $1, while the two input lines are concatenated (with the newline character between them) and assigned to $0. As another example, the following pattern definition

ptype=perlfunc pattern=sub { if ($_[0] =~ /(abc|def)/) { \ return defined($_[1]) ? $_[1] : "SEC"; } return 0; }

produces a match if the input line contains either the string "abc" or the string "def", and sets $0 to the matching line and $1 to the name of the input source.

NSubStr[<number>]

like SubStr[<number>], except the result of the match is negated.

NRegExp[<number>]

like RegExp[<number>], except the result of the match is negated.

NPerlFunc[<number>]

like PerlFunc[<number>], except the result of the match is negated.

TValue

pattern is assumed to be a truth value with TRUE and FALSE being legitimate values for the pattern. TRUE always matches an input line while FALSE never matches an input line.

Note that since Pair and PairWithWindow rules have two pattern definitions, special variables $<number> set by the first pattern are shadowed when the second pattern matches and sets the variables. In order to access the shadowed variables, they must be referred to as %<number> (e.g., instead of $1 one has to write %1). Also note that the second pattern of Pair and PairWithWindow rules may contain $<number> variables, if the second pattern is of type SubStr, NSubStr, Regexp, or NRegExp. The variables are substituted at runtime with the values set by the first pattern.

If the operand contains the arrow (->), the text following the arrow is considered to be a Perl function that will be compiled at SEC startup by calling the Perl eval() function, and eval() must return a code reference for the operand to be valid (also see VARIABLES AND EVAL section for more information). If any text precedes the arrow, it is considered to be the list of parameters for the function. Parameters are separated by whitespace and may contain $<number> and %<number> special variables. In order to evaluate the Perl function operand, the function with its parameter list will be called in scalar context. If the return value of the function is TRUE in boolean context, the truth value of the operand is TRUE, otherwise its truth value is FALSE.

If the operand begins with the equal sign (=), the following text is considered to be a Perl miniprogram. The miniprogram may contain $<number> and %<number> special variables. In order to evaluate the Perl miniprogram operand, it will be executed by calling the Perl eval() function in scalar context (also see VARIABLES AND EVAL section for more information). If the return value of the miniprogram is TRUE in boolean context, the truth value of the operand is TRUE, otherwise its truth value is FALSE.

Note that since Perl functions, miniprograms, and parameter lists may contain strings that clash with the operators of the context expression (e.g., '!'), it is strongly recommended to enclose them in parentheses, e.g.,

($1 $2) -> (sub { return ($_[0] != $_[1]); })

=({my($temp) = 0; return !$temp;})

If the operand is not a Perl function or program, it is considered to be a context name. Context name may contain $<number> and %<number> special variables. If the context name refers to an existing context, the truth value of the operand is TRUE, otherwise its truth value is FALSE.

If the whole context expression is enclosed in square brackets [ ], e.g., [MYCONTEXT1 && !MYCONTEXT2], SEC evaluates the expression _before_ pattern matching operation (normally the pattern is compared with input line(s) first, so that $<number> and %<number> variables in the context expression could be replaced with their values). Evaluating context expression first could save CPU time when many of the input lines are known to match the pattern but not the context expression, and the expression does not contain any variables.

As an example, consider the following context expressions:

-> ( sub { my(@stat) = stat("/var/log/messages"); \ return (!scalar(@stat) || time() - $stat[9] > 3600); } )

($1 $2) -> (sub { return ($_[0] != $_[1]); }) && C1

!(C1 || C2) && =("$1" eq "myhost.mydomain")

The first expression is TRUE when the /var/log/messages file does not exist or was last modified more than 1 hour ago. The second expression is TRUE when variables $1 and $2 are numerically not equal and the context C1 exists. The third expression is TRUE when contexts C1 and C2 do not exist and the $1 variable equals to the string "myhost.mydomain". Note that since && is a short-circuiting operation, the Perl code of the third expression is not evaluated if C1 and/or C2 exist.

%s - event description string (set either by the desc or desc2 parameter of the rule definition). %t - textual timestamp (as returned by date(1)). %u - numeric timestamp (as returned by (2)). %<alnum_name> - user-defined variables that can be set at SEC runtime with certain actions like assign and eval (<alnum_name> must begin with a letter and may contain letters, digits, and underscores). In order to disambiguate <alnum_name> from the following text, <alnum_name> must be enclosed in braces: %{<alnum_name>}.

In order to use semicolons inside a non-constant action parameter, the parameter must be enclosed in parentheses (the outermost set of parentheses will be removed by SEC during configuration file parsing).

The following actions are supported:

none

No action.

logonly

Event description (the value of the %s variable) is logged.

write <filename> [<event text>]

Event string <event text> and terminating newline are written to the file <filename> (<filename> may not contain spaces). File may be a regular file, named pipe, or standard output if - was specified. If the file is a regular file, <event text> is appended to the end of the file. If the file does not exist, it is created as a regular file before writing. If <event text> is omitted, %s is assumed for its value. Though write can also be expressed through the shellcmd action, write does not involve (2) like shellcmd does.

shellcmd <shellcmd>

Shell command <shellcmd> is executed. If the -quoting option was specified, %s will be converted to '%s' before supplying it to shell command (see the -quoting and -noquoting options).

spawn <shellcmd>

Identical to the shellcmd action, except the following - every line from the standard output of <shellcmd> is treated like SEC input line and matched against the rules. This is done by applying event 0 <line> to every line from standard output (see the event action). Note that if <shellcmd> outputs a large data set at once, SEC will process it all at once, so if <shellcmd> enters an endless "busy write" loop, it will block SEC from doing anything other than processing its output.

pipe '<event text>' [<shellcmd>]

Event string <event text> and terminating newline are fed to the standard input of shell command <shellcmd> (apostrophes are used to mark the beginning and end of <event text>, in order to distinguish it from <shellcmd>). If <event text> is omitted and there is nothing between apostrophes, %s is assumed for <event text>. If <shellcmd> is omitted, <event text> is written to standard output.

create [<name> [<time> [<action list>] ] ]

Context with the name <name>, with the lifetime of <time> seconds, and with empty event store is created (<name> may not contain spaces, and <time> is an integer constant). If <name> is omitted, %s is assumed for its value. Specifying 0 as <time> or omitting the value means infinite lifetime. If <action list> is specified, it will be executed when the context expires. If <action list> is made up of more than one action, semicolons must be used to separate the actions, and the list must be enclosed in parentheses. If already existing context is recreated with create, its lifetime will be extended for <time> seconds and its event store will be emptied.

delete [<name>]

Context with the name <name> is deleted (<name> may not contain spaces). If <name> is omitted, %s is assumed for its value. If non-existing context is deleted, no operation will be performed.

obsolete [<name>]

Behaves like delete, except the action list of the context <name> (if the context has an action list) will be executed before deletion.

set <name> <time> [<action list>]

Settings for the context with the name <name> will be changed (<name> may not contain spaces, and <time> is an integer constant). New lifetime of the context will be <time> seconds with optional <action list> to be executed when the context expires. Event store of the context will not be changed by set. Specifying 0 as <time> means infinite lifetime. If <action list> is made up of more than one action, semicolons must be used to separate the actions, and the list must be enclosed in parentheses.

alias <name> [<alias>]

An alias name <alias> will be created for the context with the name <name> (<name> and <alias> may not contain spaces). After the name <alias> has been created for a context, the context can be referred to by using both <alias> and <name>. If <alias> is omitted, %s is assumed for its value. If context with the name <name> does not exist, or the name <alias> already exists, the alias will not be created. Internally, SEC does not distinguish in any way between <alias> and <name> - they are both pointers to the same context data structure. Therefore, it makes no difference whether context attributes (like lifetime, event store, etc.) are changed by calling an action (e.g., set or add) for <name> or <alias>. If the delete action is called for one of the context names, the context data structure is destroyed, and all context names (which are now pointers to unallocated memory) are removed from the list of context names. Also note that if the context expires, its action list is executed only once, no matter how many names the context has.

unalias [<alias>]

Context name <alias> is removed from the list of context names, so that the name <alias> can no longer be used to refer to the context it was previously associated with (<alias> may not contain spaces). If <alias> is omitted, %s is assumed for its value. If unalias is called for non-existing context name, no operation will be performed. If the name <alias> was the last reference to a context, the unalias action behaves like delete and the context will be deleted; otherwise the context will continue to exist under other name(s) with its event store and other attributes intact.

add <name> [<event text>]

Event string <event text> is added to the event store of the context <name> (<name> may not contain spaces). Events in the store are ordered by the time they were added, and every add appends event to the end of the store. If <event text> is omitted, %s is assumed for its value. If context <name> does not exist, the context will be created with an infinite lifetime, empty action list and empty event store (as with create <name>) before adding the event. If <event text> contains newlines, it will be split into parts using the newline symbol as a delimiter, and each part is added to the event store as a separate event string.

fill <name> [<event text>]

Behaves like add, except the event store of the context <name> will be emptied before <event text> is added.

report <name> [<shellcmd>]

Event store of the context <name> is reported with shell command <shellcmd> (<name> may not contain spaces). Reporting means that events from the store are fed to standard input of <shellcmd> in the order they were added into the store, every event on a separate line. If <shellcmd> is omitted, events from the store are written to standard output.

copy <name> %<alnum_name>

Event store of the context <name> is assigned to a user-defined variable %<alnum_name> (<name> may not contain spaces). Before the assignment takes place, lines from the event store are joined into a scalar using the newline character as the separator (as with Perl join("\n", @array)).

empty <name> [%<alnum_name>]

Behaves like copy, except the event store of the context <name> will be emptied after it has been assigned to the variable %<alnum_name>. If %<alnum_name> is omitted, empty simply removes all lines from the event store.

event [<time>] [<event text>]

After <time> seconds a synthetic event <event text> is created (<time> is an integer constant). SEC treats the <event text> string exactly like line(s) read from input - it is inserted into the input buffer in order to compare it with rules. If <event text> is omitted, %s is assumed for its value. Specifying 0 as <time> or omitting the value means now. If <event text> contains newlines, it will be split into parts using the newline symbol as a delimiter, and each part is created as a separate synthetic event.

reset [<rule_number>] [<event text>]

Cancel event correlation operations that are currently detecting the composite event <event text> (<rule_number> is a string constant), i.e., SEC will terminate event correlation operations that have <event text> in their keys as event description string (if such operations exist, see RULES AND EVENT CORRELATION OPERATIONS section for more information). If there are multiple configuration files specified, reset can cancel only those correlation operations that have been started by the rules from the same configuration file where the reset action itself is defined. If <event text> is omitted, %s is assumed for its value. Since correlation operations started by different rules may detect composite events that have identical description strings, rule number can be optionally specified to point to a correlation operation that was started by a specific rule (1 means the first rule in the configuration file, 2 means the second, etc.; 0 denotes the current rule). If + or - is prepended to <rule_number>, it is considered to be an offset from the current rule (e.g., -1 means the previous rule and +1 the next rule). For example, if a rule definition with the reset action is given in the configuration file named my.conf, then reset 1 Counting linkdown events will terminate the event correlation operation with the key "my.conf | 0 | Counting linkdown events" (note that internally the SEC rule IDs start from zero), while reset Counting linkdown events will terminate event correlation operations with keys "my.conf | X | Counting linkdown events", where X runs from 0 to N-1 and N is the number of rules in the configuration file my.conf. If no operation with a given key exists, reset will take no action.

assign %<alnum_name> [<text>]

Text <text> is assigned to a user-defined variable %<alnum_name>. If <text> is omitted, %s is assumed for its value.

eval %<alnum_name> <code>

The parameter <code> is assumed to be a Perl miniprogram that will be executed by calling the Perl eval() function in list context. If the miniprogram returns a single value, it will be assigned to the variable %<alnum_name>. If the miniprogram returns several values, they will be joined into a scalar using the newline character as a separator (as with Perl join("\n", @array)), and the scalar will be assigned to the variable %<alnum_name>. If no value is returned or eval() fails, no assignment will take place. Note that before calling eval(), $<number>, %<number>, and %<alnum_name> variables in the miniprogram are replaced with their values and therefore can't be used as lvalues. Since most Perl programs contain semicolons which are also used by SEC as separators between actions, it is recommended to enclose the <code> parameter in parentheses, in order to avoid the inperpretation of semicolons inside the code by SEC. Also see VARIABLES AND EVAL section for more information.

call %<alnum_name1> %<alnum_name2> [<parameter list>]

The parameter %<alnum_name2> is assumed to be a reference to a Perl function that was created previously with the eval action. The function will be called in list context by passing optional parameter list to the function (parameters are separated by whitespace). If the function returns a single value, it will be assigned to the variable %<alnum_name1>. If the function returns several values, they will be joined into a scalar using the newline character as a separator (as with Perl join("\n", @array)), and the scalar will be assigned to the variable %<alnum_name1>. If no value is returned or %<alnum_name2> is not a code reference, no assignment will take place. Also see VARIABLES AND EVAL section for more information.

Examples:

spawn /bin/tail -f /var/log/trapd.log

Follow the /var/log/trapd.log file and feed to SEC input all lines that are appended to the file.

pipe '%t: $0' /bin/mail -s "alert message" root@localhost

Mail the timestamp and the value of the $0 variable to the local root.

add ftp_$1 $0; set ftp_$1 1800 (report ftp_$1 /bin/mail root@localhost)

Add the value of the $0 variable to the event store of the context ftp_<the value of $1>. Also extend the context's lifetime for 30 minutes, so that when the context expires, its event store will be mailed to the local root.

eval %funcptr ( sub { my(@buf) = split(/\n/, $_[0]); \ my(@ret) = grep(!/^#/, @buf); return @ret; } ); \ copy C1 %in; call %out %funcptr %in; fill C1 %out

Create a subroutine for weeding out comment lines from the input list, and use this subroutine for removing comment lines from the event store of the context C1.

action=eval %o (print ")";)

will be considered an invalid action list (however, note that

action=eval %o (print "()";)

would be passed by SEC, since now parentheses are balanced). In order to avoid such unwanted behavior, each parenthesis without a counterpart must be masked with a backslash (the backslash will be removed by SEC during configuration file parsing), e.g.,

action=eval %o (print "\)";)

type

fixed to Single (value is case insensitive, so single or sIngLe can be used instead).

continue (optional)

TakeNext or DontCont (values are case insensitive). The first specifies that search for matching rules will continue after the match (i.e., input line(s) that match pattern and context will be passed to the next rule), the second will quit the search. If continue keyword is missing from the rule definition, DontCont is assumed.

ptype

pattern type (value is case insensitive).

pattern

pattern.

context (optional)

context expression.

desc

textual description of the detected event.

action

action list that will be executed when the event is detected.

rem (optional, may appear more than once)

remarks and comments.

Note that context, desc, and action parameters may contain $<number> variables.

Examples:

type=single continue=takenext ptype=regexp pattern=ftpd\[(\d+)\]: \S+ \(ristov2.*FTP session opened desc=ftp session opened for ristov2 pid $1 action=create ftp_$1

type=single continue=takenext ptype=regexp pattern=ftpd\[(\d+)\]: context=ftp_$1 desc=ftp session event for ristov2 pid $1 action=add ftp_$1 $0; set ftp_$1 1800 \ (report ftp_$1 /bin/mail root@localhost)

type=single ptype=regexp pattern=ftpd\[(\d+)\]: \S+ \(ristov2.*FTP session closed desc=ftp session closed for ristov2 pid $1 action=report ftp_$1 /bin/mail root@localhost; \ delete ftp_$1

The first rule creates the context with the name ftp_<pid> when someone connects from host ristov2 with ftp. The second rule adds all logfile lines that are associated with the session <pid> to the event store of the context ftp_<pid> (before adding a line, the rule checks if the context exists). After adding a line, the rule extends context's lifetime for 30 minutes and sets the action list that will be executed when the context expires. The third rule mails collected logfile lines to root@localhost when the session <pid> is closed. Collected lines will also be mailed when the session <pid> has been inactive for 30 minutes (no logfile lines observed for that session).

Note that the logfile line that has matched the first rule will be passed to the second rule and will become the first line in the event store (the first rule has the continue parameter set to TakeNext). The second rule has also its continue parameter set to TakeNext, since otherwise no logfile lines would reach the third rule.

type

fixed to SingleWithScript (value is case insensitive).

continue (optional)

TakeNext or DontCont (values are case insensitive).

ptype

pattern type (value is case insensitive).

pattern

pattern.

context (optional)

context expression.

script

script that is executed after pattern and context have matched some input line(s). The names of all existing contexts are fed to the standard input of the script, a name per line. If the script returns zero for its exit value, action will be executed; if the script returns non-zero, action2 will be executed (if given).

desc

textual description of the detected event.

action

action list that will be executed when script returns zero for its exit value.

action2 (optional)

action list that will be executed when script returns non-zero for its exit value.

rem (optional, may appear more than once)

remarks and comments.

Note that context, script, desc, action, and action2 parameters may contain $<number> variables.

Also note that since the runtime of the script that is specified by the script parameter is not limited in any way, waiting for that script to complete could freeze the entire event processing for an indefinite amount of time. Therefore, once the pattern and context parameters have matched input line(s), SEC does not wait for the script to complete but rather continues its work. The exit value of the script will be fetched when the script actually terminates, and depending on the exit value, either action or action2 (if it exists) will be executed.

Examples:

type=SingleWithScript ptype=RegExp pattern=interface (\S+) down script=/usr/local/scripts/ping.sh $1 desc=Interface $1 down action=shellcmd /usr/local/scripts/notify.sh "%s"

When "interface <ipaddress> down" line appears in input, this rule acts by calling

/usr/local/scripts/ping.sh <ipaddress>

and if the script returns 0 as its exit value, the rule calls

/usr/local/scripts/notify.sh "Interface <ipaddress> down"

(note also that SEC feeds the names of all existing contexts to the standard input of the /usr/local/scripts/ping.sh script.)

type

fixed to SingleWithSuppress (value is case insensitive).

continue (optional)

TakeNext or DontCont (values are case insensitive).

ptype

pattern type (value is case insensitive).

pattern

pattern for detecting event A.

context (optional)

context expression.

desc

textual description of event A.

action

action list that will be executed when event A is first observed. Following instances of event A will be ignored.

window

time in seconds that following events A are ignored for.

rem (optional, may appear more than once)

remarks and comments.

Note that context, desc, and action parameters may contain $<number> variables.

Examples:

type=SingleWithSuppress ptype=RegExp pattern=(\S+): [fF]ile system full desc=File system $1 full action=shellcmd notify.sh "%s" window=900

When "/usr: file system full" line appears in input, this rule starts a correlation operation that calls

notify.sh "File system /usr full"

and ignores following such lines during the next 15 minutes.

Note that line "/tmp: file system full" would not be ignored, since SEC identifies correlation operations by a key that contains event description string of the composite event (given by the desc parameter). Since strings "/usr: file system full" and "/tmp: file system full" differ, SEC will start another correlation operation for the latter (see RULES AND EVENT CORRELATION OPERATIONS section for more information).

type

fixed to Pair (value is case insensitive).

continue (optional)

TakeNext or DontCont (values are case insensitive). Specifies if input line(s) that match pattern and context will be passed to the next rule.

ptype

pattern type (value is case insensitive).

pattern

pattern for detecting event A.

context (optional)

context expression.

desc

textual description of event A.

action

action list that will be executed when event A is first observed. Following instances of event A will be ignored.

continue2 (optional)

TakeNext or DontCont (values are case insensitive). Specifies if input line(s) that match pattern2 and context2 will be passed to the next rule.

ptype2

pattern type (value is case insensitive).

pattern2

pattern for detecting event B.

context2 (optional)

context expression.

desc2

textual description of event B.

action2

action list that will be executed when event B is observed. After executing action2 the event correlation operation terminates.

window (optional)

time t in seconds that is allowed to elapse between the first instance of event A and event B. If event B does not appear during t seconds, then the correlation operation started by this rule terminates. Specifying 0 as value or omitting the window parameter means setting t to infinity (i.e., if event B does not appear, event A will be ignored forever).

rem (optional, may appear more than once)

remarks and comments.

Note that context, desc, action, context2, desc2, and action2 parameters may contain $<number> variables. The pattern2 parameter may contain $<number> variables if ptype2 supports variable substitutions at runtime. If pattern2 shadows the $<number> variables of pattern, then context2, desc2, and action2 parameters may contain %<number> variables.

Examples:

type=Pair ptype=RegExp pattern=NFS server (\S+) not responding desc=$1 is not responding action=shellcmd notify.sh "%s" ptype2=substr pattern2=NFS server $1 ok desc2=$1 OK action2=shellcmd notify.sh "%s" window=3600

When "NFS server fserv is not responding" line appears in input, the correlation operation started by this rule calls

notify.sh "fserv is not responding"

and waits for the line "NFS server fserv ok" for 1 hour, ignoring all "NFS server fserv is not responding" lines. When the line "NFS server fserv ok" appears, the correlation operation executes the shell command

notify.sh "fserv OK"

and terminates. If time (1 hour) runs out, the correlation operation will terminate without doing anything.

type

fixed to PairWithWindow (value is case insensitive).

continue (optional)

TakeNext or DontCont (values are case insensitive). Specifies if input line(s) that match pattern and context will be passed to the next rule.

ptype

pattern type (value is case insensitive).

pattern

pattern for detecting event A.

context (optional)

context expression.

desc

textual description of event A.

action

action list that is executed after event A was observed and event B did not appear within the given time window.

continue2 (optional)

TakeNext or DontCont (values are case insensitive). Specifies if input line(s) that match pattern2 and context2 will be passed to the next rule.

ptype2

pattern type (value is case insensitive).

pattern2

pattern for detecting event B.

context2 (optional)

context expression.

desc2

textual description of event B.

action2

action list that is executed after event A was observed and event B appeared within the given time window. After executing action2 the event correlation operation terminates.

window

size of the time window in seconds.

rem (optional, may appear more than once)

remarks and comments.

Note that context, desc, action, context2, desc2, and action2 parameters may contain $<number> variables. The pattern2 parameter may contain $<number> variables if ptype2 supports variable substitutions at runtime. If pattern2 shadows the $<number> variables of pattern, then context2, desc2, and action2 parameters may contain %<number> variables.

Examples:

type=PairWithWindow ptype=RegExp pattern=node (\S+) interface (\S+) down desc=$1 if $2 is down action=shellcmd notify.sh "%s" ptype2=SubStr pattern2=node $1 interface $2 up desc2=$1 if $2 short outage action2=event window=600

When "node fserv interface 192.168.1.1 down" line appears in input, this rule starts a correlation operation that waits 10 minutes for "node fserv interface 192.168.1.1 up" line, and if that line does not arrive on time, the correlation operation executes the shell command

notify.sh "fserv if 192.168.1.1 is down"

and terminates. If the line arrives on time, the operation generates event "fserv if 192.168.1.1 short outage" and terminates.

type

fixed to SingleWithThreshold (value is case insensitive).

continue (optional)

TakeNext or DontCont (values are case insensitive).

ptype

pattern type (value is case insensitive).

pattern

pattern for detecting event A.

context (optional)

context expression.

desc

textual description of event B.

action

action list that is executed when thresh instances of event A have been observed within the given time window. After that all events A will be ignored during the rest of the time window.

action2 (optional)

action list that is executed when the event correlation operation terminates, if action has been previously executed by the operation.

window

size of the time window in seconds. The window is sliding - if event A has been observed less than thresh times at the end of the window, the beginning of the window is moved to the occurrence time of the second instance of event A, and the counting operation will continue. If there is no second instance of event A (i.e., the event has been observed only once), the correlation operation will terminate.

thresh

threshold value.

rem (optional, may appear more than once)

remarks and comments.

Note that context, desc, and action parameters may contain $<number> variables.

Examples:

type=SingleWithThreshold ptype=RegExp pattern=user (\S+) login failure on (\S+) desc=Repeated login failures for user $1 on $2 action=shellcmd notify.sh "%s" window=60 thresh=3

When line "user dbadmin login failure on tty1" is observed, the rule starts a correlation operation that executes the shell command

notify.sh "Repeated login failures for user dbadmin on tty1"

if additional two such lines are observed within 1 minute. Following "user dbadmin login failure on tty1" lines will be ignored by this rule until 60 seconds have elapsed since the arrival of first line.

After changing the action parameter in the rule definition to

action=shellcmd notify.sh "%s"; reset 0 %s

the correlation operation will terminate itself after calling notify.sh, and the next matching line will start a new counting operation.

type

fixed to SingleWith2Thresholds (value is case insensitive).

continue (optional)

TakeNext or DontCont (values are case insensitive).

ptype

pattern type (value is case insensitive).

pattern

pattern for detecting event A.

context (optional)

context expression.

desc

textual description of event B.

action

action list that is executed when thresh instances of event A have been observed within the time window window. After that event counting continues with the threshold value thresh2 and time window window2.

window

size of the first time window in seconds. The window is sliding.

thresh

the first threshold value.

desc2

textual description of event C.

action2

action list that is executed if no more than thresh2 instances of event A have been observed during the last window2 seconds. After executing action2 the event correlation operation terminates.

window2

size of the second time window in seconds. The window is sliding.

thresh2

the second threshold value.

rem (optional, may appear more than once)

remarks and comments.

Note that context, desc, action, desc2, and action2 parameters may contain $<number> variables.

Examples:

type=SingleWith2Thresholds ptype=RegExp pattern=(\S+): %SYS-3-CPUHOG desc=router $1 CPU overload action=shellcmd notify.sh "%s" window=300 thresh=2 desc2=router $1 CPU load normal action2=shellcmd notify.sh "%s" window2=3600 thresh2=0

When SYS-3-CPUHOG syslog message is received from a router, the rule starts a counting operation that executes the shell command

notify.sh "router <routername> CPU overload"

if additional SYS-3-CPUHOG syslog message is received from the router within 5 minutes. After that the correlation operation waits until no SYS-3-CPUHOG syslog messages have been received from the router during the last 1 hour, and then executes

notify.sh "router <routername> CPU load normal"

type

fixed to Suppress (value is case insensitive).

ptype

pattern type (value is case insensitive).

pattern

pattern for detecting event A (or its subclass) that must be suppressed (or filtered out).

context (optional)

context expression.

desc (optional)

textual description of this rule.

rem (optional, may appear more than once)

remarks and comments.

Note that the context parameter may contain $<number> variables.

Examples:

type=Suppress ptype=RegExp pattern=[Ff]ile system full context=mycontext

type=Suppress ptype=RegExp pattern=/dev/vg01/\S+: [Ff]ile system full

The first rule suppresses all "file system full" events if the context mycontext is present. The second rule filters out all "file system full" events that concern the volume group vg01.

type

fixed to Calendar (value is case insensitive).

time

crontab-style time specification (see crontab(1) for more information). Time specification consists of five fields separated by whitespace characters. First field denotes minutes (allowed values 0-59), second field hours (allowed values 0-23), third field days (allowed values 0-31, 0 denotes the last day of the month), fourth field months (allowed values 1-12), and fifth field weekdays (allowed values 0-7, 0 and 7 denote Sunday). Asterisks (*), ranges of numbers (e.g., 8-11), and lists (e.g., 2,5,7-9) are allowed as field values.

context (optional)

context expression.

desc

event description string.

action

action list to be executed.

rem (optional, may appear more than once)

remarks and comments.

Examples:

type=Calendar time=0 23 * * * desc=NightContext action=create %s 32400

This rule creates the context NightContext every day at 11PM. Context has a lifetime of 9 hours.

Suppose you have a configuration file my.conf with one rule in it:

type=SingleWithThreshold ptype=RegExp pattern=user (\S+) login failure on (\S+) desc=Repeated login failures for user $1 on $2 action=shellcmd notify.sh "%s" window=60 thresh=3

Suppose that SEC observes an input line "user admin login failure on tty1". This matches the pattern 'user (\S+) login failure on (\S+)', and after replacing $1 and $2 with their values, the desc parameter will yield "Repeated login failures for user admin on tty1". SEC will then form a key for the event correlation operation, using the configuration file name, the rule ID, and the event description string:

my.conf | 0 | Repeated login failures for user admin on tty1

Since the rule was the first one in the configuration file, its ID is 0. The ID for the second rule would be 1, for the third rule 2, etc. Note that because the configuration file name and the rule ID are present in the keys, event correlation operations started by different rules will not clash, even if the rules have identical values for the desc parameter.

After calculating the key, SEC will check if there already is an event correlation operation with that key. If such operation exists, the input line will be correlated by the existing operation. Otherwise, a new event correlation operation will be started.

By using appropriate event description string definitions, you can change the scope of individual event correlation operations. For instance, if you use "Repeated login failures for user $1" for the desc parameter in the above rule definition, the following three lines will now be correlated by the same event correlation operation:

user admin login failure on tty1 user admin login failure on tty5 user admin login failure on tty2

In other words, SEC will now just count login failures for different users, disregarding terminal names.

Finally, it should be noted that Single, SingleWithScript, Suppress, and Calendar rules never start event correlation operations, since they don't involve the matching of several events over a certain time window (i.e., their design allows them to complete their work almost immediately without starting an event correlation operation). Therefore, the desc parameter of these rules does not influence the number and the scope of event correlation operations.

When new data appear in SEC input files, only one line will be read at a time (even when more lines are available), after which the input buffer is updated with the line that was read. Input buffer contains the last N lines from the SEC input, in the order they were read (the value of N can be set with the -bufsize option). Updating the input buffer means that the last element (the oldest line) is removed from the end of the buffer, while the new line becomes the first element of the buffer. Note that when synthetic events have been created with the event action and new input data are also available in input files, synthetic events are always read first by SEC. After no more such events are available for reading, SEC will read new data from input files.

After the buffer has been updated (every update always adds only one and removes only one line), the rules from configuration files are processed, matching the rules against the new content of the input buffer. Even when a rule matches and its action list suggests an immediate change in the buffer (e.g., through the event action), the input buffer will _not_ be updated until all the rules have been compared with the current content of the buffer.

Rules from the same configuration file are compared with the input buffer in the order they were given in that file. When multiple configuration files have been specified, each file containing a distinct ruleset, events are processed virtually in parallel - the buffer is always processed by all rulesets. However, the order the rulesets are applied during event processing is determined by the order the files were given at SEC command line. If a -conf option specifies a pattern, SEC uses the Perl glob() function to expand the pattern, and the resulting file list is applied by SEC in the order returned by glob() (this is usually ascending ASCII order). E.g., if you have specified -conf options in the following order:

-conf=/home/risto/*.conf -conf=/home/risto/*.conf2

and directory /home/risto contains files A.conf, B.conf2, and C.conf, then SEC applies these ruleset files in the following order: A.conf, C.conf, and B.conf2. Also, note that even if A.conf contains a Suppress rule for a particular event, the event is still processed by rulesets in C.conf and B.conf2.

After the input buffer has been updated and its content has been processed by the rules, SEC handles the signals that have been received since the last check, and also reads possible output from its child processes.

When the timeout specified with the -cleantime option has expired, SEC also checks the contexts and the event correlation operations that have timers associated with them (e.g., SingleWithSuppress and PairWithWindow rules). This means that the value of -cleantime should be relatively small, in order to keep the event correlation operations accurate. By setting the -cleantime option value to 0, SEC will check the event correlation timers after reading every line, but this will consume more CPU time. The higher the value, the less CPU time will be consumed, so the value of the -cleantime option depends on your requirements. Note that if the -poll_timeout option value exceeds the value given with -cleantime, the -poll_timeout option value takes precedence (i.e., sleeps after unsuccessful polls will not be shortened).

If the -intevents command line option is given, SEC will generate internal events when it is started up, when it receives certain signals, and when it terminates normally. Inside SEC, internal event is treated as if it was a line that was read from a SEC input file. Specific rules can be written to match internal events, in order to take some action (e.g., start an external event correlation module with spawn when SEC starts up). The following internal events are supported:

SEC_STARTUP - generated when SEC is started (once the -intevents option has been given, this event will always be the first event that SEC observes)

SEC_RESTART - generated after SEC has received the SIGHUP signal and all internal data structures have been cleared (this event will be the first event that SEC observes after reloading its configuration)

SEC_SOFTRESTART - generated after SEC has received the SIGABRT signal (this event will be the first event that SEC observes after reloading its configuration)

SEC_SHUTDOWN - generated when SEC receives the SIGTERM signal, or when SEC reaches all EOFs of input files after being started with the -notail option. After generating SEC_SHUTDOWN event, SEC will sleep for 3 seconds before sending SIGTERM to its child processes (if there are any child processes that were started by SEC_SHUTDOWN, the processes will have enough time to set a signal handler for SIGTERM if needed).

Right before generating an internal event, SEC will also set up a context named SEC_INTERNAL_EVENT, in order to distinguish between true internal events and coincidents that look like internal events. The SEC_INTERNAL_EVENT context will be deleted immediately after the internal event has been processed by all rules.

If the -intcontexts command line option is given, or there is an -input option with a context specified, SEC will create an internal context each time a line is read from an input file, or a line is read that was created with the event action. The internal context will be deleted immediately after the line has been matched against all rules. For all input files that have the context name explicitly set with -input=<file pattern>=<context>, the name of the internal context is <context>. If the line was read from the input file <filename> for which there is no context name set, the name of the internal context is _FILE_EVENT_<filename>. If the line was created with the event action, the name of the internal context is _INTERNAL_EVENT. This will help the end user to write rules that will match data from one particular input source only, e.g., the rule

type=Suppress ptype=TValue pattern=TRUE context=[!_FILE_EVENT_/dev/logpipe]

will pass to the following rules only those lines that were read from /dev/logpipe.

The shellcmd, spawn, pipe, and report actions involve the creation of a child process. The communication between SEC and its child processes takes place through pipes (created with Perl pipe opens like open(FH, "| mycommand") or (2) system call). Note that the running time of children is not limited in any way, so long-running processes can be started from SEC. For instance, one could start a SEC agent with the spawn action that runs forever and provides SEC with additional input events. However, note that SEC sends the SIGTERM signal to all its children before termination. If some special exit procedures need to be carried out in the child process (or the child wishes to ignore SIGTERM), then the child must install a handler for the SIGTERM signal.

Note that if a rule definition includes two shellcmd actions (or other actions that call external scripts or programs), the order that these scripts or programs are executed is not determined. For instance, with the following action definition

action=shellcmd cmd1; shellcmd cmd2

cmd2 could well terminate before cmd1, or cmd2 could well start before cmd1 (e.g., when cmd1 is a complex command line and cmd2 is relatively simple, it takes more time from the shell to process and start cmd1 than cmd2).

In order to mask the variables, an extra $ or % must be prepended to them, e.g., $$1 yields $1, and not a dollar sign followed by the first value returned by the pattern. Also, all occurrences of $$ will be converted to $ when $<number> variables are substituted, and all occurrences of %% will be converted to % when %<number> or %<alnum_name> variables are substituted, so that one could use $ or % together with variable names (e.g., $$$1 will yield a dollar sign that is followed by the first value returned by the pattern).

Please note the following caveat for Pair and PairWithWindow rules when the second pattern shadows the $<number> variables of the first pattern - since both %<number> and %<alnum_name> variables may be used in the second action list and %<number> variables are substituted first (at the moment when the first pattern matches input line(s) and the event correlation operation is started), all occurrences of %% in the second action list have already been replaced with a single percent sign by the time when %<alnum_name> variables are finally substituted. (E.g., in order to express %%t in the second action list, one must actually write it as %%%%t or %%%t, since after %% to % conversion they both yield %%t.)

SEC allows the user to define patterns, context expressions, and actions which involve calls to the Perl eval() function. In addition to explicitly using %<alnum_name> variables that are global across the rules, the user can implicitly employ Perl variables created with eval() for the same purpose. E.g., when SEC has executed the following action

action=eval %a ($b = 1)

the variable $b and its value will become visible in the following context expression

context= =(++$b > 10)

(with that expression one can implement event counting implicitly). In order to avoid possible clashes with the variables inside the SEC code itself, all calls for eval() are made in the main::SEC namespace (i.e., inside the special package main::SEC). This still leaves the user with an opportunity to set the variables within the SEC code (e.g., by using the main:: prefix), and thus to alter the behavior of SEC, but this is only recommended for advanced users who have studied the code thoroughly and know what they are doing.

type=single continue=takenext ptype=regexp pattern=ftpd\[(\d+)\]: \S+ \(ristov2.*FTP session opened desc=ftp session opened for ristov2 pid $1 action=create ftp_$1

type=single continue=takenext ptype=regexp pattern=ftpd\[(\d+)\]: context=ftp_$1 desc=ftp session event for ristov2 pid $1 action=add ftp_$1 $0; set ftp_$1 1800 \ (report ftp_$1 /bin/mail root@localhost)

type=single ptype=regexp pattern=ftpd\[(\d+)\]: \S+ \(ristov2.*FTP session closed desc=ftp session closed for ristov2 pid $1 action=report ftp_$1 /bin/mail root@localhost; \ delete ftp_$1

The first rule creates the context with the name ftp_<pid> when someone connects from host ristov2 with ftp. The second rule adds all logfile lines that are associated with the session <pid> to the event store of the context ftp_<pid> (before adding a line, the rule checks if the context exists). After adding a line, the rule extends context's lifetime for 30 minutes and sets the action list that will be executed when the context expires. The third rule mails collected logfile lines to root@localhost when the session <pid> is closed. Collected lines will also be mailed when the session <pid> has been inactive for 30 minutes (no logfile lines observed for that session).

Note that the logfile line that has matched the first rule will be passed to the second rule and will become the first line in the event store (the first rule has the continue parameter set to TakeNext). The second rule has also its continue parameter set to TakeNext, since otherwise no logfile lines would reach the third rule.

type=Calendar time=59 1 * * * desc=WAITING FOR BACKUP action=event %s

type=PairWithWindow ptype=SubStr pattern=WAITING FOR BACKUP desc=Backup not ready! action=shellcmd sendtrap.sh "%s" ptype2=SubStr pattern2=BACKUP READY desc2=Backup ready action2=none window=960

The first rule generates "WAITING FOR BACKUP" event every night at 1:59AM. This event is matched by the second rule, which starts a correlation operation that will wait for "BACKUP READY" event for the next 16 minutes. If the event arrives on time, no action is executed, otherwise

sendtrap.sh "Backup not ready!"

is called.

type=SingleWithScript ptype=RegExp pattern=node (\S+) interface (\S+) down script=not_resp.sh $2 desc=NODE $1 IF $2 DOWN action=event %s

type=Pair ptype=RegExp pattern=NODE (\S+) IF (\S+) DOWN desc=Interface $2 is down at node $1 action=shellcmd notify.sh "%s" ptype2=SubStr pattern2=node $1 interface $2 up desc2=Interface $2 is up at node $1 action2=shellcmd notify.sh "%s" window=86400

If "node <node> interface <interface> down" event is observed, the interface is checked with the not_resp.sh script. If the interface is found to be down (not_resp.sh returns 0 as its exit code), event "NODE <node> IF <interface> DOWN" is generated which will be matched by the second rule. The second rule starts a correlation operation that calls

notify.sh "Interface <interface> is down at node <node>"

and then waits for the "node <node> interface <interface> up" event for the next 24 hours. When that event is observed, the correlation operation calls

notify.sh "Interface <interface> is up at node <node>"

and terminates.

Description of Error:

Disk drive (CRU: A1) has failed and can no longer be accessed. (SP Event Code 0xA07)

Probable Cause / Recommended Action:

Replace the disk module (CRU: A1).

You would like to receive an e-mail message if something happens to the disk box. You would like to use different e-mail address at night-time and also receive a report of all night events. The problem here is that useful information is scattered over 7 lines and needs to be consolidated into a single event. Consider the following rules for accomplishing this task:

type=Calendar time=0 22 * * * desc=night action=create %s 36000 \ (report %s /bin/mail root@localhost)

type=Single ptype=RegExp7 pattern=Description of Error:\n.*\n(.+)\n.*\n.*\n.*\n(.+) context=night desc=Error=$1 Recmnd=$2 action=shellcmd nightalarm.sh "%s"; add night %s

type=Single ptype=RegExp7 pattern=Description of Error:\n.*\n(.+)\n.*\n.*\n.*\n(.+) desc=Error=$1 Recmnd=$2 action=shellcmd alarm.sh "%s"

The first rule creates the context night with the lifetime of 10 hours every day at 10PM. The second rule specifies that the script nightalarm.sh must be used for sending alert messages at nights, otherwise the script alarm.sh should be used. Every night-time event is added to the context night, and collected events will be mailed to root@localhost at 8AM.

# Set up contexts NIGHT and WEEKEND for nights # and weekends. The context NIGHT has a lifetime # of 8 hours and the context WEEKEND 2 days

type=Calendar time=0 23 * * * desc=NIGHT action=create %s 28800

type=Calendar time=0 0 * * 6 desc=WEEKEND action=create %s 172800

# If a router does not come up within 5 minutes # after it was rebooted, generate event # "<router> REBOOT FAILURE". The next rule matches # this event, checks the router with ping and sends # a notification if there is no response.

type=PairWithWindow ptype=RegExp pattern=(\S+) \d+: %SYS-5-RELOAD desc=$1 REBOOT FAILURE action=event %s ptype2=RegExp pattern2=($1) \d+: %SYS-5-RESTART desc2=$1 successful reboot action2=logonly window=300

type=SingleWithScript ptype=RegExp pattern=(\S+) REBOOT FAILURE script=not_responding.sh $1 desc=$1 did not come up after reboot action=shellcmd notify.sh "%s"

# Send a notification if CPU load of a router is too # high (two CPUHOG messages are received within 5 # minutes); send another notification if the load is # normal again (no CPUHOG messages within last 15 # minutes). Rule is not applied at night or weekend.

type=SingleWith2Thresholds ptype=RegExp pattern=(\S+) \d+: %SYS-3-CPUHOG context=!(NIGHT || WEEKEND) desc=$1 CPU overload action=shellcmd notify.sh "%s" window=300 thresh=2 desc2=$1 CPU load normal action2=shellcmd notify.sh "%s" window2=900 thresh2=0

# If a router interface is in down state for less # than 15 seconds, generate event # "<router> INTERFACE <interface> SHORT OUTAGE"; # otherwise generate event # "<router> INTERFACE <interface> DOWN".

type=PairWithWindow ptype=RegExp pattern=(\S+) \d+: %LINK-3-UPDOWN: Interface (.+), changed state to down desc=$1 INTERFACE $2 DOWN action=event %s ptype2=RegExp pattern2=($1) \d+: %LINK-3-UPDOWN: Interface ($2), changed state to up desc2=$1 INTERFACE $2 SHORT OUTAGE action2=event %s window=15

# If "<router> INTERFACE <interface> DOWN" event is # received from the previous rule, send a notification and # wait for "interface up" event for the next 24 hours

type=Pair ptype=RegExp pattern=(\S+) INTERFACE (\S+) DOWN desc=$1 interface $2 is down action=shellcmd notify.sh "%s" ptype2=RegExp pattern2=($1) \d+: %LINK-3-UPDOWN: Interface ($2), changed state to up desc2=$1 interface $2 is up action2=shellcmd notify.sh "%s" window=86400

# If ten "short outage" events have been observed # in the window of 6 hours, send a notification

type=SingleWithThreshold ptype=RegExp pattern=(\S+) INTERFACE (\S+) SHORT OUTAGE desc=Interface $2 at node $1 is unstable action=shellcmd notify.sh "%s" window=21600 thresh=10

SIGHUP

SEC will reopen its log and input files, reload its configuration, and reset internal lists that contain correlation information (i.e., all active event correlation operations will be cancelled, all contexts will be deleted, and all user-defined variables will lose their values). SEC will also send the SIGTERM signal to its child processes.

SIGABRT

SEC will reopen its log and input files, and load its configuration from rule files which have been modified (file modification time returned by (2) has changed) or created after the previous configuration load. SEC will also cancel event correlation operations started from rule files that have been modified or removed after the previous configuration load. Other operations and other event correlation entities (contexts, variables, child processes, etc.) will remain intact. Note that on some systems SIGIOT is used in place of SIGABRT.

SIGUSR1

some information about the current state of SEC (content of internal lists, rule usage statistics, etc.) will be written to the SEC dumpfile.

SIGUSR2

SEC will reopen its logfile (useful for logfile rotation).

SIGTERM

SEC will terminate gracefully (all SEC child processes will receive SIGTERM).