sec(1)

NAME
sec - simple event correlator

SYNOPSIS

sec -conf=<file pattern> ...
    [-input=<file pattern>[=<context>] ...]
    [-input_timeout=<input timeout>]
    [-timeout_script=<timeout script>]
    [-reopen_timeout=<reopen timeout>]
    [-poll_timeout=<poll timeout>]
    [-check_timeout=<check timeout>]
    [-blocksize=<io block size>]
    [-log=<logfile>]
    [-syslog=<facility>]
    [-debug=<debuglevel>]
    [-pid=<pidfile>]
    [-dump=<dumpfile>]
    [-cleantime=<clean time>]
    [-bufsize=<input buffer size>]
    [-evstoresize=<event store size>]
    [-quoting | -noquoting]
    [-tail | -notail]
    [-fromstart | -nofromstart]
    [-detach | -nodetach]
    [-intevents | -nointevents]
    [-intcontexts | -nointcontexts]
    [-testonly | -notestonly]
    [-separator=<separator>]

DESCRIPTION
SEC is a tool that was designed for solving various event correlation tasks in the domains of network and system management. Event correlation is a process where a stream of primitive events is processed in order to detect composite events that correspond to event patterns in the event stream.

After startup SEC reads lines from files, named pipes, or standard input, matches the lines with patterns (regular expressions, Perl subroutines, etc.) for recognizing input events, and correlates events according to the rules in its configuration file(s). In order to detect a composite event, rules are used sequentially in the same order as they are given in the configuration file. If there are two or more configuration files, rule sequences from different files are used virtually in parallel. SEC can be configured to produce its output by executing user-specified shell commands, and by using utilities like snmptrap(1) or snmpnotify(1), SEC can generate network management events as output. Other options for producing output events are described further in this man page.

Following rule types are currently implemented in SEC for detecting composite events:

Single - match input event and immediately execute an action.

SingleWithScript - match input event and depending on the exit value of an external script, execute an action.

SingleWithSuppress - match input event and execute an action immediately, but ignore following matching events for the next t seconds.

Pair - match input event, execute an action immediately, and ignore following matching events until some other input event arrives. On the arrival of the second event execute another action.

PairWithWindow - match input event and wait for t seconds for other input event to arrive. If that event is not observed within a given time window, execute an action. If the event arrives on time, execute another action.

SingleWithThreshold - count matching input events during t seconds and if a given threshold is exceeded, execute an action and ignore all matching events during the rest of the time window.

SingleWith2Thresholds - count matching input events during t1 seconds and if a given threshold is exceeded, execute an action. Then start the counting of matching events again and if their number per t2 seconds drops below the second threshold, execute another action.

Suppress - suppress matching input events (used to keep the events from being matched by later rules).

Calendar - execute an action at specific times.

If a rule defines the correlation of several events in a certain time window, the rule will start an event correlation operation when the first matching event occurs. In order to distinguish between different event correlation operations that have been started by the rules, every operation is assigned a unique key that is generated from the rule ID, the name of the configuration file the rule belongs to, and the textual description of the composite event that the correlation operation is trying to detect (see RULES AND EVENT CORRELATION OPERATIONS section for more information).

Rules allow not only shell commands to be executed as actions, but among other options it is also possible to create and delete contexts that decide whether a particular rule can be applied at the moment, generate synthetic events that will act as input for other rules, and reset correlation operations that have been started by other rules. This makes it possible to combine several rules and form more complex event correlation schemes. Contexts can not only be used for activating or deactivating a particular rule, but they can also serve as event stores (in logsurfer(1) the contexts are used in a similar way). There are actions for adding event to a context, reporting all events associated with a context, etc. By using the features of SEC in an appropriate way, one can solve a wide range of event correlation and event consolidation tasks.

OPTIONS

-conf=<file pattern>
    expand <file pattern> to filenames (with Perl glob() function) and read event correlation rules from every file. Multiple -conf options can be specified on commandline. Each time SEC receives a signal that forces a configuration reload, <file pattern> is re-evaluated. See also TIMING AND IPC section for a discussion about the rule processing order when multiple configuration files are involved.

-input=<file pattern>[=<context>]
    expand <file pattern> to filenames (with Perl glob() function) and use the files as event sources. An input file can be a regular file, named pipe, or standard input if - was specified. Multiple -input options can be specified on commandline. Each time SEC receives a signal that forces it to reopen its input files, <file pattern> is re-evaluated. If SEC experiences a system error when reading from an input file, it will close the file. If <context> is given, SEC will set up the context <context> each time it reads a line from input files that correspond to <file pattern>. This will help the user to write rules that match data from particular input source(s) only. When there is an -input option with <context> specified, it will automatically enable the -intcontexts option. See TIMING AND IPC section for more information.

-input_timeout=<input_timeout>

-timeout_script=<timeout_script>
    if SEC has not observed new data in an input file during <input_timeout> seconds (or the file was closed <input_timeout> seconds ago), <timeout_script> will be executed with commandline parameters 1 and <the name of the input file>. If fresh data become available again, <timeout_script> will be executed with commandline parameters 0 and <the name of the input file>.

-reopen_timeout=<reopen_timeout>
    if SEC has not observed new data in an input file during <reopen_timeout> seconds (or the file was closed <reopen_timeout> seconds ago), SEC will attempt to reopen this input file. If the attempt fails, SEC will try again after every <reopen_timeout> seconds until open succeeds.

-poll_timeout=<poll_timeout>
    a real number that specifies how many seconds SEC will sleep when no new data were read from input files. Default is 0.1 seconds.

-check_timeout=<check_timeout>
    if SEC has not observed new data in an input file, SEC will not poll the file (both for status and data) during the next <check_timeout> seconds.

-blocksize=<io_block_size>
    size of the block in bytes that SEC will attempt to read from input files at once. Default is 1024 (i.e., read from input files by 1KB blocks).

-log=<logfile>
    use <logfile> for logging SEC activities. Note that if the SEC standard error is connected to a terminal, messages will be logged there, in order to facilitate debugging. If the -detach option has also been specified, no logging to standard error will take place.

-syslog=<facility>
    use syslog for logging SEC activities. All messages will be logged with the facility <facility>, e.g., local0 (see syslog(3) for possible facility values). Warning: be careful with this option if you use SEC for monitoring syslog logfiles, because it might cause message loops when SEC log messages are written to SEC input files.

-debug=<debuglevel>
    set logging verbosity for the SEC logfile. Setting debuglevel to <debuglevel> means that all messages from <debuglevel> and lower levels are logged (e.g., if <debuglevel> is 3, messages from levels 1, 2, and 3 are logged). The following levels are recognized by SEC:
    1 - critical messages (severe faults that cause SEC to terminate, e.g., a failed system call)
    2 - error messages (faults that need attention but don't cause SEC to terminate, e.g., an incorrect rule definition in the configuration file)
    3 - warning messages (possible faults, e.g., a command forked from SEC terminated with a non-zero exit code)
    4 - notification messages (normal system level events and interrupts, e.g., the reception of a signal)
    5 - informative messages (information about external commands forked from SEC)
    6 - debug messages (detailed information about all SEC activities)
    Default <debuglevel> is 6 (i.e., log everything).

-pid=<pidfile>
    use <pidfile> for storing the process ID of SEC.

-dump=<dumpfile>
    use <dumpfile> as the SEC dumpfile. Default is /tmp/sec.dump (see SIGNALS section for more information).

-cleantime=<clean_time>
    time interval in seconds that specifies how often internal event correlation and context lists are processed to perform time-related tasks and to remove obsolete elements. Default is 1 second.

-bufsize=<input_buffer_size>
    set input buffer to hold last <input_buffer_size> lines that have been read from input files. The content of input buffer will be compared with patterns that are part of rule definitions (i.e., no more than <input_buffer_size> lines can be matched by a pattern at a time). Default size is 10 (i.e., 10 lines).

-evstoresize=<event_store_size>
    set an upper limit to the size of context event stores - if an event store already contains <event_store_size> events, SEC will add no more events to that context, regardless of its configuration file directives. Specifying 0 as <event_store_size> or omitting the -evstoresize option means that there is no limit to the size of event stores.

-quoting

-noquoting
    if the -quoting option is specified, event description strings that are supplied to external shell commands of shellcmd and spawn actions will be put inside apostrophes. All apostrophes ' that strings originally contain will be masked. This option prevents the shell from interpreting special symbols that event description strings might contain. Default is -noquoting.

-tail

-notail
    if the -notail option is specified, SEC will process all data that are currently available in input files and exit after reaching all EOFs. Default is -tail - SEC will jump to the end of input files and wait for new lines to arrive. With the -tail option, SEC follows an input file by its name (and not merely by its i-node). If the input file is recreated or truncated, SEC will reopen it and process its content from the start. If the input file is removed (i.e., there is just an i-node left without name), SEC will keep the i-node open and wait for the input file to be recreated. If the -reopen_timeout=<reopen_timeout> option was specified and no data have been added to the i-node during <reopen_timeout> seconds, SEC will attempt to reopen the file without further waiting for the file to be recreated.

-fromstart

-nofromstart
    These flags have no meaning when the -notail option is also specified. When used in combination with -tail (or alone, since -tail is enabled by default), -fromstart will force SEC to read and process input files from the beginning to the end, before the 'tail' mode is entered. Default is -nofromstart.

-detach

-nodetach
    if the -detach option is specified, SEC will disassociate itself from the controlling terminal and become a daemon (note that SEC will close its standard input, standard output, and standard error, and change its working directory to the root directory). Default is -nodetach.

-intevents

-nointevents
    SEC will generate internal events when it starts up, when it receives certain signals, and when it terminates normally. Specific rules can be written to match those internal events, in order to take some action at SEC startup, restart, and shutdown. See TIMING AND IPC section for more information. Default is -nointevents.

-intcontexts

-nointcontexts
    SEC will create an internal context when it reads a line from an input file. This will help the user to write rules that match data from particular input source only. See TIMING AND IPC section for more information. Default is -nointcontexts.

-testonly

-notestonly
    if the -testonly option is specified, SEC will exit immediately after parsing the configuration file(s). If the configuration file(s) contained no faulty rules, SEC will exit with 0, otherwise with 1. Default is -notestonly.

-separator=<regexp>
    obsolete.

Note that one can introduce options both with a single dash (-) and double dash (--), and also use both an equal sign (=) and whitespace as a separator between the option name and the option value, e.g., -conf=<file pattern> and --conf <file pattern> are equivalent.

CONFIGURATION FILE
The SEC configuration file consists of rule definitions. Each rule definition is made up of key=value pairs, one key and value per line. Values are case sensitive only where character case is important (like the values specifying regular expressions). \-symbol may be used at the end of a line to continue the key=value pair on the next line. Lines which begin with #-symbol are treated as comments and ignored (whitespace characters may precede #-symbol). Any comment line, empty line, or end of the file will terminate the preceding rule definition.

Before describing each rule type in detail, patterns and pattern types, context expressions, and action lists are discussed, since they are important parts of the rule definition.

  PATTERNS AND PATTERN TYPES
SEC supports the following pattern types (if <number> is omitted, 1 is assumed):

SubStr[<number>]
    pattern is assumed to be a substring that will be searched in last <number> input lines L1, L2, ..., L<number>. The input lines are joined into a single string with the newline character acting as a separator, and the resulting string "L1\nL2\n...\nL<number>" is searched for the substring. (Note that if <number> is 1, the last input line without a terminating newline is searched.) If the substring is found, the pattern matches. The backslash constructs \t, \n, \r, \s, and \0 can be used in the pattern to denote tabulation, newline, carriage return, space character, and empty string, while \\ denotes backslash itself. As an example, consider the following pattern definition:

    ptype=substr
    pattern=Backup done:\tsuccess

    The pattern matches lines containing "Backup done:<TAB>success".

RegExp[<number>]
    pattern is assumed to be a regular expression that last <number> input lines L1, L2, ..., L<number> are compared with. The input lines are joined into a single string with the newline character acting as a separator, and the resulting string "L1\nL2\n...\nL<number>" is compared with the regular expression pattern. (Note that if <number> is 1, the last input line without a terminating newline is compared with the pattern.) If the pattern matches, backreference values will be assigned to the special variables $1, $2, ..., and the special variable $0 will be set to "L1\nL2\n...\nL<number>" (i.e., to the matching input line(s)). These special variables can be used in some other parts of the rule definition. All regular expression constructs that Perl allows are allowed in the pattern (see perlre(1) for more information). As an example, consider the following pattern definition:

    ptype=regexp
    pattern=(?i)(\S+\.mydomain).*printer: toner\/ink low

    The pattern matches "printer: toner/ink low" messages in a case insensitive manner from printers belonging to .mydomain. Note that the printer hostname is assigned to $1, while the whole message line is assigned to $0. As another example, the following pattern definition

    ptype=regexp2
    pattern=^AAA\nBBB$

    produces a match if the last two input lines are AAA and BBB, and sets $0 to "AAA\nBBB".

PerlFunc[<number>]
    pattern is assumed to be a Perl function that last <number> input lines L1, L2, ..., L<number> are submitted to. The Perl function is compiled at SEC startup by calling Perl eval() function, and eval() must return a code reference for the pattern to be valid (also see VARIABLES AND EVAL section for more information). In order to check whether the pattern matches, SEC will call the function in list context and pass lines L1, L2, ..., L<number> and the names of corresponding input sources S1, S2, ..., S<number> to the function as parameters:

    function(L1, L2, ..., L<number>, S1, S2, ..., S<number>)

    (if the input line has been generated with the event action, its input source name will be set to 'undef'). If the function returns several values or a single value that is TRUE in boolean context, the pattern matches. If the pattern matches, return values will be assigned to the special variables $1, $2, ..., and the special variable $0 will be set to "L1\nL2\n...\nL<number>" (i.e., to the matching input line(s)). These special variables can be used in some other parts of the rule definition. As an example, consider the following pattern definition:

    ptype=perlfunc2
    pattern=sub { return ($_[0] cmp $_[1]); }

    The pattern compares last two input lines in a stringwise manner ($_[1] holds the last line and $_[0] the preceding one), and matches if the lines are different. Note that the result of the comparison is assigned to $1, while the two input lines are concatenated (with the newline character between them) and assigned to $0. As another example, the following pattern definition

    ptype=perlfunc
    pattern=sub { if ($_[0] =~ /(abc|def)/) { \
    return defined($_[1]) ? $_[1] : "SEC"; } return 0; }

    produces a match if the input line contains either the string "abc" or the string "def", and sets $0 to the matching line and $1 to the name of the input source.

NSubStr[<number>]
    like SubStr[<number>], except the result of the match is negated.

NRegExp[<number>]
    like RegExp[<number>], except the result of the match is negated.

NPerlFunc[<number>]
    like PerlFunc[<number>], except the result of the match is negated.

TValue
    pattern is assumed to be a truth value with TRUE and FALSE being legitimate values for the pattern. TRUE always matches an input line while FALSE never matches an input line.

Note that since Pair and PairWithWindow rules have two pattern definitions, special variables $<number> set by the first pattern are shadowed when the second pattern matches and sets the variables. In order to access the shadowed variables, they must be referred to as %<number> (e.g., instead of $1 one has to write %1). Also note that the second pattern of Pair and PairWithWindow rules may contain $<number> variables, if the second pattern is of type SubStr, NSubStr, Regexp, or NRegExp. The variables are substituted at runtime with the values set by the first pattern.

  CONTEXT EXPRESSIONS
Context expression is a logical expression that consists of context names, Perl miniprograms, and Perl functions as operands; operands are combined with operators ! (logical NOT), && (logical AND), || (logical OR), and parentheses. Context expressions are employed for activating or deactivating rules - the truth value of the rule's context expression must be TRUE for the rule to be applicable.

If the operand contains the arrow (->), the text following the arrow is considered to be a Perl function that will be compiled at SEC startup by calling Perl eval() function, and eval() must return a code reference for the operand to be valid (also see VARIABLES AND EVAL section for more information). If any text precedes the arrow, it is considered to be the list of parameters for the function. Parameters are separated by whitespace and may contain $<number> and %<number> special variables. In order to evaluate the Perl function operand, the function with its parameter list will be called in scalar context. If the return value of the function is TRUE in boolean context, the truth value of the operand is TRUE, otherwise its truth value is FALSE.

If the operand begins with the equal sign (=), the following text is considered to be a Perl miniprogram. The miniprogram may contain $<number> and %<number> special variables. In order to evaluate the Perl miniprogram operand, it will be executed by calling Perl eval() function in scalar context (also see VARIABLES AND EVAL section for more information). If the return value of the miniprogram is TRUE in boolean context, the truth value of the operand is TRUE, otherwise its truth value is FALSE.

Note that since Perl functions, miniprograms, and parameter lists may contain strings that clash with the operators of the context expression (e.g., '!'), it is strongly recommended to enclose them in parentheses, e.g.,

($1 $2) -> (sub { return ($_[0] != $_[1]); })

=({my($temp) = 0; return !$temp;})

If the operand is not a Perl function or program, it is considered to be a context name. Context name may contain $<number> and %<number> special variables. If the context name refers to an existing context, the truth value of the operand is TRUE, otherwise its truth value is FALSE.

If the whole context expression is enclosed in square brackets [ ], e.g., [MYCONTEXT1 && !MYCONTEXT2], SEC evaluates the expression _before_ pattern matching operation (normally the pattern is compared with input line(s) first, so that $<number> and %<number> variables in the context expression could be replaced with their values). Evaluating context expression first could save CPU time when many of the input lines are known to match the pattern but not the context expression, and the expression does not contain any variables.

As an example, consider the following context expressions:

-> ( sub { my(@stat) = stat("/var/log/messages"); \
return (!scalar(@stat) || time() - $stat[9] > 3600); } )

($1 $2) -> (sub { return ($_[0] != $_[1]); }) && C1

!(C1 || C2) && =("$1" eq "myhost.mydomain")

The first expression is TRUE when the /var/log/messages file does not exist or was last modified more than 1 hour ago. The second expression is TRUE when variables $1 and $2 are numerically not equal and the context C1 exists. The third expression is TRUE when contexts C1 and C2 do not exist and the $1 variable equals to the string "myhost.mydomain". Note that the Perl code of the third expression is compiled each time the expression is evaluated.

  ACTION LISTS
Action list consists of action definitions that are separated by semicolons. Each action definition begins with a keyword specifying the action type, followed by additional parameters. Additional parameters that do not have a constant nature may contain $<number> and %<number> special variables. Also, the following variables can be used in non-constant parameters:

%s - event description string (set either by the desc or desc2 parameter of the rule definition).
%t - textual timestamp (as returned by date(1)).
%u - numeric timestamp (as returned by time(2)).
%<alnum_name> - user-defined variables that can be set at SEC runtime with certain actions like assign and eval (<alnum_name> must begin with a letter and may contain letters, digits, and underscores). In order to disambiguate <alnum_name> from the following text, <alnum_name> must be enclosed in braces: %{<alnum_name>}.

In order to use semicolons inside a non-constant parameter, the parameter must be enclosed in parentheses (the outermost set of parentheses will be removed by SEC during configuration file parsing).

The following actions are supported:

none
    No action.

logonly
    Event description (the value of the %s variable) is logged.

write <filename> [<event text>]
    Event string <event text> and terminating newline are written to the file <filename> (<filename> may not contain spaces). File may be a regular file, named pipe, or standard output if - was specified. If the file is a regular file, <event text> is appended to the end of the file. If the file does not exist, it is created as a regular file before writing. If <event text> is omitted, %s is assumed for its value. Though write can also be expressed through the shellcmd action, write does not involve fork(2) like shellcmd does.

shellcmd <shellcmd>
    Shell command <shellcmd> is executed. If the -quoting option was specified, %s will be converted to '%s' before supplying it to shell command (see the -quoting and -noquoting options).

spawn <shellcmd>
    Identical to the shellcmd action, except the following - every line from the standard output of <shellcmd> is treated like SEC input line and matched against the rules. This is done by applying event 0 <line> to every line from standard output (see the event action). Note that if <shellcmd> outputs a large data set at once, SEC will process it all at once, so if <shellcmd> enters an endless "busy write" loop, it will block SEC from doing anything other than processing its output.

pipe '<event text>' [<shellcmd>]
    Event string <event text> and terminating newline are fed to the standard input of shell command <shellcmd> (apostrophes are used to mark the beginning and end of <event text>, in order to distinguish it from <shellcmd>). If <event text> is omitted and there is nothing between apostrophes, %s is assumed for <event text>. If <shellcmd> is omitted, <event text> is written to standard output.

create [<name> [<time> [<action list>] ] ]
    Context with the name <name>, with the lifetime of <time> seconds, and with empty event store is created (<name> may not contain spaces, and <time> is an integer constant). If <name> is omitted, %s is assumed for its value. Specifying 0 as <time> or omitting the value means infinite lifetime. If <action list> is specified, it will be executed when context's lifetime is exceeded. If <action list> is made up of more than one action, semicolons must be used to separate the actions, and the list must be enclosed in parentheses. If already existing context is recreated with create, its lifetime will be extended for <time> seconds and its event store will be emptied.

delete [<name>]
    Context with the name <name> is deleted (<name> may not contain spaces). If <name> is omitted, %s is assumed for its value. If non-existing context is deleted, no operation will be performed.

obsolete [<name>]
    Behaves like delete, except the action list of the context <name> (if the context has an action list) will be executed before deletion.

set <name> <time> [<action list>]
    Settings for the context with the name <name> will be changed (<name> may not contain spaces, and <time> is an integer constant). New lifetime of the context will be <time> seconds with optional <action list> to be executed when the lifetime is exceeded. Event store of the context will not be changed by set. Specifying 0 as <time> means infinite lifetime. If <action list> is made up of more than one action, semicolons must be used to separate the actions, and the list must be enclosed in parentheses.

alias <name> [<alias>]
    An alias name <alias> will be created for the context with the name <name> (<name> and <alias> may not contain spaces). After the name <alias> has been created for a context, the context can be referred to by using both <alias> and <name>. If <alias> is omitted, %s is assumed for its value. If context with the name <name> does not exist, or the name <alias> already exists, the alias will not be created. Internally, SEC does not distinguish in any way between <alias> and <name> - they are both pointers to the same context data structure. Therefore, it makes no difference whether context attributes (like lifetime, event store, etc.) are changed by calling an action (e.g., set or add) for <name> or <alias>. If the delete action is called for one of the context names, the context data structure is destroyed, and all context names (which are now pointers to unallocated memory) are removed from the list of context names. Also note that if the lifetime of the context is exceeded, its action list is executed only once, no matter how many names the context has.

unalias [<alias>]
    Context name <alias> is removed from the list of context names, so that the name <alias> can no longer be used to refer to the context it was previously associated with (<alias> may not contain spaces). If <alias> is omitted, %s is assumed for its value. If unalias is called for non-existing context name, no operation will be performed. If the name <alias> was the last reference to a context, the unalias action behaves like delete and the context will be deleted; otherwise the context will continue to exist under another name(s) with its event store and other attributes intact.

add <name> [<event text>]
    Event string <event text> is added to the event store of the context <name> (<name> may not contain spaces). Events in the store are ordered by the time they were added, and every add adds event to the end of the store. If <event text> is omitted, %s is assumed for its value. If context <name> does not exist, the context will be created with infinite lifetime, empty action list and empty event store (as with create <name>) before adding the event. If <event text> contains newlines, it will be split into parts using the newline symbol as a delimiter, and each part is added to the event store as a separate event string.

fill <name> [<event text>]
    Behaves like add, except the event store of the context <name> will be emptied before <event text> is added.

report <name> [<shellcmd>]
    Event store of the context <name> is reported with shell command <shellcmd> (<name> may not contain spaces). Reporting means that events from the store are fed to standard input of <shellcmd> in the order they were added into the store, every event on a separate line. If <shellcmd> is omitted, events from the store are written to standard output.

copy <name> %<alnum_name>
    Event store of the context <name> is assigned to a user-defined variable %<alnum_name> (<name> may not contain spaces). Before assignment takes place, lines from the event store are joined into a scalar using the newline character as the separator (as with Perl join("\n", @array)).

empty <name> [%<alnum_name>]
    Behaves like copy, except the event store of the context <name> will be emptied after it has been assigned to the variable %<alnum_name>. If %<alnum_name> is omitted, empty simply removes all lines from the event store.

event [<time>] [<event text>]
    After <time> seconds a synthetic event <event text> is created (<time> is an integer constant). SEC treats the <event text> string exactly like line(s) read from input - it is inserted into the input buffer in order to compare it with rules. If <event text> is omitted, %s is assumed for its value. Specifying 0 as <time> or omitting the value means now. If <event text> contains newlines, it will be split into parts using the newline symbol as a delimiter, and each part is created as a separate synthetic event.

reset [<rule_number>] [<event text>]
    Cancel event correlation operations that are currently detecting the composite event <event text> (<rule_number> is a string constant), i.e., SEC will terminate event correlation operations that have <event text> in their keys as event description string (if such operations exist, see RULES AND EVENT CORRELATION OPERATIONS section for more information). If there are multiple configuration files specified, reset can cancel only those correlation operations that have been started by the rules from the same configuration file where the reset action itself is defined. If <event text> is omitted, %s is assumed for its value. Since correlation operations started by different rules may detect composite events that have identical description strings, rule number can be optionally specified to point to a correlation operation that was started by a specific rule (1 means first rule in the configuration file, 2 means second, etc.; 0 denotes the current rule). If + or - is prepended to <rule_number>, it is considered to be an offset from current rule (e.g., -1 means previous rule and +1 next rule). For example, if a rule definition with the reset action is given in the configuration file named my.conf, then reset 1 Counting linkdown events will terminate the event correlation operation with the key "my.conf | 0 | Counting linkdown events" (note that internally the SEC rule IDs start from zero), while reset Counting linkdown events will terminate event correlation operations with keys "my.conf | X | Counting linkdown events", where X runs from 0 to N-1 and N is the number of rules in the configuration file my.conf. If no operation with a given key exists, reset will take no action.

assign %<alnum_name> [<text>]
    Text <text> is assigned to a user-defined variable %<alnum_name>. If <text> is omitted, %s is assumed for its value.

eval %<alnum_name> <code>
    The parameter <code> is assumed to be a Perl miniprogram that will be executed by calling Perl eval() function in list context. If the miniprogram returns a single value, it will be assigned to the variable %<alnum_name>. If the miniprogram returns several values, they will be joined into a scalar using the newline character as a separator (as with Perl join("\n", @array)), and the scalar will be assigned to the variable %<alnum_name>. If no value is returned or eval() fails, no assignment will take place. Note that before calling eval(), $<number>, %<number>, and %<alnum_name> variables in the miniprogram are replaced with their values and therefore can't be used as lvalues. Since most Perl programs contain semicolons which are also used by SEC as separators between actions, it is recommended to enclose the <code> parameter in parentheses, in order to avoid the inperpretation of semicolons inside the code by SEC. Also see VARIABLES AND EVAL section for more information.

call %<alnum_name1> %<alnum_name2> [<parameter list>]
    The parameter %<alnum_name2> is assumed to be a reference to a Perl function that was created previously with the eval action. The function will be called in list context by passing optional parameter list to the function (parameters are separated by whitespace). If the function returns a single value, it will be assigned to the variable %<alnum_name1>. If the function returns several values, they will be joined into a scalar using the newline character as a separator (as with Perl join("\n", @array)), and the scalar will be assigned to the variable %<alnum_name1>. If no value is returned or %<alnum_name2> is not a code reference, no assignment will take place. Also see VARIABLES AND EVAL section for more information.

Examples:

spawn /bin/tail -f /var/log/trapd.log

Follow the /var/log/trapd.log file and feed to SEC input all lines that are appended to the file.

pipe '%t: $0' /bin/mail -s "alert message" root@localhost

Mail the timestamp and the value of the $0 variable to the local root.

add ftp_$1 $0; set ftp_$1 1800 (report ftp_$1 /bin/mail root@localhost)

Add the value of the $0 variable to the event store of the context ftp_<the value of $1>. Also extend the context's lifetime for 30 minutes, so that when the context's lifetime is exceeded, its event store will be mailed to the local root.

eval %funcptr ( sub { my(@buf) = split(/\n/, $_[0]); \
my(@ret) = grep(!/^#/, @buf); return @ret; } ); \
copy C1 %in; call %out %funcptr %in; fill C1 %out

Create a subroutine for weeding out comment lines from the input list, and use this subroutine for removing comment lines from the event store of the context C1.

  OTHER ISSUES
As already noted, SEC context expressions and action lists may contain parentheses which are used for grouping and masking purposes. When SEC parses its configuration, it checks whether parentheses in context expressions and action lists are balanced (i.e., whether each parenthesis has a counterpart), since unbalanced parentheses introduce ambiguity. This can cause SEC to reject some otherwise legitimate constructs, e.g.,

action=eval %o (print ")";)

will be considered an invalid action list (however, note that

action=eval %o (print "()";)

would be passed by SEC, since now parentheses are balanced). In order to avoid such unwanted behavior, each parenthesis without a counterpart must be masked with a backslash (the backslash will be removed by SEC during configuration file parsing), e.g.,

action=eval %o (print "\)";)

  SINGLE RULE
The Single rule was designed for matching input events that require immediate action to be taken. Its rule definition has the following parameters:

type
    fixed to Single (value is case insensitive, so single or sIngLe can be used instead).

continue (optional)
    TakeNext or DontCont (values are case insensitive). The first specifies that search for matching rules will continue after the match (i.e., input line(s) that match pattern and context will be passed to the next rule), the second will quit the search. If continue keyword is missing from the rule definition, DontCont is assumed.

ptype
    pattern type (value is case insensitive).

pattern
    pattern.

context (optional)
    context expression.

desc
    textual description of detected event.

action
    action list that will be executed when event is detected.

Note that context, desc, and action parameters may contain $<number> variables.

Examples:

type=single
continue=takenext
ptype=regexp
pattern=ftpd\[(\d+)\]: \S+ \(ristov2.*FTP session opened
desc=ftp session opened for ristov2 pid $1
action=create ftp_$1

type=single
continue=takenext
ptype=regexp
pattern=ftpd\[(\d+)\]:
context=ftp_$1
desc=ftp session event for ristov2 pid $1
action=add ftp_$1 $0; set ftp_$1 1800 \
         (report ftp_$1 /bin/mail root@localhost)

type=single
ptype=regexp
pattern=ftpd\[(\d+)\]: \S+ \(ristov2.*FTP session closed
desc=ftp session closed for ristov2 pid $1
action=report ftp_$1 /bin/mail root@localhost; \
       delete ftp_$1

First rule creates context with the name ftp_<pid> when someone connects from host ristov2 with ftp. The second rule adds all logfile lines that are associated with the session <pid> to the event store of the context ftp_<pid> (before adding a line, the rule checks if the context exists). After adding a line, the rule extends context's lifetime for 30 minutes and sets the action list that will be executed when context times out. The third rule mails collected logfile lines to root@localhost when the session <pid> is closed. Collected lines will also be mailed when the session <pid> has been inactive for 30 minutes (no logfile lines observed for that session).

Note that the logfile line that has matched the first rule will be passed to the second rule and will become the first line in the event store (the first rule has continue parameter set to TakeNext). The second rule has also continue parameter set to TakeNext, since otherwise no logfile lines would reach the third rule.

  SINGLEWITHSCRIPT RULE
The SingleWithScript rule was designed to integrate external scripts with SEC event flow. Its rule definition is similar to the Single rule, except of additional script parameter:

type
    fixed to SingleWithScript (value is case insensitive).

continue (optional)
    TakeNext or DontCont (values are case insensitive).

ptype
    pattern type (value is case insensitive).

pattern
    pattern.

context (optional)
    context expression.

script
    script that is executed after pattern and context have matched some input line(s). The names of all existing contexts are fed to the standard input of the script, a name per line. If the script returns zero for its exit value, action will be executed; if the script returns non-zero, action2 will be executed (if given).

desc
    textual description of detected event.

action
    action list that will be executed when script returns zero for its exit value.

action2 (optional)
    action list that will be executed when script returns non-zero for its exit value.

Note that context, script, desc, action, and action2 parameters may contain $<number> variables.

Also note that since the runtime of the script that is specified by the script parameter is not limited in any way, waiting for that script to complete could freeze the entire event processing for an indefinite amount of time. Therefore, once the pattern and context parameters have matched input line(s), SEC does not wait for the script to complete but rather continues its work. The exit value of the script will be fetched when the script actually terminates, and depending on the exit value, either action or action2 (if it exists) will be executed.

Examples:

type=SingleWithScript
ptype=RegExp
pattern=interface (\S+) down
script=/usr/local/scripts/ping.sh $1
desc=Interface $1 down
action=shellcmd /usr/local/scripts/notify.sh "%s"

When "interface <ipaddress> down" line appears in input, this rule acts by calling

/usr/local/scripts/ping.sh <ipaddress>

and if the script returns 0 as its exit value, the rule calls

/usr/local/scripts/notify.sh "Interface <ipaddress> down"

(note also that SEC feeds the names of all existing contexts to the standard input of the /usr/local/scripts/ping.sh script.)

  SINGLEWITHSUPPRESS RULE
The SingleWithSuppress rule was designed to implement the event correlation operation called compression, where multiple instances of event A are reduced into single event. Its rule definition is similar to the Single rule, except of additional window parameter:

type
    fixed to SingleWithSuppress (value is case insensitive).

continue (optional)
    TakeNext or DontCont (values are case insensitive).

ptype
    pattern type (value is case insensitive).

pattern
    pattern for detecting event A.

context (optional)
    context expression.

desc
    textual description of event A.

action
    action list that will be executed when event A is first observed. Following instances of event A will be ignored.

window
    time in seconds that following events A are ignored for.

Note that context, desc, and action parameters may contain $<number> variables.

Examples:

type=SingleWithSuppress
ptype=RegExp
pattern=(\S+): [fF]ile system full
desc=File system $1 full
action=shellcmd notify.sh "%s"
window=900

When "/usr: file system full" line appears in input, this rule starts a correlation operation that calls

notify.sh "File system /usr full"

and ignores following such lines during next 15 minutes.

Note that line "/tmp: file system full" would not be ignored, since SEC identifies correlation operations by a key that contains event description string of the composite event (given by the desc parameter). Since strings "/usr: file system full" and "/tmp: file system full" differ, SEC will start another correlation operation for the latter (see RULES AND EVENT CORRELATION OPERATIONS section for more information).

  PAIR RULE
The Pair rule was designed to implement one of the basic forms of the temporal relationship event correlation operation, where two or more events A and B are reduced into event pair (A, B) inside a given time window. Its rule definition has the following parameters:

type
    fixed to Pair (value is case insensitive).

continue (optional)
    TakeNext or DontCont (values are case insensitive). Specifies if input line(s) that match pattern and context will be passed to the next rule.

ptype
    pattern type (value is case insensitive).

pattern
    pattern for detecting event A.

context (optional)
    context expression.

desc
    textual description of event A.

action
    action list that will be executed when event A is first observed. Following instances of event A will be ignored.

continue2 (optional)
    TakeNext or DontCont (values are case insensitive). Specifies if input line(s) that match pattern2 and context2 will be passed to the next rule.

ptype2
    pattern type (value is case insensitive).

pattern2
    pattern for detecting event B.

context2 (optional)
    context expression.

desc2
    textual description of event B.

action2
    action list that will be executed when event B is observed. After executing action2 the event correlation operation terminates.

window (optional)
    time t in seconds that is allowed to elapse between first instance of event A and event B. If event B does not appear during t seconds, then the correlation operation started by this rule terminates. Specifying 0 as value or omitting window parameter means setting t to infinity (i.e., if event B does not appear, event A will be ignored forever).

Note that context, desc, action, context2, desc2, and action2 parameters may contain $<number> variables. The pattern2 parameter may contain $<number> variables if ptype2 supports variable substitutions at runtime. If pattern2 shadows the $<number> variables of pattern, then context2, desc2, and action2 parameters may contain %<number> variables.

Examples:

type=Pair
ptype=RegExp
pattern=NFS server (\S+) not responding
desc=$1 is not responding
action=shellcmd notify.sh "%s"
ptype2=substr
pattern2=NFS server $1 ok
desc2=$1 OK
action2=shellcmd notify.sh "%s"
window=3600

When "NFS server fserv is not responding" line appears in input, the correlation operation started by this rule calls

notify.sh "fserv is not responding"

and waits for the line "NFS server fserv ok" for 1 hour, ignoring all "NFS server fserv is not responding" lines. When the line "NFS server fserv ok" appears, the correlation operation executes shell command

notify.sh "fserv OK"

and terminates. If time (1 hour) runs out, the correlation operation will terminate without doing anything.

  PAIRWITHWINDOW RULE
The PairWithWindow rule was designed to implement another variant of the temporal relationship event correlation operation that checks if event A will be followed by event B inside a given time window. Its rule definition has the following parameters:

type
    fixed to PairWithWindow (value is case insensitive).

continue (optional)
    TakeNext or DontCont (values are case insensitive). Specifies if input line(s) that match pattern and context will be passed to the next rule.

ptype
    pattern type (value is case insensitive).

pattern
    pattern for detecting event A.

context (optional)
    context expression.

desc
    textual description of event A.

action
    action list that is executed after event A was observed and event B did not appear within the given time window.

continue2 (optional)
    TakeNext or DontCont (values are case insensitive). Specifies if input line(s) that match pattern2 and context2 will be passed to the next rule.

ptype2
    pattern type (value is case insensitive).

pattern2
    pattern for detecting event B.

context2 (optional)
    context expression.

desc2
    textual description of event B.

action2
    action list that is executed after event A was observed and event B appeared within the given time window. After executing action2 the event correlation operation terminates.

window
    size of the time window in seconds.

Note that context, desc, action, context2, desc2, and action2 parameters may contain $<number> variables. The pattern2 parameter may contain $<number> variables if ptype2 supports variable substitutions at runtime. If pattern2 shadows the $<number> variables of pattern, then context2, desc2, and action2 parameters may contain %<number> variables.

Examples:

type=PairWithWindow
ptype=RegExp
pattern=node (\S+) interface (\S+) down
desc=$1 if $2 is down
action=shellcmd notify.sh "%s"
ptype2=SubStr
pattern2=node $1 interface $2 up
desc2=$1 if $2 short outage
action2=event
window=600

When "node fserv interface 192.168.1.1 down" line appears in input, this rule starts a correlation operation that waits 10 minutes for "node fserv interface 192.168.1.1 up" line, and if that line does not arrive on time, the correlation operation executes shell command

notify.sh "fserv if 192.168.1.1 is down"

and terminates. If the line arrives on time, the operation generates event "fserv if 192.168.1.1 short outage" and terminates.

  SINGLEWITHTHRESHOLD RULE
The SingleWithThreshold rule was designed to implement the counting event correlation operation, where instances of event A are counted inside a given time window and the number of events is compared with a threshold value, in order to detect a composite event B. Its rule definition has the following parameters:

type
    fixed to SingleWithThreshold (value is case insensitive).

continue (optional)
    TakeNext or DontCont (values are case insensitive).

ptype
    pattern type (value is case insensitive).

pattern
    pattern for detecting event A.

context (optional)
    context expression.

desc
    textual description of event B.

action
    action list that is executed when thresh instances of event A have been observed within the given time window. After that all events A will be ignored during the rest of the time window.

window
    size of the time window in seconds. The window is sliding - if event A has been observed less than thresh times at the end of the window, the beginning of the window is moved to the occurrence time of the second instance of event A, and the counting operation will continue. If there is no second instance of event A (i.e., event has been observed only once), the correlation operation will terminate.

thresh
    threshold value.

Note that context, desc, and action parameters may contain $<number> variables.

Examples:

type=SingleWithThreshold
ptype=RegExp
pattern=user (\S+) login failure on (\S+)
desc=Repeated login failures for user $1 on $2
action=shellcmd notify.sh "%s"
window=60
thresh=3

When line "user dbadmin login failure on tty1" is observed, the rule starts a correlation operation that executes shell command

notify.sh "Repeated login failures for user dbadmin on tty1"

if additional two such lines are observed within 1 minute. Following "user dbadmin login failure on tty1" lines will be ignored by this rule until 60 seconds have elapsed since the arrival of first line.

After changing the action parameter in the rule definition to

action=shellcmd notify.sh "%s"; reset 0 %s

the correlation operation will terminate itself after calling notify.sh, and the next matching line will always start a new counting operation.

  SINGLEWITH2THRESHOLDS RULE
The SingleWith2Thresholds rule was designed to implement another variant of the counting event correlation operation, where instances of event A are counted twice - first for checking if the threshold value is exceeded (in order to detect a composite event B), and then for checking if the number of instances will stay below the second threshold value (in order to detect a composite event C). Its rule definition has the following parameters:

type
    fixed to SingleWith2Thresholds (value is case insensitive).

continue (optional)
    TakeNext or DontCont (values are case insensitive).

ptype
    pattern type (value is case insensitive).

pattern
    pattern for detecting event A.

context (optional)
    context expression.

desc
    textual description of event B.

action
    action list that is executed when thresh instances of event A have been observed within time window window. After that event counting is started again with threshold value thresh2 and time window window2.

window
    size of the first time window in seconds. The window is sliding.

thresh
    first threshold value.

desc2
    textual description of event C.

action2
    action list that is executed if no more than thresh2 instances of event A have been observed within last window2 seconds. After executing action2 the event correlation operation terminates.

window2
    size of the second time window in seconds. The window is sliding.

thresh2
    second threshold value.

Note that context, desc, action, desc2, and action2 parameters may contain $<number> variables.

Examples:

type=SingleWith2Thresholds
ptype=RegExp
pattern=(\S+): %SYS-3-CPUHOG
desc=router $1 CPU overload
action=shellcmd notify.sh "%s"
window=300
thresh=2
desc2=router $1 CPU load normal
action2=shellcmd notify.sh "%s"
window2=3600
thresh2=0

When SYS-3-CPUHOG syslog message is received from a router, the rule starts a counting operation that executes shell command

notify.sh "router <routername> CPU overload"

if additional SYS-3-CPUHOG syslog message is received from the router within next 5 minutes. After that the correlation operation waits until no SYS-3-CPUHOG syslog messages have been received from the router during last 1 hour, and then executes

notify.sh "router <routername> CPU load normal"

  SUPPRESS RULE
The Suppress rule was designed to implement event suppression and filtering, where event A is suppressed or filtered out if certain context is present or if the event parameters match the pattern specified. Its rule definition has the following parameters:

type
    fixed to Suppress (value is case insensitive).

ptype
    pattern type (value is case insensitive).

pattern
    pattern for detecting event A (or its subclass) that must be suppressed (or filtered out).

context (optional)
    context expression.

desc (optional)
    textual description of this rule.

Note that the context parameter may contain $<number> variables.

Examples:

type=Suppress
ptype=RegExp
pattern=[Ff]ile system full
context=mycontext

type=Suppress
ptype=RegExp
pattern=/dev/vg01/\S+: [Ff]ile system full

First rule suppresses all "file system full" events if context mycontext is present. Second rule filters out all "file system full" events that concern volume group vg01.

  CALENDAR RULE
The Calendar rule was designed for executing actions at specific times. Unlike all other rules, this rule reacts only to the system clock, ignoring other input. Its rule definition has the following parameters:

type
    fixed to Calendar (value is case insensitive).

time
    crontab-style time specification (see crontab(1) for more information). Time specification consists of five fields separated by whitespace characters. First field denotes minutes (allowed values 0-59), second field hours (allowed values 0-23), third field days (allowed values 0-31, 0 denotes the last day of the month), fourth field months (allowed values 1-12), and fifth field weekdays (allowed values 0-7, 0 and 7 denote Sunday). Asterisks (*), ranges of numbers (e.g., 8-11), and lists (e.g., 2,5,7-9) are allowed as field values.

context (optional)
    context expression.

desc
    event description string.

action
    action list to be executed.

Examples:

type=Calendar
time=0 23 * * *
desc=NightContext
action=create %s 32400

This rule creates the context NightContext every day at 11PM. Context has a lifetime of 9 hours.

RULES AND EVENT CORRELATION OPERATIONS
Although each event correlation operation is started by a SEC rule, there is no one-to-one relationship between rules and event correlation operations, since one rule could start several event correlation operations that run simultaneously. In order to distinguish one event correlation operation from another, SEC assigns a key to every operation that is composed from the rule file name, the rule ID, and the event description string that is derived from the desc parameter of the rule definition (by replacing variables with their values).

Suppose you have a configuration file my.conf with one rule in it:

type=SingleWithThreshold
ptype=RegExp
pattern=user (\S+) login failure on (\S+)
desc=Repeated login failures for user $1 on $2
action=shellcmd notify.sh "%s"
window=60
thresh=3

Suppose that SEC observes an input line "user admin login failure on tty1". This matches the pattern 'user (\S+) login failure on (\S+)', and after replacing $1 and $2 with their values, the desc parameter will yield "Repeated login failures for user admin on tty1". SEC will then form a key for the event correlation operation, using the configuration file name, the rule ID, and the event description string:

my.conf | 0 | Repeated login failures for user admin on tty1

Since the rule was the first one in the configuration file, its ID is 0. The ID for the second rule would be 1, for the third rule 2, etc. Note that because the configuration file name and the rule ID are present in the keys, event correlation operations started by different rules will not clash, even if the rules have identical values for the desc parameter.

After calculating the key, SEC will check if there already is an event correlation operation with that key. If such operation exists, the input line will be correlated by the existing operation. Otherwise, a new event correlation operation will be started.

By using appropriate event description string definitions, you can change the scope of individual event correlation operations. For instance, if you use "Repeated login failures for user $1" for the desc parameter in the above rule definition, the following three lines will now be correlated by the same event correlation operation:

user admin login failure on tty1
user admin login failure on tty5
user admin login failure on tty2

In other words, SEC will now just count login failures for different users, disregarding terminal names.

Finally, it should be noted that Single, SingleWithScript, Suppress, and Calendar rules never start event correlation operations, since they don't involve the matching of several events over a certain time window (i.e., their design allows them to complete their work almost immediately without starting an event correlation operation). Therefore, the desc parameter of these rules does not influence the number and the scope of event correlation operations.

TIMING AND IPC
There are several kinds of events that SEC reacts to - changes in input files (e.g., appearance of new data), the reception of a signal, status change of a child process, and time related events (expiration of the context lifetime, a Calendar rule that has become active, etc.).

When new data appear in SEC input files, only one line will be read at a time (even when more lines are available), after which the input buffer is updated with the line that was read. Input buffer contains the last N lines from the SEC input, in the order they were read (the value of N can be set with the -bufsize option). Updating the input buffer means that the last element (the oldest line) is removed from the end of the buffer, while the new line becomes the first element of the buffer. Note that when synthetic events have been created with the event action and new input data is also available in input files, synthetic events are always read first by SEC. After no more such events are available for reading, SEC will read new data from input files.

After the buffer has been updated (every update always adds only one and removes only one line), the rules from configuration files are processed, comparing the rules against the new content of the input buffer. Even when a rule matches and its action list specifies an immediate change in the buffer (e.g., by using the event action), the input buffer will _not_ be updated until all the rules have been compared against the current content of the buffer.

Rules from the same configuration file are compared against the input buffer in the order they were given in that file. When multiple configuration files have been specified, each file containing a distinct ruleset, events are processed virtually in parallel - an event is always submitted for processing to all rulesets. However, the order the rulesets are applied during event processing is determined by the order the files were given at SEC commandline. If a -conf option specifies a pattern, SEC uses Perl glob() function to expand the pattern, and the resulting file list is applied by SEC in the order returned by glob() (this is usually ascending ASCII order). E.g., if you have specified -conf options in the following order:

-conf=/home/risto/*.conf -conf=/home/risto/*.conf2

and directory /home/risto contains files A.conf, B.conf2, and C.conf, then SEC applies these ruleset files in the following order: A.conf, C.conf, and B.conf2. Also, note that even if A.conf contains a Suppress rule for a particular event, the event is still processed by rulesets in C.conf and B.conf2.

After the input buffer has been updated and its content has been processed by the rules, SEC handles the signals that have been received since the last check, and also reads possible output from its child processes.

When the timeout specified with the -cleantime option has expired, SEC also checks the contexts and the event correlation operations that have timers associated with them (e.g., SingleWithSuppress and PairWithWindow rules). This means that the value of -cleantime should be relatively small, in order to keep the event correlation operations accurate. By setting the -cleantime option value to 0, SEC will check the event correlation timers after reading every line, but this will consume more CPU time. The higher the value, the less CPU time will be consumed, so the value of the -cleantime option depends on your requirements. Note that if the -poll_timeout option value exceeds the value given with -cleantime, the -poll_timeout option value takes precedence (i.e., sleeps after unsuccessful polls will not be shortened).

If the -intevents commandline option is given, SEC will generate internal events when it is started up, when it receives certain signals, and when it terminates normally. Inside SEC, internal event is treated as if it was a line that was read from a SEC input file. Specific rules can be written to match internal events, in order to take some action (e.g., start an external event correlation module with spawn when SEC starts up). The following internal events are supported:

SEC_STARTUP - generated when SEC is started (once the -intevents option has been given, this event will always be the first event that SEC observes)

SEC_RESTART - generated after SEC has received SIGHUP signal and all internal data structures have been cleared (this event will be the first event that SEC observes after reloading its configuration)

SEC_SOFTRESTART - generated after SEC has received SIGABRT signal (this event will be the first event that SEC observes after reloading its configuration)

SEC_SHUTDOWN - generated when SEC receives SIGTERM signal, or when SEC reaches all EOFs of input files after being started with the -notail option. After generating SEC_SHUTDOWN event, SEC will sleep for 3 seconds before sending SIGTERM to its child processes (if there are any child processes that were started by SEC_SHUTDOWN, the processes will have enough time to set a signal handler for SIGTERM if needed).

Right before generating an internal event, SEC will also set up a context named SEC_INTERNAL_EVENT, in order to distinguish between true internal events and coincidents that look like internal events. The SEC_INTERNAL_EVENT context will be deleted immediately after the internal event has been processed by all rules.

If the -intcontexts commandline option is given, or there is an -input option with a context specified, SEC will create an internal context each time a line is read from an input file, or a line is read that was created with event action. The internal context will be deleted immediately after the line has been matched against all rules. For all input files that have the context name explicitly set with -input=<file pattern>=<context>, the name of the internal context is <context>. If the line was read from the input file <filename> for which there is no context name set, the name of the internal context is _FILE_EVENT_<filename>. If the line was created with event action, the name of the internal context is _INTERNAL_EVENT. This will help the end user to write rules that will match data from one particular input source only, e.g., the rule

type=Suppress
ptype=TValue
pattern=TRUE
context=[!_FILE_EVENT_/dev/logpipe]

will pass to the following rules only those lines that were read from /dev/logpipe.

The shellcmd, spawn, pipe, and report actions involve the creation of a child process. The communication between SEC and its child processes takes place through pipes (created with Perl pipe opens like open(FH, "| mycommand") or pipe(2) system call). Note that the running time of children is not limited in any way, so long-running processes can be started from SEC. For instance, you could start a SEC agent with spawn action that runs forever and provides SEC with additional input events. However, note that before termination, SEC sends the SIGTERM signal to all its children. If some special exit procedures need to be carried out in the child process (or the child wishes to ignore SIGTERM), then the child must install a handler for the SIGTERM signal.

Note that if your rule definition includes two shellcmd actions (or other actions that call external scripts or programs), the order that these scripts or programs are executed is not determined. For instance, if you have the following action definition

action=shellcmd cmd1; shellcmd cmd2

then cmd2 could well terminate before cmd1, or cmd2 could well start before cmd1 (e.g., when cmd1 is a complex commandline and cmd2 is relatively simple, it takes more time from the shell to process and start cmd1 than cmd2).

VARIABLES AND EVAL
There are two kinds of variables that can be used in SEC rule definitions - $<number> and %<number> variables that are set during pattern matching, and %<alnum_name> variables (like %s, %t, and %u) that have been designed for use in action lists. All $<number> and %<number> variables will be substituted with their values immediately after input line(s) have matched the rule, even when they are part of the action list definition and this action list is going to be executed at a later time. %<alnum_name> variables, in turn, will be substituted with their values right before the action list is actually executed. If a variable does not have a value (e.g., $4 when the pattern returned just two values), it will be left intact, i.e., it is not replaced with an empty string.

In order to mask the variables, an extra $ or % must be prepended to them, e.g., $$1 yields $1, and not a dollar sign followed by the first value returned by the pattern. Also, all occurrences of $$ will be converted to $ when $<number> variables are substituted, and all occurrences of %% will be converted to % when %<number> or %<alnum_name> variables are substituted, so that one could use $ or % together with variable names (e.g., $$$1 will yield a dollar sign that is followed by the first value returned by the pattern).

Please note the following caveat for Pair and PairWithWindow rules when the second pattern shadows the $<number> variables of the first pattern - since both %<number> and %<alnum_name> variables may be used in the second action list and %<number> variables are substituted first (at the moment when the first pattern matches input line(s) and the event correlation operation is started), all occurrences of %% in the second action list have already been replaced with a single percent sign by the time when %<alnum_name> variables are finally substituted. (E.g., in order to express %%t in the second action list, one must actually write it as %%%%t or %%%t, since after %% to % conversion they both yield %%t.)

SEC allows the user to define patterns, context expressions, and actions which involve calls to Perl eval() function. In addition to explicitly using %<alnum_name> variables that are global across the rules, the user can implicitly employ Perl variables created with eval() for the same purpose. E.g., when SEC has executed the following action

action=eval %a ($b = 1)

the variable $b and its value will become visible in the following context expression

context= =(++$b > 10)

(with that expression one can implement event counting implicitly). In order to avoid possible clashes with the variables inside the SEC code itself, all calls for eval() are made in the main::SEC namespace (i.e., inside the special package main::SEC). This still leaves the user with an opportunity to set the variables within the SEC code (e.g., by using the main:: prefix), and thus to alter the behavior of SEC, but this is only recommended for advanced users who have studied the code thoroughly and know what they are doing.

EXAMPLES

  Example 1
Consider the following rules:

type=single
continue=takenext
ptype=regexp
pattern=ftpd\[(\d+)\]: \S+ \(ristov2.*FTP session opened
desc=ftp session opened for ristov2 pid $1
action=create ftp_$1

type=single
continue=takenext
ptype=regexp
pattern=ftpd\[(\d+)\]:
context=ftp_$1
desc=ftp session event for ristov2 pid $1
action=add ftp_$1 $0; set ftp_$1 1800 \
         (report ftp_$1 /bin/mail root@localhost)

type=single
ptype=regexp
pattern=ftpd\[(\d+)\]: \S+ \(ristov2.*FTP session closed
desc=ftp session closed for ristov2 pid $1
action=report ftp_$1 /bin/mail root@localhost; \
       delete ftp_$1

First rule creates context with the name ftp_<pid> when someone connects from host ristov2 with ftp. The second rule adds all logfile lines that are associated with the session <pid> to the event store of the context ftp_<pid> (before adding a line, the rule checks if the context exists). After adding a line, the rule extends context's lifetime for 30 minutes and sets the action list that will be executed when context times out. The third rule mails collected logfile lines to root@localhost when the session <pid> is closed. Collected lines will also be mailed when the session <pid> has been inactive for 30 minutes (no logfile lines observed for that session).

Note that the logfile line that has matched the first rule will be passed to the second rule and will become the first line in the event store (the first rule has continue parameter set to TakeNext). The second rule has also continue parameter set to TakeNext, since otherwise no logfile lines would reach the third rule.

  Example 2
Suppose there is a backup job in your system that runs at 2AM every night and logs "BACKUP READY" message when it has completed its work. You want to send SNMP trap if there is no message in the log by 2:15AM.

type=Calendar
time=59 1 * * *
desc=WAITING FOR BACKUP
action=event %s

type=PairWithWindow
ptype=SubStr
pattern=WAITING FOR BACKUP
desc=Backup not ready!
action=shellcmd sendtrap.sh "%s"
ptype2=SubStr
pattern2=BACKUP READY
desc2=Backup ready
action2=none
window=960

The first rule generates "WAITING FOR BACKUP" event every night at 1:59AM. This event is matched by the second rule, which starts a correlation operation that will wait for "BACKUP READY" event for the next 16 minutes. If the event arrives on time, no action is executed, otherwise

sendtrap.sh "Backup not ready!"

is called.

  Example 3
Consider the following rules:

type=SingleWithScript
ptype=RegExp
pattern=node (\S+) interface (\S+) down
script=not_resp.sh $2
desc=NODE $1 IF $2 DOWN
action=event %s

type=Pair
ptype=RegExp
pattern=NODE (\S+) IF (\S+) DOWN
desc=Interface $2 is down at node $1
action=shellcmd notify.sh "%s"
ptype2=SubStr
pattern2=node $1 interface $2 up
desc2=Interface $2 is up at node $1
action2=shellcmd notify.sh "%s"
window=86400

If "node <node> interface <interface> down" event is observed, the interface is checked with not_resp.sh script. If the interface is found to be down (not_resp.sh returns 0 as its exit code), event "NODE <node> IF <interface> DOWN" is generated, which will be matched by the second rule. The second rule starts a correlation operation that calls

notify.sh "Interface <interface> is down at node <node>"

and then waits for the "node <node> interface <interface> up" event for the next 24 hours. When that event is observed, the correlation operation calls

notify.sh "Interface <interface> is up at node <node>"

and terminates.

  Example 4
Suppose you have a disk box that logs following error messages:

Description of Error:

Disk drive (CRU: A1) has failed and can no longer be accessed. (SP Event Code 0xA07)

Probable Cause / Recommended Action:

Replace the disk module (CRU: A1).

You would like to receive an e-mail message if something happens to the disk box. You would like to use different e-mail address at night-time and also receive a report of all night events. The problem here is that useful information is scattered over 7 lines and needs to be consolidated into single event. Consider the following rules to accomplish this task:

type=Calendar
time=0 22 * * *
desc=night
action=create %s 36000 \
       (report %s /bin/mail root@localhost)

type=Single
ptype=RegExp7
pattern=Description of Error:\n.*\n(.+)\n.*\n.*\n.*\n(.+)
context=night
desc=Error=$1 Recmnd=$2
action=shellcmd nightalarm.sh "%s"; add night %s

type=Single
ptype=RegExp7
pattern=Description of Error:\n.*\n(.+)\n.*\n.*\n.*\n(.+)
desc=Error=$1 Recmnd=$2
action=shellcmd alarm.sh "%s"

First rule creates context night with the lifetime of 10 hours every day at 10PM. The second rule specifies that script nightalarm.sh must be used for sending alert messages at nights, otherwise script alarm.sh should be used. Every night-time event is added to context night, and collected events will be mailed to root@localhost at 8AM.

  Example 5
This section presents an example rulebase for managing Cisco devices. It is assumed that the managed devices have syslog(3)-style logging enabled, and that all syslog messages are sent to a central host and written to logfile(s) that are monitored by SEC.

# Set up contexts NIGHT and WEEKEND for nights
# and weekends. Context NIGHT has a lifetime
# of 8 hours and context WEEKEND 2 days

type=Calendar
time=0 23 * * *
desc=NIGHT
action=create %s 28800

type=Calendar
time=0 0 * * 6
desc=WEEKEND
action=create %s 172800

# If a router does not come up within 5 minutes
# after it was rebooted, generate event
# "<router> REBOOT FAILURE". The next rule matches
# this event, checks the router with ping and sends
# a notification if there is no response.

type=PairWithWindow
ptype=RegExp
pattern=(\S+) \d+: %SYS-5-RELOAD
desc=$1 REBOOT FAILURE
action=event %s
ptype2=RegExp
pattern2=($1) \d+: %SYS-5-RESTART
desc2=$1 successful reboot
action2=logonly
window=300

type=SingleWithScript
ptype=RegExp
pattern=(\S+) REBOOT FAILURE
script=not_responding.sh $1
desc=$1 did not come up after reboot
action=shellcmd notify.sh "%s"

# Send a notification if CPU load of a router is too
# high (two CPUHOG messages are received within 5
# minutes); send another notification if the load is
# normal again (no CPUHOG messages within last 15
# minutes). Rule is not applied at night or weekend.

type=SingleWith2Thresholds
ptype=RegExp
pattern=(\S+) \d+: %SYS-3-CPUHOG
context=!(NIGHT || WEEKEND)
desc=$1 CPU overload
action=shellcmd notify.sh "%s"
window=300
thresh=2
desc2=$1 CPU load normal
action2=shellcmd notify.sh "%s"
window2=900
thresh2=0

# If a router interface is in down state for less
# than 15 seconds, generate event
# "<router> INTERFACE <interface> SHORT OUTAGE";
# otherwise generate event
# "<router> INTERFACE <interface> DOWN".

type=PairWithWindow
ptype=RegExp
pattern=(\S+) \d+: %LINK-3-UPDOWN: Interface (.+), changed state to down
desc=$1 INTERFACE $2 DOWN
action=event %s
ptype2=RegExp
pattern2=($1) \d+: %LINK-3-UPDOWN: Interface ($2), changed state to up
desc2=$1 INTERFACE $2 SHORT OUTAGE
action2=event %s
window=15

# If "<router> INTERFACE <interface> DOWN" event is
# received from previous rule, send a notification and
# wait for "interface up" event for the next 24 hours

type=Pair
ptype=RegExp
pattern=(\S+) INTERFACE (\S+) DOWN
desc=$1 interface $2 is down
action=shellcmd notify.sh "%s"
ptype2=RegExp
pattern2=($1) \d+: %LINK-3-UPDOWN: Interface ($2), changed state to up
desc2=$1 interface $2 is up
action2=shellcmd notify.sh "%s"
window=86400

# If ten "short outage" events have been observed
# in the window of 6 hours, send a notification

type=SingleWithThreshold
ptype=RegExp
pattern=(\S+) INTERFACE (\S+) SHORT OUTAGE
desc=Interface $2 at node $1 is unstable
action=shellcmd notify.sh "%s"
window=21600
thresh=10

SIGNALS

SIGHUP
    SEC will reload its configuration, reopen its input files and logfile, and reset internal lists that contain correlation information (i.e., all active event correlation operations will be cancelled, all contexts will be deleted, and all user-defined variables will lose their values). SEC will also send the SIGTERM signal to its child processes.

SIGABRT
    SEC will act like it has received SIGHUP, but variables and contexts (and context event stores) will not be deleted, and SEC will also not send SIGTERM to its child processes. Note that on some systems SIGIOT is used in place of SIGABRT.

SIGUSR1
    some information about the current state of SEC (content of internal lists, rule usage statistics, etc.) will be written to the SEC dumpfile.

SIGUSR2
    SEC will reopen its logfile. Useful for logfile rotating, since it does not cancel active event correlation operations like SIGHUP does.

SIGTERM
    SEC will terminate gracefully (all SEC child processes will receive SIGTERM).

AUTHOR
Risto Vaarandi (ristov at users d0t s0urcef0rge d0t net)

SEE ALSO
crontab(1), date(1), fork(2), logsurfer(1), perl(1), perlre(1), pipe(2), snmpnotify(1), snmptrap(1), syslog(3), time(2)