PIKT Logo PIKT

Reference: Miscellany

PIKT Tip Jar
PIKT is user-supported software. 
Please show your support.
 
Home
News
Introduction
Samples
Tutorial
Reference
Software
Authors
Licensing



Forum
Marketplace
Links
SiteSearch
FAQ
Contribute
Donate
ContactUs



Google
Web pikt.org


Miscellany

History Logging

Pikt can remember variable values for reference the next time an alarm script runs.  For the variable $foo (or #foo), you reference its previous value as %foo.  (Don't confuse this with @foo, which references the value tied to the preceding input line.)

Not every value is remembered.  Only variables referenced in their preceding form get remembered.  So, for example, if a script employs the variables $a, $b, $c, #x, #y, and #z, but only %b, %y, and %z are referenced in the script, only those values will be logged.  In the example, if you revise your script now also to reference %c, the next time the script runs it will have values for %b, %y, and %z, but %c won't be available until the subsequent script run.

% references within rules tie a value to the current input line, so long as you specify a lookup key.  That is to say, for example, suppose you have the following (useless, toy) script:

	Useless
	        init
	                input proc "=dfl | =behead(1)"
	                =dfdata
	                keys $fsname
	        begin
	                set #x = 0
	                set #y = #weekday(#now())
	                set #z = #hour(#now())
	        rule
	                if #cap > %cap
	                        set #x += 1
	                endif
	        rule
	                set #x *= 2
	        end
	                if    %y == 1
	                   && #z == 12
	                   && #x > %x
	                        output mail "for $fsname, cap is $text(#cap),
	                                     was $text(%cap)"
	                endif
	
Suppose, too, that the last time the alarm script was run the df output was as follows:
	/dev/dsk/c0t3d0s0    23063   16865    3898    82%    /              [line 1]
	/dev/dsk/c0t3d0s6   210895  198988       0   100%    /usr           [line 2]
	/dev/dsk/c0t3d0s3    30991   13439   14462    49%    /var           [line 3]
	/dev/dsk/c0t3d0s4    61639   45724    9755    83%    /opt           [line 4]
	/dev/dsk/c0t2d0s2  1050367  290966  706886    30%    /export/home   [line 5]
	
And the current df output is:
	/dev/dsk/c0t3d0s0    23063   16865    3898    82%    /              [line 1]
	/dev/dsk/c0t3d0s6   210895  198988       0   100%    /usr           [line 2]
	/dev/dsk/c0t3d0s3    30991   14344   13557    52%    /var           [line 3]
	/dev/dsk/c0t3d0s4    61639   45724    9755    83%    /opt           [line 4]
	/dev/dsk/c0t2d0s2  1050367  342223  655629    35%    /export/home   [line 5]
	
For the first input line of the current script run, #cap is 82%, and %cap is also 82%, both keyed on the $fsname /dev/dsk/c0t3d0s0.  For the second input line, #cap is 100%, as is %cap.  For the third input line, #cap is 52%, %cap is 49%.  And so on.

#x increments during input process whenever #cap exceeds %cap, which happens twice, in lines 3 and 5.  It also doubles in value for every input line.  Suppose that this script was run on a Tuesday, so #y is 3.  #z is set at the script beginning to the current hour.

Here is a complete accounting of what will be logged in the history file (found in =piktdir/var/histories) for possible referencing during the next script run:

	        /dev/dsk/c0t3d0s0        [$fsname for line 1]
	        82%                      [#cap    for line 1]
	        0                        [#x      for line 1]
	        /dev/dsk/c0t3d0s6        [$fsname for line 2]
	        100%                     [#cap    for line 2]
	        0                        [#x      for line 2]
	        /dev/dsk/c0t3d0s3        [$fsname for line 3]
	        52%                      [#cap    for line 3]
	        2                        [#x      for line 3]
	        /dev/dsk/c0t3d0s4        [$fsname for line 4]
	        83%                      [#cap    for line 4]
	        4                        [#x      for line 4]
	        /dev/dsk/c0t3d0s2        [$fsname for line 5]
	        35%                      [#cap    for line 5]
	        10                       [#x      for line 5]
	        10                       [#x      at the end]
	        3                        [#y      at the end]
	
The $fsname values are logged, because even though %fsname is not referenced $fsname is the required lookup key.  #cap values are logged, because %cap is referenced in the script, but only within rules.  In the end section, #cap is not logged, because there is no current input line to associate it with (we are finished with input).

#x is logged, because of the %x reference, and logged six times, because it has a particular value tied to each of the five input lines, and because its value persists after the end of input processing (after the last rule).  #y is referenced as %y, and is invariant through any input processing, so is logged just once, at the end.  The value of #z is not logged, because %z is not referenced in the script.  (Nor are any of the other variables in =dfdata.)

The rules for history logging are:

  • If it is a key variable, whether or not referenced in its history form, its values are logged.
  • If it is referenced in its history (%) form, its value(s) is (are) logged.
  • If it is a dat variable (specified in the init section), its value is logged, once for every input line.  For any given input line, the dat values for that line are invariant (cannot be changed by set statements).
  • If it is a non-dat variable set within any rule, hence within the input processing loop, its terminal value is logged, once for every input line.  "Terminal value" here means the value at the end of the final rule section (after no matter how many set statements appearing within the rule sections).
  • If it is a non-dat variable, its terminal value (here meaning the value at script conclusion) is logged.
Note that only history-referenced dat values require the specification of one or more key variables.  If a script makes history reference only to non-dat variables, no key variables need be specified.

From one script run to the next, input lines, and their associated key values, may drop out or get added.  If they drop out, their values are not logged.  (In the example above, if /dev/dsk/c0t2d0s2 were to disappear, its dat values $fsname and #cap and and its #x value would not be logged.)  Moreover, their history values may not be referenced.  (If /dev/dsk/c0t2d0s2 were to disappear from input, the associated %cap and %x for that device could no longer be referenced.)  If an input line is added (say /export/home were moved to /dev/dsk/c0t1d0s2), its values are logged, but the newly logged values cannot be referenced until the subsequent script run.

Also from one script run to the next, you might add or drop variables, change a variable's data type from, say, string ($) to number (#), change a variable from a scalar to an array (e.g., $foo becomes $foo[]), change its position within the script in a way that impacts on history logging, etc.  If Pikt detects critical changes in the script, the history mechanism is suspended for the current script run:  you may not reference % values, but current values are logged for recall during the next script run.  Pikt takes a conservative, cautious approach toward such changes.  Sometimes, if you make too many changes, the history recall mechanism can get confused, and your script will abort.  You will know about these failures by periodically reviewing (via another Pikt script) the alarm logs.  If this happens, the best recourse is to delete the blocking history file (in =piktdir/var/histories), after which there shouldn't be any more problems.

All of this is hard to put into words, but the history logging mechanism pretty much does what you would hope and expect it to.

Error & Info Logging

In general, PIKT logs daemon (piktc_svc and piktd) starts and stops, and other daemon failures to the appropriate log file (piktc_svc.log and piktd.log, both in =piktdir/var/log).

With piktc, most errors are reported to the command line, but sometimes additional, technical information is logged to its log file (piktc.log).

Every time piktc sends a service request to a client system, this is logged in the =piktdir/var/log/piktc.log file.  Likewise, piktc_svc logs, to =piktdir/var/log/piktc_svc.log, every service request it receives.  If piktc_svc receives a request from an unauthorized account or system, it flags this as an "ERROR".  You can use other log file monitoring Pikt scripts to alert you to this possible security violation.  You can also have scripts to analyze and compare the piktc.log and various piktc_svc.log files to verify the validity of service requests.

Every time piktd launches a Pikt script, it logs the invoking command in =piktdir/var/log/piktd.log.

As for pikt, each alert is assigned its own log file.  For example, the Urgent alert's log file is Urgent.log (also found in =piktdir/var/log).  In alert log files, the following things are logged:

  • alert run startups and normal/abnormal terminations
  • errors (marked by "ERROR"), whenever a serious error is detected (e.g., attempted divide by zero, or the appearance of a negative array index); depending on the seriousness of the error, the pikt run may or may not abort prematurely
  • warnings (marked by "WARNING"), whenever a possible error condition is detected (e.g., whenever interpreting a number as a string, or when a file is not found); warnings never lead to premature aborts
  • external command invocations (e.g., for every "exec" statement, or whenever #system() or $command() are invoked); the idea here is to log each time PIKT reaches out to impact on the external environment, useful for later debugging
  • anything that an exec'ed statement, $command() call or #system() call outputs to stderr; this is useful for determining why a script mysteriously fails; such log entries are not date/time-stamped--you can use this fact within log monitoring scripts to identify such entries and thus report stderr dumps
Every log entry is stamped with the current date and time, the system name, the PIKT program generating the log entry, its process id, a numerical error id, and an error level.  For some log entries, the source file, the line number in that source file, and the function where the logging occurred might also be shown.  Here is a typical log entry:
	Oct 14 09:37:25 vienna pikt[24009]: [ID 7501 WARNING] in
	  SysBakSubnetDownEmergency, resolving number 30.000000 as string 30
	
Alert logs can grow quickly in size, if a script is run often enough, if it analyzes lots of data, and if it produces many warnings (through, e.g., repeated number-to-string conversions).  You should seriously consider putting in place PIKT alarms to monitor your alert (and other) logs, truncating and/or compressing-and-archiving them when they reach a certain size.  (See the examples.)  You should also set 'verbose_log' to 'no' in PIKT.conf, except when debugging.

(When run in debug (-G) mode, PIKT binaries error and info log to stderr.)

If auto_syslog is set to 'yes' in PIKT.conf, in addition to the logging described above, all program outputs (for example, from Pikt 'output', 'output mail', etc.) are sent to syslog at facility local0.  You can, using the syslog_facility parameter, set this to any of local0 through local7 instead.

This automatic routing of all program outputs to syslog is especially useful if you have configured syslog (via the appropriate syslog.conf settings) to send along all messages to a central loghost.  You might do this to archive all PIKT messages, or to process them all at one point for purposes of graphical display.  You might also find it useful to send all EMERG-level messages to all root logins, for example.

Interfacing with Other Languages

It is possible for scripts written in other languages to invoke pikt scripts.  For example, in a Perl script you might have:

	        system("/pikt/bin/pikt +A Critical");
	
or
	        system("/pikt/lib/programs/chkdevs.pkt")
	
Pikt scripts interface with other languages in any of the following six ways:
  • in an input proc statement
  • in a filter statement
  • in an exec statement
  • within a $command() call
  • within a #system() call
  • within a #popen() call
See the sample alarms.cfg for examples.

When invoking, say, an Awk one-liner within a Pikt script, you must be mindful of how Pikt handles quotes and variable identifiers.  Consider this sample script:

	KillIdleUserSessionCritical
	        init
	                status active
	                level critical
	                task "Terminate idle user sessions"
	                input proc "=w | =nawk '/[1-9]day/ {gsub("\\/","\\\\/");
	                                        print $1 " " $2}'"
	                dat $user 1
	                dat $tty 2
	        rule
	                exec wait "=kill -9 `=ps -ef | =nawk '/$user.+$tty/
	                           {print \$2}'`"
	
In the sample above, note the single and double quotes embedded within the input proc string.  This is perfectly acceptable, but in weird circumstances you can tie yourself up in the quoting tangle common to all script languages.  Sometime you have to resort to using the $dquote(), $squote(), and/or $char() functions, possibly in conjunction with the "." concatenation operator, to undo the tangle.

In the sample, you want the end-product nawk function to be

	        gsub("\/","\\/")
	
but to achieve this you have to backslash escape each "\" going in.  (You need to transform, for example, "pts/3" into "pts\/3" because you will use the second string in the exec statement's nawk command following.)

Note that in the first nawk command, the "$1" and "$2" are not backslash-escaped, while in the second nawk command the "\$2" is.  The reason for this is that, at the point of any input proc statement, no variables have been defined yet, so no suppression of variable resolution is necessary.  With a rule (or begin or end), however, you must escape, e.g., $2 for otherwise Pikt will attempt to resolve it as a pattern-match variable.  If this inconsistency troubles you, you could easily (and needlessly) escape (using a single "\") Awk variables in the input proc statement; the effect would be the same.

Pikt was designed to interface readily with other languages.  If you prefer to use the Pikt script language minimally, and just wrap simple Pikt scripts around more elaborate scripts written in Perl, Python or whatever, that's entirely your choice.

Known Problems & Limitations

In Pikt scripts, generally every item serves a semantic purpose.  So, for example, there are no semicolons to indicate the end-of-line, nor are parentheses surrounding logical test required (they are optional).  This convenience comes at a price.  In certain rare circumstances, your script may not parse.  Here is one such circumstance:

	        for #i=-1 -#i<=10 #i-=1
	
This will break the parser, because the "-1 -#i" runs together.  The fix here is simple:  Just reverse the exit condition:
	        for #i=-1 10>=-#i #i-=1
	
For standalone Pikt scripts, the parser might interpret #foo-style numerical variables or #foo()-style numerical functions as comments, or vice-versa.

These parse errors are quite infrequent, and are usually easily fixable.  (For example, always use whitespace after the # comment character.)

Except in some arithmetical or complex logical expressions, parentheses are unnecessary.  You're sometimes better off avoiding them.

(We continue to revise and perfect the parsers.  Hopefully, someday, absolutely all ambiguous code will parse.)

In scripts, you are limited to 255 different variable names.  (That is, within an alert, each alarm script has its own separate 255-member variable name set.)  This might seem at first to be a severe limitation, but it's not really.  Very few situations would require anywhere near that many, let alone more than 255.  In those situations, just use Perl, Python or whatever other language you prefer, or resort to arrays.

For numerically indexed arrays, indices are limited to 2147483647.  This should be way more than enough for the problem domain (systems administration) for which Pikt is intended. 

For numerically indexed arrays, you are restricted to from one to three dimensions.  Here, too, this should more than suffice for most situations.  (Code is in place, but not activated, to take Pikt to a higher number of array dimensions, if ever this is needed.)

So, it is actually possible to store as many as 255 X 2147483647 X 2147483647 X 2147483647 values within a Pikt script.  (More than that, if you include $0, $1, $2, ..., and the built-in variables like $inlin.  Of course, you would probably run out of system memory before reaching these limits.)  If ever any of these limitations becomes an issue, just use Perl, Python, or whatever other language you like.

Input strings are limited to 512 characters.  This should be more than enough for most situations.  If not, again use some other language.  Internally, strings are effectively unlimited in length.

User-defined functions are not currently supported.  (Their inclusion is planned for a future release.  They will probably appear in a new, ninth config file, functions.cfg.)  For now, if you really need a custom function (and don't want to recode pikt itself), you can fake it by $command()'ing or #system()'ing Pikt scripts or scripts written in other languages, passing arguments in the script command string.  In many situations, PIKT macros-with-arguments substitute for formal user-defined functions quite nicely (see the sample macros.cfg).

In future versions, we might relax some of these limitations.  The Pikt script language is not an all-purpose programming language.  Pikt was designed to aid in systems administration, and to have scripts run quickly and in a small memory space.  As of now, we are not convinced that extending Pikt's reach is worth the added processing and memory overhead that would entail.  In the course of its development, much time and effort was spent on adding features to the Pikt script language that are, after all is said and done, little used.  We are guided by practicality and usefulness in further development, not by attainment of some lofty conceptual ideal.

Security Considerations

Finally, security.  PIKT client machines entertain piktc requests from the root account (typically) on the master control machine only.  This implies that the master control machine should be secure, ideally inaccessible to ordinary users (or accessible only to trusted users).  You should try to keep the identify of your master machine secret if at all possible (also the RPCPRGNUM you compiled into the PIKT binaries).

PIKT master-slave, client-server communications rely on a four-tier security mechanism involving (1) host authentication, (2) secret key authentication, (3) service access rights, and (4) server-client callback.

In PIKT.conf, which functions roughly like a .rhosts or .shosts file, you set various access parameters (e.g., uid, gid, master, domain, master_address, etc.) and service rights (e.g., all_services, kill_service, install_service, etc.).  By careful and judicious use of these settings alone, you can achieve a level of security sufficient in many situations.

As an option, you can use secret key authentication to prove the master's identity to the slave.  For now, PIKT uses symmetric private key encryption/decryption of the authentication/session key.  Public key encryption will be added sometime in the future.

We have selected the Blowfish cipher for our symmetric key encryption/decryption.  Blowfish is reputed to be fast, and our PIKT implementation bears that out.  Blowfish was also straightforward and relatively easy to implement.  We do plan to offer alternative ciphers where demanded and as time permits.

Needless to say, you should take every precaution to keep your private_key secret.  For example, if you store private_key in PIKT.conf, you should apply 600 or 400 permissions, and apply equally tight access rights on the PIKT etcdir (where PIKT.conf resides) and all parent directories.  When you need to update your PIKT.conf files with a revised private_key (or if you store the key in a stand-alone etcdir/private_key file), use scp or some other secure method for distributing those files.  Once you have the private_key established and PIKT encryption activated everywhere, you may of course manage secure installation of your PIKT.conf and/or etcdir/private_key like you would any other system file by way of, e.g., 'piktc -i +F PIKT.conf' or 'piktc -i +F private_key'.  Note that the new private_key won't take effect until you restart the piktc_svc daemons, either manually or via 'piktc -R'.

If one managed to crack any system in the PIKT domain, one would have access to all common secrets.  To solve this problem, you may use per-slave uid, gid, and private_key settings.  See the keys.conf section of this Reference for more detail.

For a more detailed description of PIKT's secret key authentication scheme, see the piktc_svc section of this Reference.

For the OSes that support it, PIKT optionally employs server-client callback for even greater security.  Please see the piktc_svc section of this Reference for more details.

For file installs, file fetches (to diff against the central configuration), and command executions, you can optionally encrypt all such data traffic between master and slave.  For this purpose, too, we use the Blowfish cipher.  Note that each and every service request uses its own separate "session key".  If you install five different files, for example, five separate data encryption keys are used (same as the key used for secret key authentication).

Even with encryption/decryption enabled and all the extra security overhead, PIKT master-slave, client-server communications are still surprisingly fast.

piktc and piktc_svc do complete service request logging, and piktc_svc marks denied requests as "ERROR".  It is not difficult to set up log monitoring scripts (and scripts to monitor the monitoring scripts, etc.) to spot attempted security break-ins.  Remember that you can log to syslog and special logs, and dump to a printer, besides sending out e-mail alerts.  Scripts can also call pagers in security emergencies.  You can even have a monitoring script kill the local piktc_svc under suspicious circumstances.

If you have turned on auto_syslog in PIKT.conf, you can route all log messages to a central loghost in real time.  It would not be too difficult to match legitimate piktc service log entries with client-side piktc_svc log entries, thereby identifying anomalous service requests.  Moreover, you could run this also in real time, and send out pages, dump to a printer, etc., if illegitimate activity is detected.

In firewalled environments, you should bind piktc_svc to a specific system port (by default 850, but you should change this to some other unused port number).  See the distribution INSTALL file on how to do this.  Assuming you have blocked all unused system ports at the firewall, you should unblock the designated PIKT port using the appropriate firewall rules.  In any event, note that you still need to run the portmapper, also unblock it at the firewall.

In the future, we might have piktc_svc run optionally via inetd, enabling all the tcpd protections and loggings, if you are using that excellent utility.

Through some combination of the above-described methods, we believe that you can make your PIKT setup as secure as necessary.

You can use PIKT for security in at least the following six ways:

  • As just a centrally managed scheduler.  Have piktd invoke your preferred security tools according to the schedules in piktd.conf.
  • The above, plus have PIKT manage other security tools' config files (the inevitable per-machine and per-OS customizations using #if's).
  • All of the above, plus use PIKT #ifdef's to activate/deactivate different security tools as conditions warrant.
  • All of the above, plus use PIKT to handle your security log file analysis and incident reporting.
  • All of the above, plus employ PIKT alarms and data objects as supplements to the standard tools.  (For example, have PIKT do things that other tools don't, or don't do conveniently or well; add paging capabilities to tools that lack them; etc.)
  • Use PIKT exclusively (or almost exclusively; PIKT will never replace crack!).
We submit that there are no good or widely known solutions out there for points one, two, and three above.

As for point four, there are good solutions out there already to manage your security log files, but PIKT should better them due to its more powerful and flexible built-in scripting language (Pikt).

As for point five, extending one tool requires you to learn Perl, another to get down-and-dirty with the Bourne shell, while five others require you to learn five different cryptic and proprietary command languages.  Learn those languages and modify those tools on their own terms where that makes the most sense, but when sensible add PIKT where the others leave off.

Point six, in spite of its great virtue of involving just one system and command language to learn and use, we wouldn't dare claim that PIKT can outdo every other security tool out there.

The optimum solution probably lies somewhere around point five.

Doubtless, no one can disagree that:

  • Security should be a daily, ongoing concern.
  • It should be systematic.
  • It should be flexible.
  • It should be easy to manage.
These are areas where PIKT excels.  At the very least, PIKT is a general framework on which to build your security efforts.

Please see the sample configurations, where we use the power of PIKT to implement these security principles.



Home | News | Introduction | Samples | Tutorial | Reference | Software | Authors | Licensing
Forum | Marketplace | Links | SiteSearch | FAQ | Contribute | Donate | ContactUs
Top of Page

Join pikt-users, pikt-workers, and/or the PIKT Forum. 
Open Hand Please visit our sponsors.

Page best viewed at 1024x768.   Page last updated 2005-01-07.
This site is PIKT® powered.
PIKT® is a registered trademark of the University of Chicago.
Copyright © 1998-2005 Robert Osterlund.  All rights reserved.