Log Accumulator

Download from http://www.nyangau.org/la/la.zip.

tail -f is the friend of anyone doing complicated problem investigation.

Often there are multiple logs to be watched, and some kind of sequential timed based interleaved composite log is desired. So that you can see that program X complained about something, before program Y did.

The Log Accumulator is designed to make the monitoring and correlation of multiple logs easier.

Adding logs

You tell it to monitor a variety of logs, and it uses poll to get data from them all. Completed lines of text get output to the display.

A simple example :-

% la
add /tmp/prog-X-log
LA: /tmp/prog-X-log added, opened
/tmp/prog-X-log: warning, low memory condition
add /tmp/prog-Y-log
LA: /tmp/prog-Y-log added, opened
/tmp/prog-X-log: very low memory condition, closing connections
/tmp/prog-Y-log: connection to peer lost
^D
%

As the Log Accumulator is running, you can add more logs to be watched simply by typing more add commands. This can be seen in the example above.

The Log Accumulators responses are always preceeded by LA: so that you can easily tell them apart from logged data. Logged data is always preceeded by the log name. In the examples, % or # preceeded items are things typed at the shell prompt, and other text are commands typed into the Log Accumulator itself.

The log name is the log filename by default. But log filenames can be long and cumbersome, so a shorter name can be used :-

% la
add /tmp/prog-X-log X
LA: X added, opened
add /tmp/prog-Y-log Y fg1
LA: Y added, opened
X: restarted
Y: connection to peer re-established
^D
%

Setting log options

Options may be set when the logged is added, or later changed using the opt command.

Colour

In the example above the Y log has the fg1 option, which means that its logs get displayed in ANSI colour 1 (red). Here is how we change it to ANSI colour 2 (green) :-

opt Y fg2
LA: Y options set

Only use the colour options if you're using a suitable terminal, such as vt100 or similar, ANSI, Xterm etc..

Auto-open

Sometimes when you try to add a log, it won't be possible to open it straight away (perhaps because it doesn't yet exist). You'll see something like :-

add /tmp/log-A A
LA: A added, can't open

Later on, you can ask the Log Accumulator to try to open any logs you've added that couldn't be opened by :-

open A
LA: A opened

There is an auto open option, called ao that you can apply to a log when you add it, or later using opt :-

add /tmp/to-be-created 2b ao
LA: 2b added, can't open

Later, when the log is created, you might the following :-

LA: 2b auto-opened
2b: To be or not to be, that is the question

Auto-open works by the Log Accumulator continually attemping to open any logs with the ao option. This could burn quite a few CPU cycles. Use sparingly.

Timestamps

Not every program that generates logs puts a timestamp on every line. You can make the Log Accumulator add one like so :-

add /tmp/mylog mylog time
mylog: [Sat Jul 26 17:05:31 2003] my event happened

Alternatively the opt command can be used to add this :-

add /tmp/mylog mylog
mylog: my event happened
opt mylog time
mylog: [Sat Jul 26 17:15:31 2003] my next event happened

Timestamps are in local time.

Cooked

Normally logs operate in 'cooked' mode. ie: the following character sequences are folded as indicated :-

a b \b c   ->   a c

and the \r character is ignored.

Obviously this feature is designed to accomodate program output whereby output gets continually updated on the same place on the screen. eg: A program might display 0% done, do some work, backspace and display 10% done etc..

This is quite good for program output, but programs such as shells using getline tend to output various terminal dependant escape sequences during the command line editing process, and this feature isn't sophisticated anough to cope with that.

The cooked and nocooked options can be used to change this behaviour.

Deleting logs

When you decide you no longer want to watch a log, you can use :-

del Y
LA: Y closed, deleted

Deleting doesn't actually delete the log file. It just deletes it from the Log Accumulators list of things to watch.

Show, open, close

The following example shows a listing of the logs, an attempt to open any which are closed, and finally to close them all.

show *
LA: /tmp/x (x) is open
LA: /tmp/y (y) is not open
open *
LA: y can't be opened
close *
LA: x closed

Notice that we use * as a wildcard to mean all logs. We can do this most places, just not on the add command.

File tailing

The Log Accumulator works a little like tail -f.

When the Log Accumulator opens a log file, it immediately seeks to its end. So it only reads and displays events which are appended to the log from that point in time onwards.

Also, if a log is truncated/recreated, the Log Accumulator will not explicitly notice this. Consider the following sequence :-

  1. Log file has 1000 bytes.
  2. Log Accumulator opens the file.
  3. Another 200 bytes written to the log.
  4. Log Accumulator reads and displays these bytes, and thus is now 'positioned' at 1200 bytes into the log file.
  5. Log file is truncated to 0 bytes in size.
  6. 1100 bytes are written to the log - Log Accumulator does not notice.
  7. 400 bytes are written to the log - Log Accumulator does not notice the first 100 bytes, but does notice the next 300 and reads and displays them.

Making the Log Accumulator close and reopen a log causes it to forget its 'position' into the log.

Beware programs which do automatic log rotation.

Peeking partial lines

Some programs display a row of dots (or other partial lines) as they do their job. Log Accumulator only displays lines when they are completed. Sometimes it is useful to be able to take a peek of uncompleted lines.

install.log: Decompressing data, could take a while
system.log: Disk space low (10% free)
peek *
LA: peek install.log: Decompressed ............
system.log: Disk space low (5% free)
install.log: Decompressed ........................... Done.
install.log: Installation complete.

Command line

Very often you'll set up the Log Accumulator the same way. Just put the commands in a file, and feed it in when you start. More than one file can be passed.

% cat setup.la
add /tmp/server-X-log X fg4
add /tmp/server-Y-log Y fg6 ao
tee /tmp/combined.log
% la setup.la
LA: X added, opened
LA: Y added, can't open
LA: tee-ing to /tmp/combined.log
X: low memory threshold reached

Misc

The help command shows the full command set :-

help
LA: Commands :-
LA:   add <filename> [<name> {<option>}]
LA:   opt <name> {<option>}
LA:   del <name>
LA:   show <name>
LA:   peek <name>
LA:   open <name>
LA:   close <name>
LA:   echo <text>
LA:   cat <filename>
LA:   tee [<filename>]
LA:   exit
LA:   help
LA: Where :-
LA:   <filename> is a file, pipe or device
LA:   <name> is a shorter name to refer to the log by (or *)
LA:   <option> is [no]ao, [no]time, [no]cooked, fg<col>, nofg, bg<col>, nobg
LA:   <col> is 0-7 for black,red,green,yellow,blue,magenta,cyan,white
LA:   <text> is any text

From here we can see that we can also cause our own text to be entered into the output stream (using echo).

We can also put the contents of a file into the output stream using cat.

The output stream can be sent to a file as well as the screen. Note that if you do this, the ANSI colour escape sequences are not sent to the file, they only go to the stdout output stream (ie: normally the terminal).

Putting it all together in an example :-

tee /tmp/test-log
LA: tee-ing to test-log
cat /tmp/pidfile
/tmp/pidfile: 12345
echo About to kill server
LA: echo About to kill server
server-log: SIGTERM received, aborting
cat /tmp/pidfile
LA: can't open /tmp/pidfile
tee
LA: tee-ing ended

Program logs

Some programs naturally generate logs, or can be made to.

UNIX shell

Logs of shell interactions can be made with the UNIX script command, which creates a nested shell that is logged - eg:

# script -a -f /tmp/root-shell
Script started, file is /tmp/root-shell
# install.sh
Product successfully installed
# exit
Script done, file is /tmp/root-shell
#

Afterwards :-

% cat /tmp/root-shell
Script started on Sat Jul 26 13:34:00 2003
# install.sh
Product successfully installed
# exit

Script done on Sat Jul 26 14:14:26 2003
%  

The -f argument to script is present on the Linux version, but not necessarily on others. It causes 'flushing after each write', ie: data is written to the log file as and when it is displayed. Without such an option, shell output and resulting log output do not interleave correctly in the Log Accumulator output, as the shell output gets buffered until there is a lot of it.

eTrust Access Control

CA eTrust Access Control was formerly Memco SeOS, and this is reflected in the names of the files comprising the product.

This is a security product, which records an audit log of attempted violations. Unfortunately, by default, this isn't written to an ever growing text file.

It includes a program called seaudit which can be made to display the violations in a given time interval, but once done it terminates. So it and doesn't behave like the -f in the tail -f command, and so isn't useful.

Fortunately to support centralised logging, eTrust Access Control includes a log emitter (called selogrd) and a collector (to be run on a central host).

The trick is to ensure that /usr/seos/log/selogrd.cfg contains :-

Rule#1
file /tmp/audit.log
.

What is logged can be selected via include and exclude primitives, along with other 'places' to emit the audit violations to. See the selogrd manual page for details.

selogrd checks for new audit violations every few seconds, as specified in /usr/seos/seos.ini in section [selogrd] under setting Interval.

Its normally 5, but might be better with a smaller value. When using Log Accumulator to merge logs from various commands, and from eTrust, this would ensure the security violation was display closer to the command which caused it.

Note: Despite there appearing to be no specific/visible rule covering /usr/seos/seos.ini, you can't amend this file unless eTrust is disabled.

You then ensure that selogrd is running, either by modifying /etc/inittab, or just simply running it.

You can then add the output file to the Log Accumulator.

Hints

Use a wide screen and window!

Use a filtering tool to process the resulting composite logs.

Summary

Log Accumulator is only a very small and simple program. But it does have the potential to significantly improve the recording of actions taken and results seen in most formal testing scenarios.


This documentation is written by the Log Accumulator author, Andy Key
andy.z.key@googlemail.com