When working in "offline mode" (where all of the log files exist in their entirety at the beginning of the run; presumably, they are from the past), there are a couple of gotchas that John Roulliard and I have been chatting about. I'm hoping to capture part of that here. When doing "offline" analysis of logfiles, the notion of time must come from the logfiles themselves rather than the system clock. Using the timestamps that are (generally) a part of these logs creates some issues:

• Single-host time continuity problems
• Multi-host time continuity problems
• Too low resolution for time
• Too high resolution for time + synchronization problems

Single Host Time Continutity Problems

A single host may appear to travel forwards, backwards, or more quickly/slowly in time in a logfile if the system clock was altered while the log was generated.

Multi-host Time Continuity problems

Multiple hosts without perfect time synchronization may cause time to appear to go forwards/backwards often when analyzing logfiles from two or more hosts.

Too low resolution problems

When using standard syslogd, the resolution on the timestamps is one second. This can lead to ordering problems when working with multiple files.

Too high resolution problems

Because time synchronization cannot be perfect, when working with really high resolution timestamps in files, it may be possible to order events incorrectly based solely on their timestamps.

Offline Mode/Parsing Timestamps

Supporting working with time, especially for "offline mode" seems important.

One should be able to (painlessly) tell LoGS to be in "offline mode". Offline mode seems to imply at least:

1. Not using the internal-time
2. Parse the timestamps on the log messages; use them to set *now*
3. *LoGS-INTERNAL-TIME-UNITS-PER-SECOND* should be set to the resolution of 1
   (In "online mode", this is the value of INTERNAL-TIME-UNITS-PER-SECOND or 100 under CMUCL).

In order to support this, we introduced the --parse-timestamp, --timestamp-start, and --timestamp-end flags.

--parse-timestamp causes LoGS to not use the internal-time and, instead, to set *now* by parsing the timestamp on the line. I'm not exactly sure what will happen when you parse a syslog file from, say leap day, when this year doesn't have a leap day! LoGS will probably die a flaming death and take you with it!

--timestamp-start causes LoGS to look at this position in the line for the start of the timestamp. This sets the internal variable *timestamp-start*. The default value of *timestamp-start* is 0.

--timestamp-end causes LoGS to use this position in the line for the end of the timestamp. This sets the internal variable *timestamp-end*. The default value of *timestamp-end* is 15.

Time formats:

Right now, LoGS understands only the standard syslog time format. This format is:
Mon Dt HH:MM:SS
eg:
Jun 9 12:34:56

There are several times when I would like to be able to specify times that things should happen in LoGS. I guess that what I'm after is somewhat like a "calendar event" in SEC.

LoGS, of course, already has facilities for doing things at certain times. The problem is that these times have to be specified in *LoGS-INTERNAL-TIME-UNITS-PER-SECOND* since the start of the Lisp process; isn't that convienant?!

What we would like to be able to say is something like: "write me a report on all the invalid ssh connections AT MIDNIGHT" or "... AT 12:01 PM"

While CMUCL has facilities for parsing dates that seem rather robust, these facilities are not portable. I found that Gene Michael Stover has created a library (I'm not quite sure of its name exactly!) that can be found here: http://cybertiggyr.com/gene/pdl/

I've started to do a little work to make this library support the date formats that I would like to see (such as a simple "12:34:56" to mean 12:34:56 AM today and "12:34:56 tomorrow" to mean, obviously, 12:34:56 AM tomorrow). These are the sorts of times I want to be able to use in my LoGS rulesets.

I've submitted a patch to Mr. Stover to support a couple of these formats. I hope that he will accept them so that I don't have to maintain a bunch of time parsing stuff inside of LoGS.

LoGS 0.1.2 was released yesterday. For the first time, the tarballs are available on savannah.nongnu.org!