When working in "offline mode" (where all of the log files exist in their entirety at the beginning of the run; presumably, they are from the past), there are a couple of gotchas that John Roulliard and I have been chatting about. I'm hoping to capture part of that here. When doing "offline" analysis of logfiles, the notion of time must come from the logfiles themselves rather than the system clock. Using the timestamps that are (generally) a part of these logs creates some issues:

• Single-host time continuity problems
• Multi-host time continuity problems
• Too low resolution for time
• Too high resolution for time + synchronization problems

Single Host Time Continutity Problems

A single host may appear to travel forwards, backwards, or more quickly/slowly in time in a logfile if the system clock was altered while the log was generated.

Multi-host Time Continuity problems

Multiple hosts without perfect time synchronization may cause time to appear to go forwards/backwards often when analyzing logfiles from two or more hosts.

Too low resolution problems

When using standard syslogd, the resolution on the timestamps is one second. This can lead to ordering problems when working with multiple files.

Too high resolution problems

Because time synchronization cannot be perfect, when working with really high resolution timestamps in files, it may be possible to order events incorrectly based solely on their timestamps.