Entries For: August 2007
2007-08-17
Hierarchical LoGS implementation
Filed Under:
Its been a while ...
I started on implementation of a hierarchical LoGS setup. The basic idea is a N-tiered setup (two tiers for now :) where each, say, cluster has a local instance of LoGS running analyzing the interesting stuff for that machine. This local instance is responsible for reporting on/responding to issues that are local to that cluster. If it comes across anything that is globally interesting (maybe some port scanning activity on one or more nodes of the cluster) it is reported back to another instance of LoGS that is running somewhere to correlate the globally interesting events.
This is *VERY* work in progress right now.
A related side-note: One thing I've clearly learned thus-far is that LoGS needs to handle CTRL-C in some reasonable way. I've been using an instance of LoGS spawned over an ssh connection to a remote host as a Data Source for the instance of LoGS running on my desktop. When I ctrl-c on the desktop instance of LoGS, not only does it dump me into the debugger, but when I quit out of the debugger and exit Lisp, it invariably leaves the instance of LoGS running over the ssh connection running on the remote machine. Yuck!
I'm also realizing that the RDL stuff /really/ needs to be documented in the LoGS manual!!! OMG! I'd mostly forgotten how to use the RDL :)
I wonder if my little LoGS app could somehow be changed to have a network data source and distribute SBCL-based Linux binaries to others on campus that would read from their local log file(s) and would write to this network port to correlate these worms campus-wide.
In other words, I'd like to have something where I don't have to be granted access to a department's log server in order to get access to their ssh/whatever worm data.
I'm sure I could co-opt the CS log server to do the initial testing. George is nice. They have good sys-admins at CS. :)
Back to my current implementation....
Right now I'm looking for ssh invalid/failed logins in a global sense. The ruleset on the cluster (this is nano, for now) looks for cluster-specific issues, such as Torque problems which it reports to me. It also looks for these login problems and keeps track of how many times its seen a given username and ip address that have had problems logging in. It then reports this back to the instance of LoGS running on my desktop (right now, the desktop instance is spawning the cluster instance and the cluster instance is simply printing messages about the failed logins to stdout where they are read by the desktop instance. So that there is /something/ to correlate with, the desktop instance also watches its /var/log/messages file to look for failed login attempts there. In theory... it should be simple to add other clusters to this setup and correlate across all of them.
I started on implementation of a hierarchical LoGS setup. The basic idea is a N-tiered setup (two tiers for now :) where each, say, cluster has a local instance of LoGS running analyzing the interesting stuff for that machine. This local instance is responsible for reporting on/responding to issues that are local to that cluster. If it comes across anything that is globally interesting (maybe some port scanning activity on one or more nodes of the cluster) it is reported back to another instance of LoGS that is running somewhere to correlate the globally interesting events.
This is *VERY* work in progress right now.
A related side-note: One thing I've clearly learned thus-far is that LoGS needs to handle CTRL-C in some reasonable way. I've been using an instance of LoGS spawned over an ssh connection to a remote host as a Data Source for the instance of LoGS running on my desktop. When I ctrl-c on the desktop instance of LoGS, not only does it dump me into the debugger, but when I quit out of the debugger and exit Lisp, it invariably leaves the instance of LoGS running over the ssh connection running on the remote machine. Yuck!
I'm also realizing that the RDL stuff /really/ needs to be documented in the LoGS manual!!! OMG! I'd mostly forgotten how to use the RDL :)
I wonder if my little LoGS app could somehow be changed to have a network data source and distribute SBCL-based Linux binaries to others on campus that would read from their local log file(s) and would write to this network port to correlate these worms campus-wide.
In other words, I'd like to have something where I don't have to be granted access to a department's log server in order to get access to their ssh/whatever worm data.
I'm sure I could co-opt the CS log server to do the initial testing. George is nice. They have good sys-admins at CS. :)
Back to my current implementation....
Right now I'm looking for ssh invalid/failed logins in a global sense. The ruleset on the cluster (this is nano, for now) looks for cluster-specific issues, such as Torque problems which it reports to me. It also looks for these login problems and keeps track of how many times its seen a given username and ip address that have had problems logging in. It then reports this back to the instance of LoGS running on my desktop (right now, the desktop instance is spawning the cluster instance and the cluster instance is simply printing messages about the failed logins to stdout where they are read by the desktop instance. So that there is /something/ to correlate with, the desktop instance also watches its /var/log/messages file to look for failed login attempts there. In theory... it should be simple to add other clusters to this setup and correlate across all of them.