We were wrong?
Supporting Times
Document Actions
SSH Scanning Jerks
Filed Under:
I don't know no invalid users!
So, I'm working on putting together a new cluster at the CHPC. OK, well, actually what I'm up to is ripping apart an old cluster and putting it back together.
Anyway, there's a fun dance that DHCP and PXE do that is very easy to get wrong. I like to watch /var/log/messages while I'm installing machines to make sure that, yes, they picked up the right server, have the MAC adress I'm expecting and the like. Anyway, there have been something like 3 occasions in as many days where watching /var/log/messages is going fine, then, all of a sudden, I get a flood of login attempts via SSH from like china or god knows where... Anyway, its not somewhere that my users are. Of course, there is a .0001% chance they may be on sabbatical or something, so we can't just block the rest of the unknown universe at our border firewall.
So, after the third flood of invalid login attempts (NO, I DO NOT HAVE A USER NAMED 'webmin' - That should save you bad guys a little effort
), I got to thinking - Time for some LoGS action!Here, I'm going to try to document the creation of a small ruleset to catch these baddies. For this ruleset, I'll be using the 0.2.0 proposed branch of LoGS because thats still what I'm working on/with
My first take on this:
I think that if I see someone give me a whole boatload of this crap, we can do something about it:
Oct 28 17:03:11 l096 sshd[16651]: Invalid user michael from 62.123.192.212
Oct 28 17:03:35 l130 sshd[17500]: Invalid user rr from 62.123.192.212
Oct 28 17:03:24 l099 sshd[17188]: Invalid user michael from 62.123.192.212
Oct 28 17:03:23 l120 sshd[12610]: Invalid user alexandre from 62.123.192.212
Oct 28 17:03:48 l116 sshd[16923]: Invalid user alexandre from 62.123.192.212
Oct 28 17:03:25 l112 sshd[16801]: Invalid user alexandre from 62.123.192.212
Oct 28 17:03:28 l121 sshd[11007]: Invalid user transfer from 62.123.192.212
Oct 28 17:03:27 l125 sshd[16701]: Invalid user transfer from 62.123.192.212
Oct 28 17:03:21 l123 sshd[13014]: Invalid user transfer from 62.123.192.212
Oct 28 17:04:58 ll04 sshd[2866]: Invalid user alexandre from 62.123.192.212
Oct 28 17:03:27 l081 sshd[13585]: Invalid user mike from 62.123.192.212
Oct 28 17:03:29 l083 sshd[16865]: Invalid user mike from 62.123.192.212
(IPs not anonymized to implicate the guilty)
I'm noticing here that the same IP is trying multiple Invalid users. Most likely, my users wouldn't do this. It is possible to mistype your login on your ssh command line, but you probably would realize fairly quickly that you had done so. When I tried to log in as an invalid user, in this case 'jimistesting', I was given a password prompt as usual - for some reason all 3 passwords I entered were rejected by ssh... hmmm.... Here's my log message:
Oct 30 07:53:34 ll04 sshd[16472]: Invalid user jimistesting from 127.0.0.1
I'm thinking that if I keep track of which IPs from which I have see these 'Invalid user' login attempts and which invalid user they log in as, I should be able to distinguish someone mis-remembering her username and someone trying to break in. I'm thinking if you manage to try 3 different usernames, all invalid on my system, from the same IP address you can go into my block list on the firewall :)
Note: I'm actually disreguarding the host that the bad guy is trying to log into. 3 strikes and your out against any of my hosts listening on the SSH port.
First, we need a function to identify these "Invalid user" messages:
(defun find-invalid-user-message (message &optional env)
(declare (OPTIMIZE SPEED (DEBUG 0) (SAFETY 0)))
(multiple-value-bind (matches sub-matches)
(cl-ppcre::scan-to-strings
"Invalid user (.*) from (.*)"
(message message))
(when matches
(values t
`((user ,(aref sub-matches 0))
(ip ,(aref sub-matches 1)))))))
This function uses cl-ppcre to match a regular expression. It pulls out the username and the IP address from the log message and
returns them into the LoGS 'environment' associating them with 'user and 'ip respectively.
Now, we need to handle the messages we've found:
;; a table to hold hosts we've already reported as bad
(defvar *reported-bad-hash* (make-hash-table :test #'equal))
;; 3 minute window length
(defvar *ssh-scanner-window-length* (* *LoGS-internal-time-units-per-second* 3 60))
(defun handle-invalid-user-message (message &optional env &rest xxx)
(let ((window-name (format () "invalid user from ~A"
(get-logs-env-var 'ip env))))
(unless (gethash window-name *reported-bad-hash*)
(let ((window
(ensure-window
:window-length *ssh-scanner-window-length*
:max-lines 2
:name window-name
:actions (list #'report-ssh-scanner))))
(unless
(find-in-store (get-logs-env-var 'user env) window)
(add-item window (get-logs-env-var 'user env)))))))
Here we add the username to a WINDOW object named for the IP address that these messages came from, but only if that username isn't still in the window. This window will run its actions (we'll get to that in a sec...) when it gets more than 2 usernames in the window. this window will throw away usernames older than 3 minutes (*ssh-scanner-window-length*).
Finally, we need something for the window to do when it 'blows up' and runs its action(s):
(defun report-ssh-scanner (window &optional env)
(setf
(gethash (name window) *reported-bad-hash*) t)
(format t "window: ~A blew up~%"
(name window))
(map-store
(lambda (entry &optional level)
(format t "user: ~A~%" entry))
window))
Here we report the ssh scanner, if we have not already reported this IP. Then we print out a message with the window name, then list the usernames that are in the window.
Obviously, this is a really trivial response to this. Maybe we could have this shell out and run some iptables fu or something along those lines to automate a response to these scanners.
Finally, we tie this all together into a rule. Here I'm using "RDL" syntax:
Running it, I get output something like this:
(setf *ssh-scanner-rule*
(rule
matching
#'find-invalid-user-message
doing
#'handle-invalid-user-message))
(enqueue *root-ruleset* *ssh-scanner-rule*)
window: invalid user from 202.146.92.147 blew up
user: test
user: tester
user: testing
window: invalid user from 62.123.192.212 blew up
user: a
user: b
user: c
window: invalid user from 201.88.73.66 blew up
user: admin
user: stud
user: trash
Performance note:
On my laptop, I'm processing approximately 48,000 messages/second with just this rule in the ruleset.