Linux Security Monitoring

The Barest Of The bare Minimum

Preface


Firstly, a slight caveat: this is not a guide, nor is it limited to Linux, nor is it to BSD-Unix based systems. HP-UX, Solaris, AIX etc. administrators will all recognise these same basic tools and skills. Perhaps the guide ought to be called '*NIX (although some SysV may be exempt, sorta) Security Monitoring Concepts for People Who Should Know Better', but that doesn't really roll off the tongue. This document assumes you have the following tools:

A Quiet Monday Morning

Monitoring is always the easy part, and the most quiet. Its when all hell breaks loose that we panic. This is where we use the iftop tool; this is in much the same vein as the top tool. If you're not familiar with this, take a look; though I would expect most people to have used it at least once. The simplest invocation of iftop is:
root@jcmain:~# iftop -i {interface}
though of course, I should qualify this, as always, with Don't Use Root! sudo It!. But yes, this will give us a nice easy way of seeing, in realtime, network connections with all kinds of goodies, bandwidth usage, speeds, data transfered etc. The man page will tell you more, and running the command will show you better than I can describe, but you get the idea. Easy.
If we look at this output, we can see all kinds of connections, for example the one from the home laptop I'm on now. (Like I say, Home laptop, so yours will be different, especially in a commercial environment).
As we see from that screenshot, there are five current connections (I swear, that facebook one isn't mine, I'm holding it for a friend). They all look healthy, they're resolved, apart from 192.168.1.1- but that one should be easy, and none are doing much of anything, which is good.
How, though, do we start investigating worrying connections? What constitutes something we potentially ought to look at? Can we spot this?

Panic Stations

What, then, do we do when we start seeing strange connections? A strange connection, as should be somewhat obvious, is one that is either using too much bandwidth, or is using a strange port or a strange IP. Hitting the 'p' key whilst in iftop should give us the port in use. And, as can be expected, it uses the /etc/services file (or similar) to resolve port numbers. Strange connections can also be to unresolved IP addreses, which often denotes an domestic IP address, or small business; strange connections can also be both together. Consider this line in an iftop output:
jcmain.local:52394 => jceee.local:10101 0b 0b 0b
<= 0b 0b 0b

The jcmain.local connection port is 52394, and as we know, the IANA specifies this as a dynamic port. Incidentally, that same IANA link tells us 10101 is for eZmeeting. This hasn't been resolved from our /etc/services file means its not in there, but lets take a look. There are main ways now of telling what we are running out of port 52394, or at least two different dependent on how the process which opened the port started. If the connection was instigated on our end and the IP address was specified in args, then we can just use ps. If this one happens to work, we're laughing really, means we started this purposefully and we just don't recognise it looking at this output. Luckily, this works here:

root@jcmain:~# ps aux|grep jceee
root@jcmain:~# - Returns nothing, but don't get disheartened, maybe started with the IP address. Is the IP address in /etc/hosts?
root@jcmain:~# grep jceee /etc/hosts
root@jcmain:~# - Nope. Do it the other way then, resolve it via ping
root@jcmain:~# ping -c 1 jceee.local
PING jceee.local (192.168.1.8) 56(84) bytes of data. Easy, that'll be right
root@jcmain:~# ps aux|grep 192.168.1.8
jc 22277 0.0 0.1 7336 3700 pts/15 S+ 19:17 0:00 python /usr/bin/pynk-client 192.168.1.8:10101

And there we go, the process '22277', which is pynk-client. Presumably connecting to pynk-server, but we don't care about that. Of course, the dance of grepping, and so on, could have been solved by pinging straight away, or taking it from the kernel routing table, but who cares- the above is so obvious that some people may not even think to do it and instead try to do the difficult stuff.
This, though, is not really the best way to do it, it makes a lot of assumptions, especially if there is more than one connection. There is a much better tool, and this is the lsof tool. This tool lists open files, and since Linux likes to boast how everything is a file (its not, or not purely- you want plan9 for that really) including ports and so on, we can do a lot better. Take for instance the following heavy bandwidth stealing connection:

jcmain.local:46224 => jceee.local:1234 242Kb 244Kb 244Kb
<= 0b 0b 0b

We can work out whats going on with:

root@jcmain:~# lsof -i :46224
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
vlc 26345 jc 11u IPv4 20163371 UDP jcmain.local:46224->jceee.local:1234

Oh yeah, silly me- I'm streaming music between boxes. If I want to see on a connection the PPID, perhaps to see where the command originated, or from where (useful for detecting commands run via some remote call vulnerability or so on), I can pass in the -R switch:

COMMAND PID PPID USER FD TYPE DEVICE SIZE NODE NAME
vlc 26345 4960 jc 11u IPv4 20163371 UDP jcmain.local:46224->jceee.local:1234
root@jcmain:~# ps aux|grep 4960
jc 4960 0.0 0.1 6300 3612 pts/2 Ss Aug31 0:01 /bin/bash

Should we do this to every connection we see that we don't recognise? No, that would be stupid, and potentially time wasting. My mail server, for instance, always just shows up as an IP address. For this reason particularly, whois is very handy for sussing out what address is what. This works along the same lines as testing for strange IP addresses and foreign addresses.

Pensive

Right, lets assume we've found a connection we really don't like the look of. We can't resolve the IP address, the connection isn't coming from a 'trusted' range, as it were, the PPID is perhaps from an RPC service or service that is listening online (Overflow, etc). The connection rate may or may not be off the scale yet, but the process being run is something we really don't like at any rate. Remote shells are definitely something we're looking at. What you do next is a matter of preference. A lot of administrators will kill the process, blacklist the IP range (to allow for dynamic IP addresses), some will panic and yank the network cable, and a good amount will do something similar to that which I do- throttle that connection severely and monitor what that port is doing. Personally, I like to use strace and tcpdump. This is, of course, a terrible, terrible, terrible idea on a system that contains anything personal, private or important. If you care about your system then kill it right away. There are situations and systems where it is okay to monitor, and those systems aren't them.
strace, for those not in the know, is the Systems Call Tracer. It also lists signals. strace is an extremely useful tool for working out what a process is doing, whether you run a new process in strace (a la time) or whether you attach a process to it. The following is from the strace from a simple server which listens on a port and echoes back anything it receives before being terminated.

root@jcmain:~# strace -p30568
00: bind(3, {sa_family=AF_INET, sin_port=htons(3000), sin_addr=inet_addr("0.0.0.0")}, 16) = 0
10: listen(3, 1) = 0
20: accept(3, {sa_family=AF_INET, sin_port=htons(34136), sin_addr=inet_addr("127.0.0.1")}, [16]) = 4
30: mmap2(NULL, 1003520, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7943000
40: recv(4, "hello\r\n"..., 1000000, 0) = 7
50: mremap(0xb7943000, 1003520, 4096, MREMAP_MAYMOVE) = 0xb7943000
60: send(4, "hello\r\n"..., 7, 0) = 7
70: listen(3, 1) = 0
80: accept(3,

As we can see here, apart from my BASIC style line numbering, the port is accepting from all addresses (0.0.0.0). We see at 40 it receives the word 'hello' on file descriptor 4 (the socket) and sends the same thing back on the same file descriptor. Essentially, echoing back. We can do a similar thing by using tcpdump which, as the name should suggest, dumps network traffic. This traffic can be read on STDOUT or can be read with a tool such as wireshark. Its a matter of preference, to be fair.

21:50:53.724947 IP localhost.49367 > localhost.3000: P 1:8(7) ack 1 win 513 nop,nop,timestamp 157072113 157071703
E..;^|@.@...............$...$......../.....
\.. \.Whello

21:50:53.725015 IP localhost.3000 > localhost.49367: P 1:8(7) ack 8 win 512
E..;W.@.@...............$...$..5...../.....
\.. \..hello

Is the important stuff for this, too- we see a packet with the word 'hello' was sent and echoed back in milliseconds. As I previously stated, though, this isn't a great thing to do if you're on a box that you can't afford to lose data on, though of course corporate boxes will have backups and at least this way gives the opportunity to work out exactly what needs replacing and repairing.
Once you have killed the process and blacklisted the IP address range of the attacker you need to fix the security hole. Now, many attackers are territorial, but never run with the assumption they have fixed the hole they used, and even if you do have reason to believe this after investigation, for God's sake ignore it and do the work you're self. The best way to work out where this came from is, as previously mentioned, to look at the PPID of the process running before you kill it. Running w and checking /root/~.bash_history is also a good way of making sure you catch all the changes made in a remote shell situation. Where some extra code was installed/executed which would allow for control to be taken via another manner, such as IRC, bash history isn't much good. Nor is it if you're using a different shell, of course, but that should be obvious.

Immaturity

Usually, at least in a corporate setup, you're going to want to write up a report on how you got owned, and of course fix the problem and document this, and go and grab a coffee. However, if you're like me, you're going to want to do something immature, if for anything then to make yourself feel better- lets face it, chances are its some grotty little skidbag in his parent's house somewhere, but whatever. In such a situation I like to fire up either scapy or nmap, and fire off some xmas tree packets from any number of falsified IP addresses to random ports and see how long it takes them to yank their own cable out of the wall. From what I understand, skiddies live in fear of being caught by someone on the ball, and nothing lights up more than a Christmas tree.