Monday, July 30, 2007

Explorations with adore-ng

As part of my studying for the SANS GCIH certification I thought I would take some time to really get to know several of the tools mentioned on the Certifications Bulletin. The first item is the adore rootkit. I will be studying adore-ng-0.56 on a RedHat 9 install. What I hope to understand better through this process:
  1. How to compile and 'install' the adore-ng rootkit (read the README, very well documented).
  2. How to hide processes and files.
  3. How to hide network sockets from netstat, etc.
  4. How to attempt to detect from a remote system.
  5. How to attempt to detect from a compromised system.
These posts are more or less a study guide for myself and anyone else who may stumble upon them. The authors of these tools are the real geniuses, I am simply attempting to better understand what these tools are and what I can do to protect against them in the 'wild'. Therefore, I will reemphasize what is written in the adore-ng README:
Only *YOU* are responsible for your own actions. So if you are
dumb enough to own machines and install this software on it,
only you can be blamed for it.
Task 1: Compile and Install adore-ng
After downloading the the adore-ng tar ball it was a simple matter or running ./configure and then answering the following question.
Since version 0.33 Adore requires 'authentication' for
its services. You will be prompted for a password now and this
password will be compiled into 'adore' and 'ava' so no further actions
by you are required.
This procedure will save adore from scanners.
Try to choose a unique name that is won't clash with filenames in /proc.
Password (echoed)
Next compile adore-ng. The following warning was thrown by a 'make' after configure:
/usr/include/linux/modversions.h:1:2: #error Modules should never use kernel-headers system headers,
/usr/include/linux/modversions.h:2:2: #error but rather headers from an appropriate kernel-source package.
/usr/include/linux/modversions.h:3:2: #error Change -I/usr/src/linux/include (or similar) to
/usr/include/linux/modversions.h:4:2: #error -I/lib/modules/$(uname -r)/build/include
/usr/include/linux/modversions.h:5:2: #error to build against the currently-running kernel.
So I had to modify the Makefile to use the preferred include path in /lib/modules/. I also uncommented the definition for CFLAGS+=-DREDHAT9 since this is RedHat 9 and w/o this the macro for_each_process will not be setup correctly. (These are probably RedHat 9 quirks.)

To load the adore-ng kernel module simply run ./startadore which will insert the adore module
followed by the cleaner module. I found the following article which explains the simplicity of the cleaner module: Anti-Honeypot Technology.

The first thing that I tried was an lsmod. And bingo, no adore module listed. So I thought, did it really work? A quick run of the ava utility with the 'I' option shows the following:

./ava I
Checking for adore 0.12 or higher ...
Adore 1.54 installed. Good luck.

ELITE_UID: 539905012, ELITE_GID=811693422, ADORE_KEY=qwerty CURRENT_ADORE=54


So it is installed.

Task 2: How to hide processes and files...
One of the first things I noticed was that the source directory was hidden. At first I thought it was removed or that I 'misplaced' it. An ls -l in /root/ (where I originally unrolled the tarball) shows no adore-ng directory? But if I cd /root/adore-ng/ then it is really there. You may not catch it, but the configure script does a chown to the elite UID and GID's for you. Thus you 'hide' the source. I tried a find /root/ and it also does not see it. This is because the adore-ng is in the Kernel and is intercepting my reads of the file system and doing checks on uid and gid. If both the uid and gid are not set correctly then the file will be listed. A simple test follows:

# touch /tmp/abc
# ls -l /tmp/
-rw-r--r-- 1 root root 0 Aug 1 11:12 abc

# chown 539905012 /tmp/abc
# ll /tmp/
-rw-r--r-- 1 539905012 root 0 Aug 1 11:12 abc

# chown 539905012.811693422 /tmp/abc
# ll /tmp/
total 0

# ll /tmp/abc
-rw-r--r-- 1 539905012 811693422 0 Aug 1 11:12 /tmp/abc


And we are now hidden. The ava utility appears to have the same functionality (actually runs lchown).

# ./ava h /tmp/abc
Checking for adore 0.12 or higher ...
Adore 1.54 installed. Good luck.
File '/tmp/abc' is now hidden.


To hide a process the ava utility will create and unlink a file in either /tmp/ or /proc (depending on the value of ADORE_LSM) with the following format: hide-PID or unhide-PID.

Here is an example using the ava utility to hide a shell process.

Here is a regular user's shell session:
$ ps
PID TTY TIME CMD
17321 pts/1 00:00:00 bash
19745 pts/1 00:00:00 ps

Now, using ava, we make the shell process invisible:
# ./ava i 17321
Checking for adore 0.12 or higher ...
Adore 1.54 installed. Good luck.
Made PID 17321 invisible.


Back over to the original shell session and now nothing is listed, creepy:
$ ps
PID TTY TIME CMD


What if we look for it specifically...?
$ ps -p 17321
PID TTY TIME CMD


This also appears to have hidden all child processes of the originally hidden process.

Task 3: How to hide network sockets from netstat, etc...
In this section I will explore how to use adore-ng to hide a network socket. The README for adore-ng explains that you need to update the adore-ng.h file to include any port numbers that you would like to have hidden. The array HIDDEN_SERVICES should be updated to include the decimal values for the ports you want to hide. By default ports 2222 and 7350 are both hidden. To test that port 2222 is hidden I will start a netcat listener and attempt to detect it on the compromised system.

Start a listener on port 2222...
$ /tmp/nc -l -p 2222

Try to detect it with netstat...
# netstat -an |grep -q 2222 || echo NADA
NADA


What about using lsof...?
# lsof -i :2222 || echo NADA
NADA


To prove that it is really alive and listening...
# telnet localhost 2222
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.


Task 4: How to attempt to detect from a remote system...
There are a few ways that I can think of to attempt to detect the installation of the adore-ng rootkit from a remote system. These methods are actually going to attempt to detect the changes that are likely to occur to a system after being owned.
  • Network Intrusion Detection System: A properly configured and tuned NIDS will most likely tip you off that something is awry. You may not catch the initial infection point of your system, especially if this is an insider attack. What you should catch is anomalous network behavior; as in traffic on ports that have never been seen on this host before. Let's assume that the attacker leaves the adore-ng.h header alone and opts to use port 2222 to shovel a shell to a remote system using netcat. Your NIDS should alert you that all of a sudden there is a new pattern of traffic being generated to/from this host on a port that you never expect to see traffic on.
  • Network mapping: An extension of the above NIDS approach is to maintain an up-to-date inventory of the network ports that all of your servers are listening on. This can very easily be accomplished by using nmap and comparing the results. Check out the nmap related projects page for some tools which do this already.
  • Remote Logging: I feel that all systems should utilize some type of remote logging facility. Whatever method you use please be sure that it is secure and reliable (syslog-ng is a good start). Having you logs immediately forwarded off to another host (as in real-time) will allow you to analyze signs of a possible attack and may tip you off to any subsequent rootkit installations.
Task 5: How to attempt to detect from a local system...
Attempting to use anything on a system which has been compromised is extremely dangerous. It is especially foolish to trust that the system will tell you the truth when you ask it what is going on (as was demonstrated above in the hiding of files, processes, and ports). However, if you are in a crunch and must only use the system that you suspect is under the control of adore-ng then here are some tips that I have discovered while playing with this rootkit.
[Please be aware that a system which has been owned will likely have many modifications done to it to prevent you from knowing the truth. In my example I have only modified the system with the adore-ng rootkit in the most basic ways.]
Above I demonstrated how to hide a process from a ps listing. While I was playing with this I wondered if the hidden process would still accept a kill -0 (not kill 0, notice the "dash zero"; as in send signal number 0 to a process)? The method I am about to explain is based on the fact that in my test environment a hidden process would accept this signal and therefore reveal to me that something may be amiss.

Detecting hidden processes with kill -0
In the above examples I hid a process (bash shell session) with PID 17321. Neither a ps listing or a browse through /proc/ revealed any signs of the hidden process.

But what if we try to send it a kill -0 signal?

$ kill -0 17321 && echo "it is alive"
it is alive


As an example here is what happens when the PID is not alive:

$ kill -0 1732199 && echo "it is alive"
-bash: kill: (1732199) - No such process


As an example of what happens when you try to kill a PID you do not own (init in this case):

$ kill -0 1 && echo "it is alive"
-bash: kill: (1) - Operation not permitted


So in theory we could detect a hidden process by comparing a process listing output by ps (or looking into /proc) against the results of a kill -0 against every possible system PID (this can be discovered via: #cat /proc/sys/kernel/pid_max).

Detecting hidden processes via /proc (when you are hidden too)...
Something else that I discovered is that even though the PID does not appear as a directory in /proc/ (ie. ls /proc/ will not list it) you can still reference it (as in ls -l /proc//cmdline will produce valid results). [This appears to only work if the user investigating /proc is running their shell as a hidden process also. The kill method does not appear to have this limitation.] This could be used in the same way as the above kill method to search for processes which are 'hidden'. I have written the following shell script to 'detect' hidden processes that should be run from a hidden process itself.
#!/bin/bash
# Attempt to detect hidden PID's using /proc guessing

MAXPID=`cat /proc/sys/kernel/pid_max`

PIDS_IN_PROC=`cd /proc/ && ls -d [1-9]*`

num_hidden=0

echo "Searching for hidden process [`id`]..."

# populate PID Array for all 'v'isible processes in /proc
for i in $PIDS_IN_PROC
do
PID_ARRAY[$i]="visible"
done

# now guess processes
for p in `seq 1 $MAXPID`
do
echo -ne "Checking...$p\r"

if [ "${PID_ARRAY[$p]}" = "visible" ]; then
continue
elif [ -d /proc/$p ]; then
PID_ARRAY[$p]="invisible"
echo -ne "\t\t\tFound $p\r"
((num_hidden++))
fi
done


# Report the findings
echo "We discovered a total of ${#PID_ARRAY[*]} processes ($num_hidden) are hidden"

if [ $num_hidden -eq 0 ]; then
exit 0
fi

echo "The following PIDs appear to be hidden:"

for p in `seq 1 $MAXPID`
do
if [ "${PID_ARRAY[$p]}" = "invisible" ]; then
echo -ne "$p\t"
fi
done

echo -e "\nComplete"


Of the above two process detection methods I feel that the kill -0 method may be more readily available as it does not appear to require that the user be root or running the tests under a hidden process.

Detecting hidden sockets...
If the process which is setup to listen on a hidden socket is not also itself hidden you can try to uncover it with the following technique.

# lsof | grep 'identify protocol'
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
nc 11334 aaron 3u sock 0,0 21512 can't identify protocol

Using chkrootkit
Your best method of detection may be to try the chkrootkit set of tools. The chkrootkit sites states that the suite of tools effectively detect the adore LKM. I downloaded and compiled chkrootkit-0.47 and here are the results of a lkm detection:

# ./chkrootkit lkm
ROOTDIR is `/'
Checking `lkm'... You have 2 process hidden for readdir command
chkproc: Warning: Possible LKM Trojan installed


Summary
In summary there are several tenets that a good system administrator must allows follow
that will help prevent an intruder from ever being able to install a rootkit such as adore-ng. A couple of the more important rules that I highly recommend is (1) Know thy systems and (2) Practice defense in depth. By following these two simple rules a good system administrator will typically be ahead of an intruder and immediately notice that something wrong is going on with their systems and networks.