Wednesday, April 09, 2008

Downgrade the password...

So it has been quite a while since the last post (and I promise to get back into the EC2 soon). I had to quickly post this because it really got to me...

I am on vacation for the next few days and about a 1/2 day into the first day the voice mail light goes on the phone; it is work. It is a message from one of team members stating that the password I set to authenticate access for a 3rd party to our PCI web services will not work because it has a quote in it and they need to know what might break if they change it. I would have been opposed to changing the password just because the code couldn't properly handle a 'special' character in the password. The team did the right thing though, for the short term, and changed the password so that we were not blocking access to the services which was having a serious impact on the customer (the WS is still authenticating so that is good).

I know that the application calling the service is written in .Net (probably and that they were able to quickly change the password, so I'm thinking a web or app.config. So why didn't they just encode the 'special' characters in the same .config? I'm not sure and will ask about it when I get back next week. This seems like the solution but I'm not a .net coder.

This entire situation is ironic to me. The IT team at my work has been doing some great work over the last year or so on securing our applications, especially from a PCI perspective. And then we get hit by an application that is brand new (written over the last month or so) which can't handle a password with 'special' characters.

Is this a problem with the tools? I don't think so.

Is this a problem with the coder not knowing how to do this? Doubtful too, they probably know how to encode this just fine.

What is the problem then? In my opinion this is a typical mistake that is often made. Code for what _you_ know. Too often are coding decisions based on how the developer lives and works right now and not for how the system is really going to be used or better yet, for how it may be used in the future. Being a programmer takes a certain degree of fortune telling. We should try to think about the scenarios that are down the road for this application and what it may come up against. This may be a stretch for just a password character issue but I think it probably hints at what problems may be around the bend.

Monday, February 11, 2008

Starting out with Amazon EC2 (Prelude)

This is the first in a series of posts which will cover my experiences with the Amazon EC2 (Elastic Compute Cloud) services. I had first read about EC2 a few months back and tagged in my account but for some reason never visited it again (I can tell why: too busy working on the wrong things, i.e. legacy application support). Then, I attended the February Columbus Ruby Brigade meeting and the presentation was on Amazon Services (EC2, S3, etc.).

Some background info: I have recently been working on a Data Center proposal for work which will hopefully move most of our Internet presence into a full-on ultra-redundant Data Center. But the presentation of the Amazon EC2 services has me thinking; why do I need to have physical servers in anyone's Data Center? Could I put together a plan which minimized the amount of real iron that we have to buy, maintain, and find a home for and replace most of that 'stuff' with computing in the cloud?

Well I'm going to try... It may or may not work out for this specific project but I aim to make the experience fun and definitely educational for me (and hopefully for you too).

Please check back in, as I work my way through the EC2 experience.

Monday, December 31, 2007

Simulate Human Keystrokes...

A recent problem at work resulted in the degradation of three distinct ISP links down to one. This post will explain how the great expect tool assisted with reprogramming a router's ACL when copy paste was too fast.

First some background... (Stick with me and I'll eventually get to the simulating human keystrokes).

The design of the network was that a pair of load balancing devices would use bandwidth and availability statistics to determine which of the three ISP links to send data data out on (TCP connections and UDP/DNS requests). Additionally the load balancers offered DNS services for several of our web sites and would hand out IP addresses that were associated with the ISP link that was the 'healthiest'. Thereby causing inbound HTTP requests to be somewhat balanced across the three links (a kind of intelligent round-robin DNS).

Strike one and two...

The two (HA) load balancers refused to operate one day and basically would not pass data in or out of our network. The configuration was solid and the appliances did not appear to have suffered a physical failure. After a few hours of my team tinkering with the hardware and working through the manufacturer's technical support team I decided to pull the plug on them and replaced the DNS portions with a Knoppix box and move the NAT functionality to one of our Cisco border routers (many thanks go out to a great consultant from Berbee who jumped in a configured the Cisco portions for me in a pinch).

This solution resulted in a working Internet connection for our organization again but it also resulted in cutting our effective bandwidth from the three ISP links down to one (basically cut our aggregate of 24Mbit down to 10Mbit).

After a day of running all of our traffic through a single link and a single router we started to get complaints about the responsiveness when accessing the Internet from our branch locations (the branch locations access the Internet through HQ's ISP links). In looking at the interface utilization on the router I saw that we were using less than 60% of the available bandwidth in both the inbound and outbound directions. However, when I peaked at the CPU utilization ('show processes cpu') it became immediately apparent where the problem was; were running at 99% of available CPU. I had noticed the day before that all the ACL entries on this router (and the other border routers) had a log statement attached to each; including all the allow entires. My first reaction was to strip off the logging and see if we could gain anything back.

This is where we finally get to simulating human keystrokes... (I promise)

So I did what any respectable UNIX guru would do, copy and paste the current ACL into vi and proceed to remove all the log statements; with a '%s/log//g' command of course. Now we have a brand new ACL that I can paste back into my management session to the router; I should mention that the only session available was via the serial console attached to a serial concentrator. I started the 'conf t' session and began to create a new access list, not clobbering the original. I grabbed the new ACL in one terminal window, did a command-C (copy), then switched to the console connection terminal and did a command-V (paste). A no-brainer of course, so why am I wasting your time...? We'll what I proceeded to see 'typed' into the console terminal was a bunch of ACL lines that were stepping all over themselves. The terminal was sending data to the router's serial connection faster than it could read it and as a result characters were being dropped from the lines and causing the config commands to fail.

Just great! I thought to myself, get ready to hand type in each of the ACL's (there were quite a few) and hope that you don't make a mistake! Then the dude that lurks in the inner parts of my brain said, "Hey! Use expect since it can simulate human typing characteristics." That's right, I recalled seeing something in an autoexpect generated script that referenced human-like typing. So I proceed to RTFM for expect and sure enough the 'set send_human {}' command is exactly what I need. I created a small expect script that set the send_human option and then did a send -h of the entire ACL. [At this point you should really take a pause and read through the send_human section of the expect man page.] I experimented with a few values passed to send_human until I saw something on the screen that looked like it should go without causing a problem. But now how do I get the output of expect into the router's config command line. There were two options that came to my mind immediately:
  1. Write a more elaborate expect script that spawns a session to the router's serial connection and interacts with it all the way through logging in and entering the config terminal phase.
  2. Manually start the config terminal session with the router myself and just redirect the output of the expect script at the point where I would normally type each line of the ACL.

I needed to get this done in a very short amount of time so I opted for option #2. But how do I get the output of expect as the input of a ssh/telnet session? I opted to use a fifo instead of trying to send everything through a standard UNIX pipe (I assumed, but didn't confirm, that the standard pipe would buffer all the output and have the same problem as the original past option). Here is the basic setup:

  1. Create a pipe using the mkfifo command:
    $ mkfifo myio
  2. Start the ssh/telnet session with the router (since this is a serial connection the session can be prepared ahead of time to already be in the proper config terminal location and then quit the session with ^] or ~..). The key is to take STDIN from the fifo that we just created.
    $ telnet serial-server 9999 <>
  3. Now start the expect script and send the output to the fifo.
    $ expect -f my.exp > fifo
  4. Next I sat back and watched the expect script 'type' the ACL commands into the router at a pace that was acceptable and did not overrun any buffers.
  5. After a few seconds the new ACL was ready for me to apply in place of the old one.
Hopefully, this will help some of you out if you ever find that you are in a situation where you need to provide a large set of input to one process to appear as though a real human was typing it. Even if you do not have this specific need I urge everyone to look at using expect. I have found many tremendous uses for it over the years and enjoy watching it do my work for me.

... and in case you are wondering if removing the log entries from the ACL's had a positive impact on the high CPU utilization, it only dropped it down by a few points. The real culprit turned out to be a single PC blasting out SYN packets to a site in the .ru domain (I expect to analyze this system in the days to come).

Thursday, December 06, 2007

Drawing Your Password

I came across this article and feel that this is some amazing research into how future technologies will make it easier for us to generate more secure 'passwords'. Dr. Yan has taken Draw a Secret (DAS) technology and improved upon it in a great way...

I had not seen DAS technology before reading this article and my first thoughts were: "OK, if you ask people to draw something that will become their password then you will probably end up with a bunch of smiley faces for passwords." What Dr. Yan has done is add in a background image that you then draw your password over (BDAS). This is really cool because for me this would build upon a natural tendency for me (and I imagine many others) to take an existing picture and 'doodle' on it. Obviously, you would want to avoid some common background images, such as the Mona Lisa, since most people would probably just draw a mustache on her and be done. However, you take an image of your house, dog, or favorite vacation spot and then you could get really creative (even the most artistically challenged geek could muster up something worthy).

The article explains that in testing the technology they found that passwords generated using BDAS contained 10 more bits of data than those of standard DAS. That is a significant improvement.

My only concern would be that the screens you would draw the images on may become 'tatooed' and reveal the outline of your password which would make password stealing easier. However this scenario is somewhat protected because the technology is looking at more than just the drawing. It also considers stroke counts and stroke lengths. Full details on the technology is available here.

On a side note...

This sparked a topic in my head on the security of signature data. If you walk into many retailers today and purchase items using a credit card odds are that you will be asked to sign an electronic pad. These pads convert the pin up and pin down events and stroke lengths into a binary representation that is then stored and can later be retrieved and reprinted (often on your copy of the receipt). Since the data is stored electronically in a database somewhere it could easily be stolen. I did a very brief search for signature protection legislature or standards (i.e. PCI) and came up with little worthwhile. This is an are that probably could use some research and some proper guidelines.

Tuesday, November 27, 2007

Lessons in network latency

At work were recently brought a new site online and connected it to HQ via our MPLS WAN. Almost immediately we began to notice odd behavior out of this site related to the WAN connection. At times we were seeing ping times from our HQ to the sites router go over 1 sec!

The serial interface at the remote site was clean and there were no errors on any interfaces. The problem appeared to be with the carrier. I did the standard routine, place a trouble ticket and request intrusive testing to see if they could detect any problems. The results: no problems detected. Yet we still were experiencing severe latency to this site, what's going on? I decided that we needed to collect some hard numbers to see what was happening, a simple ping output was not going to cut it to troubleshoot this problem. (I should mention that we have 20 other sites on the same MPLS network for over 3 years and none of these sites ever reported a problem this severe.)

Two great opensource tools helped track and visualize the problems:
  • Cacti (a PHP web front-end for RRDTool)
  • Smokeping (a deluxe latency measurement tool, written by the authori of RRDTool).
I had been using Cacti already to monitor the various system and network SNMP counters on much of our infrastructure so I already had a head start on seeing how much data was going into and out of our troubled remote site.

Smokeping was a tool that I had used years ago and had not setup recently. The installation is simple and detailed on the Smokeping web site. A configuration was quickly thrown together to collect and plot the latency for all of our MPLS connected sites. A few mutli-host graphs were built to compare the troubled site with other sites that were both 'newer' and with sites that were geographically farthest from the HQ. Now we wait for the problem... Waiting for this problem did not take long. Once the problem began to occur I took a look at the cacti graphs for our serial interface usage on the remote router (it was pegged at 1.5Mbit inbound) and cross referenced that with our Smokeping latency graphs. Bingo! When utilization spiked so did the latency. Well this is no big surprise! The problem with this conclusion is that all of our other 20 locations will also experience periods of circuit saturation but do not experience high latency during these times.

To test the theory of highcircuit utilization == high latency I needed to perform some additional tests. Another opensource tool to the rescue... Netperf. Netperf is a nice little application which will let you send various traffic patterns from a Netperf client to a Netperf server and provide very useful statistics. I was only interested in providing a consistent pattern of traffic from our HQ site to any one of our MPLS connected remote sites to see if they also experienced high latency during high circuit utilization. I ran tests against numerous sites at different times of the day and was very effective in maxing out the remote circuit from the inbound direction from our HQ. However, all the others sites only experience minor bumps in latency during these high circuit utilization periods. For example a site with an average ICMP echo reponse time of 40ms would jump to 60 or 80ms, max. The problem site would jump from 30 - 40ms to 1000 - 1500ms!

At this point I was convinced that we either had a configuration or hardware problem on the remote sites router or we had a severe problem with the carriers internal MPLS network. The network engineer who deployed the network at the remote site poured over the configuration and a case was opened with Cisco TAC (we use all Cisco gear). No configuration or hardware problems were discovered. Now I had the unfortunate privilege of trying to convince the very large carrier that the problem might exist in their network.

To cut this very long and painful story short, it took almost a week and a half of 'working' with the carrier to get them to recognize that we had a severe discrepancy with how this sites circuit behaved when under high load (show interface showed a load of 253/255) versus all of our other sites. Once they recognized the problem and escalated the ticket many times over and backend engineer got on the phone during a conference call and cleared the issue up in a matter of minutes... What was the problem you ask? A problem with the routing in the cloud? a faulty card in the acces router? Nope. The proble wmas that the hardware in the carriers access router was recently upgraded from a T3 to an OC48. That is a significant increase in hardware and bandwidth capacity. So why would we see poor performance on better hardware...?

The problem is that the bottleneck for the network had just been moved over to us from the carrier. The carrier is now able to get out packets from HQ (connected via a T3) to the remote sites access router faster than ever before. The problem is that we are basically taking the volume of Niagara Falls and trying to pipe it into a straw for general consumption. In researching this a little more it was also discovered that our other locations were still riding off of older access router hardware and that was why we could not reproduce the problem.

Monday, October 22, 2007

Rails: Want to change your db server... done.

I have started to develop a few pet projects using Ruby on Rails now that I have my head around the Ruby language a little more. One thing that impressed me was the ability to maintain your database schema via the db:migrate Rake tasks. Here is my two minute example of how this worked wonderfully for me...

I grabbed a very old P4 with 256MB of RAM out of the basement and cobbled together a machine that would actually boot and run. I then loaded the beta release of Ubuntu Server 7.10 (I haven't used Ubuntu too much so this was a good chance to learn this distro while also learning Rails). I loaded up the typical LAMP stack as part of the install.

I started a new site and loaded up about 5 db migration tasks into the standard MySQL db (with test data of course) when I decided that MySQL was overkill for me, a single developer just tinkering. So I thought, could I switch the whole thing over to something more lightweight without a lot of hassle? According to the Rails folks you can.

Simply adjust the config/database.yaml development section to include the new db adapter of your choice. (I decided to go with sqlite3 as there is no server and the db is a single file plus very easy backups). Then I ran rake db:migrate and bingo! All my db and test data was loaded into a sqlite3 db. I then shutdown the MySQL service (gaining back some precious RAM) and clicked around on the site. It worked, very simple, and very cool.

Saturday, September 29, 2007

Using Vi to edit textareas in Firefox... YES!

I have started to use TWiki at work to develop a more dynamic documentation store for my team and it beats the pants off of editing heavy MS Word docs. My problem in general with creating documents is that I spend a fair amount of my time in vi (on HPUX, Linux, Windows, everywhere; sometimes I wish I could use it everywhere) and then It happens... I switch to an application like Outlook of Word and I have to tell my brain that I am no longer in vi. What usually happens is that I end up with a lot of :w's peppered throughout my documents and other control sequences. (Accidentally hitting ESC in an email to switch to command-mode has caused me to loose more than a few messages.) So with this brand new Wiki I thought to myself can I somehow do my doc'ing via vi too? And you can!

Mozex to the rescue: Mozex is a Mozilla extension which allows you to use external applications to: edit content of textareas, view-source, mail-to, etc. I am interested in the editing of textareas with vim.
Here is how it can be done:

1) Go to the mozex site and install the latest version (development version to get Firefox 2.x support). Restart the browser.

2) Right-click on a page (anywhere) and select mozex -> Configuration.

3) Select the textarea button and then fill in the path to your favorite editor (vim) and make use of the %t to pass in the path to the temporary file. I also setup a hot-key combo as I really try to avoid using a mouse (why must you take you hands off of the keyboard?!).

4) While in your textarea press the hot-key combo and up pops vim!

Other tips: Start vim via a shell script that also accepts the %u which you can use to then set other vim preferences. I.E. If the URL is from your favorite Wiki then you could setup some custom syntax highlighting when launching vim. There are so many fun combinations, enjoy, and many thanks to the mozex team!

Friday, September 14, 2007

DIY Protocols are evil!

A recent project at work opened my eyes as to why you really have no good business designing your own low-level interprocess protocol. Given that there are so many good protocols already available, HTTP, SOAP, etc. why would a person be compelled to want to write their own? I should clarify what I mean by low-level in this context. In this case I am referring to a protocol which is implemented directly on top of the TCP/IP protocol. I am not arguing against higher level protocols or API's that applications expose, this is necessary nowadays unless you live in a bubble.

Anyway... I was faced with modifying this existing protocol which was implemented via a client and server communicating to each other through a C structure. The protocol went something like this:

connect to server
build struct and insert some 'request' data
write struct on socket
wait for response...

Listen/accept/fork for each client connection
read struct from socket
parse request
write response in same struct's 'response' fields
write struct on socket

Well to some of you this may seem OK, it is simple enough, very straight to the point. The problem lies in the underlying implementation of the structure. When the code is compiled to a particular machine, let's say a big-endian system, you end up with a specific byte ordering for any structure members which are integers. The problem really occurs when your server and your client are compiled on machines which have different byte orderings. What you end up with is a system sending what it thinks is a perfectly valid integer value and the other end may not order the integer bytes the same way (in some cases your number gets really big, or really small - depending on the signed-ness). Wikipedia has a good example of what all this byte-ordering means.

Anyone who has developed a network application is probably already aware of this byte-ordering difference that you must account for. That is why the de-facto byte-ordering for all network communications is big-endian. All good network programs should explicitly convert their integer data to the proper ordering before sending and then back to the native ordering on receiving (see ntoh and hton type library calls).

The second problem, which actually turns out to be the more painful, is that of byte-alignment. The concept is that different machine architectures may have a preference on how to pad the members of our C structure so that the memory regions are aligned with the integer (and shorts) of our structure. A very detailed explanation is also available on Wikipedia. What can happen is that you manually add up the size of each member of your structure and get a sum; you think great I can tell the client that the server will read exactly X bytes and write back exactly X bytes. But sometimes you would be very wrong. So to check your math you execute the sizeof() function on a variable of the type of your structure and get another _completely_ different number! The reason for this is that you compiler knows that the alignment must be adjusted to line up on a particular boundary. So you actually end up reading and writing X + y alignment bytes of data. The extra bits of data are not accessible when you reference the structure members and you never know they are there.

The project I mentioned earlier actually had the luxury of always being implemented on systems whose byte-alignment agreed and where both the client and server were implemented in C. Then the day came when a .Net application needed to communication across the wire to the server. Another team worked on the .Net implementation of a client and it was a struggle to make sure that we properly communicated the exact layout of the structure, including the padding bytes. A second problem arose when the .Net side also wanted to send over its version of a 'Character' array. What we got was a 16bytes per character array and not the old-standard 8bytes per character array.

In the end it all worked out and the systems are communicating nicely. In hindsight it probably would have been less time consuming to just toss the old DIY protocol out the door and use a more established standard protocol.

Friday, August 10, 2007

Attacking LEAP with ASLEAP

The ASLEAP tool was written by Joshua Wright and is designed to crack passwords used by wireless networks being secured with proprietary Cisco LEAP (Lightweight Extensible Authentication Protocol). I had a chance to see a BOF session at SANS San Diego 2007 that was presented by Joshua and it was amazing. He went non-stop for at least an hour on many problems with Bluetooth and even 'toyed' with sending HTML to his phone via Bluetooth to show that it could be vulnerable to a XSS attack.

Anyway, back to ASLEAP... Here are some of the features that ASLEAP has to offer (Check out for a complete list, plus PPTP support).
  • Recovers weak LEAP passwords (duh).
  • Can read live from any wireless interface in RFMON mode.
  • Can monitor a single channel, or perform channel hopping to look for targets.
  • Will actively deauthenticate users on LEAP networks, forcing them to reauthenticate. This makes the capture of LEAP passwords very fast.
  • Will only deauth users who have not already been seen, doesn't waste time on users who are not running LEAP.
  • Can read from stored libpcap files, or AiroPeek NX files (1.X or 2.X files).
Here is some information on what LEAP is: LEAP allows for clients to reauthenticate frequently; upon each successful authentication, the clients acquire a new WEP key (with the hope that the WEP keys don't live long enough to be cracked).

AFX RootKit (2003 & 2005)

Continuing my studying for my GCIH I will move on to the next Rootkit: AFX RootKit (2003 & 2005). My review of this little gem will not be as detailed as the adore-ng rootkit as I do not have a separate Windows system to trash as a full test. What I will note are the details of the rootkit that are posted on many places already.

The rootkit hides the following:
  1. Processes
  2. Handles
  3. Modules
  4. Files & Folders
  5. Registry Values
  6. Services
  7. TCP/UDP Sockets
  8. Systray Icons
The installer copies itself to the system directory and extracts 2 DLL files from it's resources. It saves the files as "iexplore.exe" and "explorer.exe". The first dll is loaded into "explorer.exe" which then installs hooks contained in "explorer.dll". The included ReadMe.txt has detailed instructions on how to install the rootkit.

Detection of the rootkit can be accomplished by the presence of
iexplore.dll and/or explorer.dll.


Monday, July 30, 2007

Explorations with adore-ng

As part of my studying for the SANS GCIH certification I thought I would take some time to really get to know several of the tools mentioned on the Certifications Bulletin. The first item is the adore rootkit. I will be studying adore-ng-0.56 on a RedHat 9 install. What I hope to understand better through this process:
  1. How to compile and 'install' the adore-ng rootkit (read the README, very well documented).
  2. How to hide processes and files.
  3. How to hide network sockets from netstat, etc.
  4. How to attempt to detect from a remote system.
  5. How to attempt to detect from a compromised system.
These posts are more or less a study guide for myself and anyone else who may stumble upon them. The authors of these tools are the real geniuses, I am simply attempting to better understand what these tools are and what I can do to protect against them in the 'wild'. Therefore, I will reemphasize what is written in the adore-ng README:
Only *YOU* are responsible for your own actions. So if you are
dumb enough to own machines and install this software on it,
only you can be blamed for it.
Task 1: Compile and Install adore-ng
After downloading the the adore-ng tar ball it was a simple matter or running ./configure and then answering the following question.
Since version 0.33 Adore requires 'authentication' for
its services. You will be prompted for a password now and this
password will be compiled into 'adore' and 'ava' so no further actions
by you are required.
This procedure will save adore from scanners.
Try to choose a unique name that is won't clash with filenames in /proc.
Password (echoed)
Next compile adore-ng. The following warning was thrown by a 'make' after configure:
/usr/include/linux/modversions.h:1:2: #error Modules should never use kernel-headers system headers,
/usr/include/linux/modversions.h:2:2: #error but rather headers from an appropriate kernel-source package.
/usr/include/linux/modversions.h:3:2: #error Change -I/usr/src/linux/include (or similar) to
/usr/include/linux/modversions.h:4:2: #error -I/lib/modules/$(uname -r)/build/include
/usr/include/linux/modversions.h:5:2: #error to build against the currently-running kernel.
So I had to modify the Makefile to use the preferred include path in /lib/modules/. I also uncommented the definition for CFLAGS+=-DREDHAT9 since this is RedHat 9 and w/o this the macro for_each_process will not be setup correctly. (These are probably RedHat 9 quirks.)

To load the adore-ng kernel module simply run ./startadore which will insert the adore module
followed by the cleaner module. I found the following article which explains the simplicity of the cleaner module: Anti-Honeypot Technology.

The first thing that I tried was an lsmod. And bingo, no adore module listed. So I thought, did it really work? A quick run of the ava utility with the 'I' option shows the following:

./ava I
Checking for adore 0.12 or higher ...
Adore 1.54 installed. Good luck.

ELITE_UID: 539905012, ELITE_GID=811693422, ADORE_KEY=qwerty CURRENT_ADORE=54

So it is installed.

Task 2: How to hide processes and files...
One of the first things I noticed was that the source directory was hidden. At first I thought it was removed or that I 'misplaced' it. An ls -l in /root/ (where I originally unrolled the tarball) shows no adore-ng directory? But if I cd /root/adore-ng/ then it is really there. You may not catch it, but the configure script does a chown to the elite UID and GID's for you. Thus you 'hide' the source. I tried a find /root/ and it also does not see it. This is because the adore-ng is in the Kernel and is intercepting my reads of the file system and doing checks on uid and gid. If both the uid and gid are not set correctly then the file will be listed. A simple test follows:

# touch /tmp/abc
# ls -l /tmp/
-rw-r--r-- 1 root root 0 Aug 1 11:12 abc

# chown 539905012 /tmp/abc
# ll /tmp/
-rw-r--r-- 1 539905012 root 0 Aug 1 11:12 abc

# chown 539905012.811693422 /tmp/abc
# ll /tmp/
total 0

# ll /tmp/abc
-rw-r--r-- 1 539905012 811693422 0 Aug 1 11:12 /tmp/abc

And we are now hidden. The ava utility appears to have the same functionality (actually runs lchown).

# ./ava h /tmp/abc
Checking for adore 0.12 or higher ...
Adore 1.54 installed. Good luck.
File '/tmp/abc' is now hidden.

To hide a process the ava utility will create and unlink a file in either /tmp/ or /proc (depending on the value of ADORE_LSM) with the following format: hide-PID or unhide-PID.

Here is an example using the ava utility to hide a shell process.

Here is a regular user's shell session:
$ ps
17321 pts/1 00:00:00 bash
19745 pts/1 00:00:00 ps

Now, using ava, we make the shell process invisible:
# ./ava i 17321
Checking for adore 0.12 or higher ...
Adore 1.54 installed. Good luck.
Made PID 17321 invisible.

Back over to the original shell session and now nothing is listed, creepy:
$ ps

What if we look for it specifically...?
$ ps -p 17321

This also appears to have hidden all child processes of the originally hidden process.

Task 3: How to hide network sockets from netstat, etc...
In this section I will explore how to use adore-ng to hide a network socket. The README for adore-ng explains that you need to update the adore-ng.h file to include any port numbers that you would like to have hidden. The array HIDDEN_SERVICES should be updated to include the decimal values for the ports you want to hide. By default ports 2222 and 7350 are both hidden. To test that port 2222 is hidden I will start a netcat listener and attempt to detect it on the compromised system.

Start a listener on port 2222...
$ /tmp/nc -l -p 2222

Try to detect it with netstat...
# netstat -an |grep -q 2222 || echo NADA

What about using lsof...?
# lsof -i :2222 || echo NADA

To prove that it is really alive and listening...
# telnet localhost 2222
Connected to localhost.
Escape character is '^]'.

Task 4: How to attempt to detect from a remote system...
There are a few ways that I can think of to attempt to detect the installation of the adore-ng rootkit from a remote system. These methods are actually going to attempt to detect the changes that are likely to occur to a system after being owned.
  • Network Intrusion Detection System: A properly configured and tuned NIDS will most likely tip you off that something is awry. You may not catch the initial infection point of your system, especially if this is an insider attack. What you should catch is anomalous network behavior; as in traffic on ports that have never been seen on this host before. Let's assume that the attacker leaves the adore-ng.h header alone and opts to use port 2222 to shovel a shell to a remote system using netcat. Your NIDS should alert you that all of a sudden there is a new pattern of traffic being generated to/from this host on a port that you never expect to see traffic on.
  • Network mapping: An extension of the above NIDS approach is to maintain an up-to-date inventory of the network ports that all of your servers are listening on. This can very easily be accomplished by using nmap and comparing the results. Check out the nmap related projects page for some tools which do this already.
  • Remote Logging: I feel that all systems should utilize some type of remote logging facility. Whatever method you use please be sure that it is secure and reliable (syslog-ng is a good start). Having you logs immediately forwarded off to another host (as in real-time) will allow you to analyze signs of a possible attack and may tip you off to any subsequent rootkit installations.
Task 5: How to attempt to detect from a local system...
Attempting to use anything on a system which has been compromised is extremely dangerous. It is especially foolish to trust that the system will tell you the truth when you ask it what is going on (as was demonstrated above in the hiding of files, processes, and ports). However, if you are in a crunch and must only use the system that you suspect is under the control of adore-ng then here are some tips that I have discovered while playing with this rootkit.
[Please be aware that a system which has been owned will likely have many modifications done to it to prevent you from knowing the truth. In my example I have only modified the system with the adore-ng rootkit in the most basic ways.]
Above I demonstrated how to hide a process from a ps listing. While I was playing with this I wondered if the hidden process would still accept a kill -0 (not kill 0, notice the "dash zero"; as in send signal number 0 to a process)? The method I am about to explain is based on the fact that in my test environment a hidden process would accept this signal and therefore reveal to me that something may be amiss.

Detecting hidden processes with kill -0
In the above examples I hid a process (bash shell session) with PID 17321. Neither a ps listing or a browse through /proc/ revealed any signs of the hidden process.

But what if we try to send it a kill -0 signal?

$ kill -0 17321 && echo "it is alive"
it is alive

As an example here is what happens when the PID is not alive:

$ kill -0 1732199 && echo "it is alive"
-bash: kill: (1732199) - No such process

As an example of what happens when you try to kill a PID you do not own (init in this case):

$ kill -0 1 && echo "it is alive"
-bash: kill: (1) - Operation not permitted

So in theory we could detect a hidden process by comparing a process listing output by ps (or looking into /proc) against the results of a kill -0 against every possible system PID (this can be discovered via: #cat /proc/sys/kernel/pid_max).

Detecting hidden processes via /proc (when you are hidden too)...
Something else that I discovered is that even though the PID does not appear as a directory in /proc/ (ie. ls /proc/ will not list it) you can still reference it (as in ls -l /proc//cmdline will produce valid results). [This appears to only work if the user investigating /proc is running their shell as a hidden process also. The kill method does not appear to have this limitation.] This could be used in the same way as the above kill method to search for processes which are 'hidden'. I have written the following shell script to 'detect' hidden processes that should be run from a hidden process itself.
# Attempt to detect hidden PID's using /proc guessing

MAXPID=`cat /proc/sys/kernel/pid_max`

PIDS_IN_PROC=`cd /proc/ && ls -d [1-9]*`


echo "Searching for hidden process [`id`]..."

# populate PID Array for all 'v'isible processes in /proc
for i in $PIDS_IN_PROC

# now guess processes
for p in `seq 1 $MAXPID`
echo -ne "Checking...$p\r"

if [ "${PID_ARRAY[$p]}" = "visible" ]; then
elif [ -d /proc/$p ]; then
echo -ne "\t\t\tFound $p\r"

# Report the findings
echo "We discovered a total of ${#PID_ARRAY[*]} processes ($num_hidden) are hidden"

if [ $num_hidden -eq 0 ]; then
exit 0

echo "The following PIDs appear to be hidden:"

for p in `seq 1 $MAXPID`
if [ "${PID_ARRAY[$p]}" = "invisible" ]; then
echo -ne "$p\t"

echo -e "\nComplete"

Of the above two process detection methods I feel that the kill -0 method may be more readily available as it does not appear to require that the user be root or running the tests under a hidden process.

Detecting hidden sockets...
If the process which is setup to listen on a hidden socket is not also itself hidden you can try to uncover it with the following technique.

# lsof | grep 'identify protocol'
nc 11334 aaron 3u sock 0,0 21512 can't identify protocol

Using chkrootkit
Your best method of detection may be to try the chkrootkit set of tools. The chkrootkit sites states that the suite of tools effectively detect the adore LKM. I downloaded and compiled chkrootkit-0.47 and here are the results of a lkm detection:

# ./chkrootkit lkm
ROOTDIR is `/'
Checking `lkm'... You have 2 process hidden for readdir command
chkproc: Warning: Possible LKM Trojan installed

In summary there are several tenets that a good system administrator must allows follow
that will help prevent an intruder from ever being able to install a rootkit such as adore-ng. A couple of the more important rules that I highly recommend is (1) Know thy systems and (2) Practice defense in depth. By following these two simple rules a good system administrator will typically be ahead of an intruder and immediately notice that something wrong is going on with their systems and networks.

Sunday, July 15, 2007

Exploring ActiveMQ via Stomp

Building on my previous post where I explored ways to get a legacy Progress application system to talk to a modern .NET based back-end, the next possible solution will involve communicating to a JMS system called AtiveMQ.

I actually came across the Apache Foundation's ActiveMQ project while searching for a message queuing back-end for a Ruby project. I was in search of a scalable message queue that was platform agnostic (I had considered SysV message queues but they are not easily portable from one OS to another). The ActiveMQ site says it all: "Apache ActiveMQ is the most popular and powerful open source Message Broker." Wow! So how can I leverage ActiveMQ to help one system talk to another when they cannot do so natively? This is the next problem I set out to solve...

The general idea is to have a legacy Progress system send and receive messages off a message queue and at the same time have a modern .NET system send and receive messages off of a message queue. The two systems do not need to talk to each other directly, they only need to talk to the messaging system.

The first step was to determine how could a legacy Progress system communicate with the ActiveMQ broker services? And this is where ActiveMQ really shines, as the web-site states: "Supports a variety of Cross Language Clients and Protocols from Java, C, C++, C#, Ruby, Perl, Python, PHP" and then there is this quote under cross languae clients: "Stomp support so that clients can be written easily in C, Ruby, Perl, Python, PHP to talk to ActiveMQ as well as any other popular Message Broker". Stomp!? What is Stomp? Check out the Stomp web-site and you will see that Stomp stands for: "Streaming Text Orientated Messaging Protocol". Take a look at the protocol definition and you will find that it is so simple that you could literally communicate to a Stomp enabled system via telnet. The folks at Stomp have produced a connector that is being used in the ActiveMQ message broker to allow a 'simple', yet fairly powerful, connection to be established to a really great JMS system without too much hassle.

So now what do we do? We have discovered a really cool JMS system that offers a simple Stomp connector and yet we are stuck with an old Progress system which has no network socket capabilities, ARGH! Well, you browse over to the Stomp web-site and check out their C client (since we already know how to extend Progress with C using the HLC mechanism this should be part of the solution). The libstomp component is only available via SVN and they only have directions on compiling on OS X with the X Code2 IDE but the code is available and should save me a couple of days worth of work. libstomp requires the APR (Apache Runtime Portable Library) so step one is to download and install this, if your development platform does not already have it (my platform, HPUX, did not but it compiled and installed w/o any problems).
[The APR library is really cool and I plan to post about it soon, this library is new to me so I was like a kid in a candy store, hyper.]
Then I compiled the stomp 'library' (only a single source and header file) which is more like just another object to be linked in later. To ease the Progress application development efforts the Stomp API that I built via the HLC was fairly basic: a function for each of the major Stomp message types that automatically built the Stomp frames and transmitted them on the wire. For simplicity and appeal I chose to keep the socket and frame level details out of Progress.
After a couple of days of massaging Progress and C code around I was able to subscribe to a JMS style queue being hosted by ActiveMQ and send and receive messages. In order to test a client other that Progress a small Ruby client was constructed to help simulate the final environment (two distinct applications sharing data via a JMS queue). Here is a small client that will send a sample message to a queue name that is given as an argument

require 'stomp'

conn = "", "", "servera", 61613, true

puts "Connection Open: #{}"

destination = $*[0] if $*[0] != nil

for i in 1..10 do
conn.send destination, "Some text to send to the consumers out there. #{i}", { "receipt" => 'something-for-me' }

I had no problems pushing 5000 messages onto the queue in a very short amount of time with this client and the ActiveMQ broker process is running on a 2-way HP9000 L-class system. I was then able to receive the messages back off of the queue almost as quickly. Here is a sample of the Progress code that is using my libstomp based API:

/* Subscribe with automatic acks. */
CALL STOMP_SUBSCRIBE "/queue/testq" "auto".

ASSIGN stomp_get_frame_tout = FALSE.

CALL STOMP_GET_FRAME 5. /* Try to read a frame, wait up to 5 seconds */

IF stomp_get_frame_tout = TRUE THEN
DO: /* unsubscribe and quit */
/* Polietly let the broker know we are leaving */

/* Read in the frame body */
create tt.



So we now have an API from Progress which can interact with a Stomp connected message broker that is JMS compliant. The next portion was to get a .NET application to speak to the same ActiveMQ server and interact with the queue. A co-worker of mine worked this piece out using the Spring.NET project (hopefully he will post some details on the client, keep watching: The Space of .Nethingness!).

Next time we will explore possible solution #3.. A fully featured ESB from Bostech Corporation. Their ChainBuilder ESB product is opensource, JBI compliant, and uses Eclipse as a GUI front-end.

Wednesday, June 27, 2007

cURL breaths life to old system

In my previous post I hinted that I would next write about how to get several legacy systems to send data to a modern .Net based application when the legacy systems only know how to talk to each other. This problem is based once again on the aged Progress applications that company A uses.

The Problem: The native language lacks the ability to make any network based connections, unless it is to a database that is of the same flavor. So what is a person to do when they want to integrate with more modern systems, many of which are being built using WCF or plain old web services? You could pack your bags and go home or better yet, start injecting more custom C code into the old dog and teach it a few new tricks.

Possible Solution (Option 1): If you need to talk to plain old web services that operate on GET's and POST's then cURL,or more specifically libcurl, is a very logical choice. cURL is just amazing; a command line tool that can do everything that a web browser can do but without the GUI. libcurl gives you the ability to put this power into your own applications. So with option 1 a libcurl based API was introduced into the legacy language.
This initial introduction of libcurl was actually done by a previous employee a while ago but it was time to update the API and support some new features. One feature that was desperately needed was support for SSL based connections (provided by the OpenSSL libraries). The next feature which we needed was the ability to pass authentication credentials with the POST requests; specifically credentials to access Windows domain systems. (I didn't mention it, but the legacy applications all run on HPUX.)
So now we have the ability to POST data to any standard web service via http/https including optional login credentials. This works very well for situations where the communication can withstand or requires being synchronous.

Next time... Possible Solution (Option 2): A JMS based messaging infrastructure...

GPG to the rescue (lipstick on the pig)

So I have been a little side tracked from my Ruby ISO8583 work for the past couple of weeks. But never fear I have been getting familiar with a really amazing API for the GPG project known as GPGME (GPG Made Easy). So why would a person be interested in this? Glad you asked.

Problem: Company A has a very, very old POS application that was written entirely in-house. The programming language/database that the app is written in is Progress (something like a 4GL). I'm not surprised if you have never heard of it. One problem is that the version of the database/language that the app is written in is now over 10 years old. Well company A needs to capture some PCI data for a very short period of time during the transaction and this data needs to be sent off to various backend systems for settlement, etc. (so this is not entirely unrelated to the ISO8583 work). The problem is that the language/database has no way to handle this, period.

The Solution: What Progress does offer is a HLC (Host Language Call) mechanism which allows custom C routines to be written and called from the application; complete with a way to pass data back and forth through shared variables and temporary tables. This is actually very cool! So what we are going to do is let the native Progress applications call a GPG encryption routine (we will be using asymmetric encryption in this case) and bingo, we have encryption in a older-than-dirt application.

The first step was to investigate the GPG libraries and see how we might integrate them and develop a very simple API for our app developers to use. And this is where GPGME came into the picture. Someone has already done a lot of the work for us and created an "Easy" API to the GPG engine. The post compile tests which are bundled with the GPGME source came in very handy when it was time to actually write our routines (it's great to see actual tests).

A few days of hacking together some C code and I had injected an encryption routine into our antique language/database.

A basic architecture and sample use case for this project would be as follows:
  1. Create a public/private key pair somewhere super-safe. Guard the private key with a pass-phrase and with your life.
  2. Distribute the public key to the legacy application systems. These applications will then use the public key to encrypt sensitive data which can be stored locally (only if you really need to) or shipped off to another super-safe system. The original 'clear-text' data must me removed after it is encrypted or the whole process is worthless!
  3. The 'super-safe' system which has the private key will then be able to access the sensitive data when it needs to.
  4. Because the legacy application systems only have the public key they cannot even decrypt the data that they themselves encrypted.
For all those out there who spend way too much time holding a well past their prime system together, I have this to say... I put lipstick on the pig. The lipstick may be pretty but it is still a pig.

Next entry teaser: " do you get several legacy applications to send data to a modern .Net based system when they only know how to talk to each other..."

Saturday, June 09, 2007

Ruby...where have you been all my life?

Ruby is beauty,
Without Ruby what to do,
Programming is dead.
OK, so that may be a Haiku if I counted correctly?

Anyway... About a year ago I was in a Barnes and Noble and saw an issue of Linux Journal which had a cover story about Ruby. I hadn't read Linux Journal for quite a while and thought that this looked like a good issue to pickup. And that was my initial exposure to the what Ruby was doing, including Ruby on Rails. I then started to read through Why's (Poignant) Guide to Ruby, a really cool mixture of programming and comic art. Next came the purchase of the Pragmatic Programmers: Agile Web Development with Rails. The problem for me was that I didn't know Ruby well enough so some of the examples were not sinking in. So the next logical step was to get the much acclaimed Pick Axe book (Pragmatic Programmers: Programming Ruby). What a great book; every time I have a question about something in Ruby I am pleasantly surprised to see that it is well covered in the Pick Axe.

I have been struggling to find a place at work that I could use Ruby. Since I am not an applications development person I have been reserved to thinking about Ruby as the general purpose toolbox to assist with simple systems and network administration tasks. Tasks that I would typically use a shell scripting language for (i.e. Bash or the Korn Shell - We use HPUX so ksh is very popular) or even to replace Perl. First off, my programming background is mostly in the non-object oriented areas using languages such as C and early PHP. So if I am responsible for developing an application at my work it will typically be written in C (we have nobody else who can program in C that I know of, or they won't admit it). Often these projects are geared toward enhancements of existing 3rd party applications or to provide legacy system integration to more modern systems (I'll post more general info on these projects later...). Therefore nothing satisfying has ever come about so the idea of using Ruby was shelved...

And then it happened... As a retail merchant we have a custom application that is used by our POS systems to authorize customer credit cards using the ISO8583 protocol with various processors, including American Express. The time had come to make some significant changes to the AMEX authorization backend process that all of our business systems utilized and my initial thought was to cvs checkout the C code and start hacking away. But then it hit me, could I use Ruby to do this? I had never considered Ruby for this kind of heavily used extremely critical application, could it scale, could it do the bit pushing that I needed (I'm sure that it can but could I mold it the right way)? So I have decided to give it a shot. Another benefit to trying Ruby is that our .Net development team could probably pickup the code and understand it a lot better than the old C code, so maintainability should be improved through the use of a modern OO language (especially with the future of IronRuby and Ruby.Net).

I plan to post on my experiences with developing a Ruby ISO8583 processing application over the next couple of weeks, so check back in often...


Monday, May 07, 2007

Broken Jeep...Part II

Alright...The Jeep is back in business. As promised I took some photos of the old parts and will post the details of the replacement process. First off here is the broken A/C Compressor still installed in the Jeep. You can see the bad pulley on the front of the Compressor which is facing the fan shroud.

To remove the compressor I had to first remove the bolt attaching the A/C hoses. Here is where you need to be ready, because any A/C gases (which are now Ozone friendly) that are in the system will rush out. In my case this wasn't too bad since the Jeep is eight years old and the A/C hasn't been very cool for the last year or two. Next I had to simply remove the four bolts at each corner or the compressor and disconnect the power connector (a simple clip at the rear of the unit). The unit then lifts right out.

The new compressor goes back in the same way the old one came out. Here is a picture of the new unit installed (nice and shiny). I also installed a new serpentine belt since I was still running on the original.

The only thing left is to check the system for leaks and refill the A/C. As I haven't used the A/C that often and prefer to have the windows down I have opted to leave the power connector disconnected and may never have the system recharged.

Sunday, May 06, 2007

Broken Jeep...Part I

I have a 1999 Jeep Cherokee that has served me very well (has about 150,000 miles). About two weeks ago the serpentine belt began to make a very loud noise and a small amount of smoke would come out from under the hood. This was a problem that I could not ignore and the thought of looking for another car made my stomach turn. I had borrowed my wife's car to get back and forth to work and that was not going to work for much longer as she needs transportation too.

So after a few minutes of tinkering with it I discovered that the pulley attached to the A/C Compressor was shot. The pulley was dragging and grinding when you tried to move it by hand and obviously needed to be replaced. Off to the local auto-parts store I went to get a replacement pulley. No luck...the pulley is not sold separately, you have to buy the entire A/C Compressor with the pulley and clutch attached. Total cost = $260.00! So before I go replacing it I needed a second opinion. A friend of mine does some auto work and offered to take a look at it and confirmed my findings, the pulley was shot.

The friend mentioned looking at some local junkyards to see if I could find another compressor which should cost much less. I called around to 2 or 3 places and no luck. I next decided to look online and see if anyone had a remanufactured compressor for luck. However, I did stumble across another interesting item: an A/C Bypass Pulley. Basically this is a pulley attached to a bracket that fits exactly in the space that the A/C compressor lives. You remove your existing compressor and put this in place and bam problem fixed. You won't have any A/C anymore but the car will still run all the same (I don't like the A/C anyway). Only one problem with this...nobody had a bypass pulley listed for my year of Jeep.

So it's off to the auto-parts store again, this time knowing that I have to buy the compressor if I want to get the Jeep fixed before I have to go back to work on Monday. I will try to take photos of the replacement process and post them after the work is all done...

Monday, April 09, 2007

SANS 2007 San Diego

First a little professional background information and what I do for a living...

I work in the IT department for a medium size retail chain who specializes in computers and electronics. My responsibilities include managing a team of system and network administrators. This also includes managing the IT security for my organization. Security is my true passion. Unfortunately, like most mid-sized IT shops it is not a primary focus and my time is split between keeping the wheels turning (i.e. making the co. money) or doing what is 'right' from a Security perspective.

So back to SANS... In February of 2006 I attended a small SANS class for MGT414 (CISSP Certification Preparation); I later gained CISSP certification in August 2006. So attending SANS 2007 San Diego was my first real security conference and it was amazing! Anyone interested in IT security should seriously consider attending a SANS event. The conference was very eye opening, so many people talking and working on security. Some of the people I talked to are from organizations that are very similar to mine; they wear several hats and are all playing catch-up while still trying to shoe-horn in Security wherever they can.

I enrolled in two tracks at SANS: i) AUD521 (PCI Compliance) and ii) SEC504 (Hackers Techniques, Exploits and Incident Handling). What follow are my opinions on each of the tracks.

AUD521 (Meeting the Minimum Standard for Protecting Credit Card and Other Private Information PCI CISP: The Visa Digital Dozen):
I enrolled in the two day AUD521 track because my organization is a level 2 merchant and the PCI standard is a very interesting animal that I wanted to learn more about and see how other IT folks were working with/through it. This track was taught by the courses author, David Hoelzer. This was a two day course which worked its way through the 12 steps of the PCI Data Security Standard. It was great to talk to the other students and see how their organizations were working toward becoming PCI compliant (or in some cases not doing much about it).

SEC504 (Hackers Techniques, Exploits and Incident Handling):
This was the real meat and potatoes... Six days of non-stop drinking from the security fire hose. The track was taught by it's author, Ed Skoudis; the man was non-stop from beginning to end. I honestly do not know how SANS fit so much information into this course, it covered everything from old school viruses to cutting edge Virtual Machine (VMWare) hacking. Anyone who is interested in system and network security will love this course. The final day of the track involves letting us, the students, hack eight systems and capture the flags to unlock the 'secret'. It was awesome! Some of the class formed teams and went after it while I decided to go at it alone. In the end I placed 3rd overall and 1st as a solo. The prize... an autographed copy of Ed's Counter Hack Reloaded. I was going to buy this book once I got home so winning it was great.

In summary... SANS training is really worth it. The tracks are great and at a large conference you will also get the benefit of attending many BOF's (Bird of a Feather) sessions in the evenings.

Friday, September 01, 2006


Welcome to my Blog. Many exciting things will begin to appear here.