If you run a Linux system using the RPM-based package management system, you might like to give your system a security check following news from the Internet Storm Center (ISC) of a SSHD rootkit affecting such distributions.

To quote from the ISC article:

The rootkit is actually a trojanized library that links with SSHD and does *a lot* of nasty things to the system.

At this point in time we still do not know what the initial attack vector is – it is unknown how the attackers get root access on the compromised servers that is needed to change the legitimate libkeyutils library with a trojanized one. We are, of course, keeping an eye on the development and will post a new diary or update this one if we receive more information about the attack vectors.

The trojanized library is very, very nasty. Upon execution it performs a number of actions, as described below.

The code first deobfuscates the text strings needed for it to work. The original text is only XORed so this is very easy to retrieve and the deobfuscated strings have already been posted on a lot of sites.

Once that has been done, the library sets up everything needed for it to work. It resolves symbols for the following functions which are used later: PEM_write_RSAPrivateKey, PEM_write_DSAPrivateKey, MD5_Init, MD5_Update, and MD5_Final. As you can already see, it is definitely messing up with the authentication mechanism.

Besides resolving the symbols, the library also hooks the following functions: pam_authenticate, pam_start and crypt as well as audit_log_user_message and audit_log_acct_message. By hooking these functions, the rootkit can modify the flow of the SSHD – as you can see, this is a user-mode rootkit, as it does not affect the kernel.

The main activity of the rootkit consists in collection of credentials of authenticated users. Notice that the rootkit can steal username and password pairs as well as RSA and DSA private keys, so no matter which authentication mechanism you use, if the target host is infected it will successfully steal your information. The hooking of audit_log* functions was done to allow the attacker to stay as low profile as possible – if the attacker uses the hardcoded backdoor password to issue any commands to the rootkit, no logs will be created.

The current version of the rootkit supports three commands: Xver, Xcat and Xbnd. The first command just prints the rootkit’s version; the Xcat commands print the collected information back in the session for the attacker while the Xbnd command allows the attacker to setup a listener.

Besides this, the rootkit can automatically send collected credentials to the attacker. In order to do this the rootkit has a DGA (Domain Generation Algorithm) implemented that will create random looking domain names in the .biz, .info and .net domains (in that order). It will then send a DNS packet containing collected credentials to the target IP address, if it was able to resolve it (meaning the attacker has registered that day’s domain). If no domains have been resolved, the DNS packet is sent to the hard-coded IP address, which in all samples we received was 78.47.139.110.

The rootkit itself looks very similar to the Ebury trojan which was detected back in 2011. In fact, I’m pretty sure that a lot of the code has been directly copied, however, the Ebury trojan patched the whole SSHD and required the attacker to change it.

This was easier to detect and prone to being overwritten with patching. The libkeyutils library, which comes as part of the keyutils-libs package is not changed that often so the chance of it being overwritten automatically is much lower.

If you run a RPM-based system you can check the integrity of the file with the rpm command:

# rpm -Vv keyutils-libs-1.2-1.el5
…….. /lib/libkeyutils-1.2.so
S.5….. /lib/libkeyutils.so.1
…….. /usr/share/doc/keyutils-libs-1.2
…….. d /usr/share/doc/keyutils-libs-1.2/LICENCE.LGPL

This will check a lot of things, the most important being the MD5 checksum so if you see the output as one above you have a trojanized library. Proper output should have all (and only) dots. Keep in mind that the RPM’s verification, of course, depends on the integrity of its database and the kernel itself.

We will keep an eye on the development and will update the diary accordingly – if you have samples or more information, especially on what the initial attack vector is please let us know.

Since the original article was posted, an important update has also been added, as follows:

Over the night (depending on where you live :), a lot of things happened. Thanks to one of our readers, Steve, I got access to a compromised server and hopefully might shed more light on what is going on here.

cPanel also posted a notice to their users that they have been compromised. What’s even worse, it appears that one of the main support servers in cPanel has been compromised which again means that the attackers got *loads* of passwords. cPanel recommends that passwords are changed, but keep in mind – if your servers are infected with the SSHD rootkit, the attackers will get your passwords/keys *anyway* (see below). So make sure that you check if your server has been compromised and that you clean it accordingly.

While analyzing the rootkit I noticed that, similarly to the Ebury trojan, it uses shared memory for communication between processes (and Steve e-mailed about that too). On Linuxes, you can check the status of shared memory with the ipcs command:

# ipcs -m

—— Shared Memory Segments ——–
key shmid owner perms bytes nattch status
0x7400845c 1769472 root 600 4 0
0x00000000 2129921 root 644 52 2
0x7400845b 1736706 root 600 4 0
0x00000000 2162691 root 644 16384 2
0x00000000 2195460 root 644 268 2
0x0052e2c1 2228229 postgres 600 10469376 16

In order to check the owner, use the –p flag:

# ipcs -mp

—— Shared Memory Creator/Last-op ——–
shmid owner cpid lpid
1769472 root 1975 1975
2129921 root 2931 2940
1736706 root 1965 1965
2162691 root 2931 2940
2195460 root 2931 2940
2228229 postgres 4011 6813
Now you can verify if SSHD owns any of the shared memory segments – normally it shouldn’t, so if it does you can try to inspect the system further.

Finally, unSpawn also wrote a ClamAV signature (a logical signature – so store this in a .ldb file):

RKH_libkeyutils.so.1.9;Target:6;(((0)&(1)&(2))&(((3)&(4)&(5))|((6)&(7)&(8))));636f6e6e656374;73656e64;736f636b6574;62696e64;746d7066696c65;77616974706964;646c636c6f7365;737472636174;737472637079

It should help detect malicious libraries. Since the attackers can store it in a different directory, make sure to use the find command to locate all potential copies. Something like this will work:

# find / -name libkeyutils*

Should sysadmins of Linux distributions that use the deb package management system also run some security checks? Your comments would be welcome below.