XOR DDOS Trojan Trouble

I have struggled over the past recent months with a clients environment becoming infected and reinfected with an XOR DDOS trojan.  The disruption and reinfection rates were costly at times.  The client in question is a small business with limited resources.  Since the systems would get reinfected, a baited system was eventually put into place to determine the incoming vector.  It was not proven, but believed that ssh brute force was the incoming vector of attack.  Once the attackers were onto the server, a root kit trojan was used.  A very intelligent one.  I highly recommend that anyone that gets nabbed by this trojan or one like it reinstall your operating system as soon as possible and execute my prevention steps outlined below.

However, there are some circumstances that require mitigation before available resources can be used for reinstall/replacement and prevention measures.  The client was in a situation where taking the system offline was not an immediate option.  I placed some really great links below. [1] [2] [3]   They review, analyze and fully confirm what we were experiencing was the same.  There were some minor differences.   However, they never really offered a short term mitigation path to follow.  Only somewhere in a comment on a forum (possibly one of the three articles below), did someone make a suggestion to change the file/directory attributes to assist in mitigation.  It was only a suggestion with no further follow-up.  Mitigation of this trojan was difficult as it was intelligent enough to always restart when it was killed, which included help from crontab entries every three minutes.  It was also observed the executable sometimes was hidden very well in the process table.

Basic MO

The victim server was CentOS 6.5 system with a basic LAMP setup, that offered ssh and VSFTP services.  Iptables was in use, but NOT SELinux.  It is my untested claim that SELinux likely would have prevented this trojan from taking hold.   I am not an SELinux user/expert so I was unable to take time to add it to this environment.   

The original malware was discovered in /lib/libgcc4.so .  This exe was perpetuated via cron in /etc/crontab every three minutes.  
( */3 * * * * root /etc/cron.hourly/udev.sh )
If crontab gets cleaned and an executable is still running, then the crontab will be repopulated on Friday night around midnight.  (remember that sometimes the exe was hidden well and was overlooked)

The malware creates random string startup scripts and places them in /etc/init.d/* .  You need only to execute ls -lrt /etc/init.d/* to discover some evidence.  Along with the use of the top utility, you can determine how many are running.  If the startups are deleted, then more executables and startup scripts will be created and begin to run as well.

The malware itself was used as a DDOS agent.  It took commands from a C&C.  The IP addresses it would communicate with were available from the strings output of the executable.  When the malware agent was called into action, the entire server and local pipe was saturated and consequently cut off from service.

Mitigation

The following steps were taken for mitigation.   The only thing that prevented the recreation of the malware was the use of the chattr command.   Adding the immutable bit to the /etc/init.d and /lib directories were helpful in preventing the malware from repopulating.  I put together the following for loop script and added the following IP addresses to IP tables to drop all communication.    The for loop consists of clean up of four running processes.  I used ls and top to determine the for loop arguments and PID's used in the kill command.   I through the following into a script called runit.sh and executed it.

For loop:
for f in zyjuzaaame lcmowpgenr belmyowxlc aqewcdyppt
do
   mv /etc/init.d/$f /tmp/ddos/
   rm -f /etc/cron.hourly/udev.sh
   rm -f /var/run/udev.pid
   mv /lib/libgcc4.so /tmp/ddos/libgcc4.so.$f
   chattr -R +i /lib
   chattr -R +i /etc/init.d
   kill -9 19897 19890 19888 19891
done

IP Addresses to drop all traffic:   103.25.9.228  103.25.9.229

Prevention

I now keep the immutable bit set on /lib on a clean system.  It turn it off before patching and software installs, in the event the /lib directory is needed for updating.  

I also recommend installing fail2ban and configuring it to watch many of your services.  I have it currently watching apache logs, ssh, vsftp, webmail, etc.   It really seems to be hitting the mark for prevention.   There is a whitelist feature to ignore traffic from a given IP or IP range.   This helps to keep the annoying customers from becoming a nag.

If you have experienced anything like the above, then please feel free to share.  This analysis is only scratching the surface.  The links below do a much deeper dive on this piece of malware.  

 
[1] https://www.fireeye.com/blog/threat-research/2015/02/anatomy_of_a_brutef.html
[2] https://blog.avast.com/2015/01/06/linux-ddos-trojan-hiding-itself-with-an-embedded-rootkit/#more-33072
[3] http://blog.malwaremustdie.org/2014/09/mmd-0028-2014-fuzzy-reversing-new-china.html


-Kevin
--
ISC Handler on Duty