Yesterday we started with the Eradication phase of the Incident Handling process. If the incident involves the usage of a rootkit, there is a first question that always needs to be answered:

To rebuild or not to rebuild, that is the question! ;)

One of the main internal ISC debates we had when we started planning the Cyber Security Awareness Month was discerning if today's and tomorrow's topics will lead to effective actions apart from rebuilding affected systems.

Almost a year ago we wrote about Family Incident Response,and provide a few links for rootkit detection tools. I took the opportunity to compile a good set of free Windows related tools at my personal blog, RaDaJo. Sometime ago, AV vendors started to add anti-rootkit capabilities to their main products, considering rootkits as another malware category. In fact, the key reason for this were the rootkit capabilities embedded in several malware specimens. On my post, I said something that I would like to ratify today:

Rootkits are one of the most complex and advanced malicious software components today, so the tools are mainly focused on the identification phase. The successful removal of a (kernel) rootkit from a system is often a really complex task.

When refering to the most prevalent type of rootkits today, kernel-based rootkits, the main issue is that even if you get full knowledge of the rootkit capabilities (imagine you are able to get a copy of its source code and had time to analyze it in-depth), the rootkit is hooked so deep in the system (at the kernel level) that the attacker was capable of performing any modification on the compromissed system. The main question in this real-world scenario is: Is it well worth to spend time trying to remove the rootkit and clean up the system versus rebuilding it from scratch?

Imagine an irreplaceable system was compromissed and a rootkit was installed. What methodology can you follow and what specific actions (and tools) can you take (and use) to eradicate the rootkit? There are a few situations were you can find yourself in this kind of scenario, dealing with high availability systems that have unique hardware components that cannot be easily installed on another node (for example, in the medical sector), or situations where a working backup is not available.

If you have been involved in incidents that required removing rootkits and have any anecdotes or ideas you can share, please send them to us via our contact page. Please, be sure to put something in the subject like "Security Tip, day 20" to make it easier for us to sort them. We will update this diary with your comments and thoughts throughout the day, so start sending them in.

--
Raul Siles
www.raulsiles.com

Thanks to all the readers submitting comments!

UPDATE 1:

From Mark:

The question to fix or rebuild is really a matter of system priority/data sensitivity/incident severity in any malware incident, in my opinion.  It would be sound advice to contain the incident (spread, communication, entry-point) of course.  Once contained, if the malware exhibits characteristics (captures, communicates, allows access, alters data) that put your critical assets (information or data) at risk, take a forensic image of the affected device(s).

If the asset is business or revenue critical and _must_ function, hopefully you have determined its Recovery Time Objective (RTO) and Recovery Point Objective (RPO) back in the preparation phase.  This gives you an operational estimate of the deadline and time available for analysis and restoration.  An estimate should be made for the length of time it will take to restore the system in a:
a)      clean rebuilt state,
b)      tape restore and tested state,
c)      cleaned (and preying) state.

A/V vendor software is notorious for not identifying and cleaning up everything from a malware infection, and if it has a rootkit, it probably has many other advanced features.  A business decision should be made by those that own the system as to the course of action to take at the critical point in RTO/RPO, and revisited as analysis is completed on the forensic image.

Again, in my opinion, if the system indeed contains critical, sensitive, or personal information, it is best to fail-safe, and restore/test or rebuild.  Your systems are only as useful and as reliable the data that they contain. You are going to pay for the incident.  The question becomes, do you want to pay now with the cost of downtime and overtime, or defer payment into the future at the cost of reputation, litigation and breach announcements?

UPDATE 2:

From Nicholas:

One observation:  ANY persistent and stealthy rootkit is actually ALWAYS detectable (a trick developed by Microsoft and not shipped as a tool, but you can do it thyself in the Linux world). You fire up your MD5-sum program and checksum EVERYTHING on the disk, and write it out.  Now you boot from trusted media, and repeat the process. Anything peristent (survives across reboots) and stealthy (presents a 'normal' view to programs within the OS that doesn't reflect reality) MUST show up as a difference in one or more of the checksums.

Microsoft built a tool in their research lab, but couldn't release it.  But you can use the trick with a Linux live-CD for the Linux world.

The other:  When dealing with a rootkit, I'm a believer in the Aliens Maxim:  "Nuke the whole side of the planet from orbit, its the only way to be sure".  I believe that, in general, if the system is known to have been compromised, "Nuke it" is likely the only reliable option.

From Patrick:

About removing a rootkit on a compromised system.

A system compromise is evidence of a system vulnerability.
If we for arguments sake say that we can properly remove the discovered rootkit we still don't know that the system is clean. There could very well be more rootkits on the system...
This could be argued for any system, but on a system that we already know is compromised the risk is so much greater. Are YOU willing to bet on those odds?