Those who've taken my Reverse-Engineering Malware class know that I am a fan of a two-phased approach to malware analysis:

1. The behavioral analysis phase examines how the malicious program interacts with its environment: the file system, the registry (if it's a Windows program), and the network.
2. The code analysis phase examines the code of the malicious program to understand what capabilities are built into it.
   

Each phase produces findings that reinforce findings from the other phase, resulting in a comprehensive understanding of the malicious program that would be harder to obtain via a single phase. The analyst typically starts with the phase that he or she is most comfortable with.

The behavioral analysis phase can be tricky when the malicious specimen exhibits rootkit tendencies--hiding its processes or files, for instance. One way to deal with this is to patch the specimen so that the concealing subroutine never executes. This is not always easy. To ease the challenge of monitoring rootkit-concealed processes, we can employ programs that can detect concealment mechanisms such as function hooking. I'd like to describe two such programs: Helios and IceSword.

The authors of Helios call it an advanced malware detection system. It attempts to tackle the task of heuristically detecting and blocking malicious programs, even rootkits, before they can embed themselves deep in the system. The program is still in the alpha development phase, but it is available as a free download for those who want to experiment with it in a laboratory environment.

I took Helios for a spin today to see whether it could help with malware analysis. I think it can be a helpful addition to the reverse-engineer's toolkit, because it can detect when the malicious program attempts to hide itself via rootkit techniques. Helios can also unhide the malicious process to to make behavioral analysis a bit easier.

For example, consider a malicious program called malware.exe. Executing normally, it is visible in the task list, as you can see in the following Task Manager screen shot:


If this program had exhibited rootkit behavior, its process would be hidden. We can simulate this by hiding the  malware.exe process using a rootkit, such as one called FU. By executing the command "fu -ph" and then supplying the process ID of the malicious program, we can hide the process from Task Manager.

If Helios is running in the background, it can detect the attempt to hide this process and alert you about it: