Linux Memory Dump with Rekall
Memory dumping for incident response is nothing new, but ever since they locked down access to direct memory (/dev/mem) on Linux, I’ve had bad experiences dumping memory. I usually end up crashing the server about 60 percent of the time while collecting data with Fmem.
A new version of Linux memory dumping utility rekall (previous called Winpmem) has recently came out. I’ve been testing it on the latest versions of Ubuntu and Redhat EL 5 and have not run into any issues with collection.
If you are fortunate enough to have an environment where you have groups of servers with the same patch levels, you should run the following steps on a non-compromised server. Additionally, if the compromised system is a VM, you can clone it and perform these actions on the clone. Make sure you collect all other volatile data (MACtimes, LSOF, PS ect..) before you dump memory as this may still cause instability to the system and you do not want to lose this data.
Preparing for collection
Install Linux Kernel Headers
Ubuntu
>sudo apt-get install linux-headers-server zip
CentOS/Redhat
>yum install kernel-headers gcc
Download and Compile rekall
When you run the makefile, it will automatically create part of the profile for the server. This will need to be copied off the server for analysis.
>wget http://downloads.rekall.googlecode.com/git/Linux/linux_pmem_1.0RC1.tgz
>tar -zxvf linux_pmem_1.0RC1.tgz
>cd linux
>make
Note:For Redhat/CentOS systems you will need to adjust the Makefile KHEADER variable.
Copy this file to your Volatility analysis machine under your volatility directory /plugins/overlays/linux/.
Load the Kernel Driver
>sudo insmod pmem.ko
>sudo lsmod |grep pmem
pmem 12680 0
Collect Memory
Now that the drive is loaded, a new device is accessible /dev/pmem. We want to copy the memory to an external device/share.
#Items in {} need to be changed per incident to be useful for analysis
>dcfldd if=/dev/pmem bs=512 conv=noerror,sync of=/{USBDRIVE}/ mount/{servername.date}.memory.dd hash=md5,sha256 hashlog=/{USBDRIVE}/{servername.date}.memory.dd-hash.log
Unload driver
>sudo rmmod pmem.ko
Analysis using Rekall
Now that collection is completed, we need to be able to examine the memory dump. Copy the memory image to your analysis workstation.
Install Rekall
>sudo apt-get install python-pip python-dev
>sudo pip install rekall
Build Rekall Profile
We now need to create a profile that will work with Rekall. Convert the file that was copied from the server and name it something useful for future analysis.
>rekal.py convert_profile 3.5.0-45-generic.zip Ubuntu3.5.0-45-generic.zip
>rekal.py --profile ./Ubuntu3.5.0-45-generic.zip -f /media/mem.dd pslist
To enter the interactive shell, you do not add a plugin on the commandline
>rekal.py --profile ./Ubuntu3.5.0-45-generic.zip -f /media/mem.dd
To list the available plugins, use the interactive shell:
>rekal.py info[tab][tab]
plugins.arp plugins.check_idt plugins.convert_profile plugins.dwarfparser plugins.info plugins.lsof plugins.null plugins.psaux plugins.vmscan
plugins.banner plugins.check_modules plugins.cpuinfo plugins.fetch_pdb
…..
To get more info about a specific plugin use a ? after plugin name
mem 12:38:31>plugins.pslist?
Some of the more useful plugins are:
-
plugins.bash -searches for bash history
-
plugins.check_modules- List loaded modules
-
plugins.dmesg - Gathers dmesg buffer
-
plugins.lsof
-
plugins.netstat
-
plugins.pslist
Optional (If you want to use Volatility for analysis)
I haven’t spent much time on this, but Volatility will not be able to use the rekall default profile. You also have to do the steps below to read the memory dump with Volatility. I’m guessing only a small change in the file is needed, but I have dug any deeper at this time.
>sudo apt-get install dwarfdump
>wget https://volatility.googlecode.com/files/volatility-2.3.1.tar.gz
>tar -zxvf volatility-2.3.1.tar.gz
>cd volatility-2.3.1/tools/linux
>make
>zip Ubuntu{Kernel ver}.zip ./module.dwarf /boot/System.map-`uname -r`
For more information on Rekall
http://docs.rekall.googlecode.com/git/tutorial.html
For more info on Volatility Linux analysis
https://code.google.com/p/volatility/wiki/LinuxMemoryForensics
--
Tom Webb
Comments
Another tool you may wish to look at for Linux memory forensics (both acquisition and analysis) is Second Look. It's got reliable memory acquisition, with pre-compiled modules for over 10000 kernels covering the major distros for the past 7+ years. Similarly, reference data is already there for you -- no need to find a clean system on which to create a profile. And it does integrity verification of the kernel and processes in memory -- so it can find much malware that Volatility and Rekall miss.
Disclaimer: I am the product lead for Second Look.
Andrew
http://secondlookforensics.com/
Anonymous
Jun 4th 2014
1 decade ago