My next class:
LINUX Incident Response and Threat HuntingOnline | Japan Standard TimeOct 21st - Oct 26th 2024

New tool: docker-mount.py

Published: 2016-10-13. Last Updated: 2016-10-13 20:00:58 UTC
by Jim Clausing (Version: 1)
4 comment(s)

In my post Forensicating Docker, Part 1 back in March (yes, I promise a Part 2 in the next couple of months, the $dayjob has slowed that down a bit), I talked a little about the AUFS layered filesystem that was used by the docker install on the system I was investigating. While I was forensicating the case I talked about in that diary, I wanted to see what the container filesystem looked like from my SIFT VM so I wrote a script to do the mounting the same way docker does (except for forensic purposes the mount is read-only). The script can be found here. Unfortunately, docker can use multiple storage drivers.  So far, I've adapted the script to handle two/three of them, AUFS and Overlay/Overlay2. AUFS is the default on (older?) versions of Ubuntu, but AUFS isn't included by default in RedHat (or derivates), you would have to compile your own kernel. Overlay2 is included in newer kernels (pretty much anything after 3.18), so I suspect it may become the default at some point in the future. These are the storage drivers that handle so called Union filesystems to handle the layering. The btrfs, zfs, and devicemapper storage drivers are all block-level rather than file-level storage drivers. In effect, they require separate devices/partitions/loop-mounted files taking advantage of filesystem features such as snapshots in the underlying filesystem drivers to handle the layering.  While I think I can get btrfs into the script, I haven't looked at zfs and I've had difficutly with devicemapper, so I may not be able to get all of these. See [3] and [4] below for more background on docker storage drivers.

Having gone through all of that, for the purpose of forensication, it is important to remember that changes made within a container will all be captured in the "top" layer of these layered or union filesystems. To find that top layer (for docker >= 1.9), you need to look at /var/lib/docker/image/storagedriver/layerdb/mounts/containerid/mount-id to find the id for the container storage. Then, the changes will be taking place in /var/lib/docker/storagedriver/containermedia-init/diff, so that is where you should expect to find any indications of malicious activity within the container.

References:
[1] https://isc.sans.edu/forums/diary/Forensicating+Docker+Part+1/20835/
[2] https://github.com/clausing/docker-scripts
[3] https://docs.docker.com/engine/userguide/storagedriver/imagesandcontainers/
[4] https://integratedcode.us/2016/08/30/storage-drivers-in-docker-a-deep-dive/

---------------
Jim Clausing, GIAC GSE #26
jclausing --at-- isc [dot] sans (dot) edu

Keywords: docker
4 comment(s)
My next class:
LINUX Incident Response and Threat HuntingOnline | Japan Standard TimeOct 21st - Oct 26th 2024

Comments

correct me if i am wrong but this overlay file system is the exact same thing that allows me to use my usb 3.0 flash drive as the root partition on my linksys wrt1900ac router? Its kind of cool actually cause if say I feel like totally reverting my linksys for whatever reason, say changing my external ip address (say i have the mac address spoofed on the root of the overlay), all i need to do is remove the overlay (ie the thumb drive) and all of the settings are reverted to whatever the actual root has. long winded i know, but its kind of cool. Still though, and ill admit this is a bit above my paygrade atm, couldn't there be an overlay on top of the overlay that you think the malicious activity is in?
The way it is used in docker, there will only be one writable layer per container and that will be the top one. It isn't clear to me if the Overlay driver can be used in the way you described, but that is an interesting idea that I'll try to look into.
Well again I'm definitely a noob at a lot of this, but I have personally crafted some strange exploit like things by just pondering those what ifs and then actually trying them. I recall someone posting that some malware can get outside of the docker container and if there actually could be an overlay of an overlay, then someone could dump their payload to the higher overlay execute the payload, then once outside the container wipe the higher up overlay and someone like you (or myself) trying to forensically analyze, would never see it. Again this is all conjecture that could have no basis in reality, but sometimes thinking outside the container (pun intended) is what is needed to be one step ahead.
Double post forgive me

Diary Archives