Here is a quick example of how OSSEC[1] can be helpful to perform threat hunting. OSSEC  is a free security monitoring tool/log management platform which has many features related to detecting malicious activity on a live system like the rootkit detection or syscheck modules. Here is an example of rules that can be deployed to track malicious processes running on a host (it can be seen as an extension of the existing rootkit detection features). What do I mean by malicious processes? Think about crypto miners. They are plenty of suspicious processes that can be extracted from malicious scripts (see my previous diary[2] about this topic). 

OSSEC has a nice feature which allows monitoring the output of a system command. A basic rule coming in any freshly deployed OSSEC agent is the disk space monitoring. OSSEC performed a ‘df’ command at regular interval and searched for ’100%’ in the output:

<rule id="531" level="7" ignore="7200">
    <if_sid>530</if_sid>
    <match>ossec: output: 'df -h': /dev/</match>
    <regex>100%</regex>
    <description>Partition usage reached 100% (disk space monitor).</description>
    <group>low_diskspace,</group>
</rule>

The idea is to search for malicious running processes on a system using the same technique. In the case of trojaned systems, commands like /bin/ps could be replaced to hide some processes. A better approach is to use the /proc virtual filesystem to list the running processes. Here is the command that I use:

# find /proc -name comm -exec cat "{}" \; 2>/dev/null |sort -u

It searches for /proc/<pid>/comm files that expose the process's command name associated with the process. Example of generated output:

accounts-daemon
acpi_thermal_pm
apache2
arpwatch
ata_sff
atd
bash
charger_manager
cpuhp/0
cpuhp/1
cron
crypto
dbus-daemon
devfreq_wq
ecryptfs-kthrea
edac-poller
ext4-rsv-conver
find
gdbus
gmain
ib-comp-wq
…

Let’s define this command in OSSEC by adding an entry in $OSSEC_HOME/etc/ossec.conf:

<localfile>
    <log_format>full_command</log_format>
    <command>find /proc -name comm -exec cat "{}" \; 2>/dev/null |sort -u</command>
    <frequency>180</frequency>
</localfile>

The ‘full_command’ type helps to return the output as a single line to be easily parsed later. Now, the create a rule in $OSSEC_HOME/rules/local_rules.xml:

<rule id="100405" level="7" ignore="7200">
    <if_sid>530</if_sid>
    <match>ossec: output: 'find /proc</match>
    <regex>Duck.sh|accounts-daemon|bonn.sh|kworker34|minerd|minergate|minexmr|mixnerdx|myatd|polkitd|rootv2.sh|jaav|jva|kw.sh|kxjd|mule|mutex</regex>
    <description>Searching for suspicious processes</description>
    <group>hunting,</group>
 </rule>

The regex has been created from a list of processes found in a crypto miner installation script. Deploy the updated config files, restart the OSSEC processes. Now, let's create a fake suspicious process on a monitored host and wait for a few minutes. You should get the following alert:

OSSEC HIDS Notification.
2018 Sep 20 08:18:20

Received From: (shiva) 192.168.254.8->find /proc -name comm -exec cat "{}" \; 2>/dev/null |sort -u
Rule: 100405 fired (level 7) -> "Searching for suspicious processes"
Portion of the log(s):

ossec: output: 'find /proc -name comm -exec cat "{}" \; 2>/dev/null |sort -u':
(sd-pam)
accounts-daemon
acpi_thermal_pm
apache2
arpwatch
ata_sff
atd
bash
charger_manager
cpuhp/0
cpuhp/1
cron
crypto
dbus-daemon
devfreq_wq
ecryptfs-kthrea
edac-poller
ext4-rsv-conver
find

--END OF NOTIFICATION

It's time to investigate!

Note that this simple alert may generate a lot of false positives! Another approach could be to check the process name combined with its working directory because many crypto miners use common process names (ex: 'apache'). But 'apache' running from /tmp is definitively suspicious! Happy hunting!

If you want to learn more about how to use OSSEC for threat hunting, I'll do a training at DeepSec (Vienna, Austria) in November about this topic[3].

[1] https://www.ossec.net
[2] https://isc.sans.edu/forums/diary/Crypto+Mining+Is+More+Popular+Than+Ever/24050/
[3] https://deepsec.net/speaker.html#WSLOT378

Xavier Mertens (@xme)
Senior ISC Handler - Freelance Cyber Security Consultant
PGP Key