Tracking HTTP POST data with ELK
The Apache web server has a very modular logging system. It is possible to customize what to log and how. But it lacks in logging data submitted to the server via POST HTTP requests. Recently, I had to investigate suspicious HTTP traffic and one of the requirements was to analyze POST data. If you already have a solution which performs full packet capture, you're lucky but it could quickly become a pain to search for information across gigabytes of PCAP files. In the past, ngrep ("network grep") helped me to capture interesting traffic like this:
This command will capture all Wordpress login attempts and write packets to a PCAP file. We have now only useful data but, here again, processing them remains a pain. Hélas, sniffing packets is not relevant against HTTPS traffic!
So, I searched for an alternative and decided to use the combination of mod_security and ELK. mod_security being available on most distributions (and I hope used!), it will reduce the need to deploy new software components. The first step is to collect the POST data. By default, mod_security logs only blocked HTTP requests. To log all the POST requests, modify/add the following lines in your mod_security.conf (/etc/apache2/mods-enabled/mod-security.conf on Ubuntu distributions):
SecRule REQUEST_METHOD "POST" "id:1000,phase:2,ctl:auditEngine=On,nolog,pass"
The first directive specify which sections will be logged. In our case, the most important if the "C" which contains the request body). Once done, restart Apache and check the log for new events. Here is a typical Wordpress login attempt:
[06/Nov/2015:10:45:09 +0100] Vjx2pcOa89sAAAbH5QsAAAAJ 62.149.143.75 40119 127.0.0.1 8080
--16656b56-B--
POST /wp-login.php HTTP/1.1
Host: leakedin.com
User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:23.0) Gecko/20100101 Firefox/23.0
Accept: */*
Accept-Encoding: gzip
Referer: http://leakedin.com/wp-login.php
Content-Length: 89
Content-Type: application/x-www-form-urlencoded
X-Forwarded-For: 62.149.143.75
X-Varnish: 1669981118
--16656b56-C--
log=admin&pwd=XXXXXXXXXXXX&redirect_to=http%3A%2F%2Fleakedin.com%2Fwp-admin%2F&wp-submit=Log+In
--16656b56-F--
HTTP/1.1 200 OK
Expires: Wed, 11 Jan 1984 05:00:00 GMT
Cache-Control: no-cache, must-revalidate, max-age=0
Pragma: no-cache
Set-Cookie: wordpress_test_cookie=WP+Cookie+check; path=/
X-Frame-Options: SAMEORIGIN
Vary: Accept-Encoding
Content-Encoding: gzip
Content-Length: 1851
Content-Type: text/html; charset=UTF-8
--16656b56-E--
--16656b56-H--
Apache-Handler: application/x-httpd-php
Stopwatch: 1446803109179197 352941 (- - -)
Stopwatch2: 1446803109179197 352941; combined=260, p1=192, p2=61, p3=1, p4=1, p5=5, sr=50, sw=0, l=0, gc=0
Response-Body-Transformed: Dechunked
Producer: ModSecurity for Apache/2.6.3 (http://www.modsecurity.org/); OWASP_CRS/2.2.8.
Server: Apache
--16656b56-Z--
The mod_security log format is not the most friendly to read but it is very complete! Hopefully, somebody developed a Logstash parser for mod_security. It is easy to deploy. Once indexed, events will be available in Kibana for investigations. Here is a Wordpress login brute force attack:
And you will find all the HTTP requests details easily browsable:
The combination mod_security/ELK will help you:
- To collect and store for a long retention period your POST data for further investigations (even if the web traffic is encrypted)
- To beautify data (ex: automatic geolocalization as seen above)
- To correlate events with other sources.
WARNING: mod_security is an Apache module and has access to all data in clear text. Keep in mind that HTTP POST requests could contain sensitive information (passwords, ID's, CC number, SSN, etc). You can sanitize sensitive data directly within mod_security via the sanitseArg directive.
Xavier Mertens
ISC Handler - Freelance Security Consultant
PGP Key
Comments