Incident responders sometimes need to investigate the nature of a website reported as being malicious. They do this by connecting to the remote site, while taking care not to infect themselves, perhaps by using a laboratory machine that isn't connected to the production network. They also take care to conceal their origin, perhaps by connecting using a non-corporate DSL line or by using an anonymizing proxy, such as Tor. There are a few other connection elements they need to account for.

When connecting to malicious websites to investigate them, take care to set your User-Agent and Referer headers according to the attacker's expectations.

The Referer Field of the HTTP Header

For instance, Websense documented a recent Kookface variant whose website looked at the Referer field of the HTTP request. If the victim visited the malicious website directly by typing the URL into the browser, the website would redirect to an apparently benign page that looked like Google news search results.

HTTP request headers when visiting the malicious page directly, without a referrer:

GET http://url-redacted HTTP/1.1
Accept-Language: en-us
User-Agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1)
Proxy-Connection: Keep-Alive
Host: hostname-redacted.com

Benign response from the website looked like this when rendered by the browser:

 In contrast, when the victim clicked on a link embedded in some page, the "Referrer" field was set:

GET http://url-redacted HTTP/1.1
Referer: http://example.com
Accept-Language: en-us
User-Agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1)
Proxy-Connection: Keep-Alive
Host: hostname-redacted.com

As the result, the website redirected the visitor to another page. This one attempted to social-engineer the person into downloading and installing malware under the guise of a Flash player update:

This particular site didn't seem to care about the specific contents of the Referer field, as long as it was set. However, many websites will only attempt to attack the visitor if the Referer field is set to a particular value, such as the one corresponding to Google. This is a defensive strategy to make it harder for security analysts to investigate the

The User-Agent Field of the HTTP Header

The Koobface website above also paid attention to the User-Agent field of the HTTP header, attacking only visitors coming from the Windows platform. For instance, this request made the site think the person is connecting from a Linux platform:

GET http://url-redacted HTTP/1.1
Referer: http://example.com
Accept-Language: en-us
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US) Gecko Ubuntu/9.10 (karmic) Firefox/3.5.1
Proxy-Connection: Keep-Alive
Host: hostname-redacted.com

The website responded by displaying the following message before redirecting to a non-malicious website http://rolly.com.

Tools for Controlling HTTP Headers

When investigating malicious websites using Firefox, you can control your Referer header by using the RefControl add-on. You can control your User-Agent header using the User Agent Switcher add-on.

Another option is to use command-line page retrieval tools, such as wget and curl. We discussed ways of controlling the headers sent by these tools in an earlier diary.

For more control over your browser's interactions with the malicious website, consider proxying your lab's browser through a local proxy tool, such as Paros Proxy, Fiddler, WebScarab, etc.

-- Lenny Zeltser

Lenny Zeltser leads a security consulting team and teaches how to analyze and combat malware. He is active on Twitter and recently launched a security blog.