We've had a few e-mails come in to the ISC now that the popular media has picked up the story of the distributed denial of service attack on CloudFlare and SpamHaus.  For instance, here is the New York Times article on the subject.  CloudFlare has their own write-up here.  I was peripherally involved (very peripherally) as were some other handlers.

Let's start with some truth.  The attack did reach upwards of 300 Gb/sec and is the largest recorded DDoS to date.  It combined already known issues on DNS open resolvers but combined it with specific targetted at a choke point which did have a real impact for SpamHaus and CloudFlare.  There also were many people who spent many hours helping deal with this problem.  A good number of those had no real connection to SpamHaus or CloudFlare, they are just fellow members of the information security community who came together to deal with a threat.  This is a Very Good Thing <tm> that this level of cooperation has built up over time and we respond to these threats as a community.

Here is what did not happen: the Internet did not come close to coming down, not much real impact was felt outside the victims and those in close Internet-proximity to them and we were all still able to get to pinterest and see cat pictures online.  The attack was significant, but not globally so despite the media reports to the contrary.  When news of the attack reached the Internet Storm Center, we did have a brief moment of panic and contemplated resorting to cannibalism.  However, we quickly decided against this option (due to a combination of calmer heads prevailing and a lack of consensus on whether people could be turned into bacon).

That's not to say it was a non-event.  It exposed some real problems, problems that each of us should take steps to help remediate so this doesn't happen again.  And by us, I mean those of you reading this that may maintain networks that unknowingly participated in this attack.  More on that shortly.  The attackers were part of a group dubbed StopHaus who decided to take down SpamHaus.  For our purposes here, we'll leave the politics surrounding the attack out of it and focus on the technical.

DNS Amplification Attacks

Accomplishing a successful denial of service attack with straight up network flooding is difficult to accomplish.  In the general case, you have to have more bandwidth than your target.  While it might be somewhat easier to take a gaming server offline over an MMO grudge match, going against a protected target requires you to have a greater amount of bandwidth than your victim which is usually not the case.

You can get around this problem two ways: control lots of machines all over the Internet (i.e. a botnet) or find ways to amplify your attacks to make the attacks much bigger in size than it took you to generate.  Those of you who remember the Smurf attack knows how this can work with ping (spoof your source and ping a broadcast address and every machines on that network will send an echo reply back to your victim).  We've fixed Smurf with default configurations.

Enter DNS resolvers.  Sending a DNS query is not generally a large request.  However, due to the security advances in DNS (such as DNSSEC), responses to requests can be quite large in comparison. Interestingly enough, CloudFlare has a pretty good write-up on DNS amplification attacks here. In their example, they have a 64 byte query that generated a 3,223 byte response.  That means they can amplify their bandwidth by ~50 times.

In short, here is how it works.  The people who were upset at SpamHaus (and by extension CloudFlare) picked a choke point inside CloudFlare that would hurt, the spoofed DNS requests to known open resolves "from" that victim IP address and they were able to generate a 300 GB/s attack.  Estimates ranged from a 30x - 100x amplification of their own bandwidth use.  When they were keeping their peak DDoS up, that's what CloudFlare was seeing.  (To see the progression to this point, you can read CloudFlare's write up linked above).  To achieve a DDoS attack of 300 Gb/s you would need access to 3-10 Gb/s of bandwidth.  Not insignificant, but also not unachievable for someone with motivation and some money.

The important takeaway is that DNS amplication has been known for some time now and that this DDoS attack was entirely preventable.  Not by CloudFlare, mind you, but by the rest of us who maintain networks and DNS servers.

So what can you do about it?

For these DDoS attacks to work, there needs to be two different components and the presence of either is not a best practice.

The first is that networks where these attacks are being launched are not filtering spoofed traffic.  In order for spoofed traffic to leave a network, the perimeter devices need to allow packets with a source address not on the internal network to be routed out.  This is a bad thing and not good network neighborhood behavior.  Everyone that has netspace should make sure all traffic leaving their network does not have a source address that is not their internal network.  The second component of this is that networks are not doing ingress filtering per BCP 38.  Namely, traffic should not be passed by upstream providers unless it is coming from a known and advertised IP spaces of their clients.  If this were adopted universally, spoofed IP traffic would all but disappear.

The second portion of this is having open recursive DNS resolvers on your network.  Rarely is this a good thing and in most cases, they are unknowingly present and being used to generate attack traffic.  The Open Recursive Project has a tool to check for Open DNS Recursive servers in your netspaces and some advice on what to do if this is an intentional choice (namely rate limiting).  Generally speaking though, most DNS servers do not need to perform recursive queries and many of the rest don't need to do it for the entire Internet. Turning off recursion is as easy as putting "recursion no;" into your named.conf file and if you need to recurse for local clients, restricting it to just your own netblocks.

If those of you who maintain networks do the above two things, this DDoS (and those that will follow) would be non-events.  So please, implement some flavor of BCP 38 and turn off open recursive DNS servers.

--
John Bambenek
bambenek \at\ gmail /dot/ com
Bambenek Consulting