Heartbleed CRL Activity Spike Found

Published: 2014-04-16. Last Updated: 2014-04-18 01:51:02 UTC
by Alex Stanford (Version: 1)
9 comment(s)

Update: CloudFlare posted in their blog twice today claiming responsibility for the majority of this spike. Quoting: "If you assume that the global average price for bandwidth is around $10/Mbps, just supporting the traffic to deliver the CRL would have added $400,000USD to Globalsign's monthly bandwidth bill."

Update: We've also seen articles from ZDNet and WIRED today in response to the below insights, with further analysis therein.

It looks like, as I had suspected, the CRL activity numbers we have been seeing did not reflect the real volume caused by the OpenSSL Heartbleed bug.

This evening I noticed a massive spike in the amount of revocations being reported by this CRL: http://crl.globalsign.com/gs/gsorganizationvalg2.crl

The spike is so large that we initially thought it was a mistake, but we have since confirmed that it's real! We're talking about over 50,000 unique revocations from a single CRL:

This is by an order of magnitude the largest spike in revocation activity seen in years, according to our current data.

I have set up a new page for everyone to monitor the activity as well as see how we are obtaining this data. The page can be found at https://isc.sans.edu/crls.html.

How will you use this page in your projects or general analysis? We'd love to hear some ideas.

If you know of other CRLs that we can add, please let us know in the comments! Additionally, if you would like to see an API call added so that you can automatically query us for this information, please let us know so that we are aware of the demand.

On a side note, we can see a clear upward trend in revocations over the past 3 or 4 years:

What do you attribute this consistent growth in revocations to? What do you think caused the previous spikes?

-- 
Alex Stanford - GIAC GWEB,
Research Operations Manager,
SANS Internet Storm Center
/in/alexstanford | @alexstanford

Keywords:
9 comment(s)

Comments

That one sure looks like an anomaly; maybe it's not related to heartbleed.

My guess though is that it will take a while for most people to go through the motions of revoking and reissuing certificates. I can see a whole bunch of reasons for that, such as:

- They can't remember how to request and renew certificates
- They're concerned about cost (and haven't looked into if their CA will issue free certs as a one-off or not)
- They're still building up their list of which systems are affected; OR
- They've decided to wear the risk and not renew their certificates

Anonymous

Apr 17th 2014
1 decade ago

I chatted with GlobalSign tech support via their online chat. The person I talked to told me that this is real and due to heartbleed.

Anonymous

Apr 17th 2014
1 decade ago

[quote=comment#30517]That one sure looks like an anomaly; maybe it's not related to heartbleed.

My guess though is that it will take a while for most people to go through the motions of revoking and reissuing certificates. I can see a whole bunch of reasons for that, such as:

- They can't remember how to request and renew certificates
- They're concerned about cost (and haven't looked into if their CA will issue free certs as a one-off or not)
- They're still building up their list of which systems are affected; OR
- They've decided to wear the risk and not renew their certificates[/quote]

While I think each bullet you posted does refer to a large portion of people, right now we have no reason to believe that this isn't accurate data. The best guesses we have seen are in the range of 800,000 Heartbleed vulnerable sites. It's possible that the bullets you point to are still the majority and that this spike is only the minority.

Anonymous

Apr 17th 2014
1 decade ago

Don't get me wrong - I was hoping it was accurate data! It seems from Dr J (whom I assume is Johannes) that it's been proven now.

In a year's time the lessons learned are going to make for great material in GSEC. That's about the only silver lining I can think of.

Anonymous

Apr 17th 2014
1 decade ago

Is the slow ramp-up of revocations simply an indicator of the increase in issuance/usage of certificates over the last 6-7 years? If we assume 3-4 year lifespans, then the beginning of the ramp-up in 2011 corresponds to certificates issued in 2006-2008. I know DoD was smack in the middle of their huge PKI push in those years (as I was the stuckee for PKI implementation at my unit), so it's not out of the realm of possibility that the gradual increase in revocations starting in 2011 reflects our growing reliance on certificates.

Anonymous

Apr 17th 2014
1 decade ago

Once a certificate expired, there is no need for it to be listed in the CRL anymore. So yes, we should see only few certificates older then 2 years or so.

There is also likely a delay in having a certificate added after it is revoked to give the organization enough time to swap certificates.

As far as Globalsign goes, I think we see a big hit from them because they are working with Cloudflare. If you "proxy" an SSL site via Cloudflare, Cloudflare is going to request a certificate for it. So as Cloudflare became aware of Heartbleed (and they were ahead of the game on it), the likely issued all new certificates which in itself could explain the 100k or such revocations that came through yesterday.

Anonymous

Apr 17th 2014
1 decade ago

Doesn't the mid-2011 spike match the DigiNotar event?

Perhaps the early 2013 spike is some large organization rolling their certificates in response to Heartbleed?

Anonymous

Apr 18th 2014
1 decade ago

The 2011 CRL tsunami is probably attributed to the DigiNotar compromise.

The 2013 CRL tsunami might be attributed to the news that Verisign had been hacked:
http://www.pcworld.com/article/249242/verisign_hacked_what_we_dont_know_might_hurt_us.html

Just my surmise on both counts.

Anonymous

Apr 18th 2014
1 decade ago

I thought the same about DigiNotar being responsible for the 2011 spike, however that spike appears to have happened on July 28th specifically in our data, about a month early. I'm not sure exactly what this means, just pointing it out. It appears to have been an influx of about ~750 revocations (seems tiny compared to the recent spike!) so many different organizations easily could have been solely responsible.

Thanks for all of the comments everyone, it's great to see more new names in the forums!

Anonymous

Apr 18th 2014
1 decade ago

Login here to join the discussion.


Top of page
Diary Archives