How did this Memcache thing happen?
As pointed out earlier (https://isc.sans.edu/forums/diary/Why+we+Dont+Deserve+the+Internet+Memcached+Reflected+DDoS+Attacks/23389/) this memcached reflected DDoS thing is pretty bad. How bad? Well, US-CERT updated its UDP-Based Amplification Attacks advistory (https://www.us-cert.gov/ncas/alerts/TA14-017A) to add Memcache to the list of potential attack vectors. The really telling bit is the chart that shows the Bandwidth Amplification Factor. Before memcache was added the largest factor was 556.9 from NTP where each byte sent in to a vulnerable server would return about 557 bytes in attack traffic. Memecache is listed as 10,000 to 51,000. That's remarkably large.
How common is an internet-facing memcache daemon? I did a little poking with Shodan (https://www.shodan.io/) and it had identified 100k or so systems. Mostly in cloud provider spaces as you would expect.
So how did this happen? I don't know much about memcache myself, so I think that makes me particulary qualified to take a stab at the answer. Since I don't know what it is, or how to install it, I turn to google to help me out. Buried in the wiki for memcached (https://github.com/memcached/memcached/wiki/ConfiguringServer) under the networking section one might read:
"By default memcached listens on TCP and UDP ports, both 11211. -l allows you to bind to specific interfaces or IP addresses. Memcached does not spend much, if any, effort in ensuring its defensibility from random internet connections. So you must not expose memcached directly to the internet, or otherwise any untrusted users. Using SASL authentication here helps, but should not be totally trusted."
But who has time for instructions, right? What's Google tell me to do? The top answer gives us:
Change the memcached configuration setting for CACHESIZE and -l :
- Open /etc/memcached.conf in a text editor.
- Locate the -m parameter.
- Change its value to at least 1GB.
- Locate the -l parameter.
- Change its value to 127.0.0.1 or localhost.
- Save your changes to memcached.conf and exit the text editor.
- Restart memcached.
Source: http://devdocs.magento.com/guides/v2.1/config-guide/memcache/memcache_ubuntu.html
Hey, that looks right. What about Digital Ocean's instructions (they have a lot of these open)? https://www.digitalocean.com/community/tutorials/how-to-install-and-use-memcache-on-ubuntu-14-04
It has this as how it's started:
/usr/bin/memcached -m 64 -p 11211 -u memcache -l 127.0.0.1
That looks fine too. So it's not a rash of bad instructions on the Internet.
Okay, how about stackoverflow/serverfault? In the top question about memcache (https://serverfault.com/questions/347621/memcache-basic-configuration) we find an illuminating answer:
"The -l option regulates to which interfaces memcached will listen to, not which hosts will be served."
I wonder how many web-admins thought they needed to open up access to all of their clients.
Comments
This is a bugfix release, but it primarily disables the UDP protocol by default.
b) https://twitter.com/dormando/status/968579781729009664
quote: if you're getting attacked by memcached's, it's pretty easy to disable them since the source won't be spoofed. They may accept "shutdown\r\n", but also running "flush_all\r\n" in a loop will prevent amplification.
/quote
Anonymous
Feb 28th 2018
6 years ago
As long as companies have their top three priorities a Availability, Availability and Availability this nonsense will keep right on happening.
An IT person at work heard me say the above and said "Availability is how the company makes its money that pays you." I agreed and replied "And IT Security is what lets the company keep the money they worked so hard for."
He's still working on his retort, I guess.
Anonymous
Feb 28th 2018
6 years ago
Anonymous
Feb 28th 2018
6 years ago
This is why software should have sane defaults.
Anonymous
Feb 28th 2018
6 years ago
https://www.heise.de/security/meldung/Rekord-DDoS-Attacke-mit-1-35-Terabit-pro-Sekunde-gegen-Github-com-3985411.html
Anonymous
Mar 2nd 2018
6 years ago