It seems that every time I do a security assessment or pentest, the findings include problems with access controls for things like routers, switches, fiber channel switches, bladecenters and the like.  And when I say "every time", I really mean EVERY TIME.  So, what kind of problems are common, and what can you do to prevent them?

Default Credentials

People love default credentials.  I've actually had a customer tell me "if we forget the password, we can just google for it" - like somehow that's a good thing.  Even for gear that comes with default credentials that force you to change them on login (for instance, fiber channel switches), I've seen other engineers then change the password back to the default - their rationalle being that it reduces their support calls later.

The number of bladecenters that I've seen that have changed the default admin user and password I think I can count without taking my shoes off (ie - it's a small number - almost nobody changes their bladecenter admin password).  This access gives the right to power off servers, decommission servers, unpatch servers, hack windows or linux passwords, almost everything is possible - the bladecenter admin access gives you the near-equivalent of physical access, and we all know that physical access trumps almost every control in the book.

People - default credentials are BAD.  If you are using default userid and password, and someone compromises you, we don't call that "hacking", we call it "logging in".  You don't need an uber-hacker to target you in this situation, ANYONE WITH GOOGLE can compromise you.  Change the admin password on your gear, and if possible, change the admin userid.  Better yet, back end admin logins with another directory (read on ...)

Prevent Access in the First Place - Access Classes
Many products will allow you to restrict administrative access in the configuration.  For instance, Cisco gear has the "access-class" config statement.  HP, Brocade, Juniper and others all have an equivalent construct.  The commands you'd put in your switch or router config might be:

Prevent Access in the First Place, Reloaded - Define a Management Network

Even with all the other controls we'll talk about, implementing a management network is a good move.  It zones all infrastructure admin into one place, you can control access to this netowrk using VPN controls or a "jump box".  The VPN approach is a neat one - it means that if you are an authorized admin, you can use an IPSEC or SSL VPN solution to get an IP that has access to admin your network from anywhere in the company.  This is really handy for admins that are mobile within the company or have to provide support from home (or the cottage, the beach, or anywhere else they can find you).

Encrypt all Administrative Accesses
I still see LOTS of admin access over standard HTTP and Telnet.  And there are LOTS of tools that will strip passwords out of this type of traffic - you can do this with CAIN if you want a GUI, but really even doing it with wireshark or tcpdump is pretty simple.  Be sure to force SSH (Version 2 if you can, Version 1 can be decrypted), or HTTPS for administration of critical network infrastructure.

Even what you might consider non-critical infrastructure should see the same protections.  I've done a pentest on a company that used telnet to administer their UPS gear (to monitor temperature and humidity as well as power characteristics and remote control of power etc).  Unfortunately for them, they used their AD admin password to login to their UPS, which they accessed using plaintext HTTP and Telnet.  Doubly unfortunate, as a standard procedure they logged in each morning to check logs etc.

Checking logs daily would normally be a really good thing if they had other controls in place (for instance, using HTTPS or SSH for administration), but since they were being pentested, I had their admin password within 10 minutes of starting the actual process !

On a cisco router or switch, the commands you need to set up SSH are:

If you allow web administrative access (that's a whole different discussion), forcing https on network instrastructure is generally even easier:

To go one better, I'd also suggest that you replace the self-signed certs that is used by default for HTTPS admin on most gear, using certs on SSH is also a really good mechanism.  Without replacing the default certificates, tools like ettercap can still be used to mount a man-in-the-middle attack and recover passwords.

Back-end Authentication and Change Logging

After everything else is said and done, there's still way too much gear out there that has a single administrative account, or no account at all (only access passwords).  This plays hob with managing change - since every config update is done using the single admin account, if a change goes bad everyone in your team can deny making that change.  (Does your team have "Ida Know" as an honorary member?)  If you don't take a stab at non-repudiation of changes, this WILL bite you eventually.

So, what to do?  Should you define a userid for each and every user on all your infrastructure gear?  Well, only as a last resort.  Most network infrastructure has the capability of back-ending authentication and access controls using some external source.  Popular back-ends are what you'd expect - RADIUS, TACACS, Kerberos, LDAP and Active Directory.  I'd say pick one and go with it.  I often go with RADIUS back-ended with AD (IAS, now NPS) because it's simple, easy to troubleshoot, and supported by almost everything.  Mind you, it's also likely that if you go with RADIUS you are then susceptible to other attacks, but you can mitigate that by setting your RADIUS server up in a private vlan or by using other intelligent design decisions to implement security controls.

Don't forget to LOG ALL ACCESSES (this is built into RADIUS and TACACS) and LOG ALL CONFIG CHANGES (lots of tools will do this for you - syslog will log that a change occurred and who made it, CATTOOLS and RANCID (thanks to our reader "Bmac" for this correction) are 2 that come to mind for more complete change logging, I've also written shell scripts to do this.  Feel free to suggest others that you use as a comment to this diary).  In some cases, you may also want to log all commands as well (most gear will let you do this in syslog). 

What does this give you?  As changes occur, you are notified that a change happened, who made it, and what it was.  You can then compare this to the CHANGE REQUEST FORM that you have in your CHANGE CONTROL SYSTEM, to be sure that:

• the change made was both requested and approved
• the change happened during the change window
• the person making the change was the one authorized to do it

if it was an unauthorized change, you have the culprit identified and can have, shall we say, a discussion that's appropriate to the situation.

Basic commands for simple logging and NTP time sync are, well, pretty simple (as with most examples in this diary, it can get more complicated)

I've phrased this discussion in the context of network infrastructure gear, but really many of these points extend to other datacenter infrastructure components as well.  Replacing default certificates used for RDP to critical Windows servers is a good move, as is certificate updates for things like VMware vSphere (both vCenter servers and ESX Hosts).  Using a Management network is an important part of designing a virtual infrastructure from any vendor, as well as bladecenters (for the same reasons).  Management networks can also be used to protect things like RADIUS authentication, syslog, NTP and SNMP based network management, all of which are sent in cleartext.  Back-end authentication to an enterprise directory (often AD) is a common solution for authentication, is useful as we discussed for managing and auditing change for both Linux and VMware servers, as well as for lots of other gear besides just routers and switches.

I'm hoping that you find these suggestions to be a helpful starting point.  There's lots more that can be done in this area, please use the comment feature to let us know if you've found this useful, have your own stories in this area, or if I've missed (or messed up) anything in this diary. 

=============== Rob VandenBrink, Metafore ===============