As most of our regular readers are aware, this is Cyber Security Awareness Month and the ISC handlers are writing diary entries about securing various ports and services.  For the die hard security professionals, this month is nothing more than a review of things you have learned over the past 10-20 years.  For those new to information security, it is our hope that you learn more about the various ports and services which are commonly used on networks.  As the old adage states,  those who do not learn from the past are doomed to repeat it.  It is my hope that you as our reader will learn from the mistakes of our collective past so that as an industry we don't have to repeat things (as often at least).

To that end, let me share a bit of a story (and personal history).

In my life prior to becoming an InfoSec professional, I was a student worker who functioned as Unix systems administrator over a small lab of Solaris workstations. As the case with many of student workers, our work space was usually some closet, or cubical where they could stash us which had 3 things, a 10bT ethernet jack, a phone line run to it, and  power.  My supervisor was located in a building halfway across campus (a good 10 minute walk) and my nearest student co-worker was at least 2 or 3 buildings away.  One of the most frustrating mysteries of my job was that my supervisor seemed to be psychic. No sooner than I sat down at my little workstation, he would call me with a list of things I needed to do.  If I was running late from class, or left my office early, he seemed to have an uncanny awareness of that fact and I would hear about it rather quickly.

As a part of this desktop Unix support team, one of the first tasks was to install our very own workstation from the ground up (with some documented guidance others had developed).  I got to have the full administrator experience of setting up accounts, and monitoring the activity of those accounts, compiling and installing new software, updating the computer, and just about everything else you could imagine. The only thing I had to do is provide the root password to my supervisor usually in a sealed envelope that he would stash in one of many books sitting on his bookshelves.

Even in these early days of my career, I had already begun that paranoid security mindset that I believe our field really requires. At the time I could tell you every application I had installed, what services I had running from the /etc/inetd.conf, and which of my users had logged in recently and what they had done. I turned on all of the logging features of which I was aware, and reviewed these audit logs routinely.  So the peculiar thing is that my supervisor never needed to telnet (these were the old days before SSH was prevalent) into my workstation to check up on me.  The question that haunted me for so long was how he could detect that I was in my office without any of the modern avenues available today (such as a central authentication system, door logs, hallway cameras and similar).

At some point, he revealed to me how he seemed so psychic to all of us student workers: PerfMeter.   Perfmeter is a tool which uses the Unix RPC (remote procedure call) services that ran on Solaris machines which would provide a number of statistics about the system including CPU usage.  At the time, the default screen savers under Solaris X windows would either be very heavy on CPU cycles, or didn't use much of anything in comparison to when you were on the workstation.  And as Perfmeter could communicate with a remote system without the need of authenticating (!!!) and left no audit log entries (!!!) of this activity occurring,  my supervisor just had to configure a graph for each of the student workers computers and know if we were at work or not.  And as a consequence of the lack of any true security perimeter, any attacker in the world could potentially use this or other RPC based applications to gather intelligence on system administrators and the systems they maintained with little to know hurdles to speak of.

PerfMeter is only one of many thousands of applications/services which communicate via RPC calls which initially use TCP or UDP port 111.  RPC is/was a very powerful technique for constructing distributed, client-server applications and is based on the notion that a called procedure need not exist in the same resource space as the calling procedure or application.  By using RPC, the programmer of these distributed applications could avoid the details of the interface with the network (and consequently avoid making as many programming mistakes in the networking code which was so prevalent at the time). 

In many cases when a client makes an RPC call for a given program, it will first connect to the rpcbind or portmapper service (tcp/udp 111) on the destination system to determine the ultimate port where the request should be sent. The active port 111 will have a list of all active services, and tell the requesting client were to connect. However, infosec professionals should know that under some versions of Unix rpcbind not only listens on the TCP/UDP port 111, but it also listens on UDP ports greater than 32770. The exact port number is dependent on the OS release and architecture. Thus, packet filtering devices, router ACL blocks, and firewalls that are configured to block access to rpcbind/portmapper at only port 111, may be subverted by sending UDP requests to rpcbind listening above port 32770. This vulnerability may allow an unauthorized user to obtain remote RPC information from a remote system even if port 111 is being blocked.

To finish a part of the story, I eventually found that Wietse Venema (author of SATAN, TCP wrappers, and other security tools) had developed a replacement to the native rpcbind/portmapper program that would correct two of the main issues I had with it at the time: access controls and logging.  With this tool, I was able to at least attempt to thwart the attacker (my supervisor) until told otherwise, and limit the amount of data my workstation was allowing out.

While Unix based RPC doesn't seem to be as prevalent today, I do expect that this construct can and will be used in the future in new and different ways.   We all have seen or heard of stories where there is an intrusion which involves some legacy application which was not properly secured.  Chances are some of you may have one of those applications running on the interior of your network.

Here are some recommendations I have for securing these applications.  If you have other suggestions, please share via our contact form.

1)  Inventory your servers to see if any of them are listening for TCP or UDP port 111 using a tool like nmap.  Also, check for high UDP ports such greater than 32770. Alternatively, get the server admins provide the output from lsof for UDP and TCP ports which are listening.
2)  For each of these servers, enable the host based logging which may be available.
3)  Verify that this log data should be flowing to your centralized log server or Security Event Management system.
4)  Configure the host based access control to limit what computers or networks can do RPC calls (and don't forget that localhost may need to be allowed for some intra system communications).  Most modern Unix/Linux based systems have incorporated Venema's tcp wrapper access control framework which makes this reasonably easy today.
5) As a defense in depth measure, make sure that your router ACLs or Firewall rules limit your exposure from outside of your organization to the portmapper service port.
6) If a server has no need for the service such as NFS, NIS/NIS+ or legacy applications which uses RPCbind, then shut off the service.
7) Solaris 10 systems can enable the host based TCP Wrapper support by running the following command:

# svccfg -s rpc/bind setprop config/enable_tcpwrappers=true
# svcadm refresh rpc/bind


So readers, what other recommendations do you have in securing the RPC protocol?

Scott Fendley, ISC Handler on Duty