Security patch management is a delicate issue in critical infrastructure. This is caused for the specific configuration, operating system version and related software required by the ICS platform. Most support contracts states that any modification outside the parameters stated by the manufacturer will void the relation and release manufacturer and seller from any responsibility about malfunction and any consequence on the industrial process.

Unfortunately, when we talk about ICS software running on windows the restriction is often applied to domain controllers as well. The case I will cover on the present diary is about an incident on a Water ICS system controlled by Emerson Open Enterprise installation and how all the servers located in the domain got modified their attributes on the domain controller without being able to tell what changes were executed.

Before this incident happened, all domain controllers and servers were configured to log the required security events to Arcsight. Now we need to find out which of the billions of logs we need to search. For this installation, we are talking about 3000 events per second. The incident began when the Open Enterprise HMI System began to loose readings from all the RTUs and after a couple of minutes it was unable to send commands to any RTU.

Since we are talking here about Windows Server 2012R2, There is an interesting repository were all event IDs for Windows 8 and Windows 2012 can be located. After looking inside the repository, the following events are interesting for searching inside the arcsight database:

Event ID 5136 states that the attribute field must be filled with the information changed for the specific object. Check the following evidence collected:

Log Name:      Security
Source:        Microsoft-Windows-Security-Auditing
Date:          23/11/2014 1:30:42 PM
Event ID:      5136
Task Category: Directory Service Changes
Level:         Information
Keywords:      Audit Success
User:          N/A
Computer:      DC.Domain.com
Description:
A directory service object was modified.
 
Subject:
 Security ID:  Domain\Administrator
 Account Name:  Administrator
 Account Domain:  Domain
 Logon ID:  0x5f8a3

Directory Service:
 Name: Domain.local
 Type: Active Directory Domain Services
 
Object:
 DN: CN=XXXXXXX,OU=XXXXX,DC=Domain,DC=Com
 GUID: CN=XXXXX,OU=XXXXX,DC=Domain,DC=Com
 Class: XXXXX
 
Attribute:
 XXXXXXXXXXX: YYYYYYYY
 Syntax (OID): X.X.X.X
 Value: ZZZZZZZZZZZZZZ
 
Operation:
 Type: Value Deleted
 Correlation ID: {ba5fa2fe-9a61-12fa-1b95-3bf03643b4e2}
 Application Correlation ID: -

For some reason, the attribute field was missing and we could not know what attributes were deleted. After researching on this issue, we found it's a bug documented as KB2458125. Six months after, this patch was not installed and because of that we were unable to know what attributes were changed on the server.

Even if we are dealing with ICS systems, the ICS admin must evaluate what patches can be applied without affecting any feature of the system like this patch. If ICS admins does not correctly control risks, incident response tasks could be useless and it will not be possible to control impact and execute containment and eradication successfully.

Want to implement patch management in your organization? Check the specific NIST guide for more information.

Manuel Humberto Santander Peláez
SANS Internet Storm Center - Handler
Twitter:@manuelsantander
Web:http://manuel.santander.name
e-mail: msantand at isc dot sans dot org