Exploration of DShield Cowrie Data with jq
There have been other diaries [1][2] showing how to explore JSON data with jq [3]. We'll review some options to understand unfamiliar JSON data and ways to filter that information. Using tools like Security Information and Event Management (SIEM) systems can help aggregate data and make it more easily searched and visualized. There are still times where being able to quickly search JSON data can be useful, especially if a SIEM option is not immediately available.
First, how do we know what data to query and how we can filter the information? One possible way to explore the data is to simply output the data and browse through the content.
# read JSON file (cowrie.json) and output
# cat cowrie.json
#
# process data output by jq
# data will be showed as "pretty" with indents and formatting, helping with readability
# | jq
#
# only display the first 100 lines
# | head -n 100
cat cowrie.json | jq | head -n 100
{
"eventid": "cowrie.session.connect",
"src_ip": "154.47.20.79",
"src_port": 62361,
"dst_ip": "192.168.68.178",
"dst_port": 2222,
"session": "d568084430b5",
"protocol": "ssh",
"message": "New connection: 154.47.20.79:62361 (192.168.68.178:2222) [session: d568084430b5]",
"sensor": "",
"timestamp": "2023-04-03T00:00:11.714471Z"
}
{
"eventid": "cowrie.client.version",
"version": "SSH-2.0-8.35 FlowSsh: FlowSshNet_SftpStress127.11.14.7335513172.220.65.6",
"message": "Remote SSH version: SSH-2.0-8.35 FlowSsh: FlowSshNet_SftpStress127.11.14.7335513172.220.65.6",
"sensor": "",
"timestamp": "2023-04-03T00:00:11.718071Z",
"src_ip": "154.47.20.79",
"session": "d568084430b5"
}
{
"eventid": "cowrie.client.kex",
"hassh": "63bec3855e65ea71385ce2634bf2f234",
"hasshAlgorithms": "curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-1.3.132.0.10,ecdh-sha2-nistp521,ecdh-sha2-nistp384,ecdh-sha2-nistp2man-group16-sha512,diffie-hellman-group15-sha512,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,diffie-hellman-group-exchange-sha256,diroup-exchange-sha1,ext-info-c;aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr,3des-ctr;hmac-sha2-256,hmac-sha1;none
"kexAlgs": [
"curve25519-sha256",
"curve25519-sha256@libssh.org",
"ecdh-sha2-1.3.132.0.10",
"ecdh-sha2-nistp521",
"ecdh-sha2-nistp384",
"ecdh-sha2-nistp256",
"diffie-hellman-group16-sha512",
"diffie-hellman-group15-sha512",
"diffie-hellman-group14-sha256",
"diffie-hellman-group14-sha1",
"diffie-hellman-group-exchange-sha256",
"diffie-hellman-group-exchange-sha1",
"ext-info-c"
],
"keyAlgs": [
Using 'jq' can make the data much more readable than just using a standard text editor, which may compress all the data on one line. This can be very difficult to read, especially when the text is wrapped.
Figure 1: Cowrie JSON data output using only 'cat' and no 'jq' processing
To more easily display the data labels, or keys [4], in the JSON data, we can use some other tools, such as 'cut', 'grep', and 'sort'. A secondary option is using 'keys' in 'jq'.
# read JSON data from Cowrie data file (cowrie.json)
# cat cowrie.json
#
# process data by 'jq', outputting data in easily readable format
# data is also conveniently output on new lines, making other tools easy to process
# | jq
#
# return every line that has a colon (":")
# data is separated by [key]:[value]
# allows us to only display lines that have a key
# | grep ":"
#
# split data into parts, separating by colon (":") and return the first part
# first part contains the key
# second part contains the value
# | cut -d ":" -f 1
#
# sort data alphabetically
# will allow the next command to work more effectively without duplicates
# | sort
#
# only display each key once and how many times it was in the data
# | uniq -c
#
# sort the data by the number of times is appeared
# | sort -n
cat cowrie.json | jq | grep ":" | cut -d ":" -f 1 | sort | uniq -c | sort -n
8 "name"
8 "value"
224 "data"
224 "id"
345 "destfile"
345 "outfile"
4240 "size"
4240 "ttylog"
4242 "arch"
4441 "compCS"
4441 "encCS"
4441 "hassh"
4441 "hasshAlgorithms"
4441 "kexAlgs"
4441 "keyAlgs"
4441 "langCS"
4441 "macCS"
4474 "version"
4585 "duplicate"
4585 "shasum"
4826 "password"
4826 "username"
5178 "protocol"
5449 "input"
5467 "src_port"
5691 "dst_ip"
5691 "dst_port"
9419 "duration"
38895 "eventid"
38895 "message"
38895 "sensor"
38895 "session"
38895 "src_ip"
38895 "timestamp"
cat cowrie.json | jq keys | sort | uniq -c | sort -n
8 "name",
8 "value"
224 "data",
224 "id",
345 "destfile",
345 "outfile",
4240 "size",
4240 "ttylog"
4242 "arch",
4441 "compCS",
4441 "encCS",
4441 "hassh",
4441 "hasshAlgorithms",
4441 "kexAlgs",
4441 "keyAlgs",
4441 "langCS",
4441 "macCS",
4474 "version"
4585 "duplicate",
4585 "shasum",
4826 "password",
4826 "username"
5178 "protocol",
5449 "input",
5467 "src_port",
5691 "dst_ip",
5691 "dst_port",
9419 "duration",
13548 "timestamp",
25347 "timestamp"
38895 [
38895 ]
38895 "eventid",
38895 "message",
38895 "sensor",
38895 "session",
38895 "src_ip",
From the data, we can see some fields that appear to show up in every JSON object and I suspect there are 38895 objects:
- eventid
- message
- sensor
- session
- src_ip
- timestamp
Most of the time the keys are self-explanatory, but it depends on the data being processed. We do see some keys that are much more unique in this particular set of data:
- name
- value
Since the count for both of these keys is the same value, I would guess that when one appears, the other appears, but let's take a look. Now, I'd like to select any data that only has the "name" key.
# read cowrie.json file
# cat cowrie.json
#
# process data with jq and select all data with 'name' key present
# jq 'select(.name)'
cat cowrie.json | jq 'select(.name)'
{
"eventid": "cowrie.client.var",
"name": "LANG",
"value": "en_GB.UTF-8",
"message": "request_env: LANG=en_GB.UTF-8",
"sensor": "",
"timestamp": "2023-04-03T08:42:53.534140Z",
"src_ip": "118.201.255.164",
"session": "49fe63c254ed"
}
{
"eventid": "cowrie.client.var",
"name": "LANG",
"value": "en_GB.UTF-8",
"message": "request_env: LANG=en_GB.UTF-8",
"sensor": "",
"timestamp": "2023-04-03T08:42:54.640685Z",
"src_ip": "118.201.255.164",
"session": "953dc5ae1e66"
}
{
"eventid": "cowrie.client.var",
"name": "LANG",
"value": "en_GB.UTF-8",
"message": "request_env: LANG=en_GB.UTF-8",
"sensor": "",
"timestamp": "2023-04-03T08:42:57.240638Z",
"src_ip": "118.201.255.164",
"session": "81d0c30a7320"
}
{
"eventid": "cowrie.client.var",
"name": "LANG",
"value": "en_GB.UTF-8",
"message": "request_env: LANG=en_GB.UTF-8",
"sensor": "",
"timestamp": "2023-04-03T08:42:58.412682Z",
"src_ip": "118.201.255.164",
"session": "f6609c750f97"
}
{
"eventid": "cowrie.client.var",
"name": "LANG",
"value": "yue_HK",
"message": "request_env: LANG=yue_HK",
"sensor": "",
"timestamp": "2023-04-03T11:49:16.140724Z",
"src_ip": "110.4.14.236",
"session": "e5bdae0ab2bd"
}
{
"eventid": "cowrie.client.var",
"name": "LANG",
"value": "yue_HK",
"message": "request_env: LANG=yue_HK",
"sensor": "",
"timestamp": "2023-04-03T11:49:16.607960Z",
"src_ip": "110.4.14.236",
"session": "e37b0e8bbf2e"
}
{
"eventid": "cowrie.client.var",
"name": "LANG",
"value": "yue_HK",
"message": "request_env: LANG=yue_HK",
"sensor": "",
"timestamp": "2023-04-03T11:49:20.174747Z",
"src_ip": "110.4.14.236",
"session": "2a723347f503"
}
{
"eventid": "cowrie.client.var",
"name": "LANG",
"value": "yue_HK",
"message": "request_env: LANG=yue_HK",
"sensor": "",
"timestamp": "2023-04-03T11:49:20.517283Z",
"src_ip": "110.4.14.236",
"session": "3f456c9ebde0"
I see that the 'name' field is related to a specific setting for cowrie. In these cases it's the "LANG" variable and that 'value' specifies the language. What if I wanted to only show items that didn't specify "en_GB.UTF-8" (UK English [5]) as the language?
# read cowrie json file
# cat cowrie.json
#
# process data by jq
# select data with value key is present
# select data where value key does not equal 'en_GB.UTF-8'
# | jq 'select(.value and .value!="en_GB.UTF-8")'
#
cat cowrie.json | jq 'select(.value and .value!="en_GB.UTF-8")'
{
"eventid": "cowrie.client.var",
"name": "LANG",
"value": "yue_HK",
"message": "request_env: LANG=yue_HK",
"sensor": "",
"timestamp": "2023-04-03T11:49:16.140724Z",
"src_ip": "110.4.14.236",
"session": "e5bdae0ab2bd"
}
{
"eventid": "cowrie.client.var",
"name": "LANG",
"value": "yue_HK",
"message": "request_env: LANG=yue_HK",
"sensor": "",
"timestamp": "2023-04-03T11:49:16.607960Z",
"src_ip": "110.4.14.236",
"session": "e37b0e8bbf2e"
}
{
"eventid": "cowrie.client.var",
"name": "LANG",
"value": "yue_HK",
"message": "request_env: LANG=yue_HK",
"sensor": "",
"timestamp": "2023-04-03T11:49:20.174747Z",
"src_ip": "110.4.14.236",
"session": "2a723347f503"
}
{
"eventid": "cowrie.client.var",
"name": "LANG",
"value": "yue_HK",
"message": "request_env: LANG=yue_HK",
"sensor": "",
"timestamp": "2023-04-03T11:49:20.517283Z",
"src_ip": "110.4.14.236",
"session": "3f456c9ebde0"
The tricky part here is in the search. Why didn't I just include the value I didn't want to appear? Why did I also look for the specific 'value' key existing again?
Well, if the search only looked for the 'value' that did not equal 'en_GB.UTF-8', it would also include any object that did not include the 'value' key. In those objects, it's still true that 'value' does not equal 'en_GB.UTF-8'.
Figure 2: Differences in lines returned with and without 'value' key specified
Now there's a basic understanding of the keys available and what some of the keys are used for. Now I'm in interested in what keys are used when a password is specified in the object.
# read JSON data from Cowrie data file (cowrie.json)
# cat cowrie.json
#
# process data by 'jq', outputting data in easily readable format
# data is also conveniently output on new lines, making other tools easy to process
# show data where the password key is present
# | jq 'select(.password)'
#
# return every line that has a colon (":")
# data is separated by [key]:[value]
# allows us to only display lines that have a key
# | grep ":"
#
# split data into parts, separating by colon (":") and return the first part
# first part contains the key
# second part contains the value
# | cut -d ":" -f 1
#
# sort data alphabetically
# will allow the next command to work more effectively without duplicates
# | sort
#
# only display each key once and how many times it was in the data
# | uniq -c
#
# sort the data by the number of times is appeared
# | sort -n
cat cowrie.json | jq 'select(.password)' | grep ":" | cut -d ":" -f 1 | sort | uniq -c | sort -n
4826 "eventid"
4826 "message"
4826 "password"
4826 "sensor"
4826 "session"
4826 "src_ip"
4826 "timestamp"
4826 "username"
If I just want to find the most common username and password combinations that are attempted, a bit more modification is needed.
# read cowrie json file
# cat cowrie.json
#
# process data with jq
# select keys 'username' and 'password' to display
# separate key data with a command and space ', '
# jq '.username + ", " + .password'
#
# sort the data
# | sort
#
# count unique values returned
# | uniq -c
#
# sort in descending order (reverse the sort)
# sort -r
#
# display 10 results
# | head -n 10
cat cowrie.json | jq '.username + ", " + .password' | sort | uniq -c | sort -r | head -n 10
9 "root, vizxv"
9 "root, user"
9 "root, GM8182"
9 "root, 7ujMko0admin"
9 "root, 666666"
9 "root, 00000000"
9 "mother, fucker"
9 "guest, guest"
9 "admin, Win1doW$"
9 "admin, vertex25ektks123"
Now for the fun part. I can see there is likely much more information associated with these particular entries. Looking at the most common keys, I suspect that 'session' is the primary key tying these objects together. What if I want to find every session, and related session data, in which the username/password combination of root/vizxv were used? First, lets see if how many unique sessions we get for that username and password combination. Based on the data and my assumptions I am anticipating 9 sessions.
# output the json data from cowrie file
# cat cowrie.json
#
# process data with jq
# select data where username is 'root' and password = 'vizxv'
# output 'session' key data
# | jq 'select(.username=="root" and .password=="vizxv") .session'
cat cowrie.json | jq 'select(.username=="root" and .password=="vizxv") .session'
"fe353544255a"
"6d2e67ed79a9"
"046a51841fa1"
"476830374333"
"76543f59c86c"
"0ec44b14798c"
"5498e01b3843"
"e6f5390528ba"
"9cc5565952c8"
#same command, but only return the number of lines of data returned
cat cowrie.json | jq 'select(.username=="root" and .password=="vizxv") .session' | wc -l
9
The data returned matches what was anticipated. 9 unique sessions were returned. One thing I'd like to be able to do in a one-liner is to take this search output and use these sessions to find all of the session data. However, I haven't quite found a good way to do that yet. If you've got an example, please put it in the comments!
We'll settle for getting collecting the data for one of these sessions and sorting the data by the timestamp so we can see the entries in chronological order.
# read cowrie json data file
# cat cowrie.json
#
# retrieve data matching session '9cc5565952c8'
# | jq '(select(.session=="9cc5565952c8"))'
#
# sort data by timestamp (
# -s (--slurp), read the data as one large array so it can be sorted by a key (timestamp)
# | jq -s 'sort_by(.timestamp)'
cat cowrie.json | jq '(select(.session=="9cc5565952c8"))' | jq -s 'sort_by(.timestamp)'
[
{
"eventid": "cowrie.session.connect",
"src_ip": "121.231.240.88",
"src_port": 37245,
"dst_ip": "192.168.68.178",
"dst_port": 2223,
"session": "9cc5565952c8",
"protocol": "telnet",
"message": "New connection: 121.231.240.88:37245 (192.168.68.178:2223) [session: 9cc5565952c8]",
"sensor": "",
"timestamp": "2023-04-03T22:09:44.692962Z"
},
{
"eventid": "cowrie.login.failed",
"username": "root",
"password": "hi3518",
"message": "login attempt [root/hi3518] failed",
"sensor": "",
"timestamp": "2023-04-03T22:09:46.676383Z",
"src_ip": "121.231.240.88",
"session": "9cc5565952c8"
},
{
"eventid": "cowrie.login.failed",
"username": "admin",
"password": "",
"message": "login attempt [admin/] failed",
"sensor": "",
"timestamp": "2023-04-03T22:09:49.290533Z",
"src_ip": "121.231.240.88",
"session": "9cc5565952c8"
},
{
"eventid": "cowrie.login.success",
"username": "root",
"password": "vizxv",
"message": "login attempt [root/vizxv] succeeded",
"sensor": "",
"timestamp": "2023-04-03T22:09:52.214314Z",
"src_ip": "121.231.240.88",
"session": "9cc5565952c8"
},
{
"eventid": "cowrie.session.params",
"arch": "linux-x64-lsb",
"message": [],
"sensor": "",
"timestamp": "2023-04-03T22:09:52.304601Z",
"src_ip": "121.231.240.88",
"session": "9cc5565952c8"
},
{
"eventid": "cowrie.command.input",
"input": "enable",
"message": "CMD: enable",
"sensor": "",
"timestamp": "2023-04-03T22:09:52.940228Z",
"src_ip": "121.231.240.88",
"session": "9cc5565952c8"
},
{
"eventid": "cowrie.command.input",
"input": "system",
"message": "CMD: system",
"sensor": "",
"timestamp": "2023-04-03T22:09:52.945289Z",
"src_ip": "121.231.240.88",
"session": "9cc5565952c8"
},
{
"eventid": "cowrie.command.failed",
"input": "system",
"message": "Command not found: system",
"sensor": "",
"timestamp": "2023-04-03T22:09:52.948144Z",
"src_ip": "121.231.240.88",
"session": "9cc5565952c8"
},
{
"eventid": "cowrie.command.input",
"input": "shell",
"message": "CMD: shell",
"sensor": "",
"timestamp": "2023-04-03T22:09:52.950730Z",
"src_ip": "121.231.240.88",
"session": "9cc5565952c8"
},
{
"eventid": "cowrie.command.failed",
"input": "shell",
"message": "Command not found: shell",
"sensor": "",
"timestamp": "2023-04-03T22:09:52.953512Z",
"src_ip": "121.231.240.88",
"session": "9cc5565952c8"
},
{
"eventid": "cowrie.command.input",
"input": "sh",
"message": "CMD: sh",
"sensor": "",
"timestamp": "2023-04-03T22:09:52.955858Z",
"src_ip": "121.231.240.88",
"session": "9cc5565952c8"
},
{
"eventid": "cowrie.command.input",
"input": "cat /proc/mounts; /bin/busybox ZROVY",
"message": "CMD: cat /proc/mounts; /bin/busybox ZROVY",
"sensor": "",
"timestamp": "2023-04-03T22:09:54.065891Z",
"src_ip": "121.231.240.88",
"session": "9cc5565952c8"
},
{
"eventid": "cowrie.command.input",
"input": "cd /dev/shm; cat .s || cp /bin/echo .s; /bin/busybox ZROVY",
"message": "CMD: cd /dev/shm; cat .s || cp /bin/echo .s; /bin/busybox ZROVY",
"sensor": "",
"timestamp": "2023-04-03T22:09:58.948773Z",
"src_ip": "121.231.240.88",
"session": "9cc5565952c8"
},
{
"eventid": "cowrie.command.input",
"input": "tftp; wget; /bin/busybox ZROVY",
"message": "CMD: tftp; wget; /bin/busybox ZROVY",
"sensor": "",
"timestamp": "2023-04-03T22:10:00.187402Z",
"src_ip": "121.231.240.88",
"session": "9cc5565952c8"
},
{
"eventid": "cowrie.command.input",
"input": "dd bs=52 count=1 if=.s || cat .s || while read i; do echo $i; done < .s",
"message": "CMD: dd bs=52 count=1 if=.s || cat .s || while read i; do echo $i; done < .s",
"sensor": "",
"timestamp": "2023-04-03T22:10:01.719670Z",
"src_ip": "121.231.240.88",
"session": "9cc5565952c8"
},
{
"eventid": "cowrie.command.failed",
"input": "while read i",
"message": "Command not found: while read i",
"sensor": "",
"timestamp": "2023-04-03T22:10:01.726645Z",
"src_ip": "121.231.240.88",
"session": "9cc5565952c8"
},
{
"eventid": "cowrie.command.input",
"input": "/bin/busybox ZROVY",
"message": "CMD: /bin/busybox ZROVY",
"sensor": "",
"timestamp": "2023-04-03T22:10:01.942564Z",
"src_ip": "121.231.240.88",
"session": "9cc5565952c8"
},
{
"eventid": "cowrie.command.input",
"input": "rm .s; exit",
"message": "CMD: rm .s; exit",
"sensor": "",
"timestamp": "2023-04-03T22:10:02.576389Z",
"src_ip": "121.231.240.88",
"session": "9cc5565952c8"
},
{
"eventid": "cowrie.log.closed",
"ttylog": "var/lib/cowrie/tty/ab19b42586e0648d2969b44c1e7541f2251d17f27efcb67976cfb2bbf07781b0",
"size": 3325,
"shasum": "ab19b42586e0648d2969b44c1e7541f2251d17f27efcb67976cfb2bbf07781b0",
"duplicate": false,
"duration": 10.278910875320435,
"message": "Closing TTY Log: var/lib/cowrie/tty/ab19b42586e0648d2969b44c1e7541f2251d17f27efcb67976cfb2bbf07781b0 after 10 seconds",
"sensor": "",
"timestamp": "2023-04-03T22:10:02.582532Z",
"src_ip": "121.231.240.88",
"session": "9cc5565952c8"
},
{
"eventid": "cowrie.session.closed",
"duration": 17.90323781967163,
"message": "Connection lost after 17 seconds",
"sensor": "",
"timestamp": "2023-04-03T22:10:02.595774Z",
"src_ip": "121.231.240.88",
"session": "9cc5565952c8"
}
]
Now that we have an easy way to search and sort, we can dig into different details that we might be interested in, such as the 'message' key.
# read cowrie json data file
# cat cowrie.json
#
# retrieve data matching session '9cc5565952c8'
# | jq '(select(.session=="9cc5565952c8"))'
#
# sort data by timestamp (
# -s (--slurp), read the data as one large array so it can be sorted by a key (timestamp)
# | jq -s 'sort_by(.timestamp)'
#
# retrieve message key
# .[] used to indicate that the key is within an array
# | jq .[].message
cat cowrie.json | jq '(select(.session=="9cc5565952c8"))' | jq -s 'sort_by(.timestamp)' | jq .[].message
"New connection: 121.231.240.88:37245 (192.168.68.178:2223) [session: 9cc5565952c8]"
"login attempt [root/hi3518] failed"
"login attempt [admin/] failed"
"login attempt [root/vizxv] succeeded"
[]
"CMD: enable"
"CMD: system"
"Command not found: system"
"CMD: shell"
"Command not found: shell"
"CMD: sh"
"CMD: cat /proc/mounts; /bin/busybox ZROVY"
"CMD: cd /dev/shm; cat .s || cp /bin/echo .s; /bin/busybox ZROVY"
"CMD: tftp; wget; /bin/busybox ZROVY"
"CMD: dd bs=52 count=1 if=.s || cat .s || while read i; do echo $i; done < .s"
"Command not found: while read i"
"CMD: /bin/busybox ZROVY"
"CMD: rm .s; exit"
"Closing TTY Log: var/lib/cowrie/tty/ab19b42586e0648d2969b44c1e7541f2251d17f27efcb67976cfb2bbf07781b0 after 10 seconds"
"Connection lost after 17 seconds"
This example also has another change since the data is now part of an array. Putting the data to an array was necessary so that we could sort it. The ".[]" is now prepended to the key value.
While are are other ways to process this kind of data, with enough practice it can be done easily through the command line with only a terminal. 'jq' is a nice addition to other tools such as 'awk', 'sed', 'cut', 'sort', etc. If you run into issues processing your cowrie files, some things to watch out for:
- Improperly formatted JSON
- Nested data and arrays (it's easy to refer to a key as '.key' when it should be '.[].key' or vice versa
[1] https://isc.sans.edu/diary/DShield+Sensor+JSON+Log+Analysis/29412/
[2] https://isc.sans.edu/diary/Filter+JSON+Data+by+Value+with+Linux+jq/27792
[3] https://stedolan.github.io/jq/
[4] https://www.w3schools.com/js/js_json_objects.asp
[5] https://docs.oracle.com/cd/E23824_01/html/E26033/glset.html
--
Jesse La Grew
Handler
Comments