How To Hunt on Sysmon Data

If you need a primer on Endpoint data, what it is and why it matters, check out this post. Otherwise, read on. In this post I’m going to specifically tackle the topic of Threat Hunting on Endpoint Data. I’ll use Sysmon as a source of endpoint data in the examples, mainly because it’s free and fairly easy to get started with. I’ll go into detail about Symon setup and configs in a future post, but you can still give it a try with a basic config and do some really effective hunting.

Basic Setup

To get started hunting, you need some data to hunt on (Sysmon) and, ideally, you need a place to store it that allows for fast and efficient search. For this I’d recommend Elasticsearch and Kibana. You can also use Splunk, but I’d prefer to steer you toward free tools just because they lower the bar for getting started. And getting started is always the hardest part. The other piece you’ll need is a way to send the Sysmon events to Elasticsearch. For that I’d recommend Winlogbeat, made by the same company as Elasticsearch, creatively named Elastic.

Elasticsearch Setup

Install Elasticsearch somewhere on your network that your Windows endpoints can reach. You could even do it right on your computer if you’re just going to hunt on that. You can find the documentation here.

You’ll also need to install Kibana, the front end to Elasticsearch. You can install it on a different box, or on the same one you installed Elasticsearch on. If you’re only going to hunt on a handful of Windows endpoints or less, then installing Kibana and Elasticsearch on the same box is perfectly fine. You can find the documentation for Kibana here.

I’m not going to go into detail on getting these installed because there are just too many ways to do it and Elastic’s documentation is pretty decent. If you are able to get these installed and working, then the rest of this how-to should be pretty easy.

Sysmon Setup

If you haven’t already, download Sysmon.

Install it with the following command:

sysmon64.exe -i -accepteula –h md5,sha256 –n

Go ahead and install Sysmon on several Windows endpoints, if you have them. Hunting is a lot more fun and interesting when you’re dealing with multiple endpoints.

To confirm that Sysmon is running, you can run the following command:

sc query sysmon64

Verify that the state is listed as running.

Winlogbeat Setup

Install and configure winlogbeat on all the endpoints that you installed Sysmon on. Rather than regurgitating the documentation for Winlogbeat, I’ll just point you to it here. One thing not listed in the documentation is the config line to pull Sysmon data from the Windows Event log. Here it is:

winlogbeat.event_logs:
  - name: Microsoft-Windows-Sysmon/Operational

To send the events to Elasticsearch, you simply need the following in the config:

output.elasticsearch:
  hosts:
    - localhost:9200 # Use the URL of your Elasticsearch server that you just set up

There are other things you can do with the Winlogbeat config, such as adding or removing fields, but for now this is all you’ll need.

To confirm that Winlogbeat is running, you can run the following command:

sc query winlogbeat

Verify that the state is listed as running.

Confirm Setup

Once Sysmon and Kibana are successfully installed, configured and running, you should start to see events in Elasticsearch.

To confirm that Elasticsearch is getting data, point your browser at the url of your Elasticsearch installation (not Kibana) with /_cat/indices on the end of the URL, for example:

http://localhost:9200/_cat/indices

This will give you a text listing of the indices currently in Elasticsearch, if you don’t see one, then you’re data isn’t making it to Elasticsearch. This is typically because of a misconfiguration of Winlogbeat, or a networking problem that prevents traffic from your endpoints from reaching your Elasticsearch server.

If you don’t see an index in Elasticsearch, go to your Windows endpoint where you installed Sysmon and Winlogbeat and check the Winlogbeat log files. They are pretty good about reporting errors encountered while sending data out. You can find that log file in [Winlogbeat Install Dir]\logs\. Open winlogbeat with a text editor and scroll down and look for errors. This should help you troubleshoot any data forwarding issues.

Once you confirm that data is making it to Elasticsearch and you see an index listed at the _cat/indices URL, then you can point your browser at the Kibana URL to load up Kibana. Once you’re in Kibana, follow these initial setup steps:

Navigate to the management page
Click on index patterns
Click Create Index Pattern
Follow the prompts to create a new index pattern that matches the index name you saw in the index listing previously

Now go to the Discover tab in Kibana and you should see some events. If you don’t, then you have some more troubleshooting to do. If you do see some events, then you’re ready to go on to the next section.

One other thing, make sure you do a few things on the target Windows computer that you’re going to hunt on. Launch a few programs, and create some activity by using the computer as you normally would for a couple of minutes, otherwise it won’t have many events to send to Elasticsearch.

Start Hunting

Ok, so now you’ve got data that you can see in Kibana. Nice work! Now the hunting begins.

Orient yourself to the data

The best way to start is to first get acquainted with the data (and Kibana). In Kibana, the data is hard to read unless you do some tweaking. To start, do a search in the Kibana search bar for:

event_id:1

Search Bar

This will filter the list of events to just Process Creation events, meaning events that were triggered when a program was launched. Click on the little arrow next to one of the events to expand it. Look through the fields in the event so you can start to understand what fields exist for a process create event. The primary field of interest is Image. Click on the little column looking icon by Image to add that field to your overall results table in Kibana. This will make it easier to look through your search results in the future.

Column Icon

Another interesting field is CommandLine. There you’ll see the exact command line that was used to launch this process.

CommandLine

A quick note on field names. Some of the field names I’m referencing will be a subfield of event_data. I won’t always reference event_data for each field for the sake of brevity and because we normalize our events to move those fields up a level. So, if you can’t find a field I’m referencing or a search isn’t working, make sure you prepend event_data. to the field name when you reference it, for example:

event_data.Image

Ok, moving on.

Another set of fields that makes Sysmon really unique and powerful are the Parent fields: ParentImage, ParentCommandLine, etc. These are interesting because they give us easy access to the most basic behavior we can hunt for, which is programs launching other programs, or more technically parent-child relationships between programs. Go ahead and click the column icon on CommandLine and ParentImage as well. Then collapse that event so you can see the results table. Your events should now be displayed as a table like this:

Kibana Results

If you launched a few programs after you installed Sysmon, make sure you see those programs in the results table. You should see the full path and filename for the programs you launched in the Image column.

At this point, it’s a lot of fun to create some activity and then see what you can find in Kibana. Remove the event_id:1 filter you did previously and look at some different event types. Look at a few of the events in detail to see if you can find anything interesting.

Just spending some time here is really helpful to get you thinking about a user action and how that gets translated into endpoint events that you can hunt on.

Some other interesting event_id’s to search for are 3, 10 and 11. You can find the human-readable name of the events in the task field. You can also find a list of all Sysmon events here.

Start Searching

Things get more interesting when you have more data. If you need to create more activity, or just let Sysmon run while you use the computer for a while, go ahead and do that. If you think you have enough data, then start doing some searches.

You can search for any value in any field. Some of the fields you might have discovered previously are event_id, which specifies the type of event using an integer, Image, CommandLine, ParentImage, ParentCommandLine. For network connect events (event_id 3) some of the interesting fields are DestinationIp, DestinationPort and DestinationHostname.

You can filter your results in Kibana by specifying a fieldname, followed by a colon, followed by a search term. Here are some examples:

Show me all Process Create events: event_id:1

Show me all Network Connect events: event_id:3

Show me all events that Google Chrome generated: Image:*chrome.exe

Show me all programs launched from a command shell: ParentImage:*cmd.exe AND event_id:1

Show me where Chrome launched a command shell: Image:*cmd.exe AND ParentImage:*chrome.exe

Also, in case it isn’t obvious, you can adjust the time range of your search in the upper-right hand corner of Kibana. I typically do the last 4 hours unless I have a specific time range I’m hunting on. If you’re dealing with a lot of data and an under-powered Elasticsearch server, then larger time ranges mean slower results.

Next Steps

Ok, now you’re hunting. You may not know what you’re hunting for yet, but as you get more familiar with the data and develop a plan, you can start to focus your hunting efforts. Depending on your environment, below are some things you might want to consider adding to your hunting regimen. While you’re not going to find most of these in your test environment, they would be worth hunting for in a larger, corporate environment.

Webservers Launching a Shell: Search for a command shell, like cmd.exe, with a webserver, like w3wp.exe, as a parent. This would be indicative of a webshell attack, often used as a mean of persistence by advanced attackers.
Svchost Launching a Shell: Search for a shell with a parent of svchost.exe. This is weird, and probably not good.
Unusual Parent of Svchost: Search for svchost.exe where services.exe is not the parent. You can see from our Insights data here that services.exe should be the parent of svchost.exe. Anything else is anomalous and worth some time investigating.
Powershell Encoded Commands: Search for powershell with a command line argument of -enc (note there are other variations of this command line switch). Powershell has a feature allowing for commands to be encoded with base64 encoding. While this is sometimes used for legitimate purposes, it can also be used by attackers to obfuscate their activities. If you find any, you can decode the command easily from the command line or a web-based tool.
Logon Bypass: Search for a shell, like cmd.exe, being launched by winlogon.exe. This is indicative a Sticky-Keys attack where an attacker is able to create persistence by modifying the registry to launch a command shell at the logon screen when they hit the shift key 5 times. This is bad.
Powershell Download: Search for powershell with the strings DownloadString or DownloadFile in the command line. This is similar to encoded command in that you might find some legitimate usage, but it is often used by attackers to pull down commands or files from a remote server to achieve their objective.

Ok, that’s all for now. There are many more things that we hunt for, but that should get you started. I hope this has been helpful in giving you some understanding of the possibilities that open up when you have access to endpoint process behavior data like that which Sysmon provides.

There is so much more I could talk about both in terms of Threat Hunting as well as the best way to use Elasticsearch and Kibana. I’ll save that for future posts, but please reach out via Twitter or any other means to let me know what you’d like to hear more about.

Happy Hunting!

EchoTrail Blog

hunting

How To Hunt on Sysmon Data

January 28 2019

general

Why We Chose to Make Insights Free

November 27 2018

hunting

Windows Process Behavior

November 02 2018

general

Welcome

November 01 2018