Windows Process Behavior
An overview of Windows process behavior and why it matters
If you need a primer on Endpoint data, what it is and why it matters, check out this post. Otherwise, read on. In this post I'm going to specifically tackle the topic of Threat Hunting on Endpoint Data. I'll use Sysmon as a source of endpoint data in the examples, mainly because it's free and fairly easy to get started with. I'll go into detail about Symon setup and configs in a future post, but you can still give it a try with a basic config and do some really effective hunting.
To get started hunting, you need some data to hunt on (Sysmon) and, ideally, you need a place to store it that allows for fast and efficient search. For this I'd recommend Elasticsearch and Kibana. You can also use Splunk, but I'd prefer to steer you toward free tools just because they lower the bar for getting started. And getting started is always the hardest part. The other piece you'll need is a way to send the Sysmon events to Elasticsearch. For that I'd recommend Winlogbeat, made by the same company as Elasticsearch, creatively named Elastic.
Install Elasticsearch somewhere on your network that your Windows endpoints can reach. You could even do it right on your computer if you're just going to hunt on that. You can find the documentation here.
You'll also need to install Kibana, the front end to Elasticsearch. You can install it on a different box, or on the same one you installed Elasticsearch on. If you're only going to hunt on a handful of Windows endpoints or less, then installing Kibana and Elasticsearch on the same box is perfectly fine. You can find the documentation for Kibana here.
I'm not going to go into detail on getting these installed because there are just too many ways to do it and Elastic's documentation is pretty decent. If you are able to get these installed and working, then the rest of this how-to should be pretty easy.
If you haven't already, download Sysmon.
Install it with the following command:
sysmon64.exe -i -accepteula –h md5,sha256 –n
Go ahead and install Sysmon on several Windows endpoints, if you have them. Hunting is a lot more fun and interesting when you're dealing with multiple endpoints.
To confirm that Sysmon is running, you can run the following command:
sc query sysmon64
Verify that the state is listed as running.
Install and configure winlogbeat on all the endpoints that you installed Sysmon on. Rather than regurgitating the documentation for Winlogbeat, I'll just point you to it here. One thing not listed in the documentation is the config line to pull Sysmon data from the Windows Event log. Here it is:
winlogbeat.event_logs: - name: Microsoft-Windows-Sysmon/Operational
To send the events to Elasticsearch, you simply need the following in the config:
output.elasticsearch: hosts: - localhost:9200 # Use the URL of your Elasticsearch server that you just set up
There are other things you can do with the Winlogbeat config, such as adding or removing fields, but for now this is all you'll need.
To confirm that Winlogbeat is running, you can run the following command:
sc query winlogbeat
Verify that the state is listed as running.
Once Sysmon and Kibana are successfully installed, configured and running, you should start to see events in Elasticsearch.
To confirm that Elasticsearch is getting data, point your browser at the url of your Elasticsearch installation (not Kibana) with /_cat/indices on the end of the URL, for example:
This will give you a text listing of the indices currently in Elasticsearch, if you don't see one, then you're data isn't making it to Elasticsearch. This is typically because of a misconfiguration of Winlogbeat, or a networking problem that prevents traffic from your endpoints from reaching your Elasticsearch server.
If you don't see an index in Elasticsearch, go to your Windows endpoint where you installed Sysmon and Winlogbeat and check the Winlogbeat log files. They are pretty good about reporting errors encountered while sending data out. You can find that log file in
[Winlogbeat Install Dir]\logs\. Open
winlogbeat with a text editor and scroll down and look for errors. This should help you troubleshoot any data forwarding issues.
Once you confirm that data is making it to Elasticsearch and you see an index listed at the _cat/indices URL, then you can point your browser at the Kibana URL to load up Kibana. Once you're in Kibana, follow these initial setup steps:
Now go to the Discover tab in Kibana and you should see some events. If you don't, then you have some more troubleshooting to do. If you do see some events, then you're ready to go on to the next section.
One other thing, make sure you do a few things on the target Windows computer that you're going to hunt on. Launch a few programs, and create some activity by using the computer as you normally would for a couple of minutes, otherwise it won't have many events to send to Elasticsearch.
Ok, so now you've got data that you can see in Kibana. Nice work! Now the hunting begins.
The best way to start is to first get acquainted with the data (and Kibana). In Kibana, the data is hard to read unless you do some tweaking. To start, do a search in the Kibana search bar for:
This will filter the list of events to just Process Creation events, meaning events that were triggered when a program was launched. Click on the little arrow next to one of the events to expand it. Look through the fields in the event so you can start to understand what fields exist for a process create event. The primary field of interest is Image. Click on the little column looking icon by Image to add that field to your overall results table in Kibana. This will make it easier to look through your search results in the future.
Another interesting field is CommandLine. There you'll see the exact command line that was used to launch this process.
A quick note on field names. Some of the field names I'm referencing will be a subfield of
event_data. I won't always reference event_data for each field for the sake of brevity and because we normalize our events to move those fields up a level. So, if you can't find a field I'm referencing or a search isn't working, make sure you prepend
event_data. to the field name when you reference it, for example:
Ok, moving on.
Another set of fields that makes Sysmon really unique and powerful are the Parent fields:
ParentCommandLine, etc. These are interesting because they give us easy access to the most basic behavior we can hunt for, which is programs launching other programs, or more technically parent-child relationships between programs. Go ahead and click the column icon on CommandLine and ParentImage as well. Then collapse that event so you can see the results table. Your events should now be displayed as a table like this:
If you launched a few programs after you installed Sysmon, make sure you see those programs in the results table. You should see the full path and filename for the programs you launched in the Image column.
At this point, it's a lot of fun to create some activity and then see what you can find in Kibana. Remove the event_id:1 filter you did previously and look at some different event types. Look at a few of the events in detail to see if you can find anything interesting.
Just spending some time here is really helpful to get you thinking about a user action and how that gets translated into endpoint events that you can hunt on.
Some other interesting event_id's to search for are 3, 10 and 11. You can find the human-readable name of the events in the task field. You can also find a list of all Sysmon events here.
Things get more interesting when you have more data. If you need to create more activity, or just let Sysmon run while you use the computer for a while, go ahead and do that. If you think you have enough data, then start doing some searches.
You can search for any value in any field. Some of the fields you might have discovered previously are
event_id, which specifies the type of event using an integer,
ParentCommandLine. For network connect events (event_id 3) some of the interesting fields are
You can filter your results in Kibana by specifying a fieldname, followed by a colon, followed by a search term. Here are some examples:
Show me all Process Create events:
Show me all Network Connect events:
Show me all events that Google Chrome generated:
Show me all programs launched from a command shell:
ParentImage:*cmd.exe AND event_id:1
Show me where Chrome launched a command shell:
Image:*cmd.exe AND ParentImage:*chrome.exe
Also, in case it isn't obvious, you can adjust the time range of your search in the upper-right hand corner of Kibana. I typically do the last 4 hours unless I have a specific time range I'm hunting on. If you're dealing with a lot of data and an under-powered Elasticsearch server, then larger time ranges mean slower results.
Ok, now you're hunting. You may not know what you're hunting for yet, but as you get more familiar with the data and develop a plan, you can start to focus your hunting efforts. Depending on your environment, below are some things you might want to consider adding to your hunting regimen. While you're not going to find most of these in your test environment, they would be worth hunting for in a larger, corporate environment.
Ok, that's all for now. There are many more things that we hunt for, but that should get you started. I hope this has been helpful in giving you some understanding of the possibilities that open up when you have access to endpoint process behavior data like that which Sysmon provides.
There is so much more I could talk about both in terms of Threat Hunting as well as the best way to use Elasticsearch and Kibana. I'll save that for future posts, but please reach out via Twitter or any other means to let me know what you'd like to hear more about.
Co-Founder and CEO of EchoTrail
Join our mailing list for updates about product launches, availability, news and more.