Threat Hunting, Process Behavior, Culture...

EchoTrail Blog

<h2 id="how-to-hunt-on-sysmon-data">How To Hunt on Sysmon Data</h2>
<p>If you need a primer on Endpoint data, what it is and why it matters, check out this <a href="/blog/understanding_windows_process_behavior">post</a>. Otherwise, read on. In this post I&#39;m going to specifically tackle the topic of <strong>Threat Hunting on Endpoint Data</strong>. I&#39;ll use <a href="https://docs.microsoft.com/en-us/sysinternals/downloads/sysmon">Sysmon</a> as a source of endpoint data in the examples, mainly because it&#39;s free and fairly easy to get started with. I&#39;ll go into detail about Symon setup and configs in a future post, but you can still give it a try with a basic config and do some really effective hunting.</p>
<h2 id="basic-setup">Basic Setup</h2>
<p>To get started hunting, you need some data to hunt on (Sysmon) and, ideally, you need a place to store it that allows for fast and efficient search. For this I&#39;d recommend <a href="https://www.elastic.co/products/elasticsearch">Elasticsearch</a> and <a href="https://www.elastic.co/products/kibana">Kibana</a>. You can also use <a href="https://www.splunk.com/">Splunk</a>, but I&#39;d prefer to steer you toward free tools just because they lower the bar for getting started. And getting started is always the hardest part. The other piece you&#39;ll need is a way to send the Sysmon events to Elasticsearch. For that I&#39;d recommend <a href="https://www.elastic.co/products/beats/winlogbeat">Winlogbeat</a>, made by the same company as Elasticsearch, creatively named Elastic.</p>
<h3 id="elasticsearch-setup">Elasticsearch Setup</h3>
<p>Install Elasticsearch somewhere on your network that your Windows endpoints can reach. You could even do it right on your computer if you&#39;re just going to hunt on that. You can find the documentation <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html">here</a>.</p>
<p>You&#39;ll also need to install Kibana, the front end to Elasticsearch. You can install it on a different box, or on the same one you installed Elasticsearch on. If you&#39;re only going to hunt on a handful of Windows endpoints or less, then installing Kibana and Elasticsearch on the same box is perfectly fine. You can find the documentation for Kibana <a href="https://www.elastic.co/guide/en/kibana/current/index.html">here</a>.</p>
<p>I&#39;m not going to go into detail on getting these installed because there are just too many ways to do it and Elastic&#39;s documentation is pretty decent. If you are able to get these installed and working, then the rest of this how-to should be pretty easy.</p>
<h3 id="sysmon-setup">Sysmon Setup</h3>
<p>If you haven&#39;t already, download <a href="https://docs.microsoft.com/en-us/sysinternals/downloads/sysmon">Sysmon</a>.</p>
<p>Install it with the following command:</p>
<p><code>sysmon64.exe -i -accepteula –h md5,sha256 –n</code></p>
<p>Go ahead and install Sysmon on several Windows endpoints, if you have them. Hunting is a lot more fun and interesting when you&#39;re dealing with multiple endpoints.</p>
<p>To confirm that Sysmon is running, you can run the following command:</p>
<p><code>sc query sysmon64</code></p>
<p>Verify that the state is listed as running.</p>
<h3 id="winlogbeat-setup">Winlogbeat Setup</h3>
<p>Install and configure winlogbeat on all the endpoints that you installed Sysmon on. Rather than regurgitating the documentation for Winlogbeat, I&#39;ll just point you to it <a href="https://www.elastic.co/guide/en/beats/winlogbeat/current/index.html">here</a>. One thing not listed in the documentation is the config line to pull Sysmon data from the Windows Event log. Here it is:</p>
<pre><code class="language-yaml">winlogbeat.event_logs:
  - name: Microsoft-Windows-Sysmon/Operational</code></pre>
<p>To send the events to Elasticsearch, you simply need the following in the config:</p>
<pre><code class="language-yaml">output.elasticsearch:
  hosts:
    - localhost:9200 # Use the URL of your Elasticsearch server that you just set up</code></pre>
<p>There are other things you can do with the Winlogbeat config, such as adding or removing fields, but for now this is all you&#39;ll need.</p>
<p>To confirm that Winlogbeat is running, you can run the following command:</p>
<p><code>sc query winlogbeat</code></p>
<p>Verify that the state is listed as running.</p>
<h3 id="confirm-setup">Confirm Setup</h3>
<p>Once Sysmon and Kibana are successfully installed, configured and running, you should start to see events in Elasticsearch. </p>
<p>To confirm that Elasticsearch is getting data, point your browser at the url of your Elasticsearch installation (not Kibana) with <em>/_cat/indices</em> on the end of the URL, for example:</p>
<p><code>http://localhost:9200/_cat/indices</code></p>
<p>This will give you a text listing of the indices currently in Elasticsearch, if you don&#39;t see one, then you&#39;re data isn&#39;t making it to Elasticsearch. This is typically because of a misconfiguration of Winlogbeat, or a networking problem that prevents traffic from your endpoints from reaching your Elasticsearch server.</p>
<p>If you don&#39;t see an index in Elasticsearch, go to your Windows endpoint where you installed Sysmon and Winlogbeat and check the Winlogbeat log files. They are pretty good about reporting errors encountered while sending data out. You can find that log file in <code>[Winlogbeat Install Dir]\logs\</code>. Open <code>winlogbeat</code> with a text editor and scroll down and look for errors. This should help you troubleshoot any data forwarding issues.</p>
<p>Once you confirm that data is making it to Elasticsearch and you see an index listed at the _cat/indices URL, then you can point your browser at the Kibana URL to load up Kibana. Once you&#39;re in Kibana, follow these initial setup steps:</p>
<ol>
<li>Navigate to the management page</li>
<li>Click on index patterns</li>
<li>Click Create Index Pattern</li>
<li>Follow the prompts to create a new index pattern that matches the index name you saw in the index listing previously</li>
</ol>
<p>Now go to the Discover tab in Kibana and you should see some events. If you don&#39;t, then you have some more troubleshooting to do. If you do see some events, then you&#39;re ready to go on to the next section. </p>
<p>One other thing, make sure you do a few things on the target Windows computer that you&#39;re going to hunt on. Launch a few programs, and create some activity by using the computer as you normally would for a couple of minutes, otherwise it won&#39;t have many events to send to Elasticsearch.</p>
<h2 id="start-hunting">Start Hunting</h2>
<p>Ok, so now you&#39;ve got data that you can see in Kibana. Nice work! Now the hunting begins.</p>
<h3 id="orient-yourself-to-the-data">Orient yourself to the data</h3>
<p>The best way to start is to first get acquainted with the data (and Kibana). In Kibana, the data is hard to read unless you do some tweaking. To start, do a search in the Kibana search bar for:</p>
<p><code>event_id:1</code></p>
<p><img src="/images/blog/search_bar.jpg" alt="Search Bar"></p>
<p>This will filter the list of events to just <strong>Process Creation</strong> events, meaning events that were triggered when a program was launched. Click on the little arrow next to one of the events to expand it. Look through the fields in the event so you can start to understand what fields exist for a process create event. The primary field of interest is <strong>Image</strong>. Click on the little column looking icon by Image to add that field to your overall results table in Kibana. This will make it easier to look through your search results in the future.</p>
<p><img src="/images/blog/column_icon.jpg" alt="Column Icon"></p>
<p>Another interesting field is <strong>CommandLine</strong>. There you&#39;ll see the exact command line that was used to launch this process.</p>
<p><img src="/images/blog/commandline.jpg" alt="CommandLine"></p>
<p>A quick note on field names. Some of the field names I&#39;m referencing will be a subfield of <code>event_data</code>. I won&#39;t always reference event_data for each field for the sake of brevity and because we normalize our events to move those fields up a level. So, if you can&#39;t find a field I&#39;m referencing or a search isn&#39;t working, make sure you prepend <code>event_data.</code> to the field name when you reference it, for example:</p>
<p><code>event_data.Image</code></p>
<p>Ok, moving on.</p>
<p>Another set of fields that makes Sysmon really unique and powerful are the Parent fields: <code>ParentImage</code>, <code>ParentCommandLine</code>, etc. These are interesting because they give us easy access to the most basic behavior we can hunt for, which is programs launching other programs, or more technically parent-child relationships between programs. Go ahead and click the column icon on CommandLine and ParentImage as well. Then collapse that event so you can see the results table. Your events should now be displayed as a table like this:</p>
<p><img src="/images/blog/kibana_results.jpg" alt="Kibana Results"></p>
<p>If you launched a few programs after you installed Sysmon, make sure you see those programs in the results table. You should see the full path and filename for the programs you launched in the Image column.</p>
<p>At this point, it&#39;s a lot of fun to create some activity and then see what you can find in Kibana. Remove the <em>event_id:1</em> filter you did previously and look at some different event types. Look at a few of the events in detail to see if you can find anything interesting. </p>
<p>Just spending some time here is really helpful to get you thinking about a user action and how that gets translated into endpoint events that you can hunt on. </p>
<p>Some other interesting event_id&#39;s to search for are 3, 10 and 11. You can find the human-readable name of the events in the task field. You can also find a list of all Sysmon events <a href="https://docs.microsoft.com/en-us/sysinternals/downloads/sysmon">here</a>.</p>
<h3 id="start-searching">Start Searching</h3>
<p>Things get more interesting when you have more data. If you need to create more activity, or just let Sysmon run while you use the computer for a while, go ahead and do that. If you think you have enough data, then start doing some searches.</p>
<p>You can search for any value in any field. Some of the fields you might have discovered previously are <code>event_id</code>, which specifies the type of event using an integer, <code>Image</code>, <code>CommandLine</code>, <code>ParentImage</code>, <code>ParentCommandLine</code>. For network connect events (event_id 3) some of the interesting fields are <code>DestinationIp</code>, <code>DestinationPort</code> and <code>DestinationHostname</code>.</p>
<p>You can filter your results in Kibana by specifying a fieldname, followed by a colon, followed by a search term. Here are some examples:</p>
<p>Show me all Process Create events:
<code>event_id:1</code></p>
<p>Show me all Network Connect events:
<code>event_id:3</code></p>
<p>Show me all events that Google Chrome generated:
<code>Image:*chrome.exe</code></p>
<p>Show me all programs launched from a command shell:
<code>ParentImage:*cmd.exe AND event_id:1</code></p>
<p>Show me where Chrome launched a command shell:
<code>Image:*cmd.exe AND ParentImage:*chrome.exe</code></p>
<p>Also, in case it isn&#39;t obvious, you can adjust the time range of your search in the upper-right hand corner of Kibana. I typically do the last 4 hours unless I have a specific time range I&#39;m hunting on. If you&#39;re dealing with a lot of data and an under-powered Elasticsearch server, then larger time ranges mean slower results.</p>
<h2 id="next-steps">Next Steps</h2>
<p>Ok, now you&#39;re hunting. You may not know what you&#39;re hunting for yet, but as you get more familiar with the data and develop a plan, you can start to focus your hunting efforts. Depending on your environment, below are some things you might want to consider adding to your hunting regimen. While you&#39;re not going to find most of these in your test environment, they would be worth hunting for in a larger, corporate environment.</p>
<ul>
<li><strong>Webservers Launching a Shell</strong>: Search for a command shell, like cmd.exe, with a webserver, like w3wp.exe, as a parent. This would be indicative of a webshell attack, often used as a mean of persistence by advanced attackers.</li>
<li><strong>Svchost Launching a Shell</strong>: Search for a shell with a parent of svchost.exe. This is weird, and probably not good.</li>
<li><strong>Unusual Parent of Svchost</strong>: Search for svchost.exe where services.exe is not the parent. You can see from our Insights data <a href="/insights/search/svchost.exe">here</a> that services.exe should be the parent of svchost.exe. Anything else is anomalous and worth some time investigating.</li>
<li><strong>Powershell Encoded Commands</strong>: Search for powershell with a command line argument of -enc (note there are other variations of this command line switch). Powershell has a feature allowing for commands to be encoded with base64 encoding. While this is sometimes used for legitimate purposes, it can also be used by attackers to obfuscate their activities. If you find any, you can decode the command easily from the command line or a web-based tool.</li>
<li><strong>Logon Bypass</strong>: Search for a shell, like cmd.exe, being launched by winlogon.exe. This is indicative a Sticky-Keys attack where an attacker is able to create persistence by modifying the registry to launch a command shell at the logon screen when they hit the shift key 5 times. This is bad.</li>
<li><strong>Powershell Download</strong>: Search for powershell with the strings <em>DownloadString</em> or <em>DownloadFile</em> in the command line. This is similar to encoded command in that you might find some legitimate usage, but it is often used by attackers to pull down commands or files from a remote server to achieve their objective.</li>
</ul>
<p>Ok, that&#39;s all for now. There are many more things that we hunt for, but that should get you started. I hope this has been helpful in giving you some understanding of the possibilities that open up when you have access to endpoint process behavior data like that which Sysmon provides.</p>
<p>There is so much more I could talk about both in terms of Threat Hunting as well as the best way to use Elasticsearch and Kibana. I&#39;ll save that for future posts, but please reach out via Twitter or any other means to let me know what you&#39;d like to hear more about.</p>
<p>Happy Hunting!</p>


Threat Hunting on Endpoint Data with Sysmon

How To Hunt on Sysmon Data

<h2 id="holiday-reflections">Holiday Reflections</h2>
<p>I wanted to take a minute to share some thoughts and reflections I had over the holiday break about our new <a href="/products/insights">Insights</a> database. We recently launched what we believe to be a cutting edge, <a href="/">searchable</a> database of analytics surrounding the behavior of Windows processes in the wild. We made much of it free in the beginning and then began to hold back some of the data and analytics in favor of paid plans. That approach made sense from a business perspective, but never really sat well with me otherwise. We want to be giving back to the community as much as possible. Cyber-security is a hard field for outsiders to break into, and there is so much to learn for entrants into the field. </p>
<p>We want to be a part of those helping to educate and give back, and we believe our Insights database can contribute a small piece to that effort. For that reason, among others, we’ve decided to make our database completely free to search for everyone, and we hope to be able to continue to do so for the foreseeable future. While we’ll still charge for automated API access, we believe there’s too much valuable insight ;) packed into our dataset to hold behind closed doors. With that said, please go forth and search away and we’ll keep adding more and more data and insights for you to uncover.</p>
<h2 id="automation">Automation</h2>
<p>Our goal is to make the Insights dataset as wide and deep as possible and to follow an API-first strategy. Nowadays, automation is a given in most security organizations and we want to enable security teams to quickly and easily bring additional context into their existing workflows and to give their analysts a helpful leg up in triaging and responding to alerts. In most detection spaces, especially endpoint detection, alerts will involve a file or process and usually include an identifying hash or filename. Having worked as analysts and assembled teams of analysts ourselves, we know firsthand that as an analyst you want all the context you can get, but you don’t necessarily want to have to open 37 tabs in your browser to get it all. With the emergence of Security Orchestration and Automation (SOAR) platforms, bringing in additional data and context is getting even easier. We&#39;re working with SOAR and SIEM providers to make our data available directly through those platforms.</p>
<h2 id="help-wanted">Help Wanted</h2>
<p>We’re a young company and while our new Insights product has existed for a few years in our minds, it only recently came into existence. We aspire to make it one of a handful of go to place for hashes and filenames and have plans to expand our platform coverage and deepen the insights that we make available. That being said, we could use your support. If you have feedback, or want to contribute data, analysis or help in another way, please reach out! We believe strongly in partnerships and know that we’ll only reach our lofty goals working with others to get there.</p>
<p>Thanks for reading!</p>


Why We Chose to Make Insights Free

<h2 id="summary">Summary</h2>
<p>This post covers Windows Process Behavior from a very high level. Look for future posts going into much more depth on a variety of topics surrounding Process Behavior and how it can be leveraged to hunt for and detect attackers on your endpoints.</p>
<h2 id="windows-process-behavior">Windows Process Behavior</h2>
<p>A technique often used by the Endpoint Detection and Response (EDR) industry is to detect attacker activity based on the behaviors exhibited by processes on your endpoints. All of that sounds pretty fancy, but I want to break it down for you here.</p>
<p>Process behavior simply refers to what a particular program does when it runs on a computer. For example, when you start Microsoft Outlook, your computer launches the Outlook executable, <a href="/insights/search/outlook.exe">outlook.exe</a>, loads it into memory and begins to execute its instructions. Since every program is different, each one behaves a little differently after it launches.</p>
<p>After Outlook.exe launches, it loads some library files that it needs to carry out its functions. After that you might see it make a network connection to retrieve emails, and read and write files to the hard drive as the user interacts with it. You might also see it launch a child process when a user clicks on an attachment. For example, if a user clicks on a Word Doc attachment, you would see Outlook launch <a href="/insights/search/winword.exe">winword.exe</a>.</p>
<p>Each of these activities that we see a process exhibit are referred to as behaviors. Here is a list of common behaviors many EDR products capture (roughly in order of importance):</p>
<ol>
<li>Parents</li>
<li>Children</li>
<li>Executable Path</li>
<li>Grandparents</li>
<li>Command Line Arguments</li>
<li>Network Connections</li>
<li>Executable Hash</li>
<li>File Writes</li>
<li>Module Loads</li>
</ol>
<p>Most EDR products use these behaviors to detect anomalous or unwanted activity, and some make the raw data available for you to analyze yourself.</p>
<p>This brings into light the concept of process parents, children, grandparents, etc. Collectively, we call this <strong>Process Ancestry</strong> and spending some time to understand it can be extremely fruitful for threat hunting and detection on endpoints.</p>
<h2 id="understanding-normal">Understanding Normal</h2>
<p>If our ultimate goal is to detect threats to our environment, then it pays to understand what normal process behavior looks like. If we understand that, then we’re much more likely to be able to pick out anomalies.</p>
<p>This understanding can be helpful in a couple of ways:</p>
<p>One, it can give you <strong>context</strong> as you interpret and investigate alerts coming from an EDR product. Despite industry efforts to reduce false positives, they’re still common and it’s up to you to investigate and make a determination as to the veracity and the severity of each alert.</p>
<p>Another, even more pertinent use case, is in <strong>Threat Hunting</strong>. Starting with an understanding of typical process behavior, you can start to build a list of behaviors that you would <strong>not</strong> expect to see in your environment and you can begin hunting for those.</p>
<h2 id="a-specific-case">A Specific Case</h2>
<p>Let’s take <a href="/insights/search/winlogon.exe">winlogon.exe</a> as an example. Click on the link to read more about what it does. Also, take a look at its children in the table below. Note that you don’t see any typical processes that a user would run, like calc.exe, winword.exe or cmd.exe. Based on the description of winlogon.exe, along with some data to back it up, we can deduce that it shouldn&#39;t launch programs like these under normal circumstances.</p>
<p><img src="/images/blog/winlogon_children.jpg" alt="Winlogon Children"></p>
<p>There is a logon bypass technique, called Sticky Keys, that allows a person to access a shell like <a href="/insights/search/cmd.exe">cmd.exe</a> without logging in, simply by modifying a registry key. Once modified, this allows a user to bring up a command prompt on top of the logon screen simply by pressing the shift key 5 times. This invokes an accessibility feature in Windows that would normally bring up a window offering logon assistance to the user, but in this case brings up a command prompt instead. Note: this is a persistence technique, not an exploit to gain initial access. </p>
<p>When this technique is leveraged by an attacker, you would see winlogon.exe launch a child of cmd.exe, or if you were looking from cmd.exe’s perspective, you’d see winlogon.exe as its parent.</p>
<p>Using a database like <a href="/products/insights">Insights</a> to confirm that winlogon.exe launching cmd.exe is either never seen, or exceedingly rare, you can feel confident adding this behavior to your future hunting efforts.</p>
<h2 id="getting-started">Getting Started</h2>
<p>If you’re wondering how to get started actually doing this. Take a look at <a href="https://docs.microsoft.com/en-us/sysinternals/downloads/sysmon">Sysmon</a> as an endpoint behavior logging tool and take a look at this <a href="https://www.vector8.io/blog/sysmon">post</a> I wrote to help get you started. Also, sign up for the mailing list below as there will be more posts on this topic to follow.</p>
<p>Check out our <a href="/blog/welcome">Welcome</a> post for a list of upcoming topics.</p>


An overview of Windows process behavior and why it matters

Windows Process Behavior

<h2 id="welcome">Welcome</h2>
<p>Welcome to the first EchoTrail blog post! I&#39;m <a href="https://www.linkedin.com/in/concannon-brian">Brian Concannon</a>, co-founder and CEO of EchoTrail. I&#39;ll be writing several posts over the next few months on Windows process behavior, Threat Hunting and Detection, Cloud Architecture, Workplace Culture and other similar topics. I&#39;m a big fan of concise and to-the-point articles that introduce a topic without the overhead of too many words. To that end, let&#39;s get started. Happy reading!</p>
<h2 id="our-mission">Our Mission</h2>
<p>Our mission is to empower security teams to operate with high efficiency and effectiveness. We aim to do that by providing a platform and tools to automate the mundane so that analysts can analyze, and really just do what humans do best. Not only do we want to build unique and innovative solutions that prioritize the people doing the work, but we also want to do it in a unique way. That leads into the second part of our mission.</p>
<p>We want to build an internal culture that values our <strong>people</strong> first. We believe that putting people first and creating and protecting a culture of respect and support, along with a growth mindset, leads to some of the best products. We accomplish this by valuing certain character traits and behaviors over others. We value authenticity, respect, honesty, collaboration and an atmosphere that promotes learning and growth. We all have many things to learn, especially when it comes to ever-evolving technology, and that doesn&#39;t happen easily in an environment where intellectual egos reign. We will put our people and our culture first, and we believe that will not only allow us to build amazing products and services for our customers, but that we will all grow and become better people in the process.</p>
<p>If that sounds like an environment you&#39;d like to help build and be a part of, we&#39;d love to talk with you. Don&#39;t hesitate to <a href="/contact">reach</a> out.</p>
<h2 id="about-me">About Me</h2>
<p>I&#39;ve been fortunate to have a variety of experiences in my career. First and foremost, I&#39;m a software engineer with a passion for creating unique solutions to security problems. I have spent much of my career architecting and developing real-time software solutions in large, complex environments. I also spent eight years as a Special Agent in the FBI, where I investigated computer intrusions and other national security matters. I have a passion for solving hard problems and building elegant solutions from the ground up, especially as they relate to keeping you safe from the many threats out there. I&#39;ve worked in both amazing environments and toxic environments. Many of these experiences are what motivate me to create a culture and an environment where everyone wins, both personally and professionally.</p>
<h2 id="upcoming-topics">Upcoming Topics</h2>
<p>Over the next several months, I&#39;ll be writing about these topics:</p>
<ol>
<li>Windows Process Behavior</li>
<li>Hunting on Endpoint Data - A How To</li>
<li>A Simple Architecture for Endpoint Threat Hunting</li>
<li>Endpoint-based vs. Network-based Detection</li>
<li>A Primer on Endpoint Events and what they look like</li>
<li>Sysmon In-Depth (including a real-world config)</li>
<li>Endpoint Agents: Active vs. Passive vs. Scan-based</li>
<li>Cloud vs. On-Prem Architectures</li>
<li>Serverless Computing</li>
<li>Culture: Why it&#39;s so important</li>
</ol>
<h2 id="reach-out">Reach Out</h2>
<p>Sign up for our mailing list below to be notified when a new post comes out.</p>
<p>If you&#39;re interested in a career or partnership with EchoTrail or Vector8, please don&#39;t hesitate to reach out. We love meeting new people and growing our team. You can reach us by any of the means listed <a href="/contact">here</a>. </p>
<p>Thanks for stopping by!</p>


EchoTrail Blog

hunting

How To Hunt on Sysmon Data

January 28 2019

general

Why We Chose to Make Insights Free

November 27 2018

hunting

Windows Process Behavior

November 02 2018

general

Welcome

November 01 2018