Skip to content

McHughSecurity

  • Home
  • Blog
  • Security Operations
  • DFIR
    • Reverse Engineering
      • Dynamic Analysis
      • Static Analysis
  • Vuln. Mgmt
  • Intelligence
    • Open-Source Intelligence
    • Threat Intelligence
      • AIL Framework
      • MISP – Open Source Threat Intelligence Platform
      • OpenCTI
  • Threat Modelling
    • Attack Libraries
    • Attack Trees
    • Mitigating Techniques
    • SaaS Threat Modelling
    • Threat Models
    • Tools
    • Validating Threat Mitigations
  • Toggle search form
  • Building a MISP Threat Feed Aggregator Blog
  • Using the workflow taxonomy in MISP MISP - Open Source Threat Intelligence Platform
  • Implementing Elastic Cloud and using Elastic Security Build
  • Auto-updating Ubuntu 20.04 in less than 2 minutes Operate
  • Deploying (and using) TheHive4 [Part 1] Build
  • Using the Estimative Language Taxonomy in MISP MISP - Open Source Threat Intelligence Platform
  • Extracting RAM from VirtualBox session Digital Forensics & Incident Response
  • [Part 3] Building a Threat Integration and Testing Lab – Splunk Enterprise Build

Threat hunting with Elasticsearch and Kibana (Part 1)

Posted on April 4, 2020 By A.McHugh 2 Comments on Threat hunting with Elasticsearch and Kibana (Part 1)

As part of my final Masters degree research component I have been collecting data from honeypots which I have seeded around the globe. The objective being to distil this data in to organisational threat data based on a fictitious business.

Part of the complication I am going to start facing, is how to how Elasticsearch and Kibana to find specific information for me from this live data set.

Previously I have indicated that a data set exists which was produced by the Canadian Institute for Cybersecurity, called IDS 2018, which contains Windows Event Logs and PCAP files relating to a set of simulated attacks generated for the purposes of teaching people how to hunt within similar datasets.

Here I will be discussing the deployment, configuration and interaction with this data set to achieve the outcome required.

Sync the bucket from Amazon S3

To begin with, I will need to download the dataset, since it is hosted on Amazon S3 I will need to sync the bucket to my local system. You will need a lot of drive space for this – estimated to be 220GB compressed… so be prepared for a large storage media requirement, and use this to forward plan the ELK stack you are going to require.

  • Install the AWS CLI, available on Mac, Windows and Linux
  • Run: aws s3 sync –no-sign-request –region <your-region> “s3://cse-cic-ids2018/” dest-dir(Where your-region is your region from the AWS regions list and dest-dir is the name of the desired destination folder in your machine)

Once the bucket has been synced, you should see a set of directories, the one we are going to look at first of all is the ‘Original Network Traffic and Log data’.

Within this directory you will see a number of sub directories with specific days being indicated – fortunately the CIC broke up their attack recordings by day for different campaigns. So looking back at the logs for specific days can change what type of attack you are looking for.

We will need to extract the zip files for each of these attack days – but for the moment I will be focusing on the logs.zip files which contain Windows EVTX files.

Preparing the Elasticsearch and Kibana Nodes

I have preemptively already built a test Elasticsearch cluster for similar purposes, so to quickly describe how this is configured see the below:

I have bootstrapped Elasticsearch to operate in a cluster between each other, this means I can lose one or two nodes, and still continue operations. It also means I can expand disks by taking nodes individually offline, and then restarting them whilst at least a single node is operational.

Off to the side I have parked a Kibana node, which is configured to interface with the three Elasticsearch instances. Again, if a single node drops, the other two nodes are still operable.

Kibana is where I will be working for the most part of this activity.

Importing EVTX files into Elasticsearch

Now that I have a working Elasticsearch cluster, Kibana is attached, and I have the CIC dataset, I will need to import the EVTX files into Kibana whilst retaining the integrity of the encrypted Windows Event Logs.

We can do this with EVTX to Elasticsearch, but keep in mind we have a lot of events log captures to import here. So I will need to write a script to get this data in over a large number of directories.

import os
from evtxtoelk import EvtxToElk
evtx_folders = [
	'D:\\...\\Friday-16-02-2018\\logs\\logs\\',
	'D:\\...\\Friday-16-02-2018\\logs\\logs\\',
	'D:\\...\\Friday-23-02-2018\\logs\\logs\\',
	'D:\\...\\Thursday-01-03-2018\\logs\\logs\\',
	'D:\\...\\Thursday-15-02-2018\\logs\\logs\\',
	'D:\\...\\Thursday-22-02-2018\\logs\\logs\\',
	'D:\\...\\Tuesday-20-02-2018\\logs\\logs\\',
	'D:\\...\\Wednesday-14-02-2018\\logs\\logs\\',
	'D:\\...\\Wednesday-21-02-2018\\logs\\logs\\',
	'D:\\...\\Wednesday-28-02-2018\\logs\\logs\\'
]
extx_folder_length = len(evtx_folders)
for folder_name in evtx_folders:
	for filename in os.listdir(folder_name):
		print ('Processing ' + filename)
		try:
			EvtxToElk.evtx_to_elk(folder_name + filename,'http://ES_NODE_ADDRESS:9200')
			print ('Finished processing ' + filename)
		except:
			print("An exception occurred." + filename)

Note: I do not write in Python natively, there are quite possibly much more elegant ways to do this. Feel free to suggest them in your comments.

Assuming this executes for you successfully in python, you should see something like the following from your Python interpreter:

Visualising the data in Kibana

Kibana should now be able to see Event Logs being parsed into Elasticsearch, if you have not already configured an index pattern for the hostlogs, we will go through that in a moment.

In Kibana, go to Management > Kibana > Index Patterns and create a new pattern. You will need to tell Kibana to select hostlogs* and then define what field is going to be used for the Time Filter

Select the [email protected] option, and then Create Index Pattern.

From here head to the Discover tab of Kibana, select the hostlogs* index pattern, and change the view scope to include from September 2017 into the timeline.
You should now see the data being imported into Elasticsearch for the attack campaigns from the CIC data set.

I will be continuing to write Part 2 of this guide once the data has completely loaded into Elasticsearch, but feel free to have a toy with the data and the other datasets available from the CIC.

Related

Digital Forensics & Incident Response, Security Operations, Threat Intelligence Tags:elasticsearch, kibana

Post navigation

Previous Post: OSINT for Threat Intelligence
Next Post: Deploying (and using) TheHive4 [Part 1]

Related Posts

  • Building the Assemblyline Analyzer for TheHive’s Cortex. Digital Forensics & Incident Response
  • Building a Cuckoo Sandbox Build
  • Loading Windows Event Logs to Elasticsearch Digital Forensics & Incident Response
  • Building a Cuckoo Malware Analysis Server Digital Forensics & Incident Response
  • External Analysis with VirusTotal Digital Forensics & Incident Response
  • Hardening TheHive4 and Cortex for public deployment Build

Comments (2) on “Threat hunting with Elasticsearch and Kibana (Part 1)”

  1. 4en6 says:
    May 18, 2020 at 8:29 am

    thanks for your effort, really helpful!

    Reply
    1. Adam McHugh says:
      May 18, 2020 at 8:33 am

      Thanks for the comment. Hopefully I will be posting some new content this week focusing on how to sift through this volume of data.

      Reply

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

two × 1 =

Recent Posts

  • Auto-updating Ubuntu 20.04 in less than 2 minutes
  • Feeding Analysis Information Leak (AIL) Framework
  • An Introduction to Threat Intelligence
  • Deploying MISP on DigitalOcean or Vultr Cloud Hosting
  • Building CCCS’ AssemblyLine for Static Analysis

Search

Recent Posts

  • Auto-updating Ubuntu 20.04 in less than 2 minutes
  • Feeding Analysis Information Leak (AIL) Framework
  • An Introduction to Threat Intelligence
  • Deploying MISP on DigitalOcean or Vultr Cloud Hosting
  • Building CCCS’ AssemblyLine for Static Analysis

Archives

  • April 2022
  • July 2021
  • June 2021
  • May 2021
  • April 2021
  • March 2021
  • February 2021
  • January 2021
  • September 2020
  • April 2020
  • March 2020
  • January 2020
  • May 2019
  • Exporting Maltego Graphs to MISP Intelligence
  • Security Orchestration with Shuffle.io Design
  • OSINT for Missing Persons (Part 1 – Intro) Open-Source Intelligence
  • Building a Cuckoo Malware Analysis Server Digital Forensics & Incident Response
  • OSINT for Threat Intelligence Blog
  • An Introduction to Threat Intelligence Threat Intelligence
  • What is Cyber Threat Intelligence? Blog
  • Feeding Analysis Information Leak (AIL) Framework AIL Framework

Categories

  • AIL Framework
  • Blog
  • Build
  • Design
  • Digital Forensics & Incident Response
  • Frameworks
  • Intelligence
  • MISP – Open Source Threat Intelligence Platform
  • Open-Source Intelligence
  • Operate
  • Security Operations
  • Static Analysis
  • Threat Intelligence

Copyright © 2022 McHughSecurity.

Powered by PressBook News Dark theme