57.

4 Monitoring Splunk Searches

4. Monitoring Splunk Searches Prev Chapter 57. Splunk Monitoring Next

4. Monitoring Splunk Searches

4.1. Monitoring Results of a Simple Search
The easiest way to get started monitoring your Splunk searches is with a simple search. The following steps will illustrate a simple way to build dynamic Splunk search monitoring. This example demonstrates how to detect brute-force password cracking attempts on all Linux servers.
1.

Build a search in Splunk to verify that you're getting the expected data. This example shows a query of host="zendev.damsel.loc" minutesago=5 "failed password".

Note
Using a time specifier such as minutesago=5 within your search can be a useful trick when it comes to monitoring searches from Zenoss. We will have Zenoss

Generated by Jive SBS on 2011-04-18-06:00 1

57.4 Monitoring Splunk Searches

2.

automatically replace zendev.damsel.loc with the appropriate hostname using a ${here/id} TALES expression. Create a Zenoss monitoring template for monitoring this Splunk search.
a.

From Advanced > Monitoring Templates, click to add a monitoring template. The Add Template dialog appears. Enter SplunkLinux in the Name field and select Linux in /Service/Linux for Template Path, and then click Submit. Select the newly created template. Add a Splunk data source to capture the count of failed passwords.
i. ii. iii.

b. c. d.

In the Data Sources area, click to add a data source. In the Add Data Source dialog, set the Name to failedPassword and the Type to Splunk, and then click OK. Double-click the data source to configure it as follows, and then click Save. Splunk Server: Hostname or IP of your Splunk server Splunk Port: 8089 Splunk Username: Splunk username (default is admin) Splunk Password: Splunk password (default is changeme) Search: host="${here/id}" minutesago=5 "failed password" Add the count data point to the failedPassword data source.

iv.

v.

Select Add Data Point from the Data Sources Action menu. Set the Name to count and click OK. Add a threshold of how many failed passwords constitutes an attack.
B.

A.

vi.

In the Thresholds area, click to add a threshold. B. Set the Name to password attack and Type to MinMaxThreshold, and then click Add..\ C. Select failedPassword_count from Data Points. D. Set the Max Value to 10. E. Set the Event Class to /Security/Login/BadPass. F. Click Save. Add a graph to visualize failed passwords per 5 minutes.
A. B. C. D.

A.

In the Graph Definitions area, click to add a graph. Set the Name to Splunk - Failed Passwords, and then click Submit. Double-click the newly created graph to edit it. Set the Units to failed/5min.

Generated by Jive SBS on 2011-04-18-06:00 2

57.4 Monitoring Splunk Searches

E. F.

Set the Min Y to 0. Select Manage Graph Points from the Action menu in the Graph Definitions area. The Manage Graph Points dialog appears. Select Data Point from the Add menu.

G.

The Add Data Point dialog appears. Select failedPassword_count from Data Point, and then click Submit. I. Click into the new count graph point. J. Set the RPN to 300,* to adjust from failed/sec to failed/5min. K. Set the Format to %6.1lf. L. Set the Legend to Count. M. Click Save. vii. Bind the SplunkLinux template to the /Server/Linux device class.
H. A. B. C. D.

From Infrastructure > Devices, navigate to the /Server/Linux device class. Click Details. Select Bind Templates from the Action menu. Move the SplunkLinux template from the Available area to the Selected area, and then click Save.

Now you will have a Failed Passwords graph on all of your Linux servers that visualizes how many failed password attempts have occurred over the last 5 minutes. You will also get a warning severity event anytime more than 10 failed password attempts are made within a 5 minute period.

4.2. Monitoring Results of a Top Search

Monitoring additional data points within a top search builds on monitoring a simple search. You can extra numeric data from the tabular results returned from a top search using the following steps. This example demonstrates how you can monitor the logs by source type for all Linux devices.
1.

Build a search in Splunk to verify that you're getting the expected data. This example shows a query of host="zendev.damsel.loc" minutesago=5 | top sourcetype.

Generated by Jive SBS on 2011-04-18-06:00 3

57.4 Monitoring Splunk Searches

Note
Take special note of the names in the sourcetype column and the names of the count and percent columns. These will be used to construct the names of the datapoints within our Splunk data source. Setup a Zenoss monitoring template just as described in the simple search example. Add a Splunk type data source named sourcetype to the template with the following settings. Splunk Server: Hostname or IP of your Splunk server Splunk Port: 8089 Splunk Username: Splunk username (default is admin) Splunk Password: Splunk password (default is changeme) Search: host="${here/id}" minutesago=5 | top sourcetype Add data points to the sourcetype data source with the following names. These names come from concatenating the data in the first column of each row with the name of the column name with the target numeric data.

2. 3.

4.

5.

linux_audit_count linux_audit_percent linux_secure_count linux_secure_percent Create a graph that will show these results within Zenoss in a useful way.

Generated by Jive SBS on 2011-04-18-06:00 4

57.4 Monitoring Splunk Searches

a. b. c. d. e. f. g.

Add a graph from the Graph Definitions area of the monitoring template. Set the ID to Splunk - Logs by Source Type then click Submit. Set the Units to percent. Set the Min Y to 0. Set the Max Y to 100. Click Save. Select Manage Graph Points from the Action menu in the Graph Definitions area. The Manage Graph Points dialog appears. Select Data Point from the Add menu. The Add Data Point dialog appears. Use SHIFT-click or CTRL-click to select the following data points from the list then click Submit. sourcetype_linux_audit_percent sourcetype_linux_secure_percent Click into each of the graph points you just added to the graph and set the following properties.

h.

i.

j.

6.

Line Type: Area Stacked: True Format: %5.1lf%% Legend: Audit or Secure respectively. Bind the monitoring template to the /Server/Linux device class just as in the simple search example.

You will now have a graph for all Linux devices that shows what percentage of logs are coming from the audit and secure logs respectively. This ability to track multiple results from a single Splunk search has many other possible uses. Experiment with the top filter in Splunk to see what other useful data you could extract.

Prev 3. Splunk Data Source Type

Up Home

Next 5. Daemons

Generated by Jive SBS on 2011-04-18-06:00 5