Professional Documents
Culture Documents
0
Search Tutorial
Generated: 11/08/2013 7:24 am
Table of Contents
Introduction..........................................................................................................1
Welcome to the Search Tutorial................................................................1
An overview of Splunk Enterprise.............................................................1
Part 1: Downloading and installing Splunk Enterprise....................................2
What you need for this tutorial..................................................................2
Where and which Splunk to download......................................................3
Install Splunk on Linux, Windows, or Mac OS X.......................................4
Start Splunk and launch Splunk Web........................................................6
Part 2: Getting started with Splunk Enterprise...............................................10
About Splunk Home................................................................................10
Navigating Splunk Web...........................................................................12
Part 3: Getting data into Splunk Enterprise....................................................15
About getting data into Splunk................................................................15
Get the tutorial data into Splunk..............................................................17
Part 4: Using Splunk Search.............................................................................21
About the Search dashboard..................................................................21
About the time range picker....................................................................24
About search actions and modes............................................................26
About the search results tabs..................................................................28
Part 5: Searching the tutorial data...................................................................31
Start searching........................................................................................31
Use fields to search.................................................................................33
Use the search language........................................................................38
Use a subsearch.....................................................................................42
Use field lookups.....................................................................................44
Part 6: Saving and sharing Reports.................................................................50
About saving and sharing reports...........................................................50
More searches and reports.....................................................................54
Part 7: Creating dashboards.............................................................................58
About dashboards...................................................................................58
Creating dashboards and dashboard panels..........................................59
Table of Contents
Next steps...........................................................................................................65
More Splunk............................................................................................65
ii
Introduction
Welcome to the Search Tutorial
What's in this tutorial?
If you're new to Splunk, this tutorial will teach you what you need to know to start
using Splunk, from a first-time download to creating rich, interactive dashboards.
This tutorial includes a sample data set composed of web server and MySQL
logs for a fictional online store. Follow the detailed instructions to add this data to
your Splunk instance. Learn the different ways you can search the data, save
reports, and create dashboards targeted to meet different business needs.
Make a PDF
If you'd like a PDF version of this manual, click the red Download the Search
Tutorial as PDF link below the table of contents on the left side of this page. A
PDF version of the manual is generated on the fly for you, and you can save it or
print it out to read later.
Note: Copying and pasting searches directly from the PDF document into Splunk
Web is not recommended. In some cases, doing so causes errors because of
hidden characters that are included in the PDF formatting.
System requirements
Splunk is a high-performance application that runs on most computing platforms:
Linux, Unix, Windows, and Mac OS. For this tutorial, you need a Windows or
Mac OS X computer or laptop that meets at least the following specifications:
Platform
Once you install Splunk on your machine, you access it using a web browser.
Splunk 6.0+ supports the latest versions of Firefox, Chrome, and Safari
browsers.
This is just a snapshot of Splunk's system requirements; for the complete list of
specifications, see the "System Requirements" topic in the Installation manual.
Note: If the Enterprise trial license expires, you can switch to the perpetual Free
license - It's included! - or purchase an Enterprise license.
Read more about "Types of Splunk licenses" in the Admin Manual.
Next steps
Now that you know what you need to run Splunk on your system, continue to the
next topic to read about downloading Splunk.
Next steps
Now that you've downloaded Splunk, continue to the next topic to install the
software on your machine.
Next steps
After the install completes, continue to the next topic to start Splunk.
# export SPLUNK_HOME=/opt/splunk
# export PATH=$SPLUNK_HOME/bin:$PATH
For more information on how to access the CLI, see "About the CLI" in the Admin
manual.
Now, to start Splunk, type:
$SPLUNK_HOME/bin/splunk start
$ splunk stop
$ splunk restart
$ splunk status
Splunk Web runs by default on port 8000 of the host on which it's installed. If you
are using Splunk on your local machine, the URL to access Splunk Web is
http://localhost:8000.
If you are using an Enterprise license, launching Splunk for the first time takes
you to this login screen. Follow the message to authenticate with the default
credentials.
If you are using a Free license, you do not need to authenticate to use Splunk. In
this case, when you start up Splunk you won't see this login screen. Instead, you
will be taken directly to Splunk Home or whatever is set as the default app for
your account.
When you sign in with your default password, Splunk asks you to create a new
password. You can either Skip this or change your password to continue.
The first page you should see is Splunk Home.
Next steps
This completes Part 1 of the Search Tutorial. Continue to Part 2: Getting started
with Splunk.
10
For an out-of-the-box Splunk Enterprise installation, you will only see one App in
the workspace. When you have more than one app, you can drag and drop the
app within the workspace to rearrange them.
Also, you can discover new apps or manage existing apps by clicking on the
buttons at the bottom of the panel to:
Find more apps to install on Splunk.
Manage apps already installed on Splunk.
When you have data in Splunk, you can see a brief summary of it here:
Click Add Data to follow the steps to get new data into Splunk.
Click Manage Inputs to view and edit existing input definitions.
Next steps
Now that you are familiar with Splunk Home, continue to the next topic to learn
how to "navigate your Splunk instance".
Return to Home
The Splunk logo on the navigation bar returns you to Splunk Home.
Apps
The Apps menu lists all the apps that you have permission to view and run. You
can Find more apps and Manage apps from this menu, too.
Settings
The Settings menu houses all the configuration pages for Knowledge objects,
Distributed environment settings, System and licensing, Data, and Authentication
settings. If you don't see some of these options, it just means that you do not
have the permissions to view or edit them.
User menu
The User menu here is named "Administrator" because that is the default user
name for a new install. You can change this display name by selecting Edit
account. You can also change the time zone settings, select a default app for
this account, and change your password. The User menu is also where you can
Logout of Splunk.
13
Messages
All system-level error messages will be listed here. You will see a notification (in
red) when there is a new message to review.
Activity
Jobs
Click on Jobs to open the search jobs manager window, where you can view and
manager currently running searches.
Triggered alerts
Alerts that you set up will display here when they are triggered. You will see a
notification for the triggered alerts.
This tutorial does not discuss saving and scheduling alerts. For more information,
read "About alerts" in the Alerting Manual.
Help
Click on Help to see links to Video Tutorials, Splunk Answers, the Splunk
Support Portal, and online Documentation.
Next steps
Now that you are more familiar with Splunk Web, let's add some data to Splunk!
14
16
Next steps
Now that you're more familiar with Splunk data inputs and indexes, continue to
the next topic and add the tutorial sample data into Splunk.
17
18
The source of a file or directory is the full pathname to the file or directory.
6. Select More settings.
The "More settings" option enables you to override Splunk's default settings for
Host, Source type, and Index. For this tutorial, you need to modify the host
settings to assign host names to the events based on the file's location in the
compressed file.
Select Segment in path from the menu.
Type in 1 for the segment number.
7. Click Save.
When it's finished, Splunk displays a message saying the upload was successful.
8. Return to Home.
19
The Data panel in Home now displays a summary of the data you just added. If
you don't have any other data in Splunk, it should look something like this:
The data summary will be discussed in more detail in the next chapter.
Next steps
Now that you've fed Splunk some data, it's time to learn about the Search app
and start searching the tutorial data.
20
The source of an event is the file or directory path, network port, or script from
which the event originated.
22
The source type of an event tells you what kind of data it is, usually based on
how it's formatted. This classification lets you search for the same type of data
across multiple sources and hosts.
For more information about how Splunk source types your data, read "Why
source types matter" in the Getting Data In manual.
You're now in the New Search page. The search bar and time range picker are
still available in this view, but the dashboard updates with many more elements:
search action buttons, and search mode selector; counts of events; job status
bar; and tabs for Events, Statistics, and Visualizations.
The next topics in this chapter will discuss each of these parts of the Search view
in more detail.
23
Next steps
Continue reading to learn about restricting searches to a time range.
By default, the time range for a search is set to All time. Usually, when you run a
search over large volumes of data, you will see faster results if you run the
search over a smaller time period.
When you're troubleshooting an issue and have a ballpark range for when the
issue occurred, it helps to narrow the time of the search to that time period. For
example, if you're investigating an incident that occurred yesterday, you could
select Yesterday or Last 24 hours. If you're investigating an incident that
24
occurred 10 minutes ago, you could select Last 15 minutes or Last 60 minutes.
For example, you can specify the earliest time to read "2 Hours Ago" and latest
time to be either "now" or "Beginning of the current hour".
Additionally, you can narrow down more precisely into the time range when you
specify a Date & Time Range.
Next steps
Continue reading to learn about search actions and search modes.
25
You can:
Edit the job settings. Select this to open the Job Settings dialog, where
you can change the job's read permissions, extend the job's lifespan, and
get a URL for the job that you can use to share the job with others or put a
link to the job in your browser's bookmark bar.
Send the job to the background. Select this if the search job is slow to
complete and you would like to run the job in the background while you
work on other Splunk activities (including running a new search job).
Inspect the job. Opens a separate window and display information and
metrics for the search job via the Search Job Inspector.
Delete the job. Use this to delete a job that is currently running, is
paused, or which has finalized. After you have deleted the job you can still
save the search as a report.
Read more about "Saving and sharing jobs in Splunk Web" in the Knowledge
Manager Manual.
See "Set search mode to adjust your search experience" in the Search Manual
for more information.
Next steps
Continue to the next topic to read a brief discussion about the format of the
search results.
27
Events
The following search just retrieves events and populates the Events results tab:
Statistics
If you clicked the the Statistics Tab for the previous search example, you would
not see any results because it does not have any transforming commands.
With a transforming search, such as one to build a chart of the top product
categories sold at the Buttercup Games online store, the Statistics Tab displays a
table of results:
Visualizations
You can also view the previous example in the Visualizations tab. It displays as a
chart visualization that you can format further.
29
Next steps
This chapter explained to you how to use and navigate the Search dashboard,
but you won't get a feel for it until you Start searching.
30
What to search
Let's return to the Search view. To do this, click Search in the App navigation
bar.
Look at the What to search panel.
Also, re-familiarize yourself with the tutorial data, which represents a fictitious
online game store, called Buttercup Games. The tutorial data includes five hosts,
eight sources, and three source types. The three source types are Apache web
access logs (access_combined_wcookie), Linux secure formatted logs (secure),
and the vendor sales log (vendor_sales).
Most of this tutorial will cover searching the Apache web access logs and
correlating it with the vendor sales logs.
31
CASE(). Read more about these methods in "Use the search command" in the
Search Manual.
Next steps
When you're ready to proceed, go to the next topic to learn how to investigate
and troubleshoot interactively using the timeline in Splunk.
33
Extracted fields
Splunk extracts fields from event data at index-time and at search-time. Read
more about "Index time versus search time" in the Managing Indexers and
Clusters manual.
Default and other indexed fields are automatically extracted for each event that is
processed when that data is indexed. Default fields include host, source, and
sourcetype (which should be familiar to you). For a complete list of the default
fields, see "Use default fields" in the Knowledge Manager Manual.
A different set of fields are extracted at search time, when you run a search.
You'll see some examples of these searches later. For more information, read
the "Overview of search-time field extractions" in the Knowledge Manager
manual.
You can also use the Interactive Field Extractor (IFX) to create custom fields
dynamically on your local Splunk instance. IFX enables you to define any pattern
for recognizing one or more fields in your events. For more information, read how
to "Extract fields interactively with IFX" in the Knowledge Manager Manual.
This tells Splunk to only retrieve events from your web access logs and nothing
else. sourcetype is a field name and access_* is a wildcarded field value used to
match any Apache web access event. Apache web access logs may formatted
as access_common, access_combined, and access_combined_wcookie.
2. In the Events tab, scroll through the list of events.
If you're familiar with the access_combined format of Apache logs, you will
recognize some of the information in each event, such as:
IP addresses for the users accessing the website.
URIs and URLs for the pages requested and referring pages.
HTTP status codes for each page request.
34
You can hide and show the fields sidebar by clicking 'Hide Fields and Show
Fields, respectively.
4. Click All Fields.
This opens the Select Fields dialog where you can edit the fields to show in the
events list.
You should see the default fields that Splunk defined--some of these fields are
based on each event's timestamp (everything beginning with date_*), punctuation
(punct), and location (index).
You should also recognize the field names that apply to the web access logs. For
example, there's clientip, method, and status. These are not default fields;
they have (most likely) been extracted at search time.
But, you should also notice other extracted fields that are related to the Buttercup
Games online store. For example, there are action, categoryId, and productId.
5. Select action, categoryId, and productId and close the Select Fields window.
35
The three fields are added under Selected Fields in the sidebar. Also, the
field/value pairs are listed under each event if it exists in the raw data for that
event.
The fields sidebar doesn't just show you what fields Splunk has captured from
your data. It also displays how many values exist for each of these fields. This
doesn't mean that these are all the values that exist for each of the fields--these
are just the values that Splunk knows about from the results of your search.
36
The selected fields are displayed under your search results if they exist in that
particular event. Different events will have different fields. If click on the arrow
next to an event, it opens up the list of all fields in that event. You can use this
panel to view all the fields in a particular event and select or deselect individual
fields for an individual event.
successful requests
You can also search for failed purchases in a similar manner using status!=200,
which looks for all events where the HTTP status code is not equal to 200.
sourcetype=access_* status!=200 action=purchase
Note that this doesn't specify a source type. The search retrieves events in both
the secure and web access logs.
Example 3: How many simulation games were bought yesterday?
Select the Preset time range, Yesterday, from the the time range picker and run:
sourcetype=access_* status=200 action=purchase categoryId=simulation
The count of events returned are the number of simulation games returned.
If you wanted to find the number purchases for each type of product sold at the
shop, you would need to run this search for each unique categoryId. If you
wanted the number of purchases for each day of the previous week, you would
37
Next steps
In Splunk, fields are searchable name/value pairings that distinguish one event
from another because not all events will have the same fields and field values.
Fields enable you to write more tailored searches to retrieve the specific events
that you want. Fields also enable you to take advantage of the search language,
create charts, and build reports.
Continue to the next topic to learn how to use the search language.
If you want to find this number for the days of the previous week, you have to run
it against the data for each day of that week. If you want to see which products
are more popular than the other, you have to run the search for each of the eight
categoryId values and compare the results.
This topic shows you how you can finds these answers using the search
language.
38
As you type in the search bar, search assistant opens with syntax and usage
information for the search command (on the right side). If search assistant
doesn't open, click the down arrow under the left side of the search bar.
You've seen before that search assistant displays typeahead for keywords that
you type into the search bar. It also explains briefly how to search. We've already
gone through retrieving events. Now, let's start using the search commands.
2. Type a pipe character, " | ", into the search bar.
The pipe indicates to Splunk that you're about to use a command, and that you
want to use the results of the search to the left of the pipe as the input to this
command. You can pass the results of one command into another command in a
series, or pipeline, of search commands.
You want Splunk to give you the most popular items bought at the online
store--from this list, the top command looks promising.
3. Under common next commands, click top.
Splunk appends the top command to your search string.
According to search assistant's description and usage examples, the top
command "displays the most common values of a field".
4. Either click the category_id field in the list or type it into the search bar to
complete your search:
sourcetype=access_* status=200 action=purchase | top categoryId
39
From this, you see that strategy games are by far the most popular item in the
online store.
The top command also returns two new fields: count is the number of times each
value of the field occurs, and percent is how large that count is compared to the
total count. Read more about the top command in the Search reference manual.
If you click on the visualization type selector, you can see that Column, Bar, and
Pie charts are recommended for this data set. Select Pie chart:
40
You can turn on drilldown to delve deeper into the details of the information
presented to you in the tables and charts that result from your search.
If you mouse over each slice of the pie, you will see the count and percentage
values for each categoryId. Click on a slice, such as "Strategy".
Read more about drilldown actions in the Splunk Data Visualizations Manual.
41
Next steps
As you run more searches, you want to be able to save them and reuse them or
share them with other people. When you're ready, proceed to the next topic to
learn how to "Save reports".
Use a subsearch
This topic walks you through examples of correlating events with subsearches.
A subsearch is a search with a search pipeline as an argument. Subsearches are
contained in square brackets and evaluated first. The result of the subsearch is
then used as an argument to the primary, or outer, search. Read more about how
subsearches work in the Search manual.
to see more than one "top purchasing customer", change this limit value. For
more information about usage and syntax, refer to the "top" command's page in
the Search Reference Manual.
This search returns one clientip value, which we'll use to identify our VIP
customer.
2. Use the stats command to count this VIP customer's purchases:
42
This search used the count() function which only returns the total count of
purchases for the customer. The dc() function is used to count how many
different products he buys.
The drawback to this approach is that you have to run two searches each time
you want to build this table. The top purchaser is not likely to be the same person
at any given time range.
Here, the subsearch is the segment that is enclosed in square brackets, []. This
search, search sourcetype=access_* status=200 action=purchase | top
limit=1 clientip | table clientip is the same as Example 1 Step 1, except
for the last piped command, | table clientip
Because the top command returns count and percent fields as well, the table
command is used to keep only the clientip value.
These results should match the previous result, if you run it on the same time
range. But, if you change the time range, you might see different results because
the top purchasing customer will be different!
43
Next steps
While this report is perfectly acceptable, you can make it better still. For example,
what is actually being purchased and how much did the customer spend? The
productId field itself is not descriptive. In the next topic, you'll learn about adding
new information to your events using field lookups.
44
This opens the Lookups editor where you can create new lookups or edit existing
ones. You can view and edit existing lookups by clicking on the links in the table
for Lookup table files, Lookup definitions, and Automatic lookups. To add
new lookups, just click Add new under Actions for that lookup item.
45
This will be the name you use to refer to the file in a lookup definition.
5. Click Save.
This uploads your lookup file to Splunk to the Search app, but now you need to
define the type of lookup you want to set up.
Note: Splunk does not recognize or cannot upload the file, check that it was
uncompressed before you attempt to upload it again.
6. Click Lookups in the breadcrumb to return to the Lookups manager.
46
The input field is the field in your event data that you are using to match the field
in the lookup table.
7. Under Lookup output fields, type in the following. Use the Add another field
link to add more fields after the first one:
47
The output fields are the field(s) in the lookup table that you want to add to your
event data based on the input field matching. Here, you are adding the fields:
price, which contains the price for each productId, and product_name, which
contains the descriptive name for each productId.
When you scroll through the fields sidebar or Fields dialog, you should see the
new fields for price and productName. Click All fields and add them to the
Selected fields list:
The result is exactly the same as in the previous subsearch example, except that
the VIP customer's purchases are more meaningful and descriptive!
48
Don't close this just yet. In the next chapter you will walk through saving this as a
report called "VIP Customer".
Next steps
When you're ready, proceed to the next chapter to learn about saving and
sharing reports.
49
Save as a report
Recall the previous search example:
sourcetype=access_* status=200 action=purchase [search
sourcetype=access_* status=200 action=purchase | top limit=1 clientip |
table clientip] | stats count AS "Total Purchased", dc(productId) AS
"Total Products", values(productName) AS "Product Names" by clientip |
rename clientip AS "VIP Customer"
If you don't have the report still open, run the search again. To save it as a report:
1. Above the search bar, click Save as and select Report.
50
When you save a new report, its Permissions are set to "Private" by default. This
means that only you can view and edit the report. You can allow other apps to
view, or edit, or view and edit the reports by changing its Permissions.
To share the VIP Customers report with everyone,
1. Under Actions, click Edit and select Edit Permissions.
51
This gives everyone who has access to this app the permission to view it.
3. Click Save.
Back at the Reports listing page, you should see the Sharing for VIP Customer
now reads App.
52
You saved this report with a time range picker, which is located to the top left.
The time range picker enables you to change the time period to run this search.
For example, you can use this time range picker to run this search for the VIP
Customer Yesterday, the day before, or last month just by selecting the Preset
time range or defining a custom time range. For more information on how to do
this, read "About the time range picker" earlier in this tutorial.
Other actions you can take on a saved report is Edit its properties, view More
Info, and Add to dashboard.
Next steps
Continue to the next topic to run more search examples and save more reports.
53
The chart command is used to count the number of events that are
action=purchase and action=addtocart. You can format the visualization as a
column chart:
Alternatively, you can use the stats command to create a table of the same
statistics, and more:
sourcetype=access_* status=200 | stats count AS views
count(eval(action="addtocart")) AS addtocart
count(eval(action="purchase")) AS purchases by productName | eval
viewsToPurchase=(purchases/views)*100 | eval
cartToPurchase=(purchases/addtocart)*100 | table productName views
addtocart purchases viewsToPurchase cartToPurchase | rename productName
AS "Product Name" views AS "Views", addtocart as "Adds To Cart",
purchases AS "Purchases"
Here, the stats command is used instead of the chart command. The eval
command is used to define new fields, which are the percentage of views and
addtocart that lead to purchases.
54
arguments to make sure the chart only counts events that have a value for
productName.
This produces the following statistics table:
If you look at the chart selection menu, notice that Line, Area, and Column
visualizations are recommended.
If you select Line and format the Y-axis and Legend, you can produce this chart:
56
This example requires the producName field from the fields lookup example.
If you didn't add the lookup, refer to that example and follow the procedure.
1. Run the following search:
sourcetype=access_* status=200 action=purchase | chart count AS Total
by categoryId | rename categoryId AS "Category"
This search is similar to the last two searches you just ran to build reports. It uses
the chart command to count the number of purchases, action="purchase",
made for each product, productName. The difference here is that the count of
purchases is now an argument of the sparkline() function.
2.
sourcetype=access_* status=200 action=purchase| chart sparkline(count)
AS "Purchases Trend" count AS Total by categoryId | rename categoryID AS
"Category"
3. Save this report and enter the Title (Purchasing trends) and Description (Count
of purchases with trending).
Next steps
Up to now, you've saved searches as Reports. Continue to the next topic to learn
about dashboards and how to save searches and reports as dashboard panels.
57
matched by the search that has been specified for the panel.
For an overview of the various visualization types offered by Splunk and
their formatting/display options, see the "Visualization reference" topic in
the Data Visualization manual.
For more information about the data structures required by of the various
visualization types see "Data structure requirements for visualizations" in
the Data Visualization manual.
2. Click the Visualizations tab and select the Pie Chart type.
59
3. Above the search bar, click Save as and select Dashboard Panel.
60
6. Click Save.
The dashboard was successfully created. Now, let's take a look at it and add
more panels to the dashboard.
7. To continue, click View dashboard.
61
If you click on the arrow under the i (information), you will see more information
about the dashboard: What app context it is in, whether or not it is scheduled,
and its permissions.
There are also quick links to edit the dashboard's Schedule and Permissions
inline with the information.
This takes you to the Buttercup Games Purchases dashboard panel editor view.
In this view, you have edit buttons: Add Panel, Add Time Range Picker, and
Edit Source.
62
2. Click Add Time Range Picker and leave the default as All time.
This time range picker will enable you to restrict all the inline searches that power
the panels to the same time range.
Let's add the another panel, one of the saved reports you created earlier, to the
dashboard.
3. Click Add Panel.
This opens the Add Panel dialog. Let's add a saved report.
4. For Content Type, click "Report" and select a saved report from the list,
Comparisons of Views, Adds to Cart, and Purchases.
5. Enter the Content Title, "Views, Adds to Cart, and Purchases".
6. Click Add Panel.
Now, when you return to the dashboard you should see two panels: "Top
Purchases by Category" and "Views, Adds to Cart, and Purchases".
While in the edit panels view, you can drag and drop a panel to rearrange it on
the dashboard.
7. Click Done.
Your dashboard should look like this:
63
Other actions you can take on a dashboard are view More Info, Export to PDF,
and Print.
Next steps
This completes the Search Tutorial. Continue to the next chapter to read about
what you can do next.
64
Next steps
More Splunk
This tutorial was a brief introduction to navigating the search interface and using
the search language. It walked you through running some basic searches and
saving the results as a report and dashboard, but it barely cut the surface of what
you can do with Splunk Enterprise. For more details refer to the following
manuals:
Search Manual: This manual explains how to search and use the Splunk
Search Processing Language (SPL?). Look here for more thorough
examples of writing Splunk searches to calculate statisitics, evaluate
fields, and report on search results.
Search Reference Manual: This manual provides a reference for the
Splunk Enterprise user who is looking for a catalog of the search
commands with complete syntax, descriptions, and examples for usage. If
you want to just jump right in and start searching, check out the Search
command cheat sheet--it's a quick guide, complete with descriptions and
examples.
Also, we encourage you to investigate the tutorial data, run more searches, and
create more dashboards! If you want to learn more about the data model and
pivot features of Splunk Enterprise, you can step through the Data Model and
Pivot Tutorial.
If you want to learn more about Splunk Enterprise features and how to use them,
check out our selection of Education videos and classes.
65