Search Report 42

Searching and Reporting with Splunk 4.
Document usage guidelines

Should be used only for enrolled students
Not meant to be a self-paced document
Not for distribution
07/12/2011
Operational Intelligence
Searching and Reporting with Splunk 4.2
Class Goals
Gain a deeper understanding of search and reporting concepts
Create efficient, well-formed searches
Perform calculations and evaluations on search results
Generate reports and charts
Correlate events with transactions
Create and use macros
Create and use lookups
Create and use summary indexes
Course scenario
Examples used in this course are based on an online retail business
Searches and reports will be based on:
- Business analytics from the web access logs and lookups
- Internal operations information from mail and internal network data
Data used in this course

Examples and labs in this course are based on a variety of data:
- Web application access logs
Events related to the online store web activity and sales
- Cisco mail logs
Events related to the internal email system
Data used in this course (cont.)

- Cisco firewall logs
Events related to employee network usage
Youll become familiar with the fields and characteristics of these data
sources as the course progresses
Course Outline
1. Search Fundamentals
2. Getting Statistics
3. Analyzing, Calculating, and Formatting
4. Creating Charts
5. Correlating Events
6. Enriching Data with Lookups
7. Summary Indexing
8. Creating and Using Macros
Section 1:
Search Fundamentals
Section objectives
Review basic search commands
Review general search practices
Examine the anatomy of a search
Understand search language syntax concepts
Review fields and use the fields command
Create a table
Extract fields
Examine multi-value fields
Basic search review

Keywords
search for error!
Phrases
web error (different than web AND error)
Fields
search for status=404
Booleans
OR, NOT, ()s. ANDs are implied
Wildcards
status=40*, matching 401, etc.
Comparisons
=,!=,<,<=,>=,> (delay > 10)!
10
General search practices

Time is the most efficient filter in Splunk narrowing by time is the most
effective thing you can do
The more you tell the search engine, the better shot you have at a good
result
- When applicable, searching for access
denied is always better than
searching for denied
If 90% of your events include the word 'error' but only 5% include the
word 'sshd', and your events have both of these words, including 'sshd' in
your search terms will help more than 'error alone
11
General search practices (cont.)

Inclusion is generally better than exclusion
- Searching for "access
denied" will be faster than NOT "access
granted" !
Apply powerful filtering commands as early in your search as possible

- Filtering to one million events and then narrowing to ten events is much slower
than filtering to one thousand events and then ten events
12
Anatomy of a Search
Disk
Intermediate
results table
Intermediate
results table
Final results
table
sourcetype=syslog ERROR | top user | fields - percent

Fetch events
from disk that
match
Summarize
into table of
top 10 users
13
Remove
column
showing
percentage
Search language syntax concepts

Searches are made up of 5 basic components
Search terms what are we looking for?
- keywords, phrases, booleans, etc.
Commands what should we do with the results?
- Create a chart, compute statistics, evaluate and format, etc.
Functions how should we chart, compute or evaluate?
- Get a sum, get an average, transform the values, etc.
Arguments Are there variables we should apply to this function?
- Get an average value for a specific field, convert milliseconds to seconds, etc.
Clauses How should we group the results?
- Get the average of values for the price field grouped by product, etc.
14
Search language example

This diagram represents a search, broken into the syntax components
Search for this
PIPE:
Take these events and
COMMAND:
Get some stats
PIPE:
Take these stats and
COMMAND:
Set values for the lost_revenue field
FUNCTION:
Get a sum
FUNCTION:
Create a string
ARGUMENT:
Get a sum of the price field
CLAUSE:
Call that sum lost_revenue
15
ARGUMENT:
Format the string from values in the
lost_revenue field, insert commas
Fields review
Splunk automatically extracts important fields in these events
A few are highlighted below (there are many more!)
web access log
clientip
product_id
method
mail log
status
MID
mailfrom
network
proxy log
cs_username
s_hostname
16
Fields command overview

Field extraction is one of the most costly parts of search
The fields command allows you to include or exclude specific fields in your
search or report
- Makes searching more efficient when traversing large amounts of data with
many fields
- Makes table-display cleaner by allowing you to remove unnecessary fields
17
Include specific fields

Improves performance only the
fields you specify are extracted
Internal fields _raw and _time
are always included in output
- The fields command does not
remove internal fields unless

explicitly specified with the remove
fields command
fields - <field name>
18
Remove specific fields

Use the remove fields
command to exclude specific
fields
As youll discover with
statistics and reporting
commands, removing fields
can be useful for display
purposes
19
Create a table
The table command returns a
table formed by only the fields in
the arguments
Columns are displayed in the
same order of fields entered in the
command
- Column headers are field names
- Rows are field values
- Each row represents an event
20
Rename fields
Use the rename command to
rename fields
Useful for giving fields more
meaningful names
Use quotes to rename to a phrase
- |
rename JSESSIONID as session ID!
Can be applied to wildcards to

rename multiple fields
- |
rename *ip as IP Address!
21
Extract fields at search time

There may be times when a field you need for a particular search has not
been extracted
The rex command allows you to extract the field at search time
- Only persists for the duration of the search
- Good for rarely-used fields
- Save the search and use it as-needed, but the extraction wont persist in ALL
your searches
You can also use the rex command to create permanent extractions via
Splunk> Manager >> Fields >> Extractions
Learn more about the regex format and using
the rex command in Splunk.com/base/
Documentation
22
Extract fields with rex

The network proxy data includes a user name for each event, but has not
been defined as a field
The rex command allows you to extract the field at search time
sourcetype="cisco_w*" | rex "\s+(?<username>[^-@]\w+)@" | top username!
!
For example purposes, the user name field has

been extracted as cs_username for this course
23
Extract fields from a table-formatted event

Many data types are formatted as large single events, formatted as a table
Each event contains fields with multiple values
- Here, the first row represents the fields, all other rows represent values
24
Extract fields from a table-formatted event (cont.)

The multikv command extracts
the fields you specify
- Field names are from the first row of
each event as displayed on

previous slide
The command creates a separate

event for each row
- This example also pipes to the
table command for better display

25
Combine multi-valued fields

The mvcombine command combines events that have a single differing
field value into one result with a multi-value field of the differing field
- For each group of results that are identical except for the given field, combine
them into a single result where the given field is a multivalue field
-delim controls how values are combined
There are many commands for use with

multi-valued fields. The complete list can
be found at
splunk.com/base/documentation
26
Lab working with fields

To become familiar with the data used in this course, examine all three
sources of data
In the web access logs, search for all events of each data type
Create tables that includes specific fields
Challenge: In the mail logs, use the rex command to extract a new field
27
Section 2:
Getting statistics
28
Section Objectives
Understand the stats command
Preview reporting and charting commands
Display top and rare values for given fields
Use the stats command
29
Reporting commands overview

The following basic reporting commands are covered in this section:
top display the most common values of a field
rare display the least common values of a field
stats calculate statistics on your events
eventstats attach summary statistics to each event
30
Scenario callouts
Most of the examples for the remainder of this course relate to a specific
scenario
For each example, a question is posed that might be asked by a colleague
or manager in the company
- The answers live in the Splunk data!
- Example:
Who are the top site visitors?
31
Getting top values

Who are the top site visitors?
The top command finds the

most common values of a given
field
- Returns top 10 results by default
Output is in table format

Automatically returns a count
and percentage
Adding limit=# after the top
command returns the specified
number of results
32
Getting top values (cont.)

This example displays the top
10 from addresses found in
company email
Are we getting an unusual amount of mail from a few senders?
Clicking Show Report

allows quick creation of a
report suitable for a
dashboard
33

Using the fields command, you can remove the percent field from the
results
What are the top 5 sites being accessed through the firewall?
34

Adding the by clause to the top
command, we can view the top
sites and their associated
acceptable use category
What are the usage categories of the top sites?
sort -count ensures the table

is sorted by count in descending
order
Note: the usage field comes from an
automatic lookup, which is covered later in
this course.
35
Getting rare values

The rare command returns the
least common field values of a
given field
What are the least frequently visited sites?
Options are identical to the top

command
36
Getting statistics
The stats command allows you to get statistics on your data
You can apply different functions to the stats command, depending on the
type of statistics youre trying to achieve
Common functions include:
- count
returns the number of occurrences of a given field within the search
results!
- distinct_count, dc returns a count of unique values for a given field
- sum returns a sum of numeric values
- avg returns an average of numeric values
- list lists all values of a given field
- values lists unique values of a given field
37
stats count
count returns the number of
occurrences of a given field
How many of each product was purchased?
The by clause returns a count

for each field value of a named
field
This example counts the
number of times each value of
product_id appears in the
events when the action field
contains purchase!
38
stats count(field)
Adding a field as an argument to
the count function returns the
number of occurrences for that
field
How many sites of each usage type are accessed?
In this example, we count how

many websites (s_hostname)
that were accessed fell into each
usage category
39
stats distinct count

distinct_count() or
dc() provides a count
of how many unique
values there are for a
given field
How many unique visitors to the online store?
This example counts how

many unique values exist
for clientip
40
stats sum(field)
For fields with a numeric value,
you can sum the actual values of
that field
Whats the total time for all web requests per site visitor?
In this example, for each user web

session, we get a sum of the
values of the time_taken field
Note: the time_taken field values are in
milliseconds. Well learn how to convert to
seconds later in this course
41
stats avg(field)
The avg function averages numeric values of a named field
This example averages all the values of the time_taken field, grouped by
the action field
Whats the average time it takes the application to perform an update vs purchase?
42
addcoltotals
The addcoltotals
command computes the
sum of all numeric values for
a given field
How many items are being removed from carts before purchase?
- Adds the total to the bottom of
the column
43
addtotals
The addtotals commands gets the
sum of numeric fields for each event
- Define a field name in which to place
the total value
- Optionally, specify only certain fields to
include in the sum
- In this example, the addtotals
command is applied to the rKB_PS
and wKB_PS fields
- By default, the command computes a
sum for all numeric fields in the event
44
stats list(field)
What websites are the users accessing?
The list function lists all field

values for a given field
This example lists the websites
each user visits
- Since the security logs generate an
event for each network request, the

same hostname will appear multiple
times
- If you want a list of unique field
values, the list function isnt ideal
45
stats values(field)
The values function creates a list
for each unique field value for a
given field
What websites are the users accessing?
46
Getting summary stats for all events

The eventstats command provides summary statistics on all events in
the search results
Makes the statistics available as a field on each event
You can then run additional stats, calculations, or reports on the field
47
eventstats
In this example, we get sum of the price field
for each product ID when a customer
removes the item from the cart
- Set a new field for that value lostSales!
Whats our revenue loss due to removal from

shopping cart?
Piping to eventstats, we create a new

field averageLosses!
- averageLosses contains the average of
the lostSales field value across ALL
events
- averageLosses value is added to each
event
Now we can add more commands to further
use the averageLosses field
48
eventstats (cont.)
Whats our revenue loss due to removal from
shopping cart?
Now that we have a summary

value on each row, we can apply
further calculations across the data
- We use the eval command to
create a new field difference!

- Use the values of the
averageLosses and
lostSales fields to calculate the
difference
The eval command is covered
later in this course
49
Create a chart
Running this in the Advanced Charting view creates a chart of the report
- Notice that all three statistics are included; lostSales, averageLosses, and
difference!
- Sales only wants to know how each product compares against an average
50
Remove unwanted fields

Once all statistics are calculated,
you can remove fields from the
chart that are no longer needed
In this example we remove the
lostSales and
averageLosses fields
- This makes the chart cleaner by
only displaying the stats important

to you
Charting options are covered
later in this course
51
Lab
Report on top and rare values
Use the stats command and associated functions
Use the eventstats command
52
Section 3:
Formatting / Calculating
53
Section Objectives
Understand the eval command
Perform calculations on values with eval!
Convert values
Round values
Format values
Use conditional statements
Further filter calculated results
54
Eval command overview

eval allows you to calculate and manipulate field values
- Does not re-write event data
Supports a variety of functions

Results of eval are written to a specified field
- can be a new or existing field
- If the destination field exists, the values of the field are replaced by the results of
eval
55
Convert values with eval

In this example, the report displays the average time it takes for the server
to process purchase and update http request actions
In the raw event, the time_taken field is stored in milliseconds
- Once the average is calculated, there are several decimal points as well
First, well use eval to convert the time_taken value into seconds
56
Convert values with eval (cont.)

The results of eval must always be set to a new or existing field
In this example:
- Set a new field reqTime with the value we wish to chart and convert
- Indicate which field the eval results should populate
- Divide the reqTime field value by 1000 to
convert milliseconds to seconds
Whats the average amount of time it takes to perform an update vs purchase?
Set new field

Define a field to set the eval results to
Divide the field value by 1000
57
Round values
But theres still too many pesky decimal points!
The round(field or number, decimals) function sets the value of a field to
the number of decimals you specify
- In this example, we divide the value of the reqTime field by 1000 then round to
2 decimal points.
Whats the average amount of time it takes to perform an update vs purchase?
Divide the value of reqTime by 1000, then round to 2 decimal points
58
Compare values with eval

How are specific products performing against
the average?
You can perform mathematical

functions against fields with
numeric field values
This example compares the
average price across all events
when action=purchase
against the actual price in a given
event
- Subtract the value of price from
avgPrice!
59
Format values with eval

eval can also be used to format values for a more meaningful display
Consider this search, that determines the lost revenue in the last 24 hours,
based on events that have a 503 status and the sum of the price field
values
How much money are we losing due to these server errors?
The result is somewhat ambiguous, especially if it will ultimately live on a

dashboard
60
Format values with eval (cont.)

The tostring function can convert a numeric field value to a string
In this example, a $ is prepended to the value
Using commas as an argument, the function automatically applies commas
appropriately
How much money are we losing due to these server errors?
Apply to lost_revenue field
Prepend with a dollar sign

Convert the value of lost_reveue to a string
Auto-apply commas appropriately
Also works with boolean values , returning true or false

61
tostring considerations
Once youve converted a numeric value to a string, the resulting value is no
longer considered numeric
- Sorting columns of numbers and timestamps may not work
- Automatic drilldown will not work with post-reporting conversions on charts
More information on formatting using the tostring
function can be found at splunk.com/base/
documentation
62
Using multiple eval commands

Remember that each new command
references the results of the previous
commands
In this example we:
- Set the sales field with total revenue for
each product ID
- Set a new field USD with the value of sales
and prepend with $
- Set a new field, GBP, with the exchange rate
(value of sales field * exchange rate)
- Prepend the GBP field with
- Remove the sales field from the final output
63
Conditional statements
The if function takes three arguments
- The first argument is a Boolean expression
- If it evaluates to TRUE, the result is the second argument
- If it evaluates to FALSE, the result evaluates to the third argument
Otherwise, group everything else into a value called Other
If the value of the usage field is Business
Apply the results to a field called Business Usage
How much network bandwidth is going

toward non-business activity?
64
Where command
Runs an eval expression to filter the results
- The result of the expression must be Boolean
- Keeps only the results for which the evaluation was successful and the Boolean
result was true
Useful in further filtering a search
65
Where command example

This example counts the
occurrences of each hostname,
then only returns a result when the
count exceeds 50
What sites were visited more than 50 times?
66
Lab
Convert field values
Round field values
Compare field values
Format field values
Use conditional statements
Filter results with the where command
67
Section 4:
Charting
68
Section Objectives
Identify chart types
Identify the chart command
Create a basic chart
Split values into multiple series
Define stacked mode
Omit null and other values from charts
Create a timechart
Chart multiple values on the same timeline
Format charts
Apply statistical functions
Group data with buckets
Create a rangemap
69
Chart types
Column and bar charts
- Compare the frequency of field
values
Stacked column and bar charts

- All columns are segments of a
single column
Line chart
- Show trends over time or another
field
70
Chart types (cont.)

Area chart
- Show trends either over time or in
comparison to another field value
Stacked area chart

- Show multiple series among the
trends in your data
Pie chart
- Show the relationship of parts of
your data to the entire set of data as

a whole
71
Chart command overview

The chart command can display any series of data that you want to plot
You decide what field is tracked on the x-axis
- Where stats uses the by clause to group data, chart uses by or over to
determine which field takes the x-axis!
chart avg(bytes) over host by date_wday the host field is the x-axis and
the series is split by day of week
chart avg(bytes) by host the host field is the x-axis since theres no split for
the series
- Because the chart command is designed to return chartable results, the value of
the y-axis should always be numeric
72
Basic chart example column

Note the following chart examples are created in
Advanced Charting view. Views > Advanced Charting
This example shows a basic chart
The count function counts the number of events for each product_name!
How many of each product are we selling?
73
Chart split by
If you want to split the series, identify the x-axis field with over, then use by
to split the series by additional fields
In this example, we split the series by host!
Which hosts are the most active?
74
Chart stacked mode

In stacked mode, all split-by values are displayed in a single column
75
Chart stacked mode

Hovering over a segment in the column displays the statistic for that segment
76
Omitting null and other

This example charts the file types that use a lot of bandwidth
Notice how the - and OTHER values skew the results
What file types use the most bandwidth?
77
Omitting null and other (cont.)

Removing the - value in the initial search, then adding the options
useother=f and usenull=f removes the empty and other field values
from the results
What file types use the most bandwidth?
78
Timechart
Time charts perform statistical aggregations against time
Trend behaviors and find anomalies over time
_time is always the x axis
You can optionally split data by another field
- each distinct value of the "split by" field is a separate series in the chart
Timecharts should typically be line or area charts
79
Basic timechart
This basic timechart displays the number of usage violations over the last
24 hours
Note the same functions and arguments youve used with stats and chart
apply
How many usage violations in
the last 24 hours?
80
Charting multiple values

This example displays the usage categories over a 1 hour period
Splitting by the usage field, each line represents a unique value of the field
The y-axis represents the count for each field value
Whats the overall usage trend

for the last 24 hours?
81
Formatting options
With the multi-series mode set to combined, all fields share the y-axis
82
Formatting options (cont.)

Setting the Multi-series mode to Split causes the y-axis to split for each field
value
The y-axis is divided into sections, each spanning the max and min count
83
Applying statistical functions

As with the stats and chart commands, statistical functions can be
applied to the timechart command
How much revenue are we
getting for each product over
the last hour?
84
Handling null values omit

Three options on handling null values in a timechart Omit, Connect, Treat
as Zero
The omit option displays gaps in the series
85
Handling null values Connect

The connect option fills the gaps with a trendline
86
Handling null values treat as zero

The treat as zero option flattens the trendline to the zero value
87
Bucket
The bucket command
puts continuous numerical
values in fields into
discrete sets, or buckets
This example displays
search results over a 7
day time span, but
buckets the results into 1
day chunks
- The bucket command is
applied to the _time field
88
Bucket (cont.)
This example displays web
requests that result in an http
status other than 200, grouped by
host.
The bucket command is applied to
the count field, and grouped into
intervals of 50.
89
Rangemap
Rangemaps are a single-value
report useful on dashboards
To create a rangemap
- Craft and save a search using the
rangemap command
- Add the saved search to a
dashboard panel
90
Rangemap (cont.)
Apply the rangemap to the
lostrevenue field
The rangemap command creates

a single-value result for a numeric
field value
Indicate a field to report on
Define conditions for the
rangemap display
- low displays green
- elevated displays yellow
What range of values are

noteable?
What range of values are

acceptable?
All other values, bad!
- severe displays red
91
Rangemap (cont.)
Once you create and save the
search, create a panel for a
dashboard
Todays Lost Revenue
- Select Single Value for the panel
type
- Select the Saved search
- Add the panel
92
Rangemap (cont.)
To display color based on your
specified ranges, you must add
the classField option to the
dashboard XML
- In the edit panel, click Edit name/
XML
The underlying XML displays
- Add
<option name="classField">range</option>
directly above the closing </single>

tag.
93
Lab
Create a basic column chart
Create a multi-series chart
Work with formatting options
Create a timechart
94
Section 5:
Correlating Events
95
Section objectives
Identify transactions
Group events using fields
Group events using fields and time
Search with transactions
Report on transactions
Determine when to use transactions vs stats
96
Transactions overview
A transaction is any group of conceptually related events that spans time
The events can come from multiple applications or hosts
- Events related to a single purchase from an online store can span across an
application server, database, e-commerce engine

- A single email message can create multiple events as it travels through various
queues
97
Basic transaction example

Each event in the network web
traffic logs represents a single user
generating a single http request.
Visiting a single website normally
generates multiple http requests
- Html file(s)
- Images
- Javascript
- Css
-
98
Group by fields
Using the transaction
command, you can group and
create a single event when
certain fields have the same
value
You can create transactions
across multiple tiers (i.e., web
server, application server,
media server) provided a
common field is shared
(accountID, sessionID)
99
Group by fields (cont.)

In this example we group by the
following fields:
- s_hostname
www.something.com
- cs_username the username
associated with the request
100
Group by fields and time (cont.)

We also Indicate the maximum
time span and pause between
individual events
- maxspan=20m!
The maximum total time between the

earliest and latest events
If not specified, default is -1 (or no limit)
- maxpause=5m!
The maximum total time between

events
If not specified, default is -1 (or no limit)
101
Examine a transaction
The example below is one of the transactions created by the search
The transaction
contains 7
individual events
Each event
shares the same
host and
username
102
Group by startswith / endswith

You can use the startswith and endswith options to form transactions
based on terms, field values, or evaluations
In this example, the first event in the transaction includes *signon* and
the last event includes purchase
103
Transaction-specific fields
The transaction command produces two additional fields, duration and
eventcount.
- The duration value is the difference between the timestamps for the first and last
event in the transaction.

- The eventcount value is the number of events in the transaction.
104
Searching with transactions

Transactions can be useful when
a single event may not provide
enough information
This example searches for the
term REJECT in the email logs.
- The events that include the term
dont provide much information

about the rejection
105
Searching with transactions (cont.)

By creating a transaction we
can then search and see
additional events related to
the rejection
- IP address of sender
- Reverse dns lookup results
- Action taken by the mail
system following the rejection
106
Reporting on transactions
You can use the same statistics and reporting commands with transactions
This example takes advantage of the duration field, showing a trend of
the mail queue slowing down over 4 hours, then beginning to correct
- Adding events to the transaction from additional hosts or sources would
uncover the cause of the slowdown
107
Transaction vs Stats
Use transaction when you need to see the events correlated together
Use stats when you just want to see the results of a calculation
Use transaction when you must define event grouping based on start/
end values
Use stats when you can group events based on a field value (eg by
src_ip)
When you can do it either way, choose stats because it is more efficient
- sourcetype=trade_entries
| transaction TradeID | table TradeID, eventcount
vs.
sourcetype=trade_entries | stats count by TradeID!
108
Lab
Correlate events with transactions
Search and report on transactions
109
Section 6:
Enriching Data with Lookups
110
Section Objectives
Discover lookups
Examine a lookup file example
Create a lookup table
Define a lookup
Configure an automatic lookup
Use the lookup in searches and reports
111
Lookups
Some data should not be indexed within Splunk
- Static or relatively unchanging data
Lookups allow you to add additional fields to your events:

- http status code descriptions (file not found, service unavailable)
- Descriptive text descriptions for errors or item IDs
- User names based on asset tags or static IP addresses
Two types of lookups

- File lookup
- Scripted lookup
112
Defining a file lookup

1. Create the lookup table
2. Define the lookup
3. Configure the lookup to run
automatically
113
Lookup file example

This example displays a lookup .csv
file used to associate a product name,
price, and competitors prices with an ID
The first row represents field

names (header)
The product_id field exists in the

access_combined events
- This is the input field
product_name, price,
flowermart, and flowersRUs fields
will be available to search once the
lookup is defined
- These are the output fields
114

The first step in creating a
lookup is creating the lookup
table
Manager >> Lookups >>
Lookup table files lists all
existing lookup tables
- From this list you can edit,
change permissions, enable/

disable, and delete
115
Create a lookup table (cont.)

1. Click new to display the
Add New form
2. Select a destination app
3. Browse and select the
.csv file to use for the
lookup table
4. Enter a name for the
lookup file
5. Save
116
Define the lookup

Lookup definitions lists all
existing lookup definitions
- From this list you can edit,
change permissions,
enable/disable, delete and
clone
117
Define the lookup (cont.)

1. Click new to display the
Add New form
2. Select a destination app
3. Enter a name for the lookup
definition
4. Select the .csv file to use
with the definition
118
Time-based lookups
lookups can also be time-based if
a field in the lookup table that
represents a timestamp
- Specify the name of the time field
You can also specify a strptime

format and offsets for the time
matching
119
Advanced Options
Under Advanced options, you
can specify:
- The minimum number of matches
for each input lookup value

- The maximum number of matches
for each input lookup value
- A default value to output if fewer
than the minimum number of
matches are present for a given
input
120
Using a non-automatic lookup

If a lookup is not configured to run automatically, use the lookup command
in your search to use the lookup fields
Format is
| lookup <lookup_name> <input field> OUTPUT <output field>, <output field>!
This example calls a lookup that includes fields for a browser and os based
on the value of the useragent field
121
Configure automatic lookup

Automatic lookups lists all
existing automatic lookup
configurations
- From this list you can edit
change permissions, enable/

disable, and clone
122
Configure automatic lookup (cont.)

1. Select the destination app
2. Enter a name
3. Select the lookup table
prod_lookup
4. Choose a sourcetype, source or

host, and identify by name
123
Configure automatic lookup (cont.)

5. Define the Lookup input fields
-
The field that exists in your events

that you are relating to the lookup
table
6. Define the Lookup output fields

-
-
The fields from your lookup table

that will be added to the events
Optionally, give the fields an
alternate name by typing a new
name after the = sign
124
Using the automatic lookup

product_id field in event
To use an automatic lookup,

simply include the fields in
your search
Resulting report
Fields from lookup in search
125
More lookup options

In addition to creating and using a file-based lookup you can also:
- Populate a lookup table with search results
- Set up a fields lookup based on an external command or script
- Set up a fields lookup based on an external database
More information on creating and configuring
lookups can be found at splunk.com/base/
documentation
126
Lab
Upload a lookup file
Use the lookup in a search
Configure the lookup to run automatically
Use the automatic lookup in a search
127
Section 7:
Summary Indexing
128
Section objectives
Define summary indexing
Create and schedule a summary search
Populate a summary index
Run searches against a summary index
Identify gaps and overlaps in the summary index
Correct gaps and overlaps in the summary index
129
Summary Indexing Overview

Efficiently report on large volumes of data
Spread the cost of a computationally expensive report over time
Common use cases include:
- Run reports over long time ranges for large datasets more efficiently
Example: Show the number of page views and visitors each of your Web sites had over
the past 30 days, broken out by site
- Build a rolling report that shows aggregated statistics over long period of time
Example: A running count of downloads for a specific file on a website

Example: Calculate the average amount spent per purchase over a year.
130
Without summary indexing.

In this example, were running a search for the top videos requested from a
website in the last 30 days
The site gets millions of hits per day
Searching the default index for a 30-day time span is not efficient as it scans
across the entire data set for the specified 30 day period
index
131
With summary indexing

The pre-calculated statistics you want to report on is incrementally added to a
summary index via a saved, scheduled search
Data is stored in a special table format that optimizes efficiency.
- Each time the saved search runs, it appends the data in the summary index.
Searches or reports that may take several minutes or more to complete can be
generated quickly
132
Summary indexing steps

1. Create a saved search using the si prefix that extracts broad statistics
you want to report on.
2. Schedule the search to run periodically over an appropriate time interval
(hourly, daily, every n-miuntes depending on your needs.
Each time the search runs, it saves the results since the last run into a summary
index that you designate.
3. Run searches and reports on this smaller, "faster" summary index

instead of working with the much larger dataset.
133
Summary indexing flow

3.
sourcetype=access_combined | sistats count by file!

Run every hour, time range 1 hour
1.
2.
index
Populating search
results are
formatted and
stored as statistics
134
Summary
index
1. Creating the summary search

Dont pipe other search operators
after the main summary indexing
reporting command
- Save that for the searches you run
against the summary indexes
Create a broad enough search to

capture the data you want
Use the proper reporting
command to get the statistics you
want for further processing later.
135
1. Creating the summary search (cont.)

Make the time span of the search
smaller than the time span of the
reports you want to run against it
- To report on a month or a year,
make the summary search for -24h

hours, and run every day
- For a day -1h and run every hour.
136
2. Schedule and save the search

If the time range you defined is 1
hour, schedule to run every hour.
- For 1 day, schedule 1 day, etc
Make sure to enable summary
indexing
Select the default summary
index, or a new index you created
- If selecting a new index, it must
be empty
137
3. Run searches against summary

Youre limited to the initial reporting command you used to create the index,
but you can pipe to additional computations or operators.
- Saved search:
sourcetype = access_combined | sistats count by clientip!
- Search on index:
index=summary search_name=searchName| stats count by clientip | <additional commands>!
138
Summary index gaps

Gaps are usually caused by two main factors:
A summary index initially only contains events from the point that you start
data collection
- There wont be data from before the summary index collection start date
- You can put it in there yourself with the backfill script.
splunkd outages
- If splunkd goes down for a significant amount of time, you could get gaps in your
summary data, depending on the populating search schedules.
139
Identifying Gaps and overlaps

To identify gaps and overlaps in your data, run a search against the
summary index that uses the overlap command.
- Identify suspected gaps/overlaps in a search by specifying a start time
and
end time or a period and a saved search name, followed by the | overlap
command in the search string.

- Full details can be found at:
http://www.splunk.com/base/Documentation/latest/Knowledge/
Managesummaryindexgapsandoverlaps
140
Correcting gaps
Use the backfill script fill_summary_index.py to correct gaps or backfill your
summary index
- Specify the App
- Specify a list of summary searches to backfill
- Or, specify to backfill all summary searches for the App
Use -dedup true to ensure the backfill script does not create duplicates
141
Correcting gaps (cont.)

To backfill all summary index that already has data in it:
./splunk cmd python fill_summary_index.py -app flowerstore -name "*" -et mon@mon -lt @mon -dedup true -auth admin:changeme !
-dedup ensures the backfill script does not create an overlap
To backfill a summary index that does not have any data, dont use dedup
More information on backfilling a summary index can
be found at splunk.com/base/documentation
142
Summary index overlaps

Overlaps are events in a summary index (from the same index-populating
search) that share the same timestamp
Overlapping events skew reports and statistics created from summary
indexes
Overlaps can occur if you set the time range of a saved search to be longer
than the scheduled search interval
- For example, don't arrange for an hourly search to gather data for the past 90
minutes
Manually delete the overlaps from the summary index by using the search
language.
143
Lab
Search a summary index
Create a report from a summary index
144
Section 8:
Macros
145
Section Objectives
Understand macros
Manage macros
Create a basic macro
Use a basic macro
Define arguments / variables for a macro
Use arguments with a macro
Add and use multiple arguments with a macro
146
Macros overview
Useful when you frequently run searches or reports with similar search
syntax
Chunks of a search you can reuse in multiple places
Can be any chunk of your search string or search command pipeline
Allows you to define one or more arguments within the search segment
- Pass values to the argument when using the macro
147
Managing Macros
As with other types of knowledge
objects, you can:
Create new
Edit
Set permissions
Enable/disable
Clone
Move
Delete
148
Creating a basic macro

Manager >> Advanced search >>
Search Macros >> Add New
1. Select the destination app
2. Enter a name
-
If the macro includes arguments,

include the number of args in the
name
3. Type the search chunk

4. Save
149
Using a basic macro

Type the macro name into the
search bar, or use it in a saved
search or report
Surround the macro name with
the left-quote (or grave accent)
character
sourcetype="cisco_esa" | transaction mid dcid icid!
- `macroname`
- Do not confuse with single-quote
character ()
Pipe to more commands, or

precede with search string
150
Adding arguments
Change one or more variables of
the macro at search time
Include the number of arguments
in parenthesis after the macro
name
- performance(1)
Within the search definition use

$arg$!
- host=$host$!
In the arguments field, enter the

name of the argument
151
Using arguments
When using a macro with
arguments, include the argument
in parenthesis following the macro
name.
sourcetype="access_*" host=$host$
| chart avg(time_taken) as reqTime by action
| eval reqTime = round(reqTime/1000, 2)!
152
Adding multiple arguments

When creating multiple arguments
in the same macro:
Include the number of variables in
parenthesis in the macro name
- performance(2)
Enter the argument names,

separated by commas
- host, status
153
Using multiple arguments

Include the arguments in
parenthesis, separated by a
comma
sourcetype="access_*" host=$host$ status=$status$
| chart avg(time_taken) as reqTime by action
| eval reqTime = round(reqTime/1000, 2)!
154
Lab
Create a basic macro
Use the basic macro in a search
Create a macro with arguments
Use the macro and pass the arguments in a search
155
Community based support

Splunk docs are constantly being updated and improved, so be sure
to select your version of Splunk to make sure the doc you are
reading applies to your version
http://www.splunk.com/base/Documentation
Splunk Answers: post specific questions and get them answered by
Splunk experts (also makes for great and informative reading)
http://answers.splunk.com
IRC Channel: Splunk maintains a channel #splunk on the EFNet
IRC server. Support engineers and many well-informed Splunk
users hang out there
156
Splunk Support
Contact Splunk Support email: support@splunk.com
File a case online
http://www.splunk.com/index.php/submit_issue
24/7 phone depending on support contract
IRC #splunk on EFNET servers
157
.conf2011:
nd
The 2 Annual Splunk Worldwide Users Conference
August 15-17, 2011: The WesFn St. Francis, San Francisco
54+ sessions : SplunkAnswers Computer Lab : 2 General
Sessions : BoF Meetups
Alumni rate: $595
Buy 3 get 1 free!
Use your Splunk EducaFon credits for Splunk University!
Register today: www.splunk.com/goto/conf2011

Thank You
Please fill out the class survey
159

Search Report 42

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Search Report 42

Uploaded by

Copyright:

Available Formats

Searching and Reporting with Splunk 4.

Document usage guidelines

Searching and Reporting with Splunk 4.2

Searching and Reporting with Splunk 4.2

Searching and Reporting with Splunk 4.2

Data used in this course

Events related to the online store web activity and sales

- Cisco mail logs

Events related to the internal email system

Searching and Reporting with Splunk 4.2

Data used in this course (cont.)

Events related to employee network usage

Searching and Reporting with Splunk 4.2

Searching and Reporting with Splunk 4.2

Searching and Reporting with Splunk 4.2

Searching and Reporting with Splunk 4.2

Basic search review

Searching and Reporting with Splunk 4.2

General search practices

denied is always better than

searching for denied

Searching and Reporting with Splunk 4.2

General search practices (cont.)

denied" will be faster than NOT "access

Apply powerful filtering commands as early in your search as possible

than filtering to one thousand events and then ten events

Searching and Reporting with Splunk 4.2

sourcetype=syslog ERROR | top user | fields - percent

Search language syntax concepts

Searching and Reporting with Splunk 4.2

Search language example

Searching and Reporting with Splunk 4.2

Searching and Reporting with Splunk 4.2

Fields command overview

Searching and Reporting with Splunk 4.2

Include specific fields

remove internal fields unless

Searching and Reporting with Splunk 4.2

Remove specific fields

Searching and Reporting with Splunk 4.2

Searching and Reporting with Splunk 4.2

rename JSESSIONID as session ID!

Can be applied to wildcards to

rename *ip as IP Address!

Searching and Reporting with Splunk 4.2

Extract fields at search time

Searching and Reporting with Splunk 4.2

Extract fields with rex

For example purposes, the user name field has

Searching and Reporting with Splunk 4.2

Extract fields from a table-formatted event

Searching and Reporting with Splunk 4.2

Extract fields from a table-formatted event (cont.)

each event as displayed on

The command creates a separate

table command for better display

Searching and Reporting with Splunk 4.2

Combine multi-valued fields

There are many commands for use with

Searching and Reporting with Splunk 4.2

Lab working with fields

Searching and Reporting with Splunk 4.2

Searching and Reporting with Splunk 4.2

Searching and Reporting with Splunk 4.2