12/6/2014 Collect & visualize your logs with Logstash, Elasticsearch & Redis | Michael Bouvy

Michael Bouvy
Yet another IT blog ?

Collect & visualize your logs with Logstash,

Elasticsearch & Redis
19 November 2013 Internet Apache, elasticsearch, iptables, kibana, logs, logstash, redis,
sysadmin, syslog
Update of December 6th : although Logstash does the job as a log shipper, you might consider
replacing it with Lumberjack / Logstash Forwarder, which needs way less resources, and keep Logstash
on your indexer to collect, transform and index your logs data (into ElasticSearch) : check out my latest
blog post on the topic.

Kibana Dashboard

Even if you manage a single Linux server, you probably already know how hard it is to keep an eye on
whats going on with your server, and especially tracking logs data. And this becomes even worse when
you have several (physical or virtual) servers to administrate.

Although Munin is very helpful monitoring various informations

from my servers / VMs, I felt the need of something more, and bit
less static / more interactive.

There are 3 kind of logs I especially wanted to track :

Apache 2 access logs

iptables logs
Syslogs

After searching arround on the internet for a great tool that would
help me, I read about the open source log management tool
Logstash which seems to perfectly suit a (major) part of my needs :
logs collecting / processing.

For the purpose of this post, I will take the following network
architecture and assume and I want to collect my Apache, iptables, system logs from servers 1/2/3
(shippers) on server 4 (indexer) and visualize them :

http://michael.bouvy.net/blog/en/2013/11/19/collect-visualize-your-logs-logstash-elasticsearch-redis-kibana/ 1/9
12/6/2014 Collect & visualize your logs with Logstash, Elasticsearch & Redis | Michael Bouvy

As you can see, I am using 4 complementary applications, the role of each one being :

Logstash : logs collector, processor and shipper (to Redis) on log shippers 1-3 ; logs indexer on
server 4 (reads from Redis, writes to Elasticsearch)
Redis : logs data broker, receiving data from log shippers 1-3
Elasticsearch : logs data persistent storage
Kibana : (time-based) logs data visualization (graphs, tables, etc.)

Installation
As shown on the schema above, I will describe how to install all of Logstash + Redis + Elasticsearch +
Kibana on the same indexer server. You may want to separate these on different servers for any
reason, just set the correct IPs / hostnames accordingly, in the examples below.

Redis

First of all, lets install Redis on our indexer server (right, thats #4 on the schema). As the versions of
Redis available in Linux distributions repositories are not up to date, well download the last stable
release from Redis website :

http://michael.bouvy.net/blog/en/2013/11/19/collect-visualize-your-logs-logstash-elasticsearch-redis-kibana/ 2/9
12/6/2014 Collect & visualize your logs with Logstash, Elasticsearch & Redis | Michael Bouvy
$sudoaptitudeinstallgcc
$wgethttp://download.redis.io/releases/redis2.6.16.tar.gz
$tarxzfredis2.6.16.tar.gz
$cdredis2.6.16
$makeMALLOC=libc
$sudocpsrc/redisserver/usr/local/bin/
$sudocpsrc/rediscli/usr/local/bin/

Launch redis-server (sudo redis-server), try to ping Redis to see if server is working :

$rediscliping

If you get a PONG reply, your Redis server works fine. You might want to install Redis more properly, if
so, follow this excellent guide at Redis.io.

Youre now ready to ship logs data from your servers to Redis. Note that Redis listens on its default port
(tcp/6379) and accepts incoming connections from any IP :

$netstattanpu|grepredis
tcp000.0.0.0:63790.0.0.0:*LISTEN16955/redisserver

Logstash (shippers)
You will need to set an instance of Logstash on each of your servers you want to collect data from, it will
act as a logs shipper.

Open a shell on one of the servers you want to collect log data from, and dowload logstash.

$sudomkdir/opt/logstash/etc/logstash
$sudocd/opt/logstash
$sudowgethttps://download.elasticsearch.org/logstash/logstash/logstash1.2.2flatjar.jar

Create a Logstash config file in /etc/logstash :

/etc/logstash/logstashtest.conf
1 input{stdin{}}
2 output{stdout{codec=>rubydebug}}

Now launch the logstash agent and type something, you should get something like this :

$javaXmx256mjarlogstash1.2.2flatjar.jaragentflogstashtest.conf
helloworld
{
"message"=>"helloworld",
"@timestamp"=>"20131117T18:35:56.672Z",
"@version"=>"1",
"host"=>"myhostname"
}

Logstash works fine, lets configure it to work with our previously-installed Redis instance. Create a new
config file :

/etc/logstash/logstashredis.conf
1 input{stdin{}}
2 output{
3 stdout{codec=>rubydebug}
http://michael.bouvy.net/blog/en/2013/11/19/collect-visualize-your-logs-logstash-elasticsearch-redis-kibana/ 3/9
12/6/2014 Collect & visualize your logs with Logstash, Elasticsearch & Redis | Michael Bouvy
3 stdout{codec=>rubydebug}
4 redis{host=>"10.0.0.5"data_type=>"list"key=>"logstash"}
5 }

Youll of course need to replace 10.0.0.5 by the IP of your server Redis is running on.

Launch logstash agent with logstash-redis.conf as config file and type something as above. Then, on
your indexer server (where Redis is installed), launch redis-cli :

redis127.0.0.1:6379>LLENlogstash
(integer1)
redis127.0.0.1:6379>LPOPlogstash
"{\"message\":\"helloredis\",\"@timestamp\":\"20131117T20:35:13.910Z\",\"@version\":\"1\",

Here it is, weve got our message transmitted by Logstash to our Redis server. Youve probably noticed
that Logstash added a few fields to our initial (minimalistic) data (@timestamp, @version and host).

Now that weve got Logstash able to send data to Redis, we can begin processing our Apache 2 and
iptables logs.

Apache 2 logs processing

Create a new config file in /etc/logstash :

/etc/logstash/logstashshipper.conf
1 input{
2 file{
3 path=>"/var/log/apache2/*access.log"
4 type=>"apache"
5 }
6 }
7
8 filter{
9 if[type]=="apache"{
10 grok{
11 pattern=>"%{COMBINEDAPACHELOG}"
12 }
13 }
14 }
15
16 output{
17 redis{host=>"10.0.0.5"data_type=>"list"key=>"logstash"}
18 }

This config is quite self-explanatory ; few things although :

type => apache allows us to use conditionals further

pattern => %{COMBINEDAPACHELOG} is a built-in regex-like used to match against our
Apache logs lines and extract fields (request, host, response, etc.)

Launch the logstash agent, and youre done. Its that simple ! You should now see the logstash list count
grow in Redis (LLEN logstash) as your Apache gets hits.

iptables logs processing

http://michael.bouvy.net/blog/en/2013/11/19/collect-visualize-your-logs-logstash-elasticsearch-redis-kibana/ 4/9
12/6/2014 Collect & visualize your logs with Logstash, Elasticsearch & Redis | Michael Bouvy

There is no built-in grok pattern available to extract data from iptables logs, but theres one available in
Logstashs cookbook config snippets.

Create a directory where you will keep your custom grok patterns (i.e. /usr/share/grok/patterns) and
create a new file called iptables :

/usr/share/grok/patterns/iptables
1 #Source:http://cookbook.logstash.net/recipes/configsnippets/
2 NETFILTERMAC%{COMMONMAC:dst_mac}:%{COMMONMAC:src_mac}:%{ETHTYPE:ethtype}
3 ETHTYPE(?:(?:[AFaf09]{2}):(?:[AFaf09]{2}))
4 IPTABLES1(?:IN=%{WORD:in_device}OUT=(%{WORD:out_device})?MAC=%{NETFILTERMAC}SRC=%{I
5 IPTABLES2(?:IN=%{WORD:in_device}OUT=(%{WORD:out_device})?MAC=%{NETFILTERMAC}SRC=%{I
6 IPTABLES(?:%{IPTABLES1}|%{IPTABLES2})

Youll also need to declare this directory in Logstashs config file (see below). Now lets process our
iptables logs, create or edit a logstash config file :

/etc/logstash/logstashshipper.conf
1 input{
2 file{
3 path=>["/var/log/syslog"]
4 type=>"iptables"
5 }
6 }
7
8 filter{
9 if[type]=="iptables"{
10 grok{
11 patterns_dir=>"/usr/share/grok/patterns/iptables"
12 pattern=>"%{IPTABLES}"
13 }
14 }
15 }
16
17 output{
18 #Checkthattheprocessedlinematchedagainstgrokiptablespattern
19 if!("_grokparsefailure"in[tags]){
20 redis{host=>"10.0.0.5"data_type=>"list"key=>"logstash"}
21 }
22 }

Actually, despite the very useful Grok Debugger, I couldnt get these this pattern working. Plus, you will
have to guess one way or another wether the log line is a REJECT, DROP, ACCEPT or whatever.

To make this simpler, you may use iptables rules like this :

iptablesNLogAndDrop
iptablesALogAndDropptcpjLOGlogprefix"RULE1DROP"loglevel=info
iptablesALogAndDropjDROP

You can also create rules for REJECT / ACCEPT rules following this one.

Good thing is that your iptables log lines will now be prefixed with a DROP (or REJECT / ACCEPT),
allowing you to process these log lines in a different way, measuring ACCEPT vs. DROP/REJECT count for
instance. Here is the grok pattern you can use :

/usr/share/grok/patterns/iptables
1 IPTABLES(.*RULE\d?(%{WORD:action})?.*SRC=(%{IP:src_ip}).*DST=(%{IP:dst_ip}).*PROT
http://michael.bouvy.net/blog/en/2013/11/19/collect-visualize-your-logs-logstash-elasticsearch-redis-kibana/ 5/9
12/6/2014 Collect & visualize your logs with Logstash, Elasticsearch & Redis | Michael Bouvy
1 IPTABLES(.*RULE\d?(%{WORD:action})?.*SRC=(%{IP:src_ip}).*DST=(%{IP:dst_ip}).*PROT

The following fields will be extracted for you iptables logs :

action = depending on what you set in your custom iptables rules, may be REJECT, DROP,
ACCEPT
src_ip = source IP address
dst_ip = destination IP address
protocol = protocol (TCP, UDP, ICMP, etc.)
src_port = source port number
dst_port = destination port number

Youll probably notice that not all the data available in the logs is exctracted, feel free to adapt the grok
pattern upon your specific needs.

Note that if you decide to create a log & accept iptables rule, its definitely
NOT a good idea to systematically use it instead of the regular ACCEPT one.
Youd rather use it to track connections from specific IP addresses ranges for
example.

system logs (syslog) processing

Edit your existing one or create a new Logstash config file :

/etc/logstash/logstashshipper.conf
1 input{
2 file{
3 path=>["/var/log/*.log","/var/log/messages","/var/log/syslog"]
4 type=>"syslog"
5 }
6 }
7
8 output{
9 redis{host=>"10.0.0.5"data_type=>"list"key=>"logstash"}
10 }

As each log line may have a different format, they will each be stored as is in the message field in
Elasticsearch. Anyway, this will not prevent you from analyzing this data (by example getting the
number of (un)successful authentications from auth.log).

Elasticsearch

http://michael.bouvy.net/blog/en/2013/11/19/collect-visualize-your-logs-logstash-elasticsearch-redis-kibana/ 6/9
12/6/2014 Collect & visualize your logs with Logstash, Elasticsearch & Redis | Michael Bouvy

Thanks to a Debian package available on Elasticsearchs official download page, a few command lines
only will be sufficient to get it up and running :

$sudoaptitudeinstallopenjdk7jreheadless
$wgethttps://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch0
$sudodpkgielasticsearch

Elasticsearch should have started automatically, open your browser and reach
http://yourhostname:9200/. If everything wen fine, you should get a JSON response looking like this
:

{
"ok":true,
"status":200,
"name":"Alibar",
"version":{
"number":"0.90.7",
"build_hash":"36897d07dadcb70a865b7f149e645ed3d44eb5f2",
"build_timestamp":"20131113T12:06:54Z",
"build_snapshot":false,
"lucene_version":"4.5.1"
},
"tagline":"YouKnow,forSearch"
}

If necessary, you can tune Elasticsearchs run parameters in /etc/default/elasticsearch and

configuration parameters in /etc/elasticsearch/[elasticsearch,logging].yml.

Note for OpenVZ users

After (too much) hours of searching and trying various configurations, I still couldnt get my
Elasticsearch running in an OpenVZ Debian container, or more precisely, it wont listen for incoming
(HTTP) connections on its default port 9200 (visible process, but nothing with netstat).

It actually seems to be a common issue with Java running in an OpenVZ container, and I finally found a
solution in this post from OpenVZ forums.

In short, edit your CT config file (usually /etc/vz/conf/CTID.conf), comment out the CPUS line and add a
CPULIMIT line as following :

/etc/vz/conf/CTID.conf
CPUUNITS="1000"
#CPUS="1"
CPULIMIT="100"

(Re)start your container, Elasticsearch should now work fine.

Logstash (indexer)
Thanks to a comment from DJP78, I realized that I forgot to explain how to configure Logstash on the
indexer side : pulling logs data from Redis and storing them into Elasticsearch.

Here is the Logstash config you can use (note that I also process local [indexer] system logs) :
http://michael.bouvy.net/blog/en/2013/11/19/collect-visualize-your-logs-logstash-elasticsearch-redis-kibana/ 7/9
12/6/2014 Collect & visualize your logs with Logstash, Elasticsearch & Redis | Michael Bouvy

/etc/logstash/logstashindexer.conf
1 input{
2 file{
3 type=>"syslog"
4 path=>["/var/log/*.log","/var/log/messages","/var/log/syslog"]
5 }
6 redis{
7 host=>"127.0.0.1"
8 data_type=>"list"
9 key=>"logstash"
10 codec=>json
11 }
12 }
13 output{
14 elasticsearch{bind_host=>"127.0.0.1"}
15 }

You can check if Logstash is correctly doing his job on the indexer, by either watching the list size
decrease in Redis (redis-cli and then LLEN logstash) or searching your Elasticsearch index via a HTTP
GET request : http://yourElasticSearchHostname:9200/_search?
q=_index%20like%20logstash%25&sort=@timestamp:desc.

Kibana
Finally, lets install Kibana. Kibana is a modern & dynamic (AngularJS based) frontend for Logstash /
Elasticsearch, allowing you to get charts, tables, etc. from your collected logs data.

All you need to use Kibana is a HTTP web server and access to Elasticsearchs port 9200 (from your
browser).

Its installation is quite straight-forward :

$sudoaptitudeinstallgit
$cd/var/www
$gitclonehttps://github.com/elasticsearch/kibana.gitkibana

Now open http://yourhostname/kibana/ in your browser. Tada !

Note that if Elasticsearch is not installed on the same server (or available through the same hostname)
as Elasticsearch, youll need to configure its hostname (and possibly port) in config.js at Kibanas root.

On first launch, Kibana offers you to use a Logstash dashboard, click on the link. You can now see your
logs data in a table, try to activate some useful fields in the left column, or create your first graph :-).

http://michael.bouvy.net/blog/en/2013/11/19/collect-visualize-your-logs-logstash-elasticsearch-redis-kibana/ 8/9
12/6/2014 Collect & visualize your logs with Logstash, Elasticsearch & Redis | Michael Bouvy

tl;dr
Download Logstash on all your shippers and your indexer
Install and launch Redis on your indexer
Install and launch Elasticsearch on your indexer
Clone Kibana git repository on your indexer in /var/www
Create Logstash config files for your shippers and indexer (see above), launch all Logstash
instances

About Michael
PHP web developer at Smile (France), passionate about programming and electronics.

http://michael.bouvy.net/blog/en/2013/11/19/collect-visualize-your-logs-logstash-elasticsearch-redis-kibana/ 9/9