Performing Indexing operation using HadoopMapReduce and Terrier
Atharva Patel
atharva_patel@daiict.ac.in
DA-IICT
Abstract
This paper introduces the map reduce
mechanism for distributing a data or
computation intensive tasks on larger
number of computation nodes to
achieve speed up in the operation. As an
example we describe how an indexing
operation of large amount of data can be
carried out using MapReduce technique.
Following that is a guide for setting up a
cluster for using hadoop and configuring
terrier to make use of this platform.
Paper also describes some results
derived from some experiments carried
out in the Glasgow University. The
paper can turn out to be a useful
resource for a beginner in the area of
Mapreduce and Hadoop platform.
1. Introduction
Indexing in real world on large data corpus
is considered to be very data and
computation intensive task. For academic
researchers especially in the field of
information Retrieval, it is very essential to
be able to verify their ideas and concepts on
actual large document collection. Carrying
out these operations on a single node
computer is very poor choice as it takes
around hours or days in performing certain
operations like indexing. The problem
becomes even worse when the size of the
data collection is very large (>100GB).
It seems to be very rational choice
to make use of the distributed computing
mechanism to perform such operations in
comparably less amount of time. For the
academic learning and research purpose,
terrier v3.0 is popular software as it
provides very easy way to change the
parameters of various operations supported
by it as well as the open source code written
Sai Dinesh
sai_dinesh@daiict.ac.in
DA-IICT
in Java language which is a very popular
programming language. HadoopMapreduce
tool developed by Yahoo and Apache
provides a very good platform to carry out
such data and computational intensive
tasks. MapReduce methodology is
developed by Google Inc.[12] which provides
great amount of abstraction to the user in
performing their computations without
getting worried about the issues involved in
parallelism, fault tolerance, load balancing,
data distribution etc.
This paper will give the viewer an
introductory level of description about the
mechanism of map and reduce methodology
with an example of performing indexing
operation on a sample data collection of
FIRE 2010in TELEGRAPH English language
of the size of around 2GB. Section 2 contains
the description of map reduce mechanism.
Section 3 describes indexing as an example
for using map-reduce methodology for
distributing the task. Section 4 of this paper
will show the setup guide and configuration
details. This section can prove to be very
useful for the people willing to make use of
Hadoop to perform various Information
Retrieval related task using terrier or any
other software tool. In section 5 we have
presented the results derived from a
experiment carried out at Glasgow
University, which demonstrate the
computational speedup that can be
achieved by deploying hadoop clusters for
indexing large data.
2. Introduction to MapReduce
method[1]
The computation takes a set of input
key/value pairs, and produces a set of
output key/value pairs. The user of the
mapReduce library expresses the
computation as two functions: Map and
Reduce.
Map, written by the user, takes an input pair
and produces a set of intermediate
key/value pairs. The MapReduce library
groups together all intermediate values
associated with the same intermediate key
Iand passes themto the Reduce function.
The Reduce function, also written by the
user, acceptsan intermediate keyI and a set
of values for that key. It merges together
3. Single pass indexing using
scalable MapReduce[2]
In a normal opeartion of single pass
indexing without mapreduce, compressed
posting lists for each term are built in
memory as the corpus is scanned. When
local memory is exhausted, the partial
indices are flushed to disk and final index is
built from the merged flushes. Elias-Gamma
compression is used to store document-
identifier (doc-id) deltas; ensuring postings
are fully compressed, both in memory and
on disk.
Now performing this single pass indexing
task using MapReduce, document
processing is split over m map tasks, with
each map task processing its own subset of
the input data. However memory runs low
or all the documents for that map have been
processed, the partial index is flushed from
Figure 1Correcting document IDs while merging. [2]
the map task, by emitting a set of <term,
these values to form a possibly smaller set
of values. Typically just zero or one output
value is produced per Reduce invocation.
The intermediate values are supplied to the
user's reduce function via an iterator.This
allows us to handle lists of values that are
too large to fit in memory.
Symbolically these operations can be
expressed like this:
map(k1,v1) →list(k2,v2)
reduce(k2,list(v2)) →list(v2)
posting list> pairs. As in single-pass
indexing, the posting lists are compressed
to minimize the data that is transferred
between map and reduce tasks. Moreover,
each map task is not aware of its context in
the overall indexing job, the doc-ids used in
the emitted posting lists cannot be globally
correct instead, these doc-ids start from 0 in
each flush.
The partial indices (flushes) are then sorted
by term, map and flush numbers before
being passed to one or more reduce tasks.
Each reducer collates the posting lists to
create the final inverted index for the
documents it processed. In particular, as the
flushes are collected at an appropriate
reduce task, the posting lists for each term
are merged by map number and flush
number, to ensure that the posting lists for
each term are in a globally correct doc-id
ordering. The reduce function takes each
term in turn and merges the posting lists for
that term into a full posting list. Figure
1presents an example for a distributed
setting MapReduce indexing paradigm of
200 documents. Note that the number of
reduce tasks therefore determines the final
number of inverted index shards created.
4. Setup guide for installing Hadoop
on a cluster
Figure 2 Visualization of the operation of
HadoopMapReduce[8]
The references I have mentioned below [7]
and [8] provide excellent step by step
guidance for installing and running hadoop
on Linux systems. There are several
corrections to them that I would like to
mention here.
As suggested in reference [7], while creating
a new user account called ‘hadoop’ one
should specify the option –m in the
command:
useradd –g hadoop_user –s /bin/bash –d
/home/hadoop –m hadoop
This will tell the system to create home
directory for the user. Without that, the
system will not generate a home directory
for new ‘hadoop’ user.
Different machines would be having
different versions and locations of Java
installed on them. It is preferable to use
same version of Java JDK on all the
machines, otherwise all the machines
should have at least Sun JDK 1.6 or more
recent version than that installed and the
JAVA_HOME path should be set correctly for
pointing to whatever installed java version’s
installation directory: two most popular
places where it is found are /usr/lib/jvm
or /usr. JAVA_HOME should be mentioned
in the hadoop-env.sh file as it is executed
every time when hadoop starts and it
overwrites all the paths set by the
.bashrc, .bash_profile or .profile files.
If the user wants to add IP addresses of all
the master and slave nodes in /etc/hosts
file then he/she only use the host names of
the IP addresses mentioned in the
/etc/hosts files at all the configuration files
like hadoop-site.xml in older versions like
0.18.x or in files like core-site.xml, hdfs-
site.xml and mapred-stie.xml etc. Violating
this may cause hadoop to throw some
exceptions saying: expected
hdfs://hostname:9000, found
hdfs://AA.BB.CC.DD:9000 etc.
While starting hadoop cluster using ./start-
all.sh file, it doesn’t clearly specify whether
all the services like namenode, jobtracker,
datanode, tasktracker, secondarynamenode
etc. started successfully or not. If the user
wants to be sure about this, he/she should
start each of these services one by one using
command like ./hadoopnamenode will start
namenode on master machine.
./hadoopjobtracker will start
mapredjobtracker on master machine. They
will also print specific messages on screen if
some exceptions were thrown while
starting up these services so that user can
try to work out them accordingly.
In the reference [8] the author has
mentioned about one exception appearing
about the namespaceID incompatible
between namenode and datanode. Two
workarounds are suggested by the author
there. Here is one more way in which one
can update namespaceIDs of all the slaves
just by running a script from the master
node. Here it is [9].
In our experiment we often failed to start or
connect jobtracker from the namenode. The
temporary solution we applied was to run
jobtracker on the same machine as
namenode by giving same hostname in
configuration files for namenode
(fs.default.name property in core-site.xml)
and jobtracker (mapred.job.tracker in
mapred-site.xml).

For carrying out operations like indexing,
we need to store and replicate document
collection on the datanodes. Namenode
stores metadata of this collection while the
datanode would be storing the actual data.
For this we can use following command to
store data from the local machine to hadoop
file system.
./hadoopfs –put <absolute path to data on
the local storage><location on hadoop file
system>
For e.g.
./hadoopfs –put
/home/hadoop/English/TELEGRAPH-
UTF8/2004_utf8
/home/hadoop/dataToBeIndexed/
If one has started the datanode individually
on slave machines then one can see the
datanode receiving blocks of different sizes
from the master nodes. After completing
transfer of these data, the datanodes and
namenode start verification of them for
each block. The verification process is slow
and time consuming as one can see the
progress going on really slowly on the
screen. Initially for testing pupose, one
should try to put small sized files in the
hadoop file system to check whether
everything is working fine. After that, one
should go for storing larger data files.
The next steps are to configure the hadoop
for running terrier jobs on top of it. The
guide provided here for terrier v3.0 [10]
and [11] are very helpful.
As described in reference [11], the
collection.spec file in the terrier v3.0 needs
to have file locations written in the format:
hdfs://master:9000/home/hadoop/dataTo
BeIndexed
Now if this file is created using the data
available on the local storage, then it didn’t
seem to us to run a command from hadoop
or terrier and collection.spec will contain
file locations in such a format. For that
purpose a trick might work for you: First
setup terrier by running the command
./trec-setup.sh <absolute path to data to be
indexed available on local machine>. This
will create collection.spec file which will
contain file locations of the format:
/
home/hadoop/English/TELEGRAPH_UTF8/
document1. Now open collection.spec file in
some text editor with find and replace
feature and replace the word /home
(whatever part of the file location) with
something like hdfs://master:9000/home
according to your desired file structure in
hadoop file system where you previously
stored data using ./hadoopfs –put
<src><target> command.Apply this replace
operation to the whole collection.spec file
and it will be ready for use by the terrier for
finding the document to be indexed.
One very important thing one should take
care of is that add the line
terrier.plugins=org.terrier.utility.io.Hadoop
Plugin
,before adding the index location path in the
terrier properties. Otherwise trec_terrier.sh
would not be able to recognize the hdfs://
scheme mentioned by the index path
property.
terrier.index.path=hdfs://master:9000/ho
me/hadoop/Indices/TELEGRAPH
5. An overview for the speedup
achieved by the MapReduce
cluster for indexing operation
An experiment similar to the one described
by us was carried out by Richard M. C.
McCreadie, Craig Macdonald and IadhOunis
at University of Glasgow.[2] Below is the
figure showing the speed up they obtained
by trying out different configurations for
the number of clusters and number of
reducers running.
Image source: [2]
Here they have used .GOV2 data corpus
which is of around 25GB size. The results of
this experiment showed that increasing
computing power and the reducers lead to
almost linear increase in the speedup of the
complete indexing operation time. Below is
the graph showing the total time required
for indexing various data collections like
WT2G, WT10G, .GOV, .GOV2 for 1 and 8
machines cluster of HadoopMapReduce
system respectively.
Image source [2]
6. Conclusion
As per the description given above, it can be
concluded that using HadoopMapreduce for
carrying out the data and computation
intensive tasks can be a very wise and
effective decision. We believe that the
references to setup guides and error
correction tricks would turn out to be a
useful resource for a beginner in setting up
a hadoop cluster and run indexing jobs or
using terrier v3.0. The discussion about the
speed up that can be achieved by adding
more machines and reducers to the cluster
can give better idea to the user about the
benefits of deploying hadoop for
performing such intensive tasks.
7. Acknowledgement
We are grateful to our instructor Prof.
PrasenjitMajumder for providing us such an
opportunity of learning HadoopMapreduce
and Terrier platform through our course on
Introduction to Information Retrieval. We
also cordially thank to my Institute DA-IICT
for providing me computational resources
in carrying out these experiment. Your
guide and support have motivated us and
have greatly contributedto the existence of
this paper.
References
1. Jeffrey Dean and Sanjay
GhemawatMapReduce: Simplified
Data Processing on Large Clusters
2. Richard M. C. McCreadie, Craig
Macdonald and IadhOunisOn Single-
Pass Indexing with MapReduce
3. Elias-Gamma compression is used to
store document-identifier deltas
4. Java download and installation
guide:
http://java.com/en/download/help
/linux_install.xml
5. Hadoop download sites:
http://www.apache.org/dyn/closer.
cgi/hadoop/core/
6. Disabling firewall and iptables:
http://www.cyberciti.biz/faq/turn-
on-turn-off-firewall-in-linux/
7. Setting up a hadoop cluster:
http://www.comp.nus.edu.sg/~shil
ei/document/Setting_Up_a_Hadoop_
Cluster.pdf
8. Setting up and running a hadoop
cluster on ubuntulinux operating
system:
http://www.michael-
noll.com/tutorials/running-hadoop-
on-ubuntu-linux-multi-node-
cluster/
9. NamespaceID incompatible in
datanodes workaround:
http://www.hadoop-
blog.com/2010/12/error-is-
starting-datanode-error.html
10. Hadoop and terrier configuration
setup guide:
http://terrier.org/docs/v3.0/hadoo
p_configuration.html
11. Guide for carrying out single pass
indexing operation using terrier 3.0
and hadoop 0.18.3:
http://terrier.org/docs/v3.0/hadoo
p_indexing.html
12. Mapreduce-simplified data
processing on large clusters
http://labs.google.com/papers/map
reduce-osdi04.pdf
13. Apache Hadoop:
http://hadoop.apache.org/