2017 International Conference on Intelligent Computing,Instrumentation and Control Technologies (ICICICT)

An Evaluation of MapReduce Framework in Cluster

Analysis
Tanvir Habib Sardar1, Ahmed Rimaz Faizabadi2, Zahid Ansari3
1,2,3,
Dept. of CSE, P.A.College of Engineering
1
tanvir_is@pace.edu.in
2
rimaz_is@pace.edu.in
3
zahid_cs@ pace.edu.in

Abstract— Data is growing exponentially due to the World
Wide Web and scientific enhancements, requires proper
strategies and techniques to deal with it. Thus, it requires
increasing the computational requirements. Efficient
processing paradigms and smart implementation
architecture is the key to meet the scalability and
performance requirements in large scale data analysis.
Clustering is one of the popular data mining techniques to
analyse datasets. MapReduce programming paradigm, on
the top of Hadoop distributed architecture, is widely used
in today’s world in order to obtain efficiency for large
dataset clustering. In this work, we have experimented and
validated the efficiency of K-means clustering algorithm in
MapReduce paradigm over hadoop architecture while
processing different sized datasets in combination of
different hadoop cluster sizes.
Keywords— Clustering, Data Mining, MapReduce, Hadoop,
Distributed Computing.
I. INTRODUCTION
Data is growing exponentially due to the digital
enhancements of the modern world, requires proper
strategies and techniques to deal with it [1]. Data
Mining is known as the process of analysing data to
extract interesting patterns and knowledge. Data
mining is used for analysis purpose to analyse
different type of data [2].
Clustering is the process by which a set of data
objects is partitioned into subsets such that the data
elements of a same cluster are similar to one
another and different from the elements of other
clusters [3]. Clustering is a classification of similar
objects into several different groups, it is usually
applied in the analysis of statistical data which can
be utilized in various fields, for example, machine
learning, data mining, pattern recognition, image
analysis and bioinformatics [4].
The method of K-means algorithm as follows [5]:
1) Determine the number of clusters k, from the
dataset. It is done with theoretical and conceptual
considerations that may be proposed to determine
how many clusters to be formed.
2) Generate K centroid (the centre point of the
cluster) beginning at random. Determination of
initial centroid is done at random from objects
provided, then to calculate the i cluster centroid
next, use the following formula:
∑
v= ,
Where v: cluster centroid, xi: the object, n: the
number of objects to be members of the cluster
3) Calculate the difference of data objects in
terms of distance of each object to each centroid.
To calculate the distance between the object with
the centroid K-means use Euclidian distance
measure.
4) Each object is allocated into the nearest
centroid. To perform the allocation of objects into
each cluster during the iteration can generally be
done explicitly where every object is declared as a
member of the cluster by measuring the distance of
the proximity of objects towards the center point of
the cluster.
5) Do iteration, and then specify a new centroid
position using equation of step 2.
6) Repeat step 3 if the new centroid position is
not the same.
With the rapid growth of data volume and
the development of information technology,
traditional clustering algorithms cannot meet the
demand [6]. Distributed Computing is a method
aimed at solving computational problems mainly by
sharing the computation over a network of
interconnected devices. Each individual system
connected on the network is called a node and the

978-1-5090-6106-8/17/$31.00 ©2017 IEEE 110

 2017 International Conference on Intelligent Computing,Instrumentation and Control Technologies (ICICICT)
collection of many nodes that form a network is
called a cluster. Apache Hadoop is open source
architecture. Hadoop is used for processing large
datasets which is developed in java. It is a
framework that helps for distributed processing of
large datasets across network of computers using
simple programming modules. MapReduce
programing paradigm is used by Hadoop
architecture to process the large datasets across a
number of connected nodes of computing systems
[7].
MapReduce programming model works as
follows [8]:
The system takes a set of input key/value pairs,
and outputs a set of output key/value pairs.
MapReduce executes the entire task as two
functions: Map and Reduce.
x Map-function takes an input pair and outputs
a set of intermediate key/value pairs. The
MapReduce collects together all intermediate
values associated with the same intermediate
key i and converts them to the Reduce
function.
x Reduce-function receives an intermediate key
i and a set of values for that key. It combines
together these values to form a possibly
reduced set of values. Generally, just zero or
one output value is formed per Reduce
invocation.
A Basic MapReduce Data Processing Scheme is
depicted in figure 1.
Split 1
reduce
Input
Split 2
dataset Output
d nodes in the Hadoop cluster.
reduce
Split n
Fig. 1: A Basic MapReduce Data Processing Scheme
Hadoop Distributed File System (HDFS) is
a distributed file system managed by hadoop. It
provides the access to large amounts of data on
Hadoop cluster [9]. HDFS is intended to run on
commodity hardware. It has many resemblances
with distributed file systems in use today. But hdfs
978-1-5090-6106-8/17/$31.00 ©2017 IEEE
has few benefits over traditional distributed file
systems such as [10]:
x Hdfs is extremely fault-tolerant.
x Can be organized on low cost hardware.
x High throughput access to data.
x Can pact with large datasets.
HDFS runs on master/slave architecture. It
consists of certain number of datanodes (slaves) and
a namenode (master). First a dataset for processing
is uploaded into namenode, then namenode divides
the dataset into blocks of data and sends to
individual datanode for processing and finally
accumulate the result obtained from different
slaves.
In this work, we proposed a modification of
traditional K-means algorithm into MapReduce
paradigm in order to execute it over hadoop
platform. Then we have evaluated the result
obtained from MapReduced K-means with different
cluster and dataset size.
II. METHODOLOGY
In our proposed methodology the traditional
k-means algorithm is modified with MapReduce
paradigm so that executing MapReduced
modification of K-means can be executed over
hadoop cluster. The objective behind this work is to
find out the efficiency gain (in terms of execution
time) of MapReduced K-means over different sized
hadoop cluster. Our modified k-means is required
to be executed over cluster of node.
Commodity machines from multiple
machines which are sharing a private LAN
managed by a switch, act as a Master node which
supervises data and flow control over all other
The experiments are conducted on
datasets consisting of 2-D data points. This dataset
is not pre-processed from any datasets and it is not
required to be pre-processed as well. Hence, the
dataset can be directly put over the master node and
hdfs would take care of it in splitting in sizes,
sending it to different nodes and accumulating
results. The master node in a hadoop cluster firstly
splits the dataset into different data chunks then it
sends each chunk to a slave node. The tracking of
data chunks is the sole responsibility of hdfs; a
MapReduce programmer is unaware of it. The
111
 2017 International Conference on Intelligent Computing,Instrumentation and Control Technologies (ICICICT)
MapReduced K-means will start mapper operation
to each slave node when a chunk of data is
available there.
The initial step of designing MapReduce
routines Kmeans algorithm is to define and analyse
the input and output of the implementation. Input is
given as pair, where “key” is the cluster centroid
and “value” is the data points in the dataset. The
input centroids (k) to the k-means are provided in
the code itself in our design. The input is, as usual,
k number of 2-D points selected from the dataset.
For each iterations, the cluster centres of reduce
operations the centroid value will be changed and
recalculated to a new value. It can be achieved by
following the algorithm to design MapReduce
routines for K-means clustering.
Each data chunk created by master node’s
hdfs (to send in slaves HDFS) is the part of data
points provided in the dataset. Each Map operation
of K-means here calculates the centroid based on
points it received in the data chunk. So, each slave
has now created their own centroid of k-means for a
particular part of data points it has received. Once
Mapper is invoked the given vector is assigned to
the cluster that is closely related to. After the
assignment, the centroid of that particular cluster is
recalculated.
The recalculation is done by the Reduce
routine and also it rearranges the cluster to avoid
creations of clusters with extreme sizes i.e. cluster
having too less data vectors or a cluster having too
many data vectors. Finally, once the centroid of the
given cluster is updated, the new set of vectors and
clusters are re-written to the disk and is ready for
the next iteration.
Proposed modified K-means: Input dataset is split
into different data chunks at master node and then
sent to different slaves. Map function at each node
adjusts the cluster centres based on input data
points in the split it has received. The reducer
gathers the data value information of each cluster
and re-computes the k cluster centers.
Algorithm:
Step1: Initially centroids are randomly selected
based on data. In our implementation we used 3
centroid values randomly chosen.
Step2: The Input file contains initial centroid and
data.
978-1-5090-6106-8/17/$31.00 ©2017 IEEE
Step3: In Mapper class, "configure" function is used
to open the file and read the centroids and store in
the data structure (used ArrayList for this)
Step4: Mapper read the data file and emits the
nearest centroid with the point to the reducer.
Step5: Reducer collects all this data and calculates
the new corresponding centroids and release.
Step6: In the job configuration, we read both files
and checked if there is a change between old and
new centroid and it is less than 0.1 then
convergence is reached else repeat step 2 with
new centroids.
Figure 2 is the depiction of the proposed algorithm.
Input
Dataset
Split 1 Split 2 Split n
Map 1 Map 2 Map n
reduce 1 reduce n
Clustered
data
Fig. 2: The Execution of Proposed Algorithm
III. RESULT AND DISCUSSION
The experiments are conducted on datasets
consisting of 2-D data points. This dataset is not
pre-processed from any datasets and it is not
required to be pre-processed as well.
The experimentation of the dataset of
different sizes is conducted with the combinations
of different cluster size to analyse the efficiency of
distributed processing provided by hadoop. For
example, a dataset with 100,000 2-D points is
experimented with 3 node, 5 node, 8 node and 10
112
 2017 International Conference on Intelligent Computing,Instrumentation and Control Technologies (ICICICT)

node cluster. Similarly, experiments with datasets minutes and 10 seconds, 7 minutes and 05 seconds,
consist of 150,000; 200,000 and 300,000 2-D points 6 minutes and 15 seconds, and 5 minutes and 10
data are conducted and execution time is recorded seconds respectively.
in seconds. A brief description of different In the fourth observation, we have fixed the
execution time with respect to different dataset is dataset to 300,000 2-D points and executed k-
provided in the following table. Our experiments to means algorithm modified using MapReduce
evaluate hadoop cluster is based on observation of paradigm against different cluster sizes.
cluster scale-up: keeping the data size constant and Experiments of 3-Node, 5-node, 8-node till 10-
increasing the cluster size. node clusters shows a reduction in execution time
Data 100,000 150,000 200,000 300,000 to 10 minutes and 30 seconds, 9 minutes and 10
Size points points points points seconds, 8 minutes and 10 seconds, and 6 minutes
3 node 270 380 490 630 and 45 seconds respectively.
5 node 210 290 370 480 We have scrutinized the result obtained
8 node 115 200 260 380
10 node 65 110 130 190
from all four experiments in order to analyse and
Table 1. Execution Time in Seconds conclude hadoop performance. We have taken
50,000 data points as a unit of performance
To obtain the performance gain by our evaluation to critically analyse the performance
modified K-means, we have conducted few obtained from different hadoop cluster. In order to
experiments which can be classified into 4 major make 50,000 data points as a unit, for example, the
observations. execution time of 100,000 points containing dataset
In the first observation, we have fixed the is divided by 2 (for example 270/2=135, for 3-node
dataset to 100,000 2-D points and executed k- cluster computing) , 150,000 points containing
means algorithm modified using MapReduce dataset is divided by 3 and so on. Therefore, for
paradigm against different cluster sizes. Then we each 50,000 data point, the time taken to execute
have executed proposed algorithm starting with the parallel K-means algorithm by hadoop clusters
node 3 then 5-node and 8-node cluster till 10- node is taken into account and ratio of their respective
cluster and experienced a reduction in execution performance is calculated.
time to 270, 240, 200 and 150 seconds respectively. Ratio is a quantitative relation between two
We have observed from the result that as the cluster amounts showing the number of times one value
size increases the execution time of the program contains or is contained within the other. The ratio
decreases. is calculated based on keeping the 10-node
In the second observation, we have fixed the execution time equals to 1 for each datasets. Thus,
dataset to 150,000 2-D points and executed k- it is very useful for determining the difference of
means algorithm modified using MapReduce performance between theorical (ideal) and practical
paradigm against different cluster sizes. (experimental) hadoop performance. The details of
Experiments of 3-Node, 5-node, 8-node till 10- our analysis are provided in the table 2. It can be
node clusters shows a reduction in execution time clearly shown that the reduction of execution time
to 6 minutes and 20 seconds, 5 minutes and 30 as per increase of cluster size is not very smooth i.e.
seconds, 5 minutes and 55 seconds, and 4 minutes it cannot be depicted using a line graph with a
respectively. We have observed from the result that straight line. This is due to the fact that there are
as the cluster size increases the execution time of many overhead nodes of a cluster contains such as
the program decreases. processing overhead of systems processes and basic
In the third observation, we have fixed the tools (like background antivirus tools execution)
dataset to 200,000 2-D points and executed k- execution with different priority, networking (data
means algorithm modified using MapReduce transfer) overhead between nodes, data overhead
paradigm against different cluster sizes. (Namenode have trivial amount of data stored in
Experiments of 3-Node, 5-node, 8-node till 10-node journal/fsimage, replication factor of hadoop, hdfs
clusters shows a reduction in execution time to 8 checksum) etc.

978-1-5090-6106-8/17/$31.00 ©2017 IEEE 113

 2017 International Conference on Intelligent Computing,Instrumentation and Control Technologies (ICICICT)

processing is not uniform in terms of speed up of

Ratio 3-Node 5-Node 8-Node 10-Node clustering, during the experiment and recorded the
Calculation various performance changes and shown in
100,000 pts 135 105 57.5 32.5
performance graph chart.
Ratio 4.15 3.23 1.76 1
150,000 pts 126.66 96.66 66.66 36.66
Ratio 3.45 2.71 1.81 1 REFERENCES
200,000 pts 122.5 92.5 65 32.5
Ratio 3.76 2.84 2 1
[1] Lovele Sharma, Vivek Srivastava, “Performance
300,000 pts 105 80 63.66 31.66
Enhancement of Information Retrieval via Artificial
Ratio 3.3 2.52 2 1
Intelligence”, International Journal of Scientific
Table 2: For Each Datasets of 50,000 points, Execution Time
Research in Science, Engineering and Technology,
(in sec) and Respective Ratio
Volume 3 | Issue 1 | January-February – 2017.
In table 2, ratio specifies the performance gain by [2] Bansal, Sharma and Goel, “Improved K-mean Clustering
larger hadoop clusters. For example, it shows that Algorithm for Prediction Analysis using Classification
for dataset consist of 100,000 datapoints, 10-node Technique in Data Mining”, International Journal of
cluster is 4.15 time faster than 3-node cluster. Computer Applications (0975 – 8887) Volume 157 – No
6, January 2017.
700 [3] Satish Chaurasiya and Ratish Agrawal, “K-Means
Clustering With Initial Centroids Based On Difference
600
Operator”, International Journal of Innovative Research
500 100,000 pts in Computer and Communication Engineering, Vol. 5,
400
150,000 pts Issue 1, January 2017.
300 [4] A. Chauhan, G. Mishra, and G. Kumar, “Survey on Data
200,000 pts
200 Mining Techniques in Intrusion Detection,” vol. 2, no. 7,
300,000 pts pp. 2–5, 2011.
100
[5] Zulfadhilah, Prayudi and Riadi, “Cyber Profiling using
0 Log Analysis and K-Means Clustering”, International
3-node 5-node 8-node 10-node
Journal of Advanced Computer Science and
Applications, Vol. 7, No. 7, 2016.
Fig. 3: The Tradeoff between cluster size and datapoints in [6] Ezhilan, Vignesh Kumar, Arvind Kumar, Afzal Khan,
MapReduced KMeans “Implementation of Optimised K-Means Clustering on
The overall combined scenario of the Hadoop Platform”, International Journal of Advanced
execution of proposed Kmeans algorithm in respect Research in Computer Science and Software
to different hadoop cluster size and data point are Engineering, Volume 6, Issue 2, February 2016.
[7] Chaturbhuj and Chaudhary, “Improved K-means
provided into the bar chart of fig. 3. clustering on Hadoop”, International Journal on Recent
IV. CONCLUSIONS and Innovation Trends in Computing and
Communication, Volume: 4 Issue: 4, pages 601 – 604,
With the huge volume of data generated April 2016.
[8] Shweta Mishra, Vivek Badhe, “Improved Map Reduce K
from the World Wide Web and scientific
Mean Clustering Algorithm for Hadoop Architecture”,
experiments, hadoop clusters are being formed and International Journal Of Engineering And Computer
widely used in order to cope with processing huge Science, Volume 5 Issues 7 July 2016, Page No. 17144-
volume of data. MapReduce programming model is 17147.
designed to give programmers develop code for [9] Raghu Garg and Himanshu Aggarwal, “Big Data
Analytics Recommendation Solutions for Crop Disease
hadoop framework. Four different datasets of
using Hive and Hadoop Platform”, Indian Journal of
different sizes are used and experimented with four Science and Technology, 9(9), 2016.
different sized hadoop clusters. From our [10] Cheng et al., “Research On Hdfs-based Web Server
experiments, it has been observed that as the Cluster”, 2011 International Conference on E-Business
number of nodes increases the execution time and E-Government (ICEE), 2011.
decreases. We also found some of the interesting
cases, which concludes to the fact that hadoop

978-1-5090-6106-8/17/$31.00 ©2017 IEEE 114