A practical approach: Comparative study of clustering techniques (DATA MINING)

Dinesh Gomber
MCA Deptt.

dinesh2_engg@pdm.ac.in

PDM University, Bahadurgarh

Abstract Prediction involves using some variable to dataset in order to predict

Data mining is used to discover patterns and relationships
in the data in order to help make better business decisions. Data
mining is a powerful technique by which we can discover the very
important information from existing data which was undiscovered
so far and can use that information for taking good decision.
unknown values of other relevant variable like classification,
regression and anomaly detection.
Description involves finding human understandable patterns and
trends in the data example. Clustering, association rule mining and
summarization.

Clustering is a technique of Data mining by which groups

are made on the basis of similarity and we call them cluster.
Images, word documents, patterns all these things can be clustered.
There are many problems which we face when choosing clustering
technique because there are a large number of dataset and many
algorithm exist so the main motive of this paper is to make this work
easy and effective so that anybody can choose the right technique
so that can take the most benefits of data mining .

Introduction:-
Data mining is a series of tools and techniques for uncovering hidden
Data Mining is a process of using tools to extract useful knowledge
patterns and relationships among data (Dunham, 2003).Data mining
from large datasets; data mining is an essential part of knowledge is an emerging discipline that focuses on applying data mining tools
management. and techniques.
Data mining is an essential step in the knowledge discovery in Baker & Yacef, 2009). Researchers within EDM focus on topics
databases (KDD) process that produces useful patterns or models ranging from using data mining to improve institutional effectiveness
from data (Figure 2) [7]. The terms of KDD and data mining are to applying data mining improving student learning processes. There
different. KDD refers to the overall process of discovering useful
is a wide range of topics within educational data mining, so this paper
knowledge from data. Data mining refers to discover new patterns will focus exclusively on ways that data mining is used to improve
from a wealth of data in databases by focusing on the algorithms to
student success and processes directly related to student learning. For
extract useful knowledge. example, student success and recommender systems, and evaluation
Main functions of data mining: of student learning within course are all topics within the broad field
of educational data mining.
Classification is finding models that analyze and classify a data item
into several predefined classes Data mining is a series of tools and techniques for uncovering hidden
patterns and relationships
1. Regression is mapping a data item to a real-valued prediction
Data mining is also one step in an overall knowledge discovery
variable.
process, where organizations want to discover new information from
2. Clustering is identifying a finite set of categories or clusters to the data in order to aid in decision-making processes.
describe the data International Journal of Data Mining & There are a variety of different data mining techniques and
Knowledge Management Process (IJDKP) Vol.2, No.5,
approaches, such as clustering, classification, and association rule
September 2012.
mining. Each of these approaches can be used to quantitatively
3. Dependency Modeling (Association Rule Learning) is finding a analyze large data sets to find hidden meaning a patterns.
model which describes significant dependencies between Related Work
variables.
Finally, [1] high dimensional data is only one issue that needs to be
4. Deviation Detection (Anomaly Detection) is discovering the considered when performing cluster analysis. In closing we mention
most significant changes in the data. some other, only partially resolved, issues in cluster analysis:
scalability to large data sets, independence of the order of input,
5. Summarization is finding a compact description for a subset of
effective means of evaluating the validity of clusters that are
data.
produced, easy interpretability of results, an ability to estimate any
Data mining has two main objective: parameters required by the clustering technique, an ability to
1. Prediction function in an incremental manner, and robustness in the presence of
2. Description different underlying data and cluster characteristics.
A fundamental issue [2] related to clustering is its stability or
consistency. A good clustering principle should result in a data
partitioning that is stable with respect to perturbations in the data.
We need to develop clustering methods that lead to stable solutions.
Comparison the various clustering algorithms of weka tools
K-Means partitioning based clustering algorithm required to
define the number of final cluster (k) beforehand. Such
algorithms are also having problems like susceptibility [3] to local
optima, sensitivity to outliers, memory space and unknown number
of iteration steps that are required to cluster. The time complexity of
the K-Means algorithm is O(ncdi) and the time omplexity of FCM
algorithm is O(ndc 2
i). From the obtained results we may conclude that K-Means
algorithm is better than FCM algorithm. FCM produces close
results to K-Means clustering but it still requires more
computation time than KMeans because of the fuzzy measures
calculations involvement in the algorithm
Automatic and Visualization Approaches
Since clustering is a process of unsupervised learning, setting
appropriate [4] parameters is a problem for lots of algorithms. most
clustering algorithms needs some parameters. Although they may be
straightforward in some cases, they are difficult to set in many
environments. Furthermore, current cluster representation
techniques can be easily understood only when the data is in low-
dimensional space.
Therefore, some algorithms are built for automatic clustering.
Meanwhile, some other efforts has been made to visualize the
process of clustering, so that the user can set the parameters easily
and the result can be more understandable.
Concept of Data Clustering
Clustering can be considered the most important unsupervised
learning technique.
Clustering is “the process of organizing objects into groups whose
members are similar in some way”.
A cluster is therefore a collection of objects which are “similar”
between them and are “dissimilar” to the objects belonging to other
clusters.
.The goal of data clustering, also known as cluster analysis, is to
discover the natural grouping(s) of a set of patterns, points, or
objects.
Find groups based on a measure of similarity such that the
similarities between objects in the same group are high while the
similarities between objects in different groups are low. But, what is
the notion of similarity? What is the definition of a cluster? clusters
can differ in terms of their shape, size, and density. The presence of
noise in the data makes the detection of the clusters even more5.
difficult. An ideal cluster can be defined as a set of points that is
compact and isolated.
Clustering algorithms can be broadly divided into two groups:
hierarchical and partitioned.
Hierarchical clustering algorithms recursively find nested clusters
either in agglomerative mode (starting with each data point in its own
cluster and merging the most similar pair of clusters successively to
form a cluster hierarchy) or in divisive (top-down) mode (starting
with all the data points in one cluster and recursively dividing each
cluster into smaller clusters). that typically involves other steps and
techniques. finding clusters in data where there are clusters of
different shapes, sizes, and density or where the data has lots of noise
and outliers. These issues become more important in the context of
high
dimensionality data sets.
For high dimensional data, traditional clustering techniques have
sometimes been used. For
example, the K-means algorithm and agglomerative hierarchical
clustering techniques [DJ88],
have been used extensively for clustering document data. While K-
means is efficient and often produces “reasonable” results, in high
dimensions, K-means still retains all of its low
dimensional limitations, i.e., it has difficulty with outliers and does
not do a good job when the clusters in the data are of different sizes,
shapes, and densities.
Clustering Techniques-
K-means:
1. It accepts the number of clusters to group data into, and the
dataset to cluster as input values.
2. It then creates the first K initial clusters (K= number of
clusters needed) from the dataset by choosing K rows of data
randomly from the dataset. For Example, if there are 10,000
rows of data in the dataset and 3 clusters need to be formed,
then the first K=3 initial clusters will be created by selecting 3
records randomly from the dataset as the initial clusters. Each
of the 3 initial clusters formed will have just one row of data.
3. The K-Means algorithm calculates the Arithmetic Mean of
each cluster formed in the dataset. The Arithmetic Mean of
a cluster is the mean of all the individual records in the
cluster. In each of the first K initial clusters, their is only one
record. The Arithmetic Mean of a cluster with one record is
the set of values that make up that record. For Example if the
dataset we are discussing is a set of Height, Weight and Age
measurements for students in a University, where a record P
in the dataset S is represented by a Height, Weight and Age
measurement, then P = {Age, Height, Weight). Then a record
containing the measurements of a student John, would be
represented as John = {20, 170, 80} where John's Age = 20
years, Height = 1.70 metres and Weight = 80 Pounds. Since
there is only one record in each initial cluster then the
Arithmetic Mean of a cluster with only the record for John as
a member = {20, 170, 80}.
4. Next, K-Means assigns each record in the dataset to only one
of the initial clusters. Each record is assigned to the nearest
cluster (the cluster which it is most similar to) using a
measure of distance or similarity like the Euclidean Distance
Measure or Manhattan/City-Block Distance Measure.
K-Means re-assigns each record in the dataset to the most
similar cluster and re-calculates the arithmetic mean of all the
clusters in the dataset. The arithmetic mean of a cluster is the
arithmetic mean of all the records in that cluster. For Example,
if a cluster contains two records where the record of the set of
measurements for John = {20, 170, 80} and Henry = {30, 160,
120}, then the arithmetic mean Pmean is represented as Pmean=
{Agemean, Heightmean, Weightmean). Agemean= (20 + 30)/2,
Heightmean= (170 + 160)/2 and Weightmean= (80 + 120)/2. The
arithmetic mean of this cluster = {25, 165, 100}. This new
arithmetic mean becomes the center of this new cluster. Following
the same procedure, new cluster centers are formed for all the
existing clusters.

6. K-Means re-assigns each record in the dataset to only one

of the new clusters formed. A record or data point is
assigned to the nearest cluster (the cluster which it is most
similar to) using a measure of distance or similarity

7. The preceding steps are repeated until stable clusters are

formed and the K-Means clustering procedure is
completed. Stable clusters are formed when new iterations
or repetitions of the K-Means clustering algorithm does
not create new clusters as the cluster center or Arithmetic
Mean of each cluster formed is the same as the old cluster
center. There are different techniques for determining
when a stable cluster is formed or when the k-means
clustering algorithm procedure is completed.
Hierarchical Clustering: Weaknesses
 The most commonly used type, single-link clustering, is
particularly greedy.
 If two points from disjoint clusters happen to be near
each other, the distinction between the clusters will
be lost.

 On the other hand, average- and complete-link

The Algorithm-Given a set of numeric points in d dimensional
space, and integer k.
Algorithm generates k (or fewer) clusters as follows:-
1. Assign all points to a cluster at random
2. Repeat until stable:
3. Compute centroid for each cluster
4. Reassign each point to nearest centroid
Step 1: Make random assignments and compute
centroids.
Density-Based Clustering Methods:
Step 2: Assign points to nearest centroids
Step 3: Re-compute centroids (in this example, solution is
now stable)
K-means: Weaknesses:-
 Must choose parameter k in advance, or try many values.
 Data must be numerical and must be compared via
Euclidean distance (there is a variant called the k-medians
algorithm to address these concerns)
 The algorithm works best on data which contains spherical
clusters; clusters with other geometry may not be found.
 The algorithm is sensitive to outliers---points which do not
belong in any cluster. These can distort the centroid
positions and ruin the clustering.
Hierarchical Clustering: The Algorithm:-
 Hierarchical clustering takes as input a set of points
 It creates a tree in which the points are leaves and the
internal nodes reveal the similarity structure of the points.
clustering methods are biased towards spherical
clusters in the same way as k-means
 Does not really produce clusters; the user must decide
where to split the tree into groups.
 Some automated tools exist for this
 As with k-means, sensitive to noise and outliers
 Clustering based on density (local cluster criterion), such
as density-connected points
 Major features:
o Discover clusters of arbitrary shape
o Handle noise
o One scan
o Need density parameters as termination condition
 Two parameters:
o Eps: Maximum radius of the neighborhood
o MinPts: Minimum number of points in an Eps-
neighborhood of that point
 Directly density-reachable: A point p is directly density-
reachable from a point q w.r.t. Eps, MinPts if
 Core Object: object with at least MinPts objects within a
radius ‘Eps-neighborhood’
 Border Object: object that on the border of a cluster

 The tree is often called a “dendogram.”

 The method is summarized below:

 Place all points into their own clusters.

 While there is more than one cluster, do
 Merge the closest pair of clusters

o p belongs to NEps(q)

o core point condition:

 | NEps (q)| >= MinPts
 Relies on a density-based notion of cluster: A cluster is
defined as a maximal set of density-connected points
 Discovers clusters of arbitrary shape in spatial databases
with noise
DBscan algorithm
 Arbitrary select a point p
 Retrieve all points density-reachable from p w.r.t. Eps
and MinPts.
 If p is a core point, a cluster is formed.
 If p is a border point, no points are density-reachable
from p and DBSCAN visits the next point of the
database.
 Continue the process until all of the points have been
processed.
 Density-reachable:
 A point p is density-reachable from a point q wrt.
Eps, MinPts if there is a chain of points p1, …, pn,
p1 = q, pn = p such that pi+1 is directly density-
reachable from pi
 Density-connected
 A point p is density-connected to a point q wrt.
Eps, MinPts if there is a point o such that both,
p and q are density-reachable from o wrt. Eps
and MinPts.
Challenges in Clustering
 Similarity Calculation
 Results of algorithms depend entirely on similarity used
 Clustering systems provide little guidance on how to pick
similarity
 Computing similarity of mixed-type data is hard
 Similarity is very dependent on data
representation. Should one
 Normalize?
 Represent one’s data numerically,
categorically, etc.?
 Cluster on only a subset of the data?
 The computer should do more to help
the user figure this out!
 Parameter selection
 Current algorithms require too many arbitrary,
user-specified parameters
Conclusion-
 Clustering is a useful way of exploring data, but is still
very ad hoc
 Good results are often dependent on choosing the right
data representation and similarity metric
 Data: categorical, numerical, boolean
 Similarity: distance, correlation, etc.
 Many different choices of algorithms, each with
different strengths and weaknesses
k-means, hierarchical, graph partitioning, etc
References:-
1:- The Challenges of Clustering High Dimensional Data
Michael Steinbach, Levent Ertöz, and Vipin Kumar
2. Data clustering: 50 years beyond K-means
Anil K. Jain*
Department of Computer Science and Engineering, Michigan State University,
East Lansing, Michigan 48824, USA
Department of Brain and Cognitive Engineering, Korea University, Anam-dong,
Seoul, 136-713, Korea.
3. Comparative Analysis of K-Means and Fuzzy CMeans Algorithms
Soumi Ghosh Sanjay Kumar Dubey
Department of Computer Science and Engineering,
Amity University, Uttar Pradesh
Noida, India.
4. Analyzing Popular Clustering Algorithms from Different Viewpoints
QIAN Wei-ning, ZHOU Ao-ying
(Department of Computer Science, Fudan University, Shanghai 200433, China)
(Laboratory for Intelligent Information Processing, Fudan University, Shanghai
200433, China).