2009 Fourth International Conference on Innovative Computing, Information and Control
A Hierarchical Clustering Algorithm based on K-means with Constraints*
GuoYan Hang, DongMei Zhang, JiaDong Ren
College of Information Science and Engineering,
Yanshan University,
Qinhuangdao, China
hgy@ysu.edu.cn
Abstract—Hierarchical clustering is one of the most important
tasks in data mining. However, the existing hierarchical
clustering algorithms are time-consuming, and have low
clustering quality because of ignoring the constraints. In this
paper, a Hierarchical Clustering Algorithm based on K-means
with Constraints (HCAKC) is proposed. In HCAKC, in order
to improve the clustering efficiency, Improved Silhouette is
defined to determine the optimal number of clusters. In
addition, to improve the hierarchical clustering quality, the
existing pairwise must-link and cannot-link constraints are
adopted to update the cohesion matrix between clusters.
Penalty factor is introduced to modify the similarity metric to
address the constraint violation. The experimental results
show that HCAKC has lower computational complexity and
better clustering quality compared with the existing algorithm
CSM.
Keywords- Hierarchical clustering; Improved Silhouette; K-
means; constraints
I. INTRODUCTION
Clustering is an important analysis tool in many fields,
such as pattern recognition, image classification, biological
sciences, marketing, city-planning, document retrievals, etc.
Hierarchical clustering is one of the most widely used
clustering methods.
At present, several existing clustering algorithms focus
on combination of the advantages of hierarchical and
partitioning clustering algorithms [1-2]. K-means, which is
one of the representative partitioning methods, obtains the
number of clusters through minimizing the objective
function. K-means has higher efficiency compared with the
hierarchical methods. However, the number of clusters K
needs to be fixed iteratively. Thus, K-means is often
required to be run many times and is computationally
expensive. How to determine the number of clusters
becomes an increasingly important problem. The common
trail-and-error method [3] generally depends on certain
clustering algorithms and is inefficient when the dataset is
large.
*
This work is supported by the National High Technology Research and
Development Program ("863"Program) of China (No. 2009AA01Z433)
and the Natural Science Foundation of Hebei Province P.R. China
(No.F2008000888).
978-0-7695-3873-0/09 $29.00 © 2009 IEEE
Authorized licensed use limited to: GOVERNMENT COLLEGE OF TECHNOLOGY. Downloaded on July 08,2010 at 20:10:42 UTC from IEEE Xplore. Restrictions apply.
JiaDong Ren, ChangZhen Hu
School of Computer Science and Technology,
Beijing Institute of Technology,
Beijing, China
jdren@ysu.edu.cn
Besides, the existing algorithms by combining
hierarchical clustering and K-means [4] ignore the existing
constraints [5]. Clustering is traditionally considered as an
unsupervised method for data analysis. However, in some
cases the background knowledge is known in addition to the
data instances. Typically, the background knowledge is in
the form of pairwise constraints (must-link and cannot-link).
Clustering quality can be improved through utilizing the
existing constraints. Carlos Ruiz enhanced the density-based
algorithm DBSCAN with constraints upon data points to get
the new algorithm C-DBSCAN [6]. C-DBSCAN has
superior performance to DBSCAN even with a small
number of constraints. However, the efficiency of C-
DBSCAN is not good. How the K-means clustering
algorithm can be profitably modified to make use of the
constraints is demonstrated in cop-kmeans [7]. Although the
clustering accuracy of cop-kmeans is improved, the
constraint violation problem has not been well addressed. I.
Davidson [8] incorporated the pairwise constraints into the
agglomerative hierarchical clustering to improve the
clustering quality. However, the problem of constraint
violation is still not solved.
In this paper, we propose HCAKC, a new method for
hierarchical clustering based on K-means with existing
pairwise constraints. In HCAKC, Improved Silhouette,
CUCMC (Constraints-based Update of Cohesion Matrix
between Clusters) are defined. The optimal number of
clusters is determined through computing the average
Improved Silhouette of the dataset such that the time
complexity can be reduced. The initial clusters of HCAKC
are obtained through running K-means. In order to improve
the clustering quality of the hierarchical clustering, the
existing pairwise must-link and cannot-link constraints are
incorporated into the agglomerative hierarchical clustering.
CUCMC is done based on the existing constraints. The
penalty factor [9] is introduced into cohesion [2] similarity
metric to address the constraint violation.
This paper is organized as follows: In section II, we
give the basic concepts and definitions. In section III, we
present our hierarchical clustering algorithm called
HCAKC. Section IV shows the experimental results of the
clustering algorithm. Finally we conclude the paper in
section V.
1479
 II. BASIC CONCEPTS AND DEFINITIONS
A silhouette [4] is a function that measures the
similarity of an object with objects of its own cluster
compared with the objects of other clusters.
For a cluster C consisting of data points p1, p2… pn, the
radius r of C is defined as formula(1), where c is centroid of
C and d(pi, c) is the Euclidean distance between pi and c.
1
⎛1 n 2 ⎞2
r = ⎜ ∑ d ( pi − c ) ⎟
⎝ n i =1 ⎠ Chs(C1, C3)=0.2, Chs(C2, C3)=0.1 respectively without
join( pi , c j ) = exp(− d ( pi , ci ) − d ( pi , c j ) / ri )
Where join(pi, cj) is the intention of pi to be joined into
Cj. The cohesion [2] of Ci and Cj is calculated as formula(3).
∑ join( p, C ) + ∑ join( p, C )
j i
Chs ( Ci , Cj ) =
p∈Ci p∈Cj
Ci + Cj
Definition 1 IS (Improved Silhouette)
Let S be a dataset consisting of clusters C1, C2…Ct. The
distance between each object oi(oi ∈ Cj, j ∈ [1,t]) and the
centroid of its own cluster is denoted as ai. bi is the
minimum distance between oi and each centroid of the
other t-1 clusters. The IS(oi) is defined as formula(4).
any two clusters. Let M= {(Ci, Cj)} be the set of must-link
constraints, indicating that cluster Ci and Cj should be in the
same class, C= {(Ci, Cj)} be the set of cannot-link
constraints, indicating that Ci and Cj should be in the
different classes. For clusters Cp, Cq, Cr(p, q, r∈ [1,n]), in
order to satisfy M, Chs(Cp, Cq) in X is updated to 1, and
Chs(Cp(Cq), Cr)is updated to max(Chs(Cp, Cr), Chs(Cq, Cr)).
And Chs(Cp, Cq) in X is updated to 0 in order to satisfy C.
In Fig. 2, we give an example to show the process of
CUCMC, where must-link constraint M (C1, C2) is known.
(1) From Fig. 2, we can obtain that Chs(C1, C2)=0.4,
considering the existing constraint. Since C1, C2 need to
satisfy M (C1, C2), Chs(C1, C2) in X, the cohesion matrix, is
(2) updated to 1, and the Chs(C1, C3) is updated to 0.2.
The CUCMC of the example is as follows:
⎡1 0.4 0.2⎤ ⎡1 1 0.2⎤
X= ⎢⎢ ⎥
1 0.1⎥ → X= ⎢⎢ 1 0.2⎥⎥
⎢⎣ 1 ⎥⎦ ⎢⎣ 1 ⎥⎦
In general, the existing constraints are in the form of
(3) m={(xi, xj)}, c={(xi, xj)}, where m indicates that point xi
and xj should be in the same cluster, and c indicates that
point xi and xj should be in the different clusters. The M=
{(Ci, Cj)} and C= {(Ci, Cj)} can be obtained through
utilizing the propagation of constraints.
Penalty factor w, w are introduced in order to address
the constraints violation.
Sim(Ci , C j ) = Chs (Ci , C j ) − w≠ M ( Ci ,C j ) − w≠ C (Ci ,C j ) (5)

IS ( oi ) = (bi − ai ) / max(ai , bi ) (4)

w≠ M ( Ci ,C j ) works as must-link (Ci, Cj) is violated,
In formula (4), the meanings of ai and bi have changed and w≠C (Ci ,C j ) works as Cannot-link (Ci, Cj) is violated.
compared with the traditional silhouette computation. Both y
ai and bi denote the distance to the cluster centroid. 8
The average IS of dataset corresponding to different
6 C2
partition is calculated. The maximal IS of the dataset
corresponds to the optimal partition of the dataset. 4
We take point A in Fig. 1 as an example to show the IS 2
C1

computation of a data point. C3

(1)Obtain the centroids of clusters C1, C2, C3 0 A(1,0) 2 4 6 8 x

respectively: Centroid1=(1.4, 1.2), Centroid2=(4.4, 4.8),
Centroid3=(6.5, 0.8333); Figure 1. The IS computation of data point
(2)Calculate aA, the distance between A and the centroid
0.4
of its own cluster: aA = (1−1.4)2 + (0 −1.2)2 =1.2649. The C1 C2 C1 C2
distances between A and each centroid of C2 and C3 can be
0.2
obtained similarly, and they are 5.8822 and 5.5628 0.2 0.1
respectively. Since bA denotes the minimum distance
according to the definition of IS, thus let bA=5.8822. C3
C3
(3)The IS of A can be obtained based on formula (4).
IS(A)=(bA-aA)/max(aA, bA)=0.7850. Figure 2. The update of cohesion matrix of cluster with constraints
Definition 2 CUCMC (Constraint-based Update of
Cohesion Matrix between Clusters)
Suppose that {Ck }k =1 is the set of given clusters and
n

X= [Chs (Cs , Ct ) ]n×n is the existing cohesion matrix between

1480

Authorized licensed use limited to: GOVERNMENT COLLEGE OF TECHNOLOGY. Downloaded on July 08,2010 at 20:10:42 UTC from IEEE Xplore. Restrictions apply.
 III. HIERARCHICAL CLUSTERING ALGORITHM BASED ON
K-MEANS WITH CONSTRAINTS
The CSM [2] needs to specify K. Different K leads to
different clustering results. Thus, how to determine the
appropriate K becomes especially important. Besides, the
existing constraints are not considered in CSM, so the
accuracy of the clustering results will not be high.
In HCAKC, we plot the curve about the average IS of
dataset to be clustered and the number of partitions. The
optimal number of clusters is determined by the maximum
of the curve, since the average IS of a dataset not only
reflects the density of clusters, but also the dissimilarity
between clusters. The cohesion matrix X is constructed
according to the cohesion between any two clusters. The
existing pairwise constraints are incorporated into the
hierarchical clustering. CUCMC is implemented based on
the existing constraints. Thus, the clustering results are
greatly optimized. In our algorithms, S is the dataset to be
clustered; K is the optimal number of clusters; n is the size
of S; m is the number of sub-clusters; M={(Ci, Cj)}, (i,
j ∈ [1,t]) is the set of existing must-link constraints; C={(Ci,
Cj)}, (i, j ∈ [1,t]) is the set of existing cannot-link
constraints.
Algorithm Find-K
Input: S
Output: K
begin
1: partition S into t clusters: C1, C2…Ct, according to the
geometry distribution of S
2: repeat{
3: for (i=1; i<=t; i++)
4: {for (each object x in Ci)
5: {calculate ISi(x), the improved silhouette of x;
6: calculate ISi , the average Improved Silhouette of S,
1 t
and ISi = ∑ ∑ ISi ( x) ;
n i =1 x∈ci
7: plot the curve about t and ISi in the 2-dimensional
coordinate system; }
8: t:=t+1; }
9: } until (the curve reaches the maximum)
10: K: =t;
end
In algorithm Find-K, dataset S is firstly partitioned into
t clusters: C1, C2…Ct, according to the geometry
distribution of S. IS is introduced into the algorithm Find-
K. The closer the improved silhouette of a cluster to 1, the
more similar the objects of the same cluster will be. The
curve, which is about cluster number t and the average IS,
is plotted. The number of clusters corresponding to the
maximum of the curve is the optimal number of clusters.
Algorithm HCAKC
Input: S, n, m, M, C
Output: The K clusters
begin
1: Call the Algorithm Find-K;
2: Initially select t points as the centroids of sub-clusters
arbitrarily (t>>K);
1481
Authorized licensed use limited to: GOVERNMENT COLLEGE OF TECHNOLOGY. Downloaded on July 08,2010 at 20:10:42 UTC from IEEE Xplore. Restrictions apply.
3: repeat
4: { for (each point x in S)
5: assign x to the closest sub-cluster based on the
distance to the centroid;
6: update the centroid of each sub-cluster;
7: } until (no points change between the t clusters) //utilize
the K-means on the S, where K equals to t
8: Compute the cohesion matrix X between the t clusters;
9: If ( (Ci Cj )∈ M or (Ci Cj )∈ C)
10: implement CUCMC;
11: If (Ci Cj ) violates M (C)
12: w, w are enforced on the cohesion matrix;
13: do{ Extract the maximal chs (Ci, Cj);
14: If (Ci and Cj do not belong to the same sub-cluster)
15: merge the two sub-clusters which they belong to a
new subcluster;
16: t:=t-1; } while (t>K).
end
In HCAKC, Find-K is firstly run in order to determine
the optimal number K of the dataset to be clustered. Then,
K-means is adopted to form t clusters initially, in which t is
more than K. The cohesion matrix named X between t
clusters is obtained base on formula (3). Afterwards, the
existing constraints sets M={(Ci, Cj)} and C={(Ci, Cj)} are
considered to implement the CUCMC. The penalty factor is
introduced to address the constraint violation. When must-
link (Ci, Cj) is violated, w≠ M ( Ci ,C j ) is forced on the similarity
metric according to formula (5). The row i column j in X is
set at Sim(Ci, Cj).
IV. EXPERIMENTAL RESULTS
All of our experiments have been conducted on a
computer with 2.4Ghz Intel CPU and 512M main memory.
The operating system of the computer is Microsoft
Windows XP. HCAKC is compared with CSM to evaluate
the clustering quality and time performance of HCAKC.
The algorithms are all implemented in Microsoft Visual
C++6.0.
We performed our experiments on the UCI datasets:
Ionosphere, iris, breast-cancer, credit-g, page-blocks. The
must-link and cannot-link constraints are generated
artificially by utilizing the same method with [7]. The
details of datasets are shown in Tab.1. For instance, D1 is
the Ionosphere dataset consisting of 355 instances from two
clusters. Accuracy [7], one of the clustering quality
measures, is computed to compare the clustering results
between HCAKC and CSM. We averaged the measures for
100 trails on each dataset. Fig. 3 and Fig. 4 show the
experimental results comparing HCAKC with CSM.
HCAKC and CSM are run on D1 (i.e. Ionosphere
dataset) with constraints respectively. Fig. 3 shows the
accuracy results comparing HCAKC with CSM on the
Ionosphere dataset. From Fig. 3, we can get the conclusion
that CSM is lower in accuracy compared with HCAKC
with varying size of constraints.
In CSM, the constraints are not considered. In HCAKC,
the constraints are incorporated into the hierarchical
clustering to update the cohesion matrix, and the constraints
 violation is addressed as well. Thus, HCAKC is better in
terms of clustering accuracy.
The experiments have been conducted on the datasets:
iris, breast-cancer, credit-g and page-blocks respectively to
compare the time efficiency of HCAKC with that of CSM.
From Fig. 4, we can conclude that HCAKC is better than
CSM in CPU running time on different datasets.
The cluster number K needs to be specified as a
parameter before the CSM algorithm. The time cost of the
parameter setting is expensive, since the K-means needs to
be run iteratively. HCAKC finds out the optimal K via
computing the average IS of the points in datasets, and the
time cost of this process is insignificant. The time efficiency
of HCAKC is obvious even when the scale of the dataset is
large.
TABLE.1 PARAMETERS IN TESTING DATASET
V. CONCLUSION
In order to improve the time efficiency and clustering
quality of CSM, a new method named HCAKC is proposed
in this paper. In our proposed algorithm, the curve graph
about average IS of the dataset and different partition
number has been plotted. The optimal number of clusters is
determined by locating the maximum of the curve graph. As
a result, the complexity of the process when determining the
number of clusters has been improved. Thereafter, the
existing constraints have been incorporated to complete the
CUCMC during the hierarchical clustering process. The
penalty factor is introduced into our algorithm to address the
constraints violation. Hence, the clustering quality has been
improved. The results of the experiments have demonstrated
that the HCAKC algorithm is efficient in reducing the time
complexity and increasing the clustering quality.

Dataset Name Size Clusters REFERENCES

[1] L. Sun, T. C. Lin, H. C. Huang, B. Y. Liao, and J. S. Pan, “An
D1 Ionosphere 355 2
optimized approach on applying genetic algorithm to adaptive cluster
D2 Iris 150 3 validity index”, 3rd International Conference on International
Information Hiding and Multimedia Signal Processing, Kaohsiung,
D3 breast-cancer 277 2 Taiwan, Nov. 2007, vol. 2, pp. 582-585.
D4 credit-g 1000 2 [2] C.R. Lin, M.S. Chen, “Combining Partitional and Hierarchical
Algorithms for Robust and Efficient Data Clustering with Cohesion
D5 page-blocks 5473 5 Self-Merging”, IEEE Transaction On Knowledge and Data
Engineering, 2005, 17(2): 145-159.
[3] H. J. Sun, S. R. Wang and Q. S. Jiang, “FCM-Based model selection
algorithms for determining the number of cluster”, Pattern
CSM HCAKC Recognition, 2004, vol. 37(10), pp. 2027−2037.
1 [4] S. Lamrous, M. Taileb, “Divisive Hierarchical K-Means”, CIMCA
0. 9 2006: International Conference on Computational Intelligence for
0. 8 Modeling, Control and Automation, Jointly with IAWTIC 2006:
Accuracy(%)

0. 7 International Conference on Intelligent Agents Web Technologies

0. 6
0. 5 and Internet Commerce, Sydney, NSW, Australia, 2006, pp. 18-23.
0. 4 [5] S. C. Chu, J. F. Roddick, C. J. Su and J. S. Pan, “Constrained Ant
0. 3 Colony Optimization for data clustering”, 8th Pacific Rim
0. 2
0. 1 International Conference on Artificial Intelligence, PRICAI 2004:
0 Trends in Artificial Intelligence, Auckland, New Zealand, 2004, vol.
5 10 15 20 25 30 35 40 3157, pp. 534-543.
Constraints Ratio/Size(%) [6] C. Ruiz, M. Spiliopoulou and E. Menasalvas, “C-DBSCAN:
Density-based clustering with constraints”, In 11th International
Conference on Rough Sets, Fuzzy Sets, Data Mining and Granular
Figure 3. HCAKC and CSM comparison in terms of accuracy Computing, Toronto, Canada, 2007, pp. 216-223.
[7] K. Wagstaff, C. Cardie, S. Rogers and S. Schroedl, “Constrained K-
HCAKC CSM means clustering with background knowledge”, Proceeding of the
17th International Conference on Machine Learning, 2001, pp. 577-
584.
80
Running time(%)

70 [8] I. Davidson and S. S. Ravi, “Agglomerative hierarchical clustering

60 with constraints: Theoretical and empirical results”, 9th European
50 Conference on Principles and Practice of Knowledge Discovery in
40
30 Databases, Porto, Portugal, 2005, pp. 59-70.
20 [9] M. Bilenko, S. Basu and R. J. Mooney, “Integrating constraints and
10
0 metric learning in semi-supervised clustering”, Proc. of the 21st Int’l
D2 D3 D4 D5 Conf. on Machine Learning, New York, ACM Press, 2004,
Datasets pp.81−88.

Figure 4. HCAKC and CSM comparison in terms of running time

1482

Authorized licensed use limited to: GOVERNMENT COLLEGE OF TECHNOLOGY. Downloaded on July 08,2010 at 20:10:42 UTC from IEEE Xplore. Restrictions apply.