~
Proceedings of ICSP2000
Classification Mechanism of Support Vector
Machines
Chen Junli
Key Lab. for Radar Signal Processing, Xidian Univ., Xian 710071, China
Email: chenj1@rsp .xidian.edu.cn
Abstract
The purpose of this paper is to
provide an introductory tutorial on the
basic ideas behind Support Vector
Machines (SVMs). The paper starts with
an overview of Structural Risk
Minimization (SRM) principle, and
describes the mechanism of how to
construct SVMs. For a two-class pattern
recognition problem, we discuss in detail
the classification mechanism of SVMs in
three cases of linearly separable, linearly
non-separable and non-linear. Finally, for
non-linear case, we give a new function
mapping technique: By choosing an
appropriate kernel function, the SVMs can
map the low dimensional input space into
the high dimensional feature space, and
construct an optimal separating hyperplane
with maximum margin in the feature
space.
Key words
Support Vector Machines (SVM)
Structural Risk Minimization (SRM)
Kernel function
Feature space
1. Introduction
For a given learning task with a finite amount of
training data, the best generalization performance will
be achieved if the right balance is struck between the
accuracy attained on that particular training set, and
the capacity of the machine, that is, the ability of the
machine to learn any training set without error. In
practically application, traditional neural network
0-7803-5747-7/00/$10.00@2000IEEE.
Jiao Licheng
approaches had suffered difficulties with
generalization, producing models that can overfit the
data, especially for small data. Vapnik had proposed
SVMs algorithm in the late seventies, which
combined with several techniques from statistics,
machine learning and neural networks. Its formulation
embodies the Structural Risk Minimization (SRM)
principle. By use of kernel function mapping
technique, SVMs can achieve good ability of
classification generalization through small data
learning. Moreover, SVMs have been receiving
increasing attention in the areas of theoretic and
engineering application due to their many attractive
features. For the pattern recognition case, SVMs have
been used for isolated handwritten digit recognition,
object recognition, speaker identification, face
detection in images, and text categorization etc[5],[6].
For the regression estimation, SVMs have been
compared on benchmark time series prediction tests
and linear operator inversion etc[7].
2.Classifiction Mechanism of SVMs
2.1 Structural Risk Minimization (SRM)
SVMs algorithm is based on statistics learning
theory. It embodies the S R M principle. Firstly we
describe the S R M principle.
The idea of SRh4 principle is to create a structure
such that S, is hypothesis space of VC dimension h ,
where the VC dimension is a scalar value that
measures the capacity of a set of functions,
s, c s, c .* c s, c * * *
SRM consists in solving the following problem
 could solve the following optimization problem:
1
Minimize Q, = --llwll (4)

where RemIf]is empirical risk, I is training data,

Subjectto > = I, ...,m
y i ( ( w * x j ) + b ) 1,i (5)
and 1- 6 is confidence.
Equation (1) shows the conshuction idea of (w.x,)+b = +1
SVMs is that optimizing structural to make the second \ t J$w.f)+b=+l} (w.$ ) + b = -1
term of (1). minimize, constrained by the first term
unchanged. This method is substantially different
from Empirical Risk Minimization (EM) which is
adopted in tradition neural network.

2.2 Classification Mechanism of SVMs

We in detail discuss classification mechanism of
SVMs in three cases of linearly separable, linearly
non-separable and non-linear through a two-class Figure 1. A binary classification problem
pattem recognition problem. This constrained optimization problem is dealt with by
2.2.1 Linearly Separable Example introducing Lagrange multiplier ai2 0 , and a
Assume that we are given a set of training data, Lagrangian
which belong to two separate classes,
1
(4yl),.**(xm ym x E R~ Y E {-1>+1]
The goal is to find some decision function
g (x) = sgn (f( x ) ) that accurately predicts the labels The Lagrangian L has to be minimized with respect to
of unseen data ( x , y ) , and minimizes the w and b and maximized with respect t o a , . Classical
classification error. f ( x ) is a linear function, Lagrangian duality enables the primal problem, (6),to
f ( x ) = ( x - w ) + b , for W E R N , b E R (2) be transformed to its dual problem, which is easier to
This gives a classification rule whose decision solve. The dual problem is given by,
boundary i x l f ( x ) = 0)is an N - 1dimensional maxW(a) = m a x b n L(w,b,a)] (7)
a w.6
hyperplane separating the classes +1 and -1 from
each other. Figure 1 depicts the situation. The The minimum with respect towand b o f the
problem of learning from data can be formulated as Lagrangian, L, is given by,
finding a set of parameters (w,b) such that aL
sgn((w.xi)+b)= y f o r d 1I i I m .
-
ab
=o * Z a j Y i
i=l
m
Fig. lshows margin p(w,b) is : aL
-=o
aw
* w =C a , x , y ,
i=l

(3) Hence from (6), (7) and (8), dual problem is

max ~ ( a ) -
= max- 1 ixia y y (xi x >+2ai
The optimal separating hyperplane is given by
a 2 i=] j=] i=l
maximizing margin, because a large margin can make
estimate reliable on the training set, and also make (9)
estimate perform well on unseen examples. Thus we with constraints,
 [a;
2 0,i= l,...ym
The second term of (8) shows that the solution vector
has an expansion in terms of a subset of the training
patterns, namely those patters whose Lagrange
multiplier aiis non-zero. By the Karush-Kuhn-Tucker
complementarily conditions, these training patterns
are the ones for which
ai(yi((xi *w)+b)-l)=O, i = l , ...,m (11)
and therefore they correspond precisely to the Support
Vectors(SV). If the data is linearly separable, all the
determined by a small subset of the training set; the
other points could be removed from the training set
and recalculating the hyperplane would produce the
same answer.
Thus the decision function can be written as
where b is computed using (11).
2.2.2 Linearly Non-Separable Example
For linearly non-separable example, one can
introduce slack variables (5 2 0 ). The constraint of
( 5 ) are modified to,
y i ( ( w - x i ) + b ) >l-ti,i= l,...,m (13)
The generalized optimal separation hyperplane is
determined by the vector w,that minimizes the
following functional,
1 space. K(x,y ) is the kernel function performing the
@(w,t)= fllwl12 + C C t i
i=l
(14)
subject to the constraint of (13).
The solution to the optimization problem of (14)
under the constraint of (13) is given by the saddle
point of the Lagrangian
L(wb,{,a,P)=1/2(w*w)+ czti
i=l
(15)
m
- &((Xi
i=l
w)+ b3yi - 1+ ti >- 2pisi
i=l
where ai,piare the Lagrange multipliers. As before,
Lagrangian duality enables the primal problem, (16),
to be transformed to its dual problem. The dual
problem is given by,
By minimum with respect to w,b,ti of the
Lagrangian L, the dual problem is,
(17)
with constraints,
CO I aiI c,i = l,...,m
The solution to this minimization problem is identical
0
to the separable case except for a modification of the
bounds of the Lagrange multipliers. The parameter C
introduces additional capacity control within the
classifier. In some circumstances C can be directly
related to a regularization parameter.
2.2.3 Non-Linear Example
In the case where a linear boundary is
inappropriate, SVMs can map the input vector into a
high dimensional feature space. By choosing a non-
linear mapping, the SVMs construct an optimal
separation hyperplane in this higher dimensional
non-linear mapping into feature space.
Y$T,t +zIKj
Figure 2. The non-linear mapping mechanism
The optimization problem of (17) becomes,

(19)
and the constraints (18)are unchanged.
Solving ( 19) with constraint (18) determines the
Lagrange multipliers, and a classifier implementing
the optimal separation hyperplane in the feature space
is given by,
[ )
f ( x ) = sgn C a , y , ~ ( x ~ , x ) + b
SVS
(20)
3. Feature Space and Mapping
Mechanism of Kernel Function
By the use of reproducing kernels, we can
construct a mapping into a high dimensional feature
space. The idea of the kernel function is to enable
operations to be performed in the input space rather
than the potentially high dimensional feature space.
Hence the inner product does not need to be evaluated
in the feature space.
The following theory is based upon Reproducing
Kernel Hilbert Spaces (RKHS). If K is a symmetric
positive definite function, which satisfies Mercers
Conditions,
then the kernel represents a legitimate inner product in
feature space. Valid functions that satisfy Mercers
conditions are partially given,
@Gaussian Radial basis function
@ExponentialRadial basis function
@B splines K(x,y ) = B2n+l(X - y )
. 1559 -
The obvious question that arises is that with so many
different mappings to choose from, which is the best
for a particular problem? However, there has not been
strong theoretical method for selecting a kernel at
present. Unless this can be validated using
independent test sets on a large number of problems,
method such as cross-validation will remain the
preferred method for kernel selection.
4.Conclusion
SVMs are attractive approach to data modeling.
They combine statistical learning theory with
generalization control. The formulation results in a
global quadratic optimization problem with convex
constraints, which is readily solved by interior point
methods. The kernel mapping provides a unifying
framework for most of the commonly employed
model architectures. A technique for choosing the
kernel function and additional capacity control
remains to be further research.
REFERENCE
1.Bueges C J C. A Tutorial on Support Vector
Machines for Pattern Recognition. Data Mining and
Knowledge Discovery, Boston, 1998.
2..Gunn S . Support Vector Machines for Classification
and Regression. Image Speed and Intelligent
Systems Technical Report. 1998.
3.Lu Zengxiang, etc. Supervised Support Vector
Machines Learning Algorithm and Application.
Journal of TSINGHUA University. No.7,1999.
4.Weston J, Herbrich R. Adaptive Margin Support
Vector Machines. London: the MIT Press, 1999.
5.Blaze V, etc. Comparison of View-based Object
Recognition Algorithms Using Realistic 3d Models.
ICAN96,1996.
6.Cortes C, Vapnik V. Support Vector networks.
Machine Learning, 1995.
7.Osuma E, etc. An improved training algorithm for
support vector machines. In proceedings of the 1997
IEEE Workshop on Networks for Signal Processing,
Principe E J, etc. 1997.