Fuzzy Rule-Based Approaches

to Dimensionality Reduction

Nikhil Ranjan Pal

Indian Statistical Institute, 203 B.T. Road, Calcutta, India

nikhil@isical.ac.in, nrpal59@gmail.com

Abstract. In this talk we deal with the problem of dimensionality re-

duction in a fuzzy rule-based framework. We consider dimensionality re-
duction through feature extraction as well as through feature selection.
For the former approach, we use Sammon’s stress function as a criterion
for structure-preserving dimensionality reduction. For feature selection
we propose an integrated framework, which embeds the feature selection
task into the classifier design task. This method uses a novel concept of
feature modulating gate and it can exploit the subtle nonlinear interac-
tion between the tool (here a fuzzy rule based system), the features and
the task at hand. This method is then extended to Takagi-Sugeno (TS)
model for function approximation/prediction problem. The effectiveness
of these methods is demonstrated using several data sets.

Keywords: Dimensionality reduction, Fuzzy Rule Based Systems, Sam-

mon’s Stress, Supervised method, Unsupervised method.

1 Introduction
Success of any decision making system, designed based on data, usually de-
pends on the features used to design the system and use of more features is not
necessarily good. This is true irrespective of whether the system is a pattern clas-
sification system or a function approximation type system. Hence a lot of effort
is given to reduce the dimensionality of the data before they are used to design
the system. Dimensionality reduction can be realized through either feature ex-
traction or feature selection. Given a data set X = {x1 , x2 , · · · , xn }; xi ∈ Rp ,
dimensionality reduction can be viewed as a mapping from the p dimensional
input space to a q dimensional output space ( q < p), M : Rp → Rq , such
that M is obtained by optimizing some criterion. Note that, the task of feature
selection can also be expressed through M . If the task at hand is of classifica-
tion, then typically the classification error is used as the criterion, while when
the task is of function approximation (or prediction), the prediction error is the
common criterion of choice. Such dimensionality reduction methods are known
as supervised methods as each data point is associated with a target output. Di-
mensionality reduction can also be done when we do not know the task for which
the reduced data will be used. In other words, dimensionality reduction can also
be done when the given data points are not associated with any target values as

M.K. Kundu et al. (Eds.): PerMIn 2012, LNCS 7143, pp. 23–27, 2012.

c Springer-Verlag Berlin Heidelberg 2012
24 N.R. Pal

in case of classification. Dimensionality reduction methods for such data sets are
called unsupervised methods. For an unsupervised method one needs to use a
task independent criterion for feature extraction/selection. In this talk we shall
consider both unsupervised and supervised methods.

2 Fuzzy Rule Based Approaches

The problem of dimensionality reduction can be solved using many computa-

tional approaches such as neural networks, fuzzy logic, and evolutionary com-
puting [2007, 1997, 2006, 2004]. Here we shall restrict ourselves to only fuzzy
rule based approaches [2002]. The main advantage of a fuzzy rule based system
lies in its interpretability. Moreover, it is less likely to make poor generalization.
Here, first we shall discuss how fuzzy rule based systems can be used for di-
mensionality reduction through feature extraction when the data points are not
associated with any target values.

2.1 Dimensionality Reduction via Feature Extraction

A good criterion for dimensionality reduction would be to preserve the “struc-

ture” of the original data into the lower dimensional data. This can be achieved if
we can preserve the neighborhood relation in the projected data. The neighbor-
hood relation can be preserved to a great extent if we can preserve the inter-point
distances. This is what we do here. Specifically, we use Sammon’s stress func-
tion [1969] as our criterion to reduce dimensionality through extraction. The
sammon’s stress function is defined as

1  (d∗ij − dij )2
SE =  ∗ . (1)
i<j dij i<j d∗ij

In (1) d∗ij = d(xi , xj ), xi , xj ∈ X and dij = d(yi , yj ), yi , yj ∈ Y , where

d(xi , xj ) be the Euclidian distance between xi and xj , and yi is the q-dimensional
projection of xi . Sammon’s algorithm finds Y minimizing the error function SE
using the gradient descent algorithm. Note that, Sammon’s algorithm does not
have prediction ability.
Now we deal with the unsupervised feature extraction problem using a su-
pervised framework and equip Sammon’s method with prediction ability. First,
we generate a lower dimensional representation Y of X minimizing SE. In this
way we get a data set (X, Y ) such that xi ∈ X ⊂ Rp is associated with a
yi ∈ Y ⊂ Rq . The data set (X, Y ) is then used to find a fuzzy rule base RB
with a view to realizing RB(xi ) = yi . The rules could be of Mamdani-Assilian
(MA) [1975] type or Takagi-Sugeno (TS) [1985] type. The MA and TS rules are
of the forms:
MA rule : If x is CLOSE to vix then y is CLOSE to viy ;
TS rule : If x is CLOSE to vix then y = ui (x).