Assignment # 1

By: Muhamad Sohaib Jamal email:

msohaibjamal@gmail.com

Introduction to Pattern Recognition

Pattern Recognition is defined differently in different books by various authors like, one
author defined it as The assignment of a physical object or event to one of several pre-specified
categories [1]. While another [2] defined the pattern recognition as A problem of estimating
density functions in a high-dimensional space and dividing the space into the regions of
categories or classes. Somewhere [3] it is defined as Given some examples of complex signals
and the correct decisions for them, make decisions automatically for a stream of future
examples. Whereas [4] defined it as The science that concerns the description or classification
(recognition) of measurements. The [5] defines pattern recognition as The process of giving
names to observations x. According to Morse, Pattern Recognition is concerned with
answering the question What is this?.
In other words, the identification of implicit objects, types or relationships in raw data by
an animal or machine is known to be Pattern Recognition. After analyzing the raw data based on
a certain purpose and method, we can do actions such as classification or clustering on those raw
data. Classification is to classify a data into known categories, while clustering is to create new
categories from a set of data. In the sense of training, classification is also called supervised
classification and clustering as unsupervised classification. There are some interesting
explanations about the term pattern [6]. Norbert Wiener [7] said that A pattern is essentially
an arrangement. It is characterized by the order of the elements of which it is made, rather than
by the intrinsic nature of these elements. Satoshi Watanabe [8] defined a pattern as "the opposite
of chaos; it is an entity, vaguely defined, that could be given a name." A pattern is an entity,
vaguely defined, that could be given a name, e.g., fingerprint image, handwritten word, human
face, speech signal, DNA sequence, etc...

Applications

Although there are a lot of pattern recognition application but I will give a few examples
of pattern recognition applications like Optical Character Recognition (OCR) applications:
different languages handwriting recognition, Internet search, Junk mail filtering, Telephone
directory assistance, Information extraction, Personal identification, Computer aided diagnosis,
Automatic target recognition, Printed circuit board inspection, Fruit sorting, Forecasting crop
yield, Sequence analysis, Clustering of microarray data, License plate recognition in eTAG
(Motorways of Pakistan Service), Building and building group recognition using satellite data,
Land cover classification using satellite data, Cancer detection and grading using microscopic
tissue data, Autonomous navigation, Biometric recognition and Searching for meaningful
patterns. Some other Related fields of study and Applications of Pattern Recognitions are given
below:
Table 1. Some applications of pattern recognition (from the pattern recognition course website [9])
Machine Based Pattern Recognition

Machine based pattern recognition is the study of how machines can observe the
environment, learn to distinguish patterns of interest, make sound and reasonable decisions about
the categories of the patterns.

Components of Pattern Recognition System

A typical pattern recognition system is composed of:

1. Physical Environment
2. Data Acquisition/Sensing
3. Pre-Processing
4. Feature Extraction
5. Features
6. Classification
7. Post-Processing
8. Decision Making
As shown below in Figure 1. We will now briefly go through each and every topic and its sub-
topics.

Figure 1. Object/process diagram of a pattern recognition system.

1. Physical Environment
Physical environment gives data to the Data Acquisition/Sensing devices for unknown
patterns to be recognized. For example, in case of bio-metric verification during SIM card
registration requires a finger of the user who want to register the SIM card in Pakistan.
2. Data Acquisition/Sensing
Data acquisition and sensing is used for measurement of physical variables like finger will
need a biometric sensor for recognizing the finger. The most important issues in the data
acquisition and sensing devices are the bandwidth, resolution, sensitivity, distortion, SNR
and latency, etc.
3. Pre-Processing
Pre-processing involves the removal of noise in data received from the acquisition and
sensing devices. This step also isolates the patterns of interest from the background.
4. Feature Extraction
Feature extraction is used for finding a new representation in terms of features.
5. Features
The feature used for extraction should be discriminative (i.e. similar values for similar
patterns and different values for different patterns) and invariant with respect to translation,
rotation and scaling, robust features with respect to occlusion, distortion, deformation, and
variations in environment etc We might also select multiple features but we have to be
careful about the unreliable features, domain dependence and prior information,
computational costs and feasibility, noise in the measurements.
6. Classification
Classification learn parameters for function from model inputs to classes
7. Post-Processing
Post-Processing is used for the evaluation of confidence in decisions, exploitation of context
to improve performance and combination of experts.
8. Decision Making
It is a process through which different samples are decided to be in different classes. It is the
final step of the pattern recognition. One might call it the output or result of the pattern
recognition.
The Design Cycle

The Design Cycle of a pattern recognition system is shown in the Figure 2.

Figure 2. The Design Cycle.

Data collection: Collecting training and testing data.

Feature selection: Domain dependence and prior information. Computational cost and
feasibility. Discriminative features. Similar values for similar patterns. Different values
for different patterns. Invariant features with respect to translation, rotation and scale.
Robust features with respect to occlusion, distortion, deformation, and variations in
environment.
Model selection: Domain dependence and prior information. Definition of design
criteria. Parametric vs. non-parametric models. Handling of missing features.
Computational complexity. Types of models: templates, decision-theoretic or statistical,
syntactic or structural, neural, and hybrid.
Training: Training can be done by:
o Supervised learning: a teacher provides a category label or cost for each pattern in
the training set.
o Unsupervised learning: the system forms clusters or natural groupings of the input
patterns.
o Reinforcement learning: no desired category is given but the teacher provides
feedback to the system such as the decision is right or wrong.
Evaluation: It helps in estimating the performance with training samples, predicting the
performance with future data and problems of generalization.

Other Important Terms

Representation. This is the way individual real world objects and phenomena are
numerically described (or encoded) such that they can be related to each other in some
meaningful mathematical framework. This framework has to allow the generalization to
take place.
Design set. This is the set of objects available or selected to develop the recognition
system.
 Adaptation. This is usually a reduction of the representation such that it becomes more
suitable for the generalization step.
Generalization. This is the step in which objects of the design set are related such that
classes of objects can be distinguished and new objects can be accurately classified.
Evaluation. This is an estimate of the performance of a developed recognition system.

Issues and Potential Research Areas of Pattern Recognition

Given below are few issues regarding the different areas of Pattern Recognition [10]:

Representation
1. The use of vector spaces.
a. How to overcome the fundamental inadequacy of vector space
representations?
2. Compactness
a. What is the distribution of classification problems that is in agreement with
the compactness hypothesis?
3. Representation types
a. Can representations be found that offer a good basis for modeling object
structure and which can also be used for generalizing from examples?
4. Missing data problem
a. Can similarities solve the missing data problem?
5. Optimized and trainable representations
a. Can good representations be learnt?
6. Spatial connectivity
a. How to incorporate contextual relations into the representation?

Design Set
1. Multiple use of the training set
a. What is a general theory on the re-use of datasets for training?
2. Representativeness of the training set
a. When is the training set sufficiently well sampled and representative for the
recognition problem? Should a classifier develop over time?
3. Unknown or undetermined class distributions
a. Is domain sampling possible? Can from a given dissimilarity matrix be
determined whether the sampling is sufficiently dense?
 Adaptation
1. Problem complexity
a. How can the complexity of a recognition problem be characterized?
2. Selection or combining
a. What are the advantageous ways of optimizing a set of representations?
3. Nonlinear transformations
a. When are nonlinear transformations of the representation useful?
4. Class structure or class distribution
a. How to constitute classifiers making use of the fact that classes constitute
non-linear structures when represented in high-dimensional spaces?

Generalization
1. Classifier selection or classifier combination
a. How to decide between a selection or combining a set of classifiers?
b. Which are good sets of classifiers to be combined?
2. Trained or fixed combining
a. When and how should a combining classifier be trained?
3. Sequential or parallel training
a. When should a recognition system be trained part by part and when in its
entirety?
4. Classifier typology
a. A library of problems corresponding to the library of classifiers.
5. Generalization principles
a. Bayes or Occam?
6. The use of unlabeled objects and active learning
a. How to make use of unlabeled data to construct classifiers?
7. Multi-class problems
a. Should multi-class recognition be performed by detection or by classification?
8. One-class problems
a. What is a proper one-class classifier?
9. Domain based classification
a. How to construct domain-based classifiers?
10. Object structure
a. How to learn the structure in objects?

Evaluation
1. Recognition system performance
a. How to evaluate the performance for ill-defined class distributions?
2. Prior probability of problems
a. How to judge the expected performance of a recognition procedure?

References
[1] Duda, Richard O., Peter E. Hart, and David G. Stork. Pattern classification. John Wiley &
Sons, 2012.
[2] Fukunaga, Keinosuke. Introduction to statistical pattern recognition. Academic press, 2013.
[3] Ripley, Brian D. Pattern recognition and neural networks. Cambridge university press, 2007.
[4] Schalkoff, Robert J. Pattern recognition. John Wiley & Sons, Inc., 1992.
[5] Schrmann, Jrgen. Pattern classification: a unified view of statistical and neural
approaches. New York: Wiley, 1996.
[6] [Chao, Wei-Lun. "Introduction to pattern recognition." National Taiwan University, Taiwan
(2009): 1-31.]
[7] Wiener, Norbert, and Johannes Petrus Schad, eds. The Cybernetics of the Nervous
System. Vol. 17. Elsevier Pub. Company, 1965.
[8] Watanabe, S. (1985). Pattern Recognition: Human and Mechanical. Wiley.
[9] Shyh-Kang Jeng, Introduction, Pattern recognition Course Website, 2009. [online]
Available: http://cc.ee.ntu.edu.tw/~skjeng/PatternRecognition2007.htm. [Accessed Sep. 30,
2009].
[10] Duin, Robert PW, and Elbieta Pekalska. "Open issues in pattern recognition." Computer
Recognition Systems. Springer Berlin Heidelberg, 2005. 27-42.