Chapter 1

Overview of Pattern Recognition

Introduction to Pattern Recognition

What is Pattern?
something exhibiting certain characteristics (regularity)
something able to serve as a model (Greek Beauty, Kyoto Beauty)
something representing a concept of what was observed or experienced (Sticky dish)
In engineering, usually represented as a set of measurements or observations in vector or mat
rix forms
Examples:

Introduction to Pattern Recognition

Pattern Recognition
to recognize given information (usually measures or observation) via information reduction, infor
mation mapping, or information labeling process.
Usually involves the concepts of classification, recognition, or description.

Definition
Classification
assigns input data into one or more of C pre-specified classes based on the extracted features.

Description
an alternative to classification where a conceptual or structural description of the input pattern is desire
d.

Recognition
ability to classify , even identify in many cases.

Preprocessing
filtering and transformation of raw input to minimize noise and to extract rather reliable features.

Postprocessing
Refining the classification or recognition results by utilizing properties of the pattern classes

Introduction to Pattern Recognition

Definition
Desirable Feature
distinct characteristics of data (measurements or observations).
allows representation of high dimensional raw measurements in a lower dimensional space
while preserving essential information, i.e., reducing the redundancy.
provides a good separable space for pattern classification.
examples: mean, variance, distribution type, etc.
desirable feature selection is the most essential one for success.
feature selection is a problem dependent task (the concept of tuning).

Noise
undesirable components in data
distortions or errors in raw input data (preprocessing can increase the distortion)
errors in feature extraction, errors in training data
More Classification Errors
Searching for Noise Robust Features

Introduction to Pattern Recognition

Major Four types of pattern recognition approaches
Tree-structure (or decision) based pattern recognition

Just imagine how to classify apples from oranges; usually intuitional tasks
But very powerful in many applications
Still rather hard to generalize the algorithm, i.e., training data dependency
Recommended to design ones own decision based pattern recognizer before imple
menting the main PR algorithm as a practice.

Statistical pattern recognition

statistical or decision theoretic approach
parametric and non-parametric methods; Most engineering tasks
Easy to generalize and usually produces rather good results for new unknown inputs:
Why?
Not easy to understand the inner operation of the PR algorithm

Introduction to Pattern Recognition

Major Four types of pattern recognition approaches
Syntactic pattern recognition
structural analysis on interrelationships or interconnections of features
requires quantification and extraction of structural information and then assesses structural similarit
y of patterns;
Very powerful for bio-pattern, character recognition, and NLP

Neural Network based Pattern recognition

Somehow a block box approach
involves neural networks for pattern recognition
(Given input Trained multilayer NN Classified output)
Almost impossible to analyze the internal decision behaviors; Hard to improve

Fuzzy Pattern recognition

Based on the Fuzzy theory
Not easy to implement but motion compensation for cameras can be an example.

Introduction to Pattern Recognition

Mappings in an abstract representation of pattern generation/classificati
on/interpretation systems

p4
class i
p1
class j

class
k
Class membership or
description space: C

m1

p2

m2

p3

m3

pattern space: P

Observation or
measurement space: F

Introduction to Pattern Recognition

Pattern recognition procedures

Preprocessing
Usually the first step after acquiring raw data from which pattern recognitio
n is performed.
Usually involves:
Noise reduction; Filtering
Adjusting or deduction of the average value of data (In many cases, including normal
ization)
Segmentation of region (object) of interest from the acquired raw data
Processing of the raw data and mapping the results into different representation dom
ains
Various transforms can be used such as Wavelet, Fast Fourier transforms

Introduction to Pattern Recognition

Pattern recognition procedures

Preprocessing noise reduction in speech signals

Primary signal

Noise reference signal

After spectral subtraction

After noise reduction

Introduction to Pattern Recognition

Pattern recognition procedures

Preprocessing noise reduction in image

Before noise reduction

After noise reduction

Introduction to Pattern Recognition

Pattern recognition procedures

Preprocessing segmentation
Region Tree

Object Tree

Root

R1

Region
Segmentation

R3

R2
R4
R5
R8

R9
R12

Body

TV

Jacket

Hair

Face

Torso

R7

R6
R10

R 11
R13

Introduction to Pattern Recognition

Pattern recognition procedures
Feature extraction
needs to be computationally feasible by manual or by automated procedure
s
Feature vector and feature space
Features are formed as a d-dimensional vector, called the feature vector.
Feature vectors are typically used in statistical and neural network based pattern reco
gnition.

Observation
environment

Feature extraction
process

X (feature vector)
X (string)
P (structure)

Input data preprocessing/feature extraction

Introduction to Pattern Recognition

Pattern recognition procedures

Pattern classification (Assuming we already have pre-trained patterns)

action of classifying the raw data into one of the pre-determined classes or patterns
What can we do if a raw datum from a totally unknown class is met?
Signal (information)
Data acquisition
Measure mi
Preprocessing and
enhancement
Feature extraction

Classification

Description

diagram of typical pattern recognition systems

Selection problem of features

An example of fish classification [textbook, Figure 1.1]

Selection problem of features

Which feature is better for good classification between salmo
n and sea bass classes?

Histogram of length feature for salmon and sea

bass classes

Histogram of lightness feature for salmon and sea

bass classes

Changing the position of the threshold affects the classification error!

Representation of Feature Vectors

Scatter Plot
plot of feature vectors in feature space
difficult to visualize feature vectors with greater than 3-D.

A two-class scatter plot

Some Terminologies
Classifiers
partitions feature space into class-labeled decision regions.
Separable or inseparable
If inseparable, any compromise is possible and allowed?

A decision region
represents the corresponding class in feature space.

Discriminant function
computes the class-likelihood value of a feature (vector) with respect to a certain class.

Adaptive or Self-learning .vs. Fixed

Classification Rule
Classification on features can be done based on compari
ng a set of discriminant functions
Decision rules: Assign
gm x gi x

to class

(region

i 1,2, , c and i m

Likelihood, Nearest, Measure, Similarity, .

Rm

) if

Classification Rule
Decision boundary divides decision regions in feature space where

g k x gl x
Linear
discriminant
functions
R1
R2

Decision
boundary

R3

Quadratic
discriminant
function
R1

R2

arbitrary
discriminant
function
R1
R2

R2

Decision
region

R3

R2
R1

R4

Classification Rule
Which decision boundary is the best linear, quadra
tic or arbitrary decision boundary?

Classification Rule
Generalization problem
For a robust classification performance with test data, the decision boundary needs to
be conformed in a robust way.
So, the training samples should represent the nature of the underlying data successful
ly
Sufficient number of training data?
The more, the better. But not practical if too much.
Experimentally, or intuitively, we have to decide the number with some
restraints such as small number of available data.
The classifiers should be sufficiently trained! (Not always possible!)
There are some tricky methods to recover the shortage of the training data but sufficie
nt training data are better!
More insight for the tasks, better pattern recognizers!

Development Procedure of PR systems

Step 1: Study the classes of patterns under consideration to develop possible characteri
zations. (ideation, logic process)
Step 2: Determine the availability of feature/measurement data
Step 3: Consider constraints on desired system performance,
computational resources, and the implementation cost (Engineering)
Step 4: Consider the availability of training data
Step 5: Consider the availability of suitable and known PR technologies
Step 6: Develop a PR system simulation
Step 7: Train the system
Step 8: Simulate system performance
Step 9: Iterate among the above steps until desired performance is
achieved.
If still not successful, then ?

Development Procedure of PR systems

Procedure of designing a pattern classifier

Typical pattern classifier

Summary
Relationship of pattern recognition to other areas

(Adaptive) signal processing and systems

training and learning of adaptive systems

Artificial Intelligence, Expert System

Optimization/estimation theory
parameter estimation towards optimization of systems

Fuzzy sets
Structural Modeling
Formal language, ..

Summary
Pattern recognition applications
Image preprocessing, segmentation and analysis, coding
Seismic /weather/hurricane analysis and forecast
Radar, Sonar signal classification/analysis
Face, Gesture recognition
Speech/Audio recognition/understanding, Speaker and language recognitio
n
Fingerprint, DNA identification
Character (letter or number) recognition
Handwriting analysis; Electronic Signature for e-commerce
Human bio-signals such as ECG and EEG signal analysis/understanding
Medical diagnosis (tumor, mammogram, germ, virus, cancer,.)

Others to be mentioned
Terms

Novel Patterns (New unknown input)

Supervised, Unsupervised, Reinforced(-ment) Learning
Good Features

Translation, Rotation, Scale Invariant

Noise, Environment Robust
Statistically Independent (No Redundancy)

Authors are saying in p.12 that: Although the pattern classification techniques presented in this
book cannot substitute for domain knowledge, they can be helpful in making the feature values
less sensitive to noise.