[Facial Recognition System

CE)]

Chapter 1
1.1

Alpa Rupala (ME-

INTRODUCTION

What is Facial Recognition System (FRS)?

In our body, the face is main attention in social life playing an important role in
conveying identity. We can recognize a number of faces learned throughout our lifespan
and identify faces at a glance even after years of separation. This skill is quite robust
despite of large variations in visual stimulus due to changing condition, aging and
distractions such as beard, glasses or changes in hairstyle.

Face recognition, an attractive and full of challenges research field of Computer

Vision and Biometrics concerning theoretical methods and software systems for
machines to recognize people based on their digital face images, has been fuelled by
many academic scientists and industrial developers for over twenty years.[ ] Image
processing (IP), Pattern recognition (PR), Machine learning (ML), Visual perception,
psychophysics and neuroscience, is one of the most successful studies of biometrics is
Face recognition, which exploits the knowledge from this research disciplines.
Biometrics (Bio=Pertaining to biology & Metrics=Science and art of measurement)
is the one branch of practical science where the physical characteristics of the human
body are considered for the identification of person. Some of the biometrics areas are
iris recognition, fingerprint recognition, face recognition, hand gesture recognition,
age estimation [2]. Usually face recognition methods extract high-dimensional face
information from face data and project such data to a lower dimensionality space,
where new face samples are classified. In present, face recognition plays a major role
in surveillance, personal information accesses, and improved human machine
interaction.
The Face Recognition can be divided in two Problems [8]:
1. Identification (one to- one matching): When presented with a face image
of an unknown individual along with a claim of identity, ascertaining whether
the individual is who he/she claims to be.

FRS

Page 1

[Facial Recognition System

CE)]

Alpa Rupala (ME-

2. Verification (one to many matching): Given an image of an unknown

individual, determining that persons identity by comparing (possibly after
encoding) that image with a database of (possibly encoded) images of known
individuals.

1.2

Challenges of FRS:
The different Factors which affects the face recognition accuracy are listed
below: [8][5][]

Fa c e
Reco
g n it i
on

O cc
lu s i
on
Im a g e
O r ia t a t
io n

Pose
Va r ia ti
ons

Ve r y
L a rg e
S c a le

Fa c ia l
E x p re ss
io n
Low
R e s o lu
t io n

A g in g
C o n d it i
on

Illu m in a tio n
Va r ia t io n s

Fig. 1.2.1 Challenges of Face Recognition

Pose variations: Face images are greatly varied by pose (frontal, 45 degree,
profile, upside down) angles of people. When the pose angles are large (not in
the range of 45), the system accuracy is drastically degraded. some facial
features such as an eye or the nose may become partially or wholly occluded
Facial expression. The appearances of faces which deformed the crucial facial
features such as eyes, eyebrows, mouth, nose are directly affected by a
persons facial expression rate.
Illumination variations:

The acquired images are strongly affected by

environment. When the image is formed, factors such as lighting (spectra,

source distribution and intensity) and camera characteristics (sensor response,
lenses) affect the appearance of a face
FRS

Page 2

[Facial Recognition System

CE)]

Alpa Rupala (ME-

Aging condition: As human faces are vastly changing over time, the
identification process of face images under long term aging effects is a real
challenge, even by human beings.
Low resolution images: Equipped with limited memory, usually average
quality lens, and having to operate in real time, surveillance cameras
consequently produce blur, small and low quality images that are too
challenging to obtain high recognition rates.
Very large scale systems: Face database of a countrys population can reach
up to hundreds of millions or over one billion images. How to make a FR
system working efficiently with such huge databases in real time is a hard
question to answer.
Image orientation. Face images directly vary for different rotations about the
cameras optical axis.
Occlusion. Faces may be partially occluded by other objects. In an image with
a group of people, some faces may partially occlude other faces
Main Focus of this report is on occlusion challenge.
OCCLUSION is defined as an Obstacle or any unnecessary-object in image
disturbing the matching sometimes called recognizing process. Occlusion in an
image refers to hindrance in the view of an object.
Occlusion observed can normally be natural as well as synthetic.
1. Natural occlusion indicates to obstruction of views between the two
image objects without any intension.
2. Synthetic occlusion refers to artificial blockade of intentionally
covering the images view with a white/black solid rectangular block.
Even in real time application face image intentionally becomes occluded via
accessories such as
Sunglasses, scarf, beards, hat.
Hand on face
Face dirt
Face behind Fence
Texture on Face images, etc.
FRS

Page 3

[Facial Recognition System

CE)]

1.3

Alpa Rupala (ME-

Application of FRS:

There are some applications of FER :

1. Security (access control, computer/ network security; email).
2. Surveillance (CCTVs to monitor criminals, drug offenders).
3. Criminal justice systems (mug-shot/booking systems, post-event analysis,
forensics).
4. Image database investigations (searching image databases of licensed
drivers, benefit recipients, missing children, immigrants and police
bookings).
5. Smart Card
6. Multi-media environments that use adaptive human computer interfaces
(part of ubiquitous or context aware systems, behavior monitoring at
childcare or old peoples centers, recognizing a customer and assessing his
needs).
7. Video indexing (labeling faces in video).
8. Witness faces reconstruction.
9. Checking the criminal record by input image in branch of investigation

1.4

Motivation

Face recognition has recently received a blooming attention and interest from the
scientific community as well as from the general public. The interest from the general
public is mostly due to the recent events of terror around the world, which has
increased the demand for useful security systems.
Face recognition is very helpful in many applications such as video surveillance,
forensic applications criminal investigations, and in many other fields.

FRS

Page 4

[Facial Recognition System

CE)]

Alpa Rupala (ME-

As far as we know, the most important facial parts of the human face are the eyes and
the mouth. The natural shapes of human eyes and mouth helps as to extract the
features of the face and then compare them with already had database for recognition
purpose but if this main features are occluded then the recognition becomes somewhat
difficult and challenging. This work will be helpful in some identification or
verification cases of terrorists like persons who wore any disguise and still can be
identified.

1.5

Objective of this work

The main objective of the present work is to implement the Face recognition system
which verify the occluded image form the database with the use of Local Pattern Of
Gradient (LPOG) technique which is the feature extraction technique, the most
important in face recognition system

1.6

FRS

Report Origination

Page 5

[Facial Recognition System

CE)]

Chapter 2

Alpa Rupala (ME-

Background Theory

2.1 Generic Framework of Face Recognition

FRS proceed in mainly three steps as shown in Figure 2.1,
1. Face Detection
2. Feature Extraction
3. Classification

Fig. 2.1.1 Generic Framework of Face Recognition

Image Acquisition:
Image acquisition means the procedure of obtaining images from an already
available dataset or standard facial expression Database like JAFFE, MUG,
FEEDTUM, CMU, ORL.
Face detection:
It is combination of two processes. One is finding the face from the image and
another one is processing of detected face image like cropping face, resizing,
normalization, and sampling etc.
FRS

Page 6

[Facial Recognition System

CE)]

Alpa Rupala (ME-

Image Pre-Processing:
In Pre-processing, basically removing noise and normalize the face image also
if image in any format need to convert in specified format, resize in the specified size
and remove unnecessary features from it. There is a different technique for PreProcessing like Histogram equalization, filtering and RGB image to Grayscale
conversion, etc. As per the requirement of further recognition and depended on the
input image, the pre-processing technique is applied.
Features Extraction:
Feature Extraction is a crucial stage of FER where the performance of
recognition is depending. It extracts the meaning full set of information which call
feature vector. Feature Vector represents characteristics of faces in images. There are
three techniques used for feature extraction in pattern recognition commonly which
are Geometric-base (Feature-base), Appearance-based and hybrid method. Based on
Feature we want to extract the techniques are used.
Dimensionality Reduction:
In this Step the Higher dimensional images is needs to be converted into the
lower dimensional one for the better comparison in the classification process
Classification:
Classification applies on the feature vector of training and testing image. It is
used for result of outcome in recognition. There is a different classifier like a Support
Vector Machine (SVM), Euclidean Distance, Artificial Neural Network (ANN) and
cross co-relation.

FRS

Page 7

[Facial Recognition System

CE)]

Chapter 3
3.1

Alpa Rupala (ME-

Face Recognition Approach

Face Recognition Approaches Comparisons

In many applications like the surveillance and monitoring, the traditional biometric
techniques will fail as for obvious reasons one cannot ask everyone to come and put
his/her thumb on a slide or something similar. So we need a system which is similar to
the human eye in some sense to identify a person. To cater this need and using the
observations of human psychophysics, face recognition as a field emerged. Different
approaches have been tried by several groups, working world wide, to solve this
problem.
The research on face recognition categorizes it into the following approaches: [8-9]
[14] [16]
1.

Feature-based approach

2.

Holistic or global approach

3.

Hybrid approach

FRS

Page 8

[Facial Recognition System

CE)]

Alpa Rupala (ME-

F
ea
ac
e t
u
r
e
B
a
s
e
d
Fig. 3.1.1 Classification of Face Recognition Approaches

3.2

Feature Based Approach

Feature-based face recognition uses a priori information or local features of faces to

select a number of features to exclusively identify individuals. Local features include
the eyes, nose, mouth, chin and head outline, which are selected from face images.
Topological graphs are used to represent relations between features, and a simple
deterministic graph-matching scheme that exploits the basic structure is used to
distinguish familiar faces from a database. [14]
Some Features Based Techniques are shown below:
3.2.1

Elastic Bunch Graph Matching:

Elastic Bunch Graph Matching algorithm which recognizes faces by matching the
probe set represented as the input face graphs, to the gallery set that is represented
as the model face graph. Fundamental to the Elastic Bunch Graph Matching is the
FRS

Page 9

[Facial Recognition System

CE)]

Alpa Rupala (ME-

concept of nodes. Essentially, each node of the input face graph is represented by
a specific feature point of the face. For example, a node represents an eye and
another node represents the nose and the concept continues for representing the
other face features. Therefore, the nodes for the input face graph are
interconnected to form a graph-like data structure which is fitted to the shape of
the face as illustrated in Fig. 3.2.1.1.[14]

Fig. 3.2.1.1 Face Topology Graph [14]

Like PCA, the EBGM algorithm was also designed for accuracy and had no
need to run in real time. The execution of this algorithm is currently performed
by running three separate executables on the data.

3.2.2

Convolution Neural Network:

The neural networks are among the most successful decision-making systems
which can be trained to perform complex functions in various fields of
applications including pattern recognition, optimization, identification and
classification. There are methods which perform feature extraction using neural
networks. [14]
The neural networks algorithm for face recognition is biologically inspired and
based on the functionality of neurons. The perceptron is the neural network
equivalent to a neuron. Just like a neuron sums the strengths of all its electric
inputs, a perceptron performs a weighted sum on its numerical inputs. Using these
perceptrons as a basic unit, a neural network is formed for each person in the
database. [8]
FRS

Page 10

[Facial Recognition System

CE)]

Alpa Rupala (ME-

The neural networks usually consist of three or more layers. An input layer takes
in a dimensionally reduced (using PCA) image from the database. An output layer
produces a numerical value between 1 and -1. In between these two layers, there
usually exist one or more hidden layers. For the purposes of face recognition, one
hidden layer usually provides a good balance between complexity and accuracy.
Including more than one such layer exponentially increases the training time,
while not including any results in poor recognition rates. Once this neural network
is formed for each person, it must be trained to recognize that person. [8]

Fig 3.2.2.1 Neural Network Topology [14]

The most common training method is the back propagation algorithm. This
algorithm sets the weights of the connections between neurons such that the
neural network exhibits high activity for inputs that belong to the person it
represents and low activity for others. During the recognition phase, a reduced
image is placed at the input of each of these networks, and the network with
the highest numerical output would represent the correct match. [8]
3.2.3

Hidden Markov Model:

In HMM-based face recognition system, in which a scanning strategy is employed

to simulate a human-like saccadic sequence, computed on the basis of the concept
of saliency. The approach converts a face image into an attention based
scanpath, that is, a sequence composed of two types of information: Where
FRS

Page 11

[Facial Recognition System

CE)]

Alpa Rupala (ME-

information, the coordinates of the salient region in the face, and What
information, local features detected in there. [s] Since HMMs require a onedimensional observation sequence and images are two-dimensional, the images
should be converted into either 1D temporal sequence or 1D spatial sequence. [8]
At the core of the scanning mechanism is the calculation of saliency. This
calculation should be cheap enough that it can be applied to the whole image
without significantly increasing time and space requirements, and it should be
informative. With this approach, a cheap and parallel search for salient features
will drive a serial and detailed analysis.
In [8], a spatial observation sequence was extracted from a face image by using a
band sampling technique. Each face image was represented by a 1D vector series
of pixel observation. Each observation vector is a block of L lines and there is an
M lines overlap between successive observations. An unknown test image is first
sampled to an observation sequence. Then, it is matched against every HMMs in
the model face database (each HMM represents a different subject). The match
with the highest likelihood is considered the best match and the relevant model
reveals the identity of the test face.
One of the major drawbacks of HMM is that it is sensitive to geometrical shape
[8].
3.2.4

Dynamic link Architecture:

DLAs attempt to solve some of the conceptual problems of conventional artificial

neural networks, the most prominent of these being the representation of
syntactical relationships in neural networks. DLAs use synaptic plasticity and are
able to form sets of neurons grouped into structured graphs while maintaining the
advantages of neural systems.
The DLA architecture was recently extended to Elastic Bunch Graph Matching.
This is similar to the graph described above, but instead of attaching only a single
jet to each node, the authors attached a set of jets (called the bunch graph
representation), each derived from a different face image. To handle the pose
FRS

Page 12

[Facial Recognition System

CE)]

Alpa Rupala (ME-

variation problem, the pose of the face is first determined using prior class
information [Kruger et al. 1997], and the jet transformations under pose
variation are learned [Maurer and Malsburg 1996a]. Systems based on the EBGM
approach have been applied to face detection and extraction, pose estimation,
gender classification, sketch-image-based recognition, and general object
recognition. The success of the EBGM system may be due to its resemblance to
the human visual system. [16]

3.3

Holistic Approach:

In holistic approach, the whole face region is taken as an input in face detection
system to perform face recognition. Holistic face recognition utilizes global
information from faces to perform face recognition. The global information from
faces is fundamentally represented by a small number of features which are directly
derived from the pixel information of face images. These small numbers of features
distinctly capture the variance among different individual faces and therefore are used
to uniquely identify individuals.
Some Holistic Based Techniques are shown below:

3.3.1 PCA
It is a mathematical procedure that performs a dimensionality reduction by
extracting the principal components of the multi-dimensional data. The first
principal component is the linear combination of the original dimensions that has
the highest variability. The n-th principal component is the linear combination
with the maximum variability, being orthogonal to the n-1 first principal
components. [8]
The scheme is based on an information theory approach that decomposes face
images into a small set of characteristic feature images called Eigenfaces, which
are actually the principal components of the initial training set of face images. [8]
An important feature of PCA is that one can reconstruct any original image from
the training set by combining the Eigenfaces. The technique used in creating
Eigenfaces and using them for recognition is also used outside of facial
recognition. This technique is also used for handwriting analysis, lip reading,
FRS

Page 13

[Facial Recognition System

CE)]

Alpa Rupala (ME-

voice recognition, sign language/hand gestures interpretation and medical imaging

analysis. [8]

3.3.2 LDA
Linear Discriminant Analysis is a well-known scheme for feature extraction and
dimension reduction. LDA is widely used to find linear combinations of features
while preserving class separability. Unlike PCA, LDA tries to model the
differences between classes. Classic LDA is designed to take into account only
two classes. Specifically, it requires data points for different classes to be far from
each other, while points from the same class are close. Consequently, LDA obtains
differenced projection vectors for each class. [8]
The face space created in LDA gives higher weight to the variations between
individuals than those of the same individual. As a result, LDA is less sensitive to
lighting, pose, and expression variations. The drawback is that this algorithm is
significantly more complicated than PCA. [8]

3.3.3 ICA
Independent Component Analysis aims to transform the data as linear
combinations of statistically independent data points. Therefore, its goal is to
provide an independent rather that uncorrelated image representation. ICA is an
alternative to PCA which provides a more powerful data representation. Its a
discriminant analysis criterion, which can be used to enhance PCA. [8]
Whereas PCA depends on the pair wise relationships between pixels in the image
database, ICA strives to exploit higher-order relationships among pixels. That
is, PCA can only represent second-order inter-pixel relationships, or relationships
that capture the amplitude spectrum of an image but not its phase spectrum. On
the other hand, ICA algorithms use higher order relationships between the pixels
and are capable of capturing the phase spectrum. Indeed, it is the phase spectrum
that contains information which humans use to identify faces. [8]

3.4

FRS

Hybrid Approach:

Page 14

[Facial Recognition System

CE)]

Alpa Rupala (ME-

In hybrid approach, both local features and the whole face is used as the input to the
face detection system. It is more similar to the behavior of human being to recognize
the face.[8] The concept of eigenfaces can be extended to eigen features, such as
eigen eyes, eigen mouth, etc. Since most feature extraction methods have no special
requirement on the input feature, it is reasonable to combine it with feature design
procedure in order to seek more effective face representation. [14]
Some Hybrid Techniques are shown below:

3.4.1 Component Based:

The basic idea of component-based methods [Heisele et al. 2001] is to decompose
a face into a set of facial components such as mouth and eyes that are
interconnected by a flexible geometrical model. (Notice how this method is
similar to the EBGM system [Okada et al. 1998;Wiskott et al. 1997] except that
gray-scale components are used instead of Gabor wavelets.) The motivation for
using components is that changes in head pose mainly lead to changes in the
positions of facial components which could be accounted for by the flexibility of
the geometric model. However, a major drawback of the system is that it needs a
large number of training images taken from different viewpoints and under
different lighting conditions. To overcome this problem, the 3D morphable face
model [Blanz and Vetter 1999] is applied to generate arbitrary synthetic images
under varying pose and illumination. [16]
Fourteen facial components were used for face detection, but only nine
components that were not strongly overlapped and contained gray-scale structures
were used for classification. In addition, the face region was added to the nine
components to form a single feature vector (a hybrid method), which was later
trained by a SVM classifer [Vapnik 1995]. Training on three images and testing on
200 images per subject led to the following recognition rates on a set of six
subjects: 90% for the hybrid method and roughly 10% for the global method that
used the face region only; the false positive rate was 10%. [16]

3.4.2 Modular Eigenfaces:

FRS

Page 15

[Facial Recognition System

CE)]

Alpa Rupala (ME-

In mugshot applications, usually a frontal and a side view of a person are

available; in some other applications, more than two views may be appropriate.
One can take two approaches to handling images from multiple views. The first
approach pools all the images and constructs a set of eigenfaces that represent all
the images from all the views. The other approach uses separate eigenspaces for
different views, so that the collection of images taken from each view has its own
eigenspace. The second approach, known as view-based eigenspaces, performs
better. [16]
The concept of eigenfaces can be extended to eigenfeatures, such as eigeneyes,
eigenmouth, etc. Using a limited set of images (45 persons, two views per person,
with different facial expressions such as neutral vs. smiling), recognition
performance as a function of the number of eigenvectors was measured for
eigenfaces only and for the combined representation. For lower-order spaces, the
eigenfeatures performed better than the eigenfaces [Pentland et al. 1994]; when
the combined set was used, only marginal improvement was obtained. [16]

FRS

Page 16

[Facial Recognition System

CE)]

Chapter 4

Alpa Rupala (ME-

LITERATURE REVIEW

4.1. Summary of used Techniques

4.1.1. Skin Color Segmentation
Color is an important feature for the facial features detection. Skin color
processing is faster than other methods. Use of skin color for facial feature
detection has some advantages. Skin Color Segmentation can classify every pixel
as skin or non-skin from its neighbors. By this method, at first the face has been
detected from the images by the proposed skin color segmentation method,
because skin color segmentation is faster process for facial features detection. [6]
The YCbCr color space is used for the skin color segmentation of the facial
features. YCbCr is a color space that has red, blue components. In case of YCbCr
color space, Cr is bigger than Cb components. Eyes and mouth regions are
extracted from the face image, after the face detection. Facial regions are
eliminated by applying threshold value. After the facial region elimination, facial
features are detected from the whole images by color segmentation but it has
small unwanted pixels, which are reduced by image erosion and image dilation
through morphological process. Then image is converted to binary image and
connects the existing components to detect facial features like eyes, mouth and
lip. [6]

4.1.2.

Fast Weighted Principle component analysis

Face Recognition and Identified FW-PCA providing appropriate weights to

sample images. Facial Occlusion is a critical problem for Face recognition. Fast
Weighted Principal Component Analysis (FW-PCA), which computes PCA only
with effective pixels. The adoption of FW-PCA makes it possible to detect and
restore occluded regions in face images. [4]

4.1.3.

Exemplar Based Image Inpainting

Inpainting is done by sampling, and by copying or joining together patches

(called exemplar) taken from the known part of the image, So, the corresponding
methods used are known as exemplar-based techniques. Exemplar-based
FRS

Page 17

[Facial Recognition System

CE)]

Alpa Rupala (ME-

inpainting symphonizing entire patches by learning from patches in the familiar

part of the image and these methods are known to be faster than PDE pixel based
approaches. [4]
The normalized input image x which is reconstructed using the method of FWPCA which consist of
a)Training process :- The eigenspace E is computed using all the training
images, The standard deviation of the reconstruction error for each pixel
= [1 2 N] T is also computed.
b) Reconstruction process: - First, the initial weight for FW-PCA is determined
using the sub-samplings of the input image. M pixels are extracted from the input
image according to S random sampling patterns. Considering the extracted pixels
as the effective pixels, the reconstructed image xs (s = 1, , S) for each
sampling pattern is computed by using FW-PCA i.e., Then, the reconstruction
error rs between x and xs is obtained by
rs

|X ^X s|

(4.1.3.1)

Note that the pixel values of x is normalized so that they have the same mean and
variance of x during each iteration. The pixel values of effective pixels in the
final reconstructed image x is normalized so that they have the same mean and
variance of the normalized input image x.
c) Occluded-region restoration: - As mentioned above, image reconstruction
for the whole input image is done using the eigenspace E. The reconstructed
image x may result in the smoothed image of the input image x. Therefore, only
the missing pixels of the input image x are replaced by the pixels of the
reconstructed image x. The restored image x is obtained by

(4.1.3.2)
Finally, restoration is done only on the occluded regions of the input image.

4.1.4.

LPOG

LPOG is the combination of BELBP (Block wised Elliptical Local Binary

Pattern) and LPQ (Local Phase Quantization).

FRS

Page 18

[Facial Recognition System

CE)]

Alpa Rupala (ME-

Fig 4.1.4.1 LPOG technique Framework [1]

The steps of the proposed feature extraction method called local patterns of
gradients (LPOG) in this work are illustrated in Fig. 3. From the input image I,
two directional gradient images along x-axis and y-axis (Gx, Gy) are
generated using the formulas given below: [1]
Gx =

1
1
I ( x1, y )+ 0I ( x , y ) + I (x +1, y)
2
2

(4.1.4.1)

1
1
I ( x , y1 ) +0I ( x , y )+ I (x , y +1)
2
2

(4.1.4.2)

G y=

Then, a symmetric pair of BELBP and a LPQ operator is used to extract local
patterns from each gradient image under the form of BELBP and LPQ images
(we call them local patterns images). Next, each local pattern image is divided
into disjoint rectangular sub regions to calculate their histogram sequences,
which are then concatenated to constitute a description of the image. Uniform
patterns [11], which are binary strings with no more than two bitwise transitions
from 0 to 1 or 1 to 0, are utilized to reduce the BELBP descriptions length. As a
last step, all the local patterns images representations are aggregated to form a
global LPOG feature vector of the given image. [1]

FRS

Page 19

[Facial Recognition System

CE)]

Alpa Rupala (ME-

The discriminant power of LPOG is built upon the usage of BELBP and LPQ
operators to extract local texture patterns from gradient images. The novel
BELBP operator encodes macroscopic relation between one gradient pixel and its
neighbors. The ideas of ELBP about the nature orientation (horizontal) of
important facial features (such as eyes, eyebrows, mouth) and using a symmetric
pair of ELBP for integrating both horizontal and vertical information from a face
image are augmented in BELBP with the oriented block patterns when
computing the accumulated images as they are not only present on the used
ellipse patterns but also on the block-wised neighborhoods. [1]
Beside the role of BELBP, LPQ operator has a significant impact on the
robustness of LPOG method as well. While BEBP is based on binary
thresholding of each pixel from an accumulated gradient image with its neighbor,
LPQ is based on quantization of STFT phase responses and it extracts the local
phase patterns from gradient images. LPQ was proved to be strong against
blurred faces; as a result its presence in LPOG equips our approach with blur
invariant property. On the other hand, LPOG is also strong to uniform
illumination, an attribute derived from LPQ. [1]

4.1.5.

OLPP

To preserve the data discriminative properties, the Laplacianfaces method uses

locality preserving projections (LPPs) to learn a lower dimensionality subspace,
which represents the intrinsic geometry of the data and its local structure. To
preserve better the data geometry in the lower dimensionality space, the
orthogonal LPP (OLPP) method adds orthogonal property to the transformation
matrix obtained by the LPP method. [2]
To define data locality in the OLPP method , it is built a locality graph where two
face images xab and xa_b_ are neighbors only if xa_b_ is among the k NN of
xab, and vice versa. [2]

4.1.6.

Structure Sparse Error Coding

Incorporating the two aspects of the error structure, we propose the structured
sparse error coding for face recognition with occlusion. The method is more
FRS

Page 20

[Facial Recognition System

CE)]

Alpa Rupala (ME-

stable and has higher breakdown point in dealing with the occlusion problems in
face recognition as compared to the related state-of-the-art methods, especially
for the extreme situation, such as the high level occlusion and the low feature
dimension. The intrinsic error structure of contiguous occlusion from two sides:
i) the structure of the error support, including its spatial continuity and its shape;
ii) the structure of the error distribution under different error metrics.
The study of the error structure makes our work contribute mainly in the
following aspects:
1) A morphological graph model for the error support structure. Recently,
structured sparse representation (SSR) has attracted a lot of attention, since
the measurements used to recover the structured sparse signal can be
substantially decreased compared to the standard compressive sensing
method. Using the graph theory, SSR also offers a scheme to use the
coefficient structure. Inspired by these works, in order to explore the
geometrical structure of the occlusion error, we propose a new graph model,
called morphological graph, by integrating the morphology information of
occlusion into graph.
2) An exponential probabilistic model for the error distribution structure.
Although there must exist some close relationship between the error metric
and error distribution, to the best of our knowledge, none of the existing
methods takes the two factors together. We study the error distribution under
different error metrics, especially, under the entropy-induced metrics. With
the assumption that the unoccluded region of the test image y can be
approximated sufficiently by the corresponding regions of the training
samples, we show that the error e measured by the combination operator of
the correntropy induced metric and the dilation invariance metric, might
present a special sparse structure: both the error in the unoccluded region and
the error in the occluded region follow the exponential distribution, which
induces good source-and-error separation.
3) Using the structured information of the error support and error distribution,
we propose a new scheme, called Structured Sparse Error Coding (SSEC),
for FR with occlusion. Experiments show that SSEC is a high breakdown

FRS

Page 21

[Facial Recognition System

CE)]

Alpa Rupala (ME-

point method which achieves much stable recognition performance for

various occlusion levels and various feature dimensions.

4.2. Summary of Literature Review Papers

Huu-Tuan Nguye et al. [1] proposed a method named Local pattern of gradient
(LPOG) which is the combination of BELBP and LPQ. The researchers experiments
on the different databases like FERET, AR and SC Face, they used the proposed
LPOG technique with WPCA (Whitened Principle component Analysis) for
dimensionality reduction and KNN (Kernel Nearest Neighbour) for classification.
John Solder et al. [2] Proposed a novel method of dimensionality reduction
say OLPPs, a lower dimensional subspace. On the public databases (namely, PUT,
FEI, FERET, YALE and ORL) proposed scheme was tested on sparse and dense face
image representation under various conditions and used the SVM Classification with
soft margin techniques for higher accuracy.
Kathryn Bonnen et al. [12] propose The component-based framework which
consists of the following major steps: 1) landmark extraction using Active Shape
Models (ASM), 2) alignment and cropping of components using Procrustes Analysis,
3) representation of components with Multiscale Local Binary Patterns (MLBP), 4)
per-component measurement of facial similarity, and 5) fusion of per-component
similarities. On the databases consisting of face images of 8000 subjects, that the
proposed component-based representation provides higher recognition accuracies
over holistic-based representations. The proposed component-based representations:
1) are more robust to changes in facial pose and 2) improving recognition accuracy on
occluded face images in forensic scenarios.
Mehedi Hasan et al. [6] proposed the skin color segmentation method which localizes
the different components of the face by their color region segmentation. In this
experiment they have compared their technique with RGB and HIS color space. The
proposed method has eliminated the problem arise due to different pose, position of
image, expressions and illumination variation problem. The average accuracy of the
proposed algorithm is 97.69% and extraction of the facial features becomes easy
using the proposed method from the color images.

FRS

Page 22

[Facial Recognition System

CE)]

Alpa Rupala (ME-

Xiao-Xin Li et al. [11] proposed the Structure Sparse Error coding technique
which is mainly used for the classification purpose. Proposed Method SSEC is the
combination of Morphological Graph Model for morphological error support structure
and Probabilistic Model for the error distribution structure, which mainly focused on
shape of the occlusion.
Weihong Deng et al. [15] propose an Extended Sparse Representation Based
Classifier (ESRC) for undersampled face recognition, which is effective even when
there is only a single training image per subject.
Bindu A et al. [13] Proposed Algorithm is an initiative towards extracting the
logical relations existing between the different facial features which serve as a key
towards the accomplishment of efficient reconstruction under heavily occluded cases.
Logical Inpainting in evolving the occluded face width (with 10% visibility) has
motivated to evolve the face length using the logical relationships. The response time
of the Inpainting Algorithm can be drastically reduced mainly because of the reduced
database size
Kanokmon Rujirakul et al. [7] proposed the recognition optimization utilizing
histogram equalization and weighted Euclidean Distance classification derived from
Eigenvalues to enhance the recognition precision. These mechanisms, the so-called
Weighted Histogram Equalized PEM-PCA or whe-PEM-PCA, result in higher
accuracy and lower complexity compared to a traditional PCA and its derivatives
leading to a high speed face recognition system.
Sandesh V Khamele et al. [4] in this paper they classify the occlusion into Two
Categories first, One is for major occlusion(goggle, scarf, masks) and another is for
Minor occlusion (tattoo, beard, moustache). In Minor Occlusion Exemplar Based
inpainting Technique With KNN Classifier is used, but in Major occlusion this
technique is not efficient one so the image reconstruction processor need to be done.

FRS

Page 23

[Facial Recognition System

CE)]

Alpa Rupala (ME-

4.3. Comparison of Implemented Techniques

Table 4.3.1 Comparison of different Implemented Techniques

RESEARC Year
HERS
And
Public
-ation
Huu-Tuan
2015
Nguye, and
IEEE
Alice Caplier
[1]

Method
Used

Database
Used

Advantages

Disadvantages

LPOG
(BELBP
+LPQ)
WPCA
KNN
OLPP
+SVM

FERET,
AR,
SCFace

More fast

Requires High
Computation
Done on Identification
only

PUT,
FEI,
FERET,
Yale,
ORL

It can handle
Noisy, Non
Linear and high
dimensional data

Low Accuracy with

Gray scale images

More efficient
and robust

John Solder,
Carlos
Alberto
Ramirez
Behaine, and
Jacob
Scharcanski
[2]
Kathryn
Bonnen,
Brendan F.
Klare,
Anil K. Jain
[12]

2015
IEEE

2013
IEEE

Compon
ent Base
(Active
Shape
Model)
MLBP

PCSO
LFW
AR,
FERET

More Robust
and accurate
on Pose
variation and
occlusion

Misalignment of
Component can be
lead to wrong result,
New technique of
effectively align and
extract component
need to be used

Mehedi
Hasan,
Foisal
Hossain [6]

2014
IEEE

Own
database

Easy

High Quality Image is

required

Xiao-Xin Li
Dao-Qing Dai
Xiao-Fei
Zhang
Chuan-Xian
Ren[11]
Weihong
Deng,
Jiani Hu,
Jun Guo [15]

2013
IEEE

Skin
Color
Segment
ation
Morphol
ogy
Techniqu
e
Structure
sparse
error
coding

AR
Extended
Yale B

More accurate
when high level
occlusion and low
feature
dimensions

Not optimal

Extended
Sparse
Represen
tation
Based
Classifier

AR
FERET

more accurate
when number
of training
images
decreases

universal intraclass
variant dictionaries
for unconstrained
face recognition is
not optimal

FRS

2012
IEEE

More Efficient

Page 24

[Facial Recognition System

CE)]

Alpa Rupala (ME-

(ESRC)

Bindu A
C N Ravi
Kumar [13]

2013
IEEE

Inpaintin
g
Algorith
m

Own
Database

Kanokmon
Rujirakul,
Chakchai SoIn, Banchar
Anonkijpanic
h [7]

2014
IEEE

Weighted
Histogra
m
Equalize
d PEMPCA

Senthil
Jaffe
FEI

Sandesh V
Khamele,
Shyamal G
Mundada [4]

2015
ijarcce

INPAIN
TING
Fast
Weighted
Principal
compone
nt
analysis
(FWPCA
)

own
database

FRS

very effective
for
heavily
occluded
images
with
90% occlusion
higher
accuracy and
lower
complexity
compared to a
traditional
PCA
save
processing
time

Page 25

response
time
drastically reduced
by reduce in the
database size
Lower number of
cores yield more
computational time

For
Major
Occlusion requires
more computation

[Facial Recognition System

CE)]

Chapter 5

Alpa Rupala (ME-

PROPOSED WORK

5.1. Problem Statement

The authors Nguyen, Huu-Tuan, and Alice Caplier, they have done the LPOG
technique on the Face identification, but they stated that this work can be extended to
the verification purpose, facial expression recognition purpose like other face related
application. And they have done this experiment on aligned images, so the result can
be varied on misaligned images which can be obtained by any automatic face
detection algorithm like voila-Jones or Haar-detector.

5.2. Proposed Flow Work

I would like to extend Nguyen et al. [1] work on face verification with the use of
automatic face detection algorithm named voila Jones and combination of OLPP and
PCA for dimensionality reduction and for classification Structure Sparse Error coding
(SSEC) can be used.

Fig 5.2.1 Proposed Flow Chart

FRS

Page 26

[Facial Recognition System

CE)]

Alpa Rupala (ME-

In Future Plan, I would like to extend the LPOG technical on verification On JPEG
Occluded Image. In which Firstly the input image will be fetch on that image the
automatic face detection algorithm named Voila Jones will be applied for face
detection. In this detected face the firstly the occlusion should be detected and that
will be done by the skin color segmentation technique which is basically used to find
the component of body part by their color segmentation. Then the reconstruction of
main test image will be done by the inpainting and FWPCA techniques. Now the test
image is ready for the next processor of verification.
On test image the feature extraction need to be done that will be done by the LPOG
technique which is the best than recent state-of-art techniques for feature extraction.
Then the OLPP technique will be used for the dimensionality reduction purpose. Then
the reduced dimensioned test image will be compared with the already had database
one by one for matching and verifying. And this comparison will be done by SSEC
technique.

FRS

Page 27

[Facial Recognition System

CE)]

Chapter 6

Alpa Rupala (ME-

TOOLS AND TECHNOLOGY

The implementation of proposed method will be done using MATLAB programming

language.
The basic step involved in executing our ideology is as follows:

Write code for in the MATLAB Script File

Run the file and check the OUTPUT of verified Face

6.1 Tools Used

MATLAB (R2013b) : MATLAB (matrix laboratory) is a multi-paradigm
provides numerical environment and fourth generation programming language
which is developed by Math Works. It allows matrix manipulations, plotting
of functions and data, implementation of algorithm, provide GUI, and
interfacing with programs written in other languages, including C, C++, Java,
FORTRAN and Python

FRS

Page 28

[Facial Recognition System

CE)]

Chapter 7

Alpa Rupala (ME-

CONCLUSIONS

The proposed Report briefly discussed a critical survey of existing literature on Face
Recognition System. Also, Different approaches of face recognition system with brief
description of all of them. Face recognition system is very important in our daily life.
Table provides the different researchers research on Face Recognition System with
some of the face recognition algorithms and techniques along with their advantage
and disadvantage in tabular form. As we know that the occlusion is very big challenge
in face recognition so the proposed flow chart make it more robust to recognition the
face when it is occluded using LPOG, OLPP techniques for verification.

FRS

Page 29

[Facial Recognition System

CE)]

Alpa Rupala (ME-

REFERENCES
[1] Nguyen, Huu-Tuan, and Alice Caplier, Local Patterns of
Gradients (LPOG)
for Face Recognition, Information Forensics and Security, IEEE Transactions
on (Volume:10 , Issue: 8 ), ISSN :1556-6013, INSPECAccessionNumber:
15232971, DOI: 10.1109/TIFS.2015.2426144 ,pp: 1739 - 1751 , IEEE, Aug. 2015

[2] Soldera, John, Carlos Alberto Ramirez Behaine, and Jacob Scharcanski,
Customized Orthogonal Locality Preserving Projections With Soft-Margin
Maximization for Face Recognition , Instrumentation and Measurement, IEEE
Transactions
on (Volume:64
, Issue:
9 ),
ISSN
:0018-9456,
DOI:10.1109/TIM.2015.2415012, pp: 2417 2426, IEEE, Sept. 2015
[3] Sheethal Rao K, Steven L Fernandes, Patel Haniben, Prajna, Pratheek,Rakshan V
Devadiga, Shivani Kottary, Surakshitha, Thilak P, Review on Various State of
Art Technique to Recognize Occluded Face Images, 2ND International
Conference On Electronics And Communication System, ISBN: 978-1-47997224-1, DOI: 10.1109/ECS.2015.7124977, pp: 595 601, IEEE, Feb. 2015

[4] Sandesh V Khamele, Shyamal G Mundada, An Approach for Restoring Occluded

Images for Face-Recognition, International Journal of Advanced Research in
Computer and Communication Engineering (Volume:4, Issue: 5), ISSN : 22781021, DOI: 10.17148/IJARCCE.2015.45122, pp: 571-575, IJARCCE , May 2015
[5] Jyoti Reddy, Rajesh Kumar Gupta, Dr. Mohan Awasthy , A Survey: Face
Recognition by Sparse Representation International Journal of Engineering and
Applied Sciences (Volume-2, Issue-2) ISSN: 2394-3661,pp: 27-29, IJEAS ,
February 2015

[6] Md. Mehedi Hasan, Md. Foisal Hossain, Facial Features Detection in Color
Images Based on Skin Color Segmentation, 3rd INTERNATIONAL
CONFERENCE ON INFORMATICS, ELECTRONICS & VISION ISSN : 9781-4799-5180-2/14, pp: 1-5, IEEE , 2014
[7] Kanokmon Rujirakul, Chakchai So-In and Banchar Anonkijpanich, Weighted
Histogram Equalized PEM-PCA Face Recognition, International Computer
Science and Engineering Conference (ICSEC), ISBN: 978-1-4799-4965-6, DOI:
10.1109/ICSEC.2014.6978185, pp: 144 150,IEEE, Aug. 2014

FRS

Page 30

[Facial Recognition System

CE)]

Alpa Rupala (ME-

[8] Deepali H. Shah1, Dr. J. S. Shah2 and Dr. Tejas V. Shah, The Exploration of
Face Recognition Techniques, International Journal of Application or Innovation
in Engineering & Management (Volume 3, Issue 2), ISSN 2319 4847, PP- 238246, IJAIEM, February 2014
[9] Ms. B. Saranya Bargavi, Ms C. Santhi, Global and Local Facial Feature
Extraction using Gabor Filters, International Journal of Science, Engineering and
Technology Research (Volume: 3 ,Issue: 4), ISSN: 2278 7798, pp: 1020-1023,
IJSETR, April 2014
[10] Kaushik R. Makwana, A Survey on Face Recognition Eigen face and PCA
method, International Journal of Advance Research in Computer Science and
Management Studies (Volume: 2, Issue: 2), ISSN: 2321-7782, pp: 287-291,
IJARCSMS , Feb.2014

[11] Xiao-Xin Li, Dao-Qing Dai, Xiao-Fei Zhang, and Chuan-Xian Ren,
Structured sparse error coding for face recognition with occlusion, Image
Processing, IEEE Transactions on (Volume:22 , Issue:5), ISSN :1057-7149,
INSPEC Accession Number:13370726, DOI: 10.1109/TIP.2013.2237920, pp:
1889-1900, IEEE, May 2013

[12] Kathryn Bonnen, Brendan F. Klare, Anil K. Jain, Component-Based

Representation in Automated Face Recognition, Information Forensics and
Security, IEEE Transactions on (Volume:8 , Issue: 1 ) , ISSN : 1556-6013, DOI:
10.1109/TIFS.2012.2226580, pp: 239 253, IEEE, Jan. 2013

[13] Bindu A, C N Ravi Kumar, Novel Inpainting Algorithm for Heavily

Occluded Face Reconstruction, Advances in Computing, Communications and
Informatics (ICACCI), 2013 International Conference on, ISBN: 978-1-47992432-5, DOI: 10.1109/ICACCI.2013.6637458, pp: 1822 1826, IEEE, Aug. 2013

[14] M. Parisa Beham, S. Mohamed Mansoor Roomi, A Review Of Face

Recognition Methods, International Journal of Pattern Recognition and Artificial
Intelligence (Volume:27, Issue: 4), DOI: 10.1142/S0218001413560053, pp:
1356005- (1 -35), IJPRA, July 2013

[15] Weihong Deng, Jiani Hu, and Jun Guo, Extended SRC: Undersampled Face
Recognition via Intraclass Variant Dictionary, Pattern Analysis and Machine
Intelligence, IEEE Transactions on (Volume:34 , Issue: 9 ), ISSN : 0162-8828,
DOI: 10.1109/TPAMI.2012.30, pp: 1864 1870, IEEE, Sept. 2012

FRS

Page 31

[Facial Recognition System

CE)]

Alpa Rupala (ME-

[16] W. Zhao, R. Chellappa, P. J. Phillips, And A. Rosenfeld, Face Recognition: A

Literature Survey, ACM Computing Surveys, Vol. 35, No. 4, 3,ISSN: ACM
0360-0300/03/1200-0399, pp. 399458,ACM, Dec. 2003

FRS

Page 32