FYP Thesis

Vision-Based Biometric Authentication
System
Padraic o hIarnain
B.E. Electronic and Computer Engineering Project Report
Project Supervisor: Mr. Frank Callaly
March 2007
Vision-Based Biometric Authentication System
ii
Declaration of Originality
I hereby declare that this thesis is my
original work except where stated.
______________
____________________
Date
Signature
Final Year Project, Dept. of Electronic Engineering, NUI Galway
iii
Abstract
PC security is a big issue in the world today. Password and PIN authentication systems are outdated and
insecure. Recent advances in Biometric technology pave the way for a new vision-based type of
authentication system that is more secure and easier to use.
The aim of this project was to design and develop a vision based biometric authentication system for PCs.
This system would enable users to sit down at a PC and be authenticated by their facial features. This face
recognition authentication system involves capturing images using a webcam, processing the captured
images and detecting the presence of a face. This face is then analysed using Principal Component
Analysis (PCA) in order to compare the face to known users. Verifying a face as a user allows that user to
be logged on to the PC. This is achieved by integrating the face recognition program with the PCs
authentication system.
iv
Acknowledgements
I would firstly like to thank my supervisor, Mr Frank Callaly for his help throughout the course of the
project. I would also like to thank Mr. Martin Burke for his help and thanks to all my friends and
classmates for their assistance.
Table of Contents
Declaration of Originality...........................................................................................................................ii
Abstract......................................................................................................................................................iii
Acknowledgements....................................................................................................................................iv
Table of Contents.........................................................................................................................................v
List of Figures...........................................................................................................................................vii
Introduction.................................................................................................................................................1
Chapter 1 Software and Tools......................................................................................................................7
1.1 Linux.................................................................................................................................................7
1.1.1 Overview....................................................................................................................................7
1.1.2 Why Linux?..............................................................................................................................10
1.1.3 Red Hat Fedora Core.............................................................................................................10
1.2 GNOME..........................................................................................................................................10
1.3 GCC (GNU Compiler Collection)...................................................................................................12
1.4 C Programming Language...............................................................................................................13
1.4.1 Overview..................................................................................................................................13
1.4.2 Characteristics of C...................................................................................................................13
1.5 OpenCV...........................................................................................................................................14
1.5.1 Overview..................................................................................................................................14
1.5.2 OpenCV contents......................................................................................................................15
1.6 Web camera.....................................................................................................................................16
Chapter 2 Face Detection...........................................................................................................................18
2.1 Method of Face Detection................................................................................................................18
2.1.1 ViolaJones Algorithm for Face Detection...............................................................................18
2.1.2 Face Detection Program............................................................................................................20
2.1.3 Face Detection with a USB camera...........................................................................................22
2.2 Testing and implementation.............................................................................................................23
Chapter 3 Authentication...........................................................................................................................26
3.1 PAM.................................................................................................................................................26
3.1.1 Configuring an application for PAM.........................................................................................26
3.1.2 Creating a PAM module............................................................................................................28
3.2 Implementing Biometric programs with Authentication..................................................................30
3.3 Integrating PAM with Biometric system..........................................................................................30
3.4 Testing and Implementation.............................................................................................................31
vi
Chapter 4 Face recognition........................................................................................................................34

4.1 PCA.................................................................................................................................................34
4.1.1 Why use PCA?..........................................................................................................................34
4.1.2 Theory of PCA..........................................................................................................................35
4.1.3 Mathematics of PCA.................................................................................................................36
4.2 PCA using OpenCV.........................................................................................................................38
4.2.1 Eigenvector computation..........................................................................................................38
4.2.2 Calculating decomposition coefficients....................................................................................40
4.2.3 Calculating the Euclidean distance...........................................................................................40
4.3 Testing and Implementation.............................................................................................................41
Discussion.................................................................................................................................................45
Conclusion.................................................................................................................................................48
Appendix...................................................................................................................................................49
References.................................................................................................................................................50
vii
List of Figures
Figure 0-1 : Example of face recognition [4]...............................................................................................2
Figure 0-2 : Biometric Market Technology [7]............................................................................................3
Figure 0-3 : Overall Project Structure..........................................................................................................5
Figure 1-1 : Linux Logo..............................................................................................................................7
Figure 1-2 : CLI being used from a terminal...............................................................................................8
Figure 1-3 : Fedora logo............................................................................................................................10
Figure 1-4 : GNOME Logo.......................................................................................................................11
Figure 1-5 : GNOME Desktop environment..............................................................................................11
Figure 1-6 : GNOME login screen.............................................................................................................12
Figure 1-7 : The C programming specification..........................................................................................13
Figure 1-8 : Labtec Quickcam...................................................................................................................17
Figure 2-1 : 14 different Haar-like features [18]........................................................................................19
Figure 2-2 : Formula for Summed Area Table [18]....................................................................................19
Figure 2-3 : Two node classifier [18].........................................................................................................19
Figure 2-4 : Three node classifier [18].......................................................................................................20
Figure 2-5 : Result of face detection in an image......................................................................................23
Figure 2-6 : Result of face detection from a camera..................................................................................24
Figure 2-7 : Improved face detection result...............................................................................................25
Figure 4-1 : Results of face recognition.....................................................................................................42
Figure 4-2 : Another result of face recognition..........................................................................................43
Figure 4-3 : Result of face recognition using a camera..............................................................................43
Figure 4-4 : Result of face recognition using a camera..............................................................................44
Introduction
Biometrics consist of technologies that support automatic identification or verification of identity based
on behavioral or physical traits [3]. Biometrics validates identities by calculating unique individual
characteristics. The most predominant areas of biometrics involve fingerprints, iris and facial
characteristics, hand geometry, retina, voice and touch. Over the last twenty years or so there has not
been a significant market for Biometric technologies. They have been consigned to infrequent use in films
and in some high-security government or military installations. Nowadays, Biometrics is increasing its
stronghold in many aspects of both public and private life. For example, in some cases in the computer
industry, Biometrics is replacing the more conventional personal identification numbers (PIN) and
passwords. Although Password/Pin systems and Token systems are still the most common person
verification and identification methods, trouble with forgery, theft and lapses in users memory pose a
very real threat to high security environments which are now turning to biometric technologies to
alleviate this potentially dangerous threat. The areas in which Biometrics are gaining the most support is
in the protection of restricted areas, both commercial and domestic.
This project is based on the face recognition method of Biometric authentication. Although other
Biometric methods such as fingerprints and iris scans are more accurate methods of identification; facial
recognition provides an ingrained human backup because we instinctively recognize one another. Face
recognition is the process in which the facial features of someone are recognized and then matched to one
of the many faces in a database. The intuitive way to do face recognition is to look at the major features
of the face and compare them to the same features on other faces.[2].
The history of face recognition dates back to the 1960s when a semi-automated method was used to
compare facial features. First the key features in the photograph were marked by hand; key features
included eyes, ears, nose and mouth. Then the distances and ratios between these marks and a common
reference point were computed and these values were then compared to reference data of other faces. In
the early 1970s Goldstein, Harmon and Lesk [2] created a face recognition system using 21 particular
markers e.g. hair color and lip thickness. This method was less computerized then the previous method
because many of the measurements had to be made entirely by hand.
The next step in face recognition was made by Fisher and Elshlagerb [2] in the early 1970s. They
measured the key features in a face using templates of the features of the different parts of the face. They
then plotted all the pieces on to a general template. Even though this method was more automated then
the previous it proved to be too inconclusive as the features used did not include enough distinctive data
to represent a face.
Kirby and Sirovich pioneered the eigenface approach in 1988 at Brown University. It was the first
genuinely successful system for automatic recognition of human faces. The system functions by
projecting face images onto a feature space that projects the significant variations among known face
images. These significant features are known as Eigenfaces. This eigenface approach is used in this
project.
Fingerprints, hand geometry, iris scans, DNA analysis and to some extent personal signatures are all
biometric identifiers. But the only one that does not delay or interfere with access is face recognition .
Humans identify others by their face and voice and therefore are likely to be more comfortable with a
system that uses face and voice recognition.
Figure 0-1 : Example of face recognition [4]
This makes face recognition ideal for high traffic areas which are open to the general public for e.g.
airports and railway stations, ATMs, public transportation and businesses of all kinds. Face recognition
gives a record of who was there. Since the record is stored in a database, known persons can be detected
automatically and unknown persons checked quickly.
Concerns following recent terror attacks and constant threats to safety have created a pressing need for
advanced security. As a result, merchants of sophisticated face recognition biometrics solutions have
found their products very much in demand.
According to the World Face Recognition Biometrics Market, the face recognition
market earned
revenues of $186 million in 2005 and is likely to grow at a compound annual growth rate (CAGR) of 27.5
percent to reach $1021.1 million in 2012 [6].
Figure 0-2 : Biometric Market Technology [7]
However, to compete effectively with alternative biometric technologies, price reduction of face
biometric solutions is critical. A major competitor that matches the face biometric system in terms of
accuracy and performance is the non-automated fingerprint identification systems (non-AFIS). A
reluctance to lower prices may hamper the adoption of face recognition biometrics.
The purpose of this project is to create a vision based biometric authentication system for PCs.
Nowadays, most PCs use a password based authentication system to determine access rights. The reason
for creating a vision based authentication system is because the password based system:
Less secure Anybody may enter anybody elses password
Remembering passwords This may become a problem when a user is accessing a number of
different systems
Tedious A user has to enter his/her passwords every time the workstation needs to be locked
Because of the advances in image processing techniques, particularly in the areas of face detection and
face recognition, coupled with the low cost of digital imaging hardware, make a vision based
authentication system quite practical.
The aim of this project is to build a face recognition authentication system that:
Retrieves images from a camera in real-time
Detects the presence of a face in the image
Identifies the face against some enrolled images
Through integration with the PCs authentication system, logs on the user corresponding to the
identified face
Figure 0-3 : Overall Project Structure
The initial plan for this project was:
Get a face detection program up and running
Implement that program with the authentication system to create an initial working demo
Get a face recognition program working
Detect a users face with the face recognition program and log the user on
Add a utility to add new users to the database of authorized faces
Integrate the above programs into a fully functional face recognition authentication system
During the course of the year this plan was updated to include additional goals with more specific
requirements e.g. as well as writing the code to detect a face, the face detection system required the
installation of Linux, OpenCV and a webcam driver.
In the first chapter, a background and detailed description will be provided on the various software,
hardware and tools used to implement the many steps of the project. In chapter two, the face detection
methodology will be presented, and excerpts of the face detection code will be explained. We will
conclude chapter two with a brief summary on the testing and implementation of the face detection
process. Chapter three is concerned with Authentication. A brief discussion on PAM is followed by
biometric implementation with PAM. Chapter three will conclude with a summary on the testing and
implementation of the authentication process. In Chapter four, the Principal Component Analysis (PCA)
method of face recognition is introduced and we will discuss why the PCA method was chosen, the theory
of face recognition using the PCA method and how it is used in this system. Also we will describe the
mathematics behind face recognition and outline the relevant formulae. This chapter will close with a
description of the tests and results. This project will finish with a discussion on the results and a
conclusion driven from these results.
Chapter 1
Software and Tools
1.1 Linux
1.1.1 Overview
Linux is a Unix-like computer operating system. It is an exceptional example of open source development
and of free software. Linuxs core source code is available for anyone to use, alter and redistribute freely.
The first Linux systems were created in 1992 when system utilities and libraries from the GNU project
were joined with the Linux kernel. A few years later Linux was gaining the support of big corporations
such as IBM, Hewlett-Packard and Sun Microsystems [8]. Now Linux is used as an operating system for
a wider variety of computer hardware than any other operating system. Primarily known for its use in
servers, it is also used in such hardware as desktop computers, mainframes, supercomputers and
embedded devices such as cell phones.
Figure 1-4 : Linux Logo
A Linux distribution is a package containing the Linux kernel and a variety of other Linux based software
packages. They initially make possible the installation and configuration of the Linux operating system. A
distribution is also responsible for system security and the assimilation of thousands of different software
packages into a coherent whole. A general purpose distribution usually includes: a boot loader (e.g.
GRUB), the Linux kernel, GNU Compiler Collection (GCC), GNU C library, Desktop Environment (e.g.
GNOME), X window system display and networking protocol, a GNU bash shell and thousands of other
software applications such as text editing, media players, office suites, scientific programs and web
servers [8]. There are many distributions available for download; the most commonly know ones are:
Gentoo
Ubuntu
Sus
Fedora Core
One of the main components of a Linux system is its command line interface (CLI). It is possible to
completely run a Linux system from the CLI as it provides accessibility to all of the underlying source
code. Some distributions use the CLI as their only interface, thus increasing security and reducing system
resource consumption. The CLI is universal in the Linux world, it does not change from machine to
machine or from distribution to distribution, which is why it is the preferred interface among
programmers and Linux power users. The CLI was used continually throughout the project because it was
the easiest way to:
Create, move and copy files and folders.
Install applications.
Execute programs.
Change file/folder permissions
The CLI also helped a lot in the debugging of program code. The standard output of a program could be
redirected to a text file for later observation. It was especially useful in the debugging of the face
detection program. One terminal was used to run the program and another terminal was used to
continually concatenate the output file. So while the program was running it was possible to check if it
was working.
Figure 1-5 : CLI being used from a terminal
Nowadays most Linux distributions come with a desktop environment; They combine a window
manager with a suite of standard applications that adhere to human interface guidelines [8]. The two
most predominant desktop environments are KDE and GNOME which were released in 1996 and 1997
respectively. Later in the report the GNOME desktop environment will be described in detail.
The number of Linux users is greatly increasing every year and because of that the number of high quality
applications being developed for the Linux operating system is also increasing. Here are some of the
major software applications available:
Mozilla Firefox Internet
VLC player Multimedia
OpenOffice Office suite
The GIMP Graphics
Linux still lacks in the kind of application support provided by the Windows operating system but
projects such as WINE [9] are trying to remedy this by providing a way to run Windows compatible
applications on a Linux system.
Most Linux distributions support a lot of different programming languages. Here are some of the
languages Linux supports:
C++
Java
Python
Fortran
Perl
Ruby
The bulk of the programming utilities are found within the GNU tool chain, including the GNU Compiler
Collection (GCC) and the GNU Build System. GCC provides compilers for C, C++, Java, FORTRAN and
10
other less common languages. Core system software such as libraries and utilities are usually written in C
and compiled with GCC. The Linux kernel itself is built the same way. Later in the report, the C language
and GCC are described in greater detail as they are a major part in this project.
1.1.2 Why Linux?
In order for this project to work, it required the underlying source code to be accessible and editable to the
user. The programs included in this project needed to be integrating with the system core which is not
possible using the Windows operating system but is possible with Linux because it is an open source
development. Linux also provided the necessary programming language compiler and application
support. For those reasons Linux was the best choice of operating system for this project.
1.1.3 Red Hat Fedora Core
Red Hat Linux was a popular Linux distribution first released in 1994. It was the first Linux distribution
to use RPM as its packaging format. Since 2003 it has been discontinued in favor of its new Red Hat
Enterprise Linux for enterprise environments and Fedora Core for the free version.
Figure 1-6 : Fedora logo
Fedora Core is a complete, general purpose operating system that contains only free and open source
software. It is available for distribution [10] in a five-CD set or on a single DVD. That includes the
requirements for basic installation and a lot of optional applications, such as office, programming and
graphics applications. There are also several software repositories available online containing thousands
of additional software packages [11].
Fedora Core 5 was the release used for this project. It includes the GNOME desktop environment, the
GRUB boot loader (gives the option to boot different operating systems) and GCC; all of which are
required for this project.
11
1.2 GNOME
The GNOME project is an easy to use computing platform built entirely from free software. It is a
desktop environment that includes programs necessary for application launching, window management,
file handling and task management. According to the GNOME website [12]: The GNOME project
provides two things: The GNOME desktop environment, an intuitive and attractive desktop for users, and
the GNOME development platform, an extensive framework for building applications that integrate into
the rest of the desktop.
Figure 1-7 : GNOME Logo
GNOME also includes a lot of useful applications such as:
Nautilus a file manager
Totem A music player
GNOME Terminal a terminal emulator
gedit a text editor
12
Figure 1-8 : GNOME Desktop environment
Included in the GNOME desktop environment are the GNOME login screen and GNOME screensaver;
both of which are adapted to work in the authentication part of this project.
Figure 1-9 : GNOME login screen
1.3 GCC (GNU Compiler Collection)

The GCC is a set of free software, programming language compilers produced by the GNU project. It is
the standard compiler for all Linux operating systems. It was first released in 1987 when it only handled
13
the C programming language. Now it handles many programming languages including: C, C++, Java,
Pascal, and FORTRAN. Today, GCC has been ported to more kinds of operating systems and processors
than any other compiler.
In a Linux system, users invoke a driver program named gcc, which interprets command arguments,
decides which language compilers to use for each input file, runs the assembler on their output, and then
possibly runs the linker to produce a complete executable binary [13]. Each of the programming
language compilers is a separate program that is called using the gcc command.
In this project the GCC compiler was used to compile all the source code for the C programs which
included the face detection, face recognition and PAM programs. The face detection/recognition programs
were compiled from the command line using the g++ command (pretty much the same thing as the gcc
command). The PAM modules were compiled with the gcc command but were implemented using a
makefile.
1.4 C Programming Language

1.4.1 Overview
C is a general purpose, procedural, imperative computer programming language developed in 1972 by
Dennis Ritchie at the Bell Telephone Laboratories for use with the UNIX operating system [14]. Now it
has extended to a lot of other platforms and is one of the most commonly used programming languages. C
is the programming language used most in the writing of system software and is also widely used in the
writing of applications.
Figure 1-10 : The C programming specification
14
1.4.2 Characteristics of C
Here are some of the more important characteristics of the C programming language:
Has facilities for structured programming
Allows lexical variable scope and recursion
Static type system prevents many unintended operations
C has many keywords:

o
Declaring data types using struct, union or enum.
Declaring variables using char, int or double.
Control flow statements used for conditional or iterative execution such as if, else,
do, while and for.
Arbitrary jumps are possible with goto.
Parameters of C functions are passed by value
Pass-by-reference is achieved by explicitly passing pointer values
The struct data type allows associated data elements to be linked and controlled as a unit
Low level admission to computer memory via machine addresses and typed pointers
Intricate functionality such as mathematical and I/O functions included in library routines
Pointers allows an object or function to be referenced and recorded in memory by their address or
location
Source text is free-format, using semicolon as a statement stopper
All the programs in this project were written in the C programming. In order for the face
detection/recognition programs to compile they needed to be written in C because the OpenCV libraries
were compiled in C. The PAM modules written for this project were also written in C because PAM
modules, implementing the PAM libraries, will not compile in any other language.
1.5 OpenCV
1.5.1 Overview
OpenCV stands for an open source computer vision library. Originally developed by Intel, it runs on both
Windows and Linux and is focused mainly towards real-time image processing. Other areas that the
library covers are:
Object Identification
Face recognition
Segmentation and recognition
Motion understanding and tracking
Mobile robotics
Ego-motion
Gesture recognition
Structure from motion (SFM)
Human-computer interface (HCI)
15
OpenCV was downloaded from the sourceforge OpenCV project page [15].
1.5.2 OpenCV contents
In order to deal with all sorts of image processing applications, there are many functions available in the
OpenCV library. Here are some of the basic functions
Basic Structures
Operations on arrays copying, filing, arithmetic, linear algebra, math functions, discrete
transforms
Dynamic structures memory storages, sets, trees
Drawing functions Curves and shapes, text, points and contours
Data persistence file storage, writing data, reading data
Error handling and system functions
All of these basic functions are used through out the face detection/recognition code.
As well as the basic functions listed above, OpenCV has many high level functions that deal with specific
applications in image processing. Here are some of the most commonly used high level functions:
Image processing
o
Gradients, edges and corners
Filters and colour conversion
Special image transforms
Histograms
Matching
Sampling, interpolation and geometric transforms
Structural analysis
o
Contour processing
Computational geometry
Planar subdivisions
Pattern recognition
o
Object detection
Camera Calibration
The process of face detection in this project is carried out by a combination of functions. The keys ones
are related to pattern recognition and image processing.
Then there are experimental functions also included in the OpenCV libraries:
3D tracking functions
o
3D tracker calibrate camera
3D tracker locate objects
Eigen objects (PCA) functions

o
Calculate covariance matrix
Calculate eigen objects and values
Calculate decomposite coefficients
Projection of eigen vectors in eigen space
Embedded hidden markov models functions
The functions used in this projects face recognition are the Eigen objects or PCA functions.
16
17
1.6 Web camera

The face detection process in this biometric authentication system required the use of a USB web camera.
When a camera was being selected for this project there were a few details that had to be taken into
consideration first:
Camera resolution How accurate must it be for face recognition?
Size If this camera is to be used in a biometric authentication system for a PC then it must not
be obstructive or awkward.
Price If this authentication system were to be implemented on a large scale system with many
PCs then the price must be considered. The system would be more practical if the camera costs
were low.
Taking these details into account, the camera obtained for this project was a Labtec Quickcam.
Figure 1-11 : Labtec Quickcam
The reasons for selecting this particular camera were:
Resolution: Considering that the camera will be close to the users face, the resolution would not
need to be excellent in order for face detection or face recognition to work, so this camera was
picked because it has good resolution.
Size: This camera is very small and can easily be placed on top of the PC monitor.
18
Price: The normal retail price for this model of camera is 20. Since the rest of the tools used in
this project, software and hardware, are free; the total cost of implementing this improved
authentication system would only be 20 per PC.
Availability: Frank Callaly, project supervisor, had one of these cameras already in his possession
so it was borrowed from him for the duration of the project.
After obtaining the camera, a driver had to be installed in order for the Linux system to recognise it as
USB camera device. A universal camera driver, called the gspca5xx driver, was recommended by the
project supervisor. This driver, specifically designed for Linux, is able to work on a large number of USB
cameras and was downloaded from the spca5xx site [16]. There was a complication with the gspca.c
source file when the driver was being installed. The source file was trying to locate a higher version of the
Linux kernel then what was installed on the PC. The gspca.c file was edited to include the kernel version
2.6.15-1.2054_FC5.
19
Chapter 2
Face Detection
Face detection is a process that determines the location of a human face in a digital image. In this project,
face detection is the first step. It involves capturing images in real-time from a camera and then
determining whether or not the image contains facial features. If a face is detected in an image it is then
highlighted in the image and passed on to face recognition.
2.1 Method of Face Detection

In this project, The OpenCV library is used in the detection of faces. This library uses the statistical
approach originally developed by Paul Viola and Michael Jones [17] for face object detection. This
algorithm is implemented in this project using a modified face detection program contained in the library.
2.1.1 ViolaJones Algorithm for Face Detection
The Viola-Jones method of face detection uses simple Haar-like features and a cascade of boosted tree
classifiers. The Haar-like features are computed for the input image and then these features are passed
through a cascade of classifiers in order to determine if they are facial features.
The features are called Haar-like features because they are calculated similar to the coefficients in the
Haar wavelet transform. The Haar wavelet is the first ever wavelet and is also the simplest. The detected
Haar-like features are described by their shape, their coordinates and their size. The shape of the feature is
described by a certain template. Figure below shows a set of fourteen Haar-like templates.
20
Figure 2-12 : 14 different Haar-like features [18]
But a face, because of its complexity, cannot be classified by such a small number of templates. In real
classifiers, hundreds of features are used. Using hundreds of features is indeed more accurate but the time
taken to compute the pixel sums of a lot small rectangles would slow down detection. Paul Viola
introduced a method to speed up these computations. An Integral Image, Summed Area Table (SAT), is
first computed for the whole image, where
Figure 2-13 : Formula for Summed Area Table [18]
Using SAT a given rectangles pixel sum can be computed by using only its corners.
The computed feature value is then passed through a simple classifier that might have two or three nodes.
Figure 2-14 : Two node classifier [18]
21
Figure 2-15 : Three node classifier [18]

The classifier will respond with either a +1, meaning the feature passed or with a -1 meaning the feature
does not satisfy the criteria of the classifier. Such a classifier is known as a weak classifier, it is not used
to detect a face but rather to distinguish if the image contains a feature that may relate to a face. E.g. the
Haar-like feature 3(a) displayed in figure 2-1 will respond to an eye once properly positioned and scaled.
The next step is to chain a number of weak classifiers together into a more complex classifier know as a
boosted classifier. In this chain of weak classifiers, the ones with the smallest return of error are given
more weight in the boosted classifier. But for this boosted classifier to give a high success rate and small
false alarm rate, it would require a very large number of weak classifiers, resulting in slow execution.
Viola has instead built a cascade of boosted classifiers. The simpler boosted classifiers are first in line
because they are more effective at eliminating non-face candidates, therefore most of the detection time is
spent on real faces. A face is only detected when a candidate passes through all classifiers without being
rejected. The cascade of boosted classifiers used in this project was trained using hundred of frontal face
images and is supplied with the OpenCV library.
2.1.2 Face Detection Program

Apart from installing software and tools, face detection was the first step in this project. The aim was to
write a program to process images from a camera in real-time and then detect if any face objects are
present in that image. This program was written using the OpenCV libraries.
The face detection program is able to read in from several different outputs an image, a file containing
the names of images, an AVI file or from a USB camera. The initial goal was to get the face detection
program running with a simple image containing a face. The important parts of the face detection code is
displayed and explained below.
When running the program, the user must pass the classifier location and the input image. An example of
the command used to run the program from the command line is:
22
./facedetect --cascade= "/usr/src/redhat/SOURCES/opencv1.0.0/data/haarcascades/haarcascade_frontalface_alt2.xml" input.jpg
The following code in facedetect.c reads in the classifier name and the image name from the input and
loads it:
if( argc > 1 && strncmp( argv[1], "--cascade=", optlen ) == 0 )
{
cascade_name = argv[1] + optlen;
input_name = argc > 2 ? argv[2] : 0;
}
cascade = (CvHaarClassifierCascade*)cvLoad( cascade_name, 0, 0, 0 );
The input image is then passed into the face detection function, converted from a colour image to a
greyscale image and then resized as a smaller image. These steps are taken to make the Haar-like features
easier to identify.
cvCvtColor( img, gray, CV_BGR2GRAY );
cvResize( gray, small_img, CV_INTER_LINEAR );
cvEqualizeHist( small_img, small_img );
The new image is then checked for face objects; if a face object is detected it is stored as a rectangular
object, the coordinates for which are then used to draw a circle around the detected face. It is then
displayed in a window:
CvSeq* faces = cvHaarDetectObjects( small_img, cascade, storage,
1.2, 2, 0/*CV_HAAR_DO_CANNY_PRUNING*/, cvSize(40, 40));
for( i = 0; i < (faces ? faces->total : 0); i++ )
{
CvRect* r = (CvRect*)cvGetSeqElem( faces, i );
CvPoint center;
int radius;
center.x = cvRound((r->x + r->width*0.5)*scale);
center.y = cvRound((r->y + r->height*0.5)*scale);
radius = cvRound((r->width + r->height)*0.25*scale);
cvCircle( img, center, radius, colors[i%8], 3, 8, 0 );
23
}
cvNamedWindow( "result", 0 );
cvShowImage( "result", img );
The cvHaarDetectObjects function is the most important function in the face detection code. It
basically does all the work explained earlier in Viola-Jones algorithm [17]. Here are the function
arguments details in order to better understand what it is doing. Arguments listed in order:
1. small_img : this is the reduced grayscale image used as the input to the function.
2. cascade : the cascade of boosted classifiers is located at this location.
3. storage : temporary memory storage.
4. 1.2 : scaling the cascade by 20% each pass.
5. 2 : groups of at least 3 (2+1) neighbor face rectangles are grouped into a single face, smaller
groups are rejected.
6. 0 : passing no arguments here. Passing CV_HAAR_DO_CANNY_PRUNING into this argument
will speed up detection by reducing number of false alarms but at the cost of accurate detection.
7. cvSize(40, 40) : minimum face size allowed by classifier.
2.1.3 Face Detection with a USB camera

The face detection program is also designed to read in images from a camera. If the user doesnt specify
an input to the program then the program checks if a USB camera device is connected.
if( !input_name || (isdigit(input_name[0]) && input_name[1] == '\0') )
capture = cvCaptureFromCAM( !input_name ? 0 : input_name[0] - '0' );
The feed from the camera is stored in the CvCapture variable capture. Using OpenCVs frame
manipulation, a frame is captured from the camera and transferred to an image. The code then checks if
that images origin (top left corner) is in the right place (and corrects it if its not) and copies the image to
new image. This new image is then used as the input for the detect_and_draw function.
if( !cvGrabFrame( capture ))
break;
frame = cvRetrieveFrame( capture );
24
if( !frame )
break;
if( !frame_copy )
frame_copy = cvCreateImage( cvSize(frame->width,frame->height),
IPL_DEPTH_8U, frame->nChannels );
if( frame->origin == IPL_ORIGIN_TL )
cvCopy( frame, frame_copy, 0 );
else
cvFlip( frame, frame_copy, 0 );
detect_and_draw( frame_copy );
So the program captures frames from the camera in real-time and if there is a face object in the image
then it is circled and displayed. The average detection time is around 20ms which is more than fast
enough.
2.2 Testing and implementation

The usual rate of false alarms (detecting a face in an image where there is none) is very low. Therefore the
testing was not very rigorous. First the program was tested with different images e.g. jpegs. Below is an
example of one of these images:
Figure 2-16 : Result of face detection in an image
25
The program was tested using other faces and even images with no faces. In each case the face detection
program was successfully i.e. detected faces in facial images and rejected non-facial images. The results
of which are in the Appendix.
The program was then tested with the input from the camera. The program was run for 10 minutes at a
rate of 1 frame every 20ms without detecting any other object than the face of the user. So the program
processed approximately 30000 frames without a single error. A sample of the output is displayed below.
Figure 2-17 : Result of face detection from a camera
When the program was first run, there were issues with the quality of the camera input. Since the
attributes (brightness, contrast, hue, etc) of the camera input were not set in the program or in the
OpenCV configuration files the image was way too bright and blurry. The first approach to solve this
problem was to edit the configuration files in the OpenCV source folder. This proved to be a problem as
changing the values in the configuration also changed the size of the image, making it smaller. So instead,
the attribute values were passed manually from the program.
cvSetCaptureProperty(capture, CV_CAP_PROP_FRAME_HEIGHT, 240);
cvSetCaptureProperty(capture, CV_CAP_PROP_FRAME_WIDTH, 320);
cvSetCaptureProperty(capture, CV_CAP_PROP_BRIGHTNESS, 0.4);
cvSetCaptureProperty(capture, CV_CAP_PROP_CONTRAST, 0.08);
cvSetCaptureProperty(capture, CV_CAP_PROP_HUE, 0);
cvSetCaptureProperty(capture, CV_CAP_PROP_SATURATION, 0.2);
26
This greatly improved the quality of picture which in turn returned better results when the face was
passed onto face recognition.
Figure 2-18 : Improved face detection result
In conclusion, the face detection program was a complete success. The program detected faces in images
where there were faces with a high degree of accuracy.
27
Chapter 3
Authentication
The chapter describes the authentication process, precisely how it is done and how it is implemented with
the vision based biometric system. The first aim was to create a basic authentication system. The next step
was to gradually integrate it with the face detection program and then later with face recognition.
3.1 PAM
PAM or Pluggable Authentication Modules are a method to assimilate multiple low-level authentication
systems into high-level applications. PAM is based on a set of library modules located in the /lib/security
directory. Applications that rely on PAM have their PAM configuration files stored in /etc/pam.d/. There
are many applications that depend on PAM. In this project the ones that are looked at are the GNOME
login and screensaver applications. These applications request authentication from the PAM modules
listed in their PAM configuration files. In this chapter we discuss how to create and compile PAM
modules, how to configure application PAM settings and then how to integrate the biometric system with
PAM.
3.1.1 Configuring an application for PAM

Each application dependant on PAM has their matching configuration file located in /etc/pam.d/. In order
for that application to be authenticated it must call the modules listed in this file. An example of a PAM
configuration file is shown below.
#%PAM-1.0
# Fedora Core
auth
include
system-auth
account
include
system-auth
password
include
system-auth
session
include
system-auth
This configuration file is for the GNOME screensaver. The file is pointing to another configuration file
containing more lists of PAM modules. All configuration files come in the format:
Module-type
control-flag
module-path
28
arguments
Where,
Module-type refers to the type of PAM module

o
auth authenticates users, usually asking a password and then checking it.
account checks if account details are correct for authentication. E.g. has the account
expired or has the system resources reached maximum users.
password used to set passwords.
session makes it possible for the user to use their account once authenticated
Control flag describes the importance of module

o
required in order for the module-type to succeed this PAM module must return a
success.
requisite if the module fails, control is restored to the application.
sufficient if no previous required module has failed and this module succeeds, then no
more of this module-type is invoked.
optional this module is not crucial to the success or failure of the applications access.
This module will be invoked if all other modules have not returned successes.
Module path is the name of the module to be invoked.
Arguments only some modules use arguments. E.g. passing shadow to a password moduletype will invoke the use of shadow passwords [19].
As mentioned above, the two applications that had to be altered so they depend on face
detection/recognition are the login and screensaver applications. In order to do this their PAM
configuration files needed to be changed so they pointed at PAM modules designed specifically to rely on
the biometric programs. Here is what the new configuration files look like:
#%PAM-1.0 GNOME screensaver
auth
required
pam_face_screensaver.so
account
required
password
required
session
required
29
#%PAM-1.0 GNOME Desktop Manager login

auth
required
pam_env.so
auth
required
pam_face_login.so
account
required
pam_nologin.so
account
required
pam_face_login.so
password
required
pam_face_login.so
session
required
pam_face_login.so
session
required
pam_loginuid.so
session
optional
pam_console.so
Now when GNOME runs the login application or brings up the screensaver, they will only be
authenticated using the biometric system.
3.1.2 Creating a PAM module

A PAM module needed to be created that relied on the output being produced by the biometric programs.
The solution for this was easy. Create a PAM module that read a user name in from a file. Since one of the
initial aims of this project was to authenticate a user by just detecting a face, therefore the face detection
program could be modified to write a default users name to a file every time a face was detected and as a
result authenticating that user. Reading a user name from a file would also eventually work with face
recognition because the recognition program could output the user name of a face that has been
recognised to be in the user database.
The two applications, login and screensaver, need two different types of authentication:
Login requires the module to read in a user name from a file and then log that user on. If there is
no name in the file, it will fail. If the user name is wrong, it will fail.
Screensaver requires that the module reads a user name in from a file, checks if that name is the
same as the current user, then grants access if it is and returns a failure if it doesnt.
For each of these tasks a separate PAM module was made although they are very similar.
30
Here are excerpts from the pam_face_screensaver.so module. First a PAM function gets the current user
name. If there is an error retrieving it then it will return an error. This should not happen because the user
is already logged on.
retval = pam_get_user(pamh, &user, NULL);
if (retval != PAM_SUCCESS) {
D(("get user returned error: %s", pam_strerror(retval)));
return retval;
The next step is to transfer that user name to a temporary string. Then read in a string from the output file
of the face detection/recognition code, compare the two and return a success if theyre equal or return a
fail if theyre not.
sscanf(user, "%s", temp);
ifp = fopen("/lib/security/user.txt", "r");
fscanf(ifp, "%s", name);
if( *name == *temp )
{
return PAM_SUCCESS;
}
else
{
return PAM_AUTH_ERR;
}
For the login module, pam_face_login.so, only a few lines are changed from the screensaver module.
Instead of comparing two strings, the module tries to log the user read in from the output file of the
biometric programs.
ifp = fopen("/lib/security/user.txt", "r");
fscanf(ifp, "%s", name);
pam_set_item(pamh, PAM_USER, (const void *) name);
return PAM_SUCCESS;
31
The code returns PAM_SUCCESS no matter what the outcome is because if the inputted user name
doesnt exist, the authentication fails anyway. According to the GNOME login PAM configuration file the
application is not authenticated by pam_face_login.so alone. So it must also pass the other modules, e.g.
pam_loginuid.so will fail if the user name set to PAM doesnt exist in the database.
3.2 Implementing Biometric programs with Authentication

As mentioned above in previous chapter, the PAM modules depend on the output of the face
detection/recognition system. So the first step was to get the face detection program to write a default user
name to a file every time a face is detected (any face) and get that much working with the PAM modules.
The next step would be to write the user name retrieved from the face recognition function to the same
file to see if that would work with the PAM modules.
Writing a default user name to a file each time a face is detected was implemented by the following code:
if(r)
{
rewind(ofp);
fprintf(ofp, "pohiarnain");
}
The face recognition would print out the user name instead of pohiarnain after it has finished with its
face recognition.
fprintf(ofp, "%s", user_name);
3.3 Integrating PAM with Biometric system

Every time the GNOME screensaver or the login application runs its corresponding PAM module, the
output from the biometric program is read in. Because of this the biometric programs must commence on
start up and continue to run so the input to the PAM modules is constantly updated. In order to do this a
script was written to call the program in run level 5. This script was written and moved to /etc/inti.d/. In
the run level 5 directory, /etc/rc.d/rc5.d/ there is a list of all programs to be run at start up. The script to
32
run the biometric programs is included in here (see appendix A for script). The key statement in the script
is when it calls the program; the whole path name must be included:
start() {
echo -n $"Starting Face Detection"
/root/Desktop/OpenCV/facedetect --cascade=/usr/src/redhat/SOURCES/opencv1.0.0/data/haarcascades/haarcascade_frontalface_alt2.xml" &
echo_success
echo
}
Now the program is able to run indefinitely as long as the PC is on.
3.4 Testing and Implementation

While implementing and testing with PAM it was established that working with the login screen and
screensaver would be a problem because if the program failed then the user would have to restart the
computer, thus making it very time consuming to debug the program. So instead of experimenting with
the login or screensaver it was more efficient to use the switch user application. This application is run
solely from the command line so any errors occurring with the PAM module would just return an error to
the command line and not freeze the system. Once the module was completely debugged then porting it to
the other applications was easy.
Compiling the module was another issue because it had to be compiled in a way so it was recognised as a
PAM shared library (.so file). So first step was trying to get a standard PAM module to compile and run.
The module pam_permit.so was selected for this testing because its an easy to understand program with a
very simple output i.e. it returns a PAM_SUCCESS no matter what. The source code for all the standard
PAM modules was downloaded from the fedora rpm repository [11]. After researching alternative PAM
modules in hope to find a suitable Makefile to compile the module, a project was found called
pam_chroot which was downloaded from the pam_chroot sourceforge site [20]. This module is not
included in the standard PAM libraries so the source code is included with the download as well as the
Makefile. After editing the Makefile to compile pam_permit instead, it compiled the module without
error. Then after including the new pam_permit module in the switch user configuration file and moving
the module to where the modules are stored, /lib/security/, the switch user application ran without error
and always returned PAM_SUCCESS.
33
At this stage the appropriate application and Makefile were selected for testing the pam_face modules.
The main issues with debugging the pam_face modules were:
Comparing the constant value user with the string value name was returning segmentation
faults (segmentation faults occur at run-time, the error does not show up when compiling. These
faults are usually due to the program trying to access a memory location that it is not allowed to
access or when it is trying to access a memory location in the wrong way). The solution was to
copy the user value to a temporary string and then compare that to the name.
At first, the program was comparing the two pointer values and returning a segmentation fault.
This happened because of a simple error with the code syntax, the code was comparing the
pointer values not the values the pointers were referencing i.e. the strings.
The authentication module at first was written to return PAM_SUCCESS if the right conditions
were met. What if the wrong conditions occur? If there are conditions set in a PAM module, then
each condition should return a value. Due to this the module was failing at first because it didnt
return anything when the conditions were not met. Adding in another value, PAM_AUTH_ERR,
which would be returned if the conditions were not met, solved this issue.
The final stage in testing the Authentication process was to test if the biometric system would intialise on
start-up and then authenticate the user every time either the screensaver or login applications are run. The
first attempt to get the program running on start-up was done by adding it to the run level 5 script,
rc.local. This is the final script that is called during start-up; it basically runs any programs listed in the
file rc.local. This didnt work because there were errors due to permission issues. The next attempt
involved writing a new start-up script to initiate the program. This was carried out by using the simple
readahead script as a template and added the path name of the face detection/recognition program. The
first attempt to get this to operate worked but with one problem. The face detection/recognition program
was called during start-up but was not told to run in the background and since this program is designed to
run indefinitely, the PC never left the start-up stage. The Linux rescue CD had to be used to boot the PC
directly to the command line so the script could be changed to run the program in the background. This is
done by adding a & at the end of the program path.
So now the biometric system was constantly running in the background and continuously updating the
output text file user.txt. The PAM module for login was checking this file every time the login
application was implemented. At first the login application worked perfectly, logging in the user listed in
34
user.txt. So did the screensaver application, unlocking the screen when the current users name was the
same as the name listed in user.txt. But after 2 minutes the face detection/recognition program would
stop running. Since it was running in the background it was not returning any error data either. In order to
find out the error details the start-up script was edited to print out the output of the program to a
temporary text file as shown below.
cascade=/usr/src/redhat/SOURCES/opencv1.0.0/data/haarcascades/haarcascade_frontalface_alt2.xml" > /tmp/output.txt &
Having done this the program was now printing out the standard output information to this text file.
According to the file after two minutes of running the biometric program would shut down because there
was too much data being stored in some of the image variables. In order to remedy this, the image
variables
had
to
be
released
after
theyve
been
processed.
Adding
in
this
function,
cvReleaseImage(image_name), solved the problem. The biometric system was then run for around 5
hours straight to see if it could operate for very long periods of time. The results were good; the face
detection authentication system ran for hours with fail. I next tested it by putting different objects (book,
hand, bag, etc.) in front of the camera and see if it would log on. The system didnt detect any faces in
those cases so it didnt authenticate.
So in conclusion, the authentication system worked perfectly with the face detection system and would
eventually work with face recognition.
35
Chapter 4
Face recognition
The process of face recognition involves the examination of facial features in an image, recognizing those
features and matching them to one of the many faces in the database. There are many algorithms capable
of performing face recognition; such as:
Principal Component Analysis
Discrete Cosine Transform
3D recognition methods
Gabor Wavelets method
Hidden Markov Models
Kernel methods
The Principal Component Analysis (PCA) method of face recognition is used in this biometric
authentication system. In this chapter we will discuss why the PCA method was chosen, the theory of face
recognition using the PCA method and how it is used in this system.
4.1 PCA
4.1.1 Why use PCA?
There were many issues to consider when choosing a face recognition method. The keys ones were:
Accuracy
Time limitations
Process speed
Availability
With these in mind the PCA method of face recognition was selected for this project because:
36
Simplest and easiest method to implement due to project deadlines this method seemed the
most practical
Very fast computation time
Accurate this method is definitely not the most accurate of face recognition algorithms but
considering the requirements of this project it was judged to be accurate enough
PCA is supported within the OpenCV library this was key because it made integration with the
face detection program very easy
There are disadvantages with the PCA method but they were deemed to be inconsequential. PCA is:
Translation variant if the image is shifted or tilted then it will not recognise the face. This flaw
doesnt affect this system as the camera will always be located upright and in front of the user.
Scale variant scaling the images will affect the performance of face recognition. This was not a
big problem in this project as new users are added using the camera at the PC so the face to be
recognised and the faces in the database are captured in the same way; making them very similar
in size. Also the user will always be relatively close to the camera eliminating the possibility of a
blurry, out of focus face.
Background variant recognising a face with a different background is difficult. The face
detection program is altered so it not only detects a face but extracts it to a new image, thus
cutting out almost all the background. The residual sections of background are insignificant since
the cameras resolution will only keep the face in focus because of the distance.
Lighting variant if the light intensity changes then the accuracy of face recognition drops. This
authentication system involves capturing images in front of a PC and usually the room is lighted
when a user is using a PC.
Taking all this into consideration, the PCA method of face recognition was deemed to be the most
practical for this project.
4.1.2 Theory of PCA

Principal Component Analysis is a process that extracts the most relevant information contained in a face
and then tries to build a computational model that best describes it. The basic theory of PCA can be
described in the following steps:
37
1. Eigenvectors or eigenfaces of the covariance matrix are found. This is done by training a set of
face images.
2. These eigenvectors become the eigenspace (a multi-dimensional subspace comprised of the
Eigenvectors) in which every face is projected on.
3. Recognition is performed by comparing the location of a face in the eigenspace with the location
of known users. In other words calculating the Euclidean distance.
This is only the basic theory of PCA. In order to fully understand PCA it needs to be explained
mathematically which is done below.
4.1.3 Mathematics of PCA

The mathematics of PCA [21] describes how an image is broken down, how the eigenvectors are
calculated and eventually how images can be compared to each other by calculating the Euclidean
distance.
First a 2D face image is broken down into a 1D vector by concatenating each row into one long vector.
Say M equals the number of input images there are and N (= rows of image x columns of image) equals
the size of the images. Then an image can be represented by the vector:
[21]
where the pjs represent an individual pixel value. Before analysis these image vectors have to be mean
centred by subtracting the mean image from the image vector. Let m be the mean image and wi be the
mean centred image:
[21]
The next step is to find a set of eigenvectors that best describe the principal information in each of the
mean centred images. The set of eigenvectors (ei) with the highest eigenvalues (i) are the ones needed.
Where the eigenvalues are equal:
38
[21]
It has been proven that these eigenvectors and eigenvalues are obtained by getting the eigenvectors and
eigenvalues of the covariance matrix. The covariance matrix is given the equation:
[21]
W is the matrix composed of the mean centred image vectors (wi) and WT is its transpose. Trying to solve
the eigenvectors of C directly is not practical as the size of it is potentially enormous. E.g. a 100 x 100
image would create a mean centred image 10000 coefficients long, so the covariance matrix would be
10000 x 10000 in size. There is another way of calculating them using the M x M matrix WTW. Let di and
i be the eigenvectors and eigenvalues of WTW, respectively:
[21]
Then multiply both sides by W:
[21]
Now the first M - 1 eigenvectors (ei) of the covariance matrix (WWT) are represented by the normalised
eigenvectors Wdi. Eigenvalues (i) of the covariance matrix (WWT) are given by the eigenvalues i. The
covariance matrix can only be of rank M 1 because there is a finite amount of input image vectors and
the -1 comes from the initial subtraction of the mean image.
The eigenvectors corresponding to nonzero eigenvalues of the covariance matrix produce an
orthonormal basis for the subspace within which most image data can be represented with a small amount
of error. [21] So these eigenvectors contain the most relevant information from the set of input images
and are used to construct an eigenspace within which other images can be analysed. They are arranged
from high to low according to their matching eigenvalues. The reason for this is because the eigenvector
with the highest eigenvalue exhibits the greatest variance in the image.
39
A face image can then be projected onto this eigenspace and the location of the image in this subspace is
determined by the formula:
[21]
T
i
Where vi = e x wi and M is the number of dimensions (smaller than M). describes the input of each
eigenvector in the representation of the input face image. Face recognition can be implemented by
projecting an input face image () onto the eigenspace and then finding a face in the database (k) that is
closest to it. The distance between the faces in the subspace is called the Euclidean distance (k).
[21]
A threshold, , is predefined so if the Euclidean distance between an input face and a users face is
smaller than this threshold, it is recognised as a face belonging to that user.
4.2 PCA using OpenCV

Now that the theory and method of PCA in face recognition has been thoroughly explained it is time to
describe how it was implemented in this project. First the goals were listed:
Create an eigenspace using a set of training faces e.g. use the database of user faces as the
training set. This eigenspace is constructed by calculating the eigenvectors and eigenvalues of this
set.
Calculate the location of each user face in the eigenspace. This is done by calculating the
decomposition coefficients of each face in reference to the eigenvectors.
Calculate the location of the inputted face image in the eigenspace.
Determine the Euclidean distance between the input face and each one of the user faces. If one of
the user faces is close i.e. Euclidean distance under the threshold, then print that users name to
the output file.
OpenCV supports PCA and sample code for PCA was obtained from the yahoo OpenCV group website
[22]. The eigenvector functions available in OpenCV are able to perform all the goals listed above. The
implementation is described below.
40
4.2.1 Eigenvector computation

The OpenCV library has functions specifically designed to calculate the eigenvectors and eigenvalues of a
set of input images. The first one used is the function that Calculates orthonormal eigen basis and
averaged object for group of input objects [23].
void cvCalcEigenObjects( int nObjects, void* input, void* output,
int ioFlags, int ioBufSize, void* userData,
CvTermCriteria* calcLimit,IplImage* avg, float* eigVals );
Here is the description of each argument in order to better understand how the function works:
1. nObjects : number of input images to be used.
2. input : pointer to an array of input images.
3. output : pointer to an array eigen objects (vectors).
4. ioFlags : input/output flags.
5. ioBufSize : input/output buffer size in bytes.
6. userData : pointer to structure containing the data for callback functions.
7. calcLimit : criteria used to determine when to stop computation of eigen objects.
8. eigVals : pointer to array of eigenvalues in descending order.
Before this function was called there were a few variables that needed initializing. The number of input
objects was set to the number of users in the system. The eigenvectors are stored as an IplImage variable
which is the same as the input images. The criteria are also defined before the function is called. Here are
the constraints contained in the criteria:
criteria.type = CV_TERMCRIT_ITER|CV_TERMCRIT_EPS;
criteria.max_iter = 10;
criteria.epsilon = 0.1;
This shows that the criteria variable contains two constraints. The max iteration variable set at 10 means
that the maximum number of principal eigenvectors computed is 10. Epsilon set at 0.1 means that the
function will stop calculating eigenvectors when the current eigenvector contains 10 times less variance
then the eigenvector with the highest variance i.e. eigenvector with the highest eigenvalue. The
eigenvalues are initialized as a floating point variable.
41
Here is the implementation of the function in the face recognition code:

cvCalcEigenObjects( n, images, eigens, 0, 0, 0, &criteria, avg, vals );
This function returns the eigenvalues and the eigenvectors of the set of images.
4.2.2 Calculating decomposition coefficients
This function follows the cvCalcEigenObjects function. It calculates the decomposition coefficients of
each input image i.e. it determines where the input image is located in the eigenspace by expressing them
as coefficients of the eigenvectors.
void cvEigenDecomposite( IplImage* obj, int eigenvec_count, void* eigInput,
int ioFlags, void* userData, IplImage* avg, float* coeffs );
This functions arguments are described here:

1. obj : input object (only one input image may be processed at a time).
2. eigenvec_count : the number of eigenvectors used.
3. eigInput : pointer to an array of input eigen objects (outputted from the function in 4.2.1).
4. ioFlags : input/output flags.
5. userData : pointer to structure containing the data for callback functions.
6. avg : averaged object (outputted from the function in 4.2.1).
7. coeffs : pointer to an array of coefficient values.
This is how this function is employed in this project:
for( i=0; i<n; i++ )
{
cvEigenDecomposite( images[i], nEigens, eigens, 0, 0, avg, coeffs[i] );
}
Now the decomposition coefficients of each image are stored. Before these coefficients are used in the
next function they must be normalized so each vector coefficient is given equal importance.
42
4.2.3 Calculating the Euclidean distance

The normalized decomposition coefficients describe the location of a face in the eigenspace. In order to
calculate the Euclidean distance between one face and another, their corresponding coefficients are
compared e.g. coefficient 1 of image A is subtracted from coefficient 1 of image B. The difference
between each coefficient is added up. The overall value is equal to the distance between the faces in the
eigenspace. So in order to recognize a face, the distance between it and another face must be under a
certain threshold. If the distance is under that threshold then those faces are recognized to be from the
same person.
4.3 Testing and Implementation

The first step was to decide what images to use for a training set and how to load them into the face
recognition program. The following decisions were made:
The training set should be comprised of the faces of all the users. The probability of verifying a
users face greatly increases if that users face was included in the construction of the eigenspace.
The user database should contain 3 things:

o
A text file containing the current number of users in the database. This number will
decide how many images in the training set. Every time a new user is added the value in
this file will be incremented.
A directory containing the face images of all the users. The name of these face images
will be the same as their corresponding user names e.g. /lib/security/faces/pohiarnain.jpg.
A text file containing a list of names of all the users e.g. pohiarnain. This file would be
read in by the face recognition program and each name listed would point to its
corresponding face image. This would be done by adding the name string to the path of
the face image to be read in e.g. /lib/security/faces/%s.jpg, user_name.
Each time the face recognition function is called (face recognition program is in a loop, so face
recognition function is called every time a face is detected) the list of users will be read in and
loaded to the PCA functions. By doing it this way a new user can be added to the database and
loaded to the face recognition function right away. If the images were only read in at the start of
the program then the new user would have to wait until the PC reboots to log on.
The performance of the face recognition algorithm was improved by improving on the face detection
function. The background of the input image may interfere with PCA because it is background variant. So
43
the face detection program was altered to not only detect the face but set it as the region of interest (ROI)
in the image. Once a ROI is set it is the only part of the image that is processed. The ROI improvement
was implemented using the following code:
cvSetImageROI(other, cvRect(r->x, r->y, (r->x + r->width),
(r->y + r->height)));
A database of user faces had to be compiled so they can be inputted into the PCA functions. PCA is
background and lighting variant, therefore adding new user faces should be done with the camera to
improve the accuracy of the program (4.1.1). The user faces were captured using a modified face
detection program. This program would detect a face and save that face to the user database under the
name of the user e.g. pohiarnain.jpg. 6 faces were added to the database for testing purposes.
Most OpenCV functions will only operate correctly if the input and output images are the same size.
Therefore, as a precaution, all images are scaled to the same size before being analysed.
After the code was compiled and run successfully it was decided that the first test should be comparing
two identical images. The eigenvectors of 6 users would be calculated as the basis of an eigenspace. Then
the location of two identical images in the eigenspace would be calculated and compared. Therefore, 6
faces were loaded from the face database and equivalent eigenvectors computed. Then a 7 th image is
added to the list and the coefficients of all images are calculated. The Euclidean distance is then
calculated from the 7th image to each one of the other images. The results are as follows:
44
Figure 4-19 : Results of face recognition
Figure 4-20 : Another result of face recognition
The two figures above are examples of the results. The small screen on the left is the face in front of the
camera. The small black and white screen is the projection of the image onto the eigenspace. The
conclusive results are displayed on the command line. As expected the distance between two identical
images is zero. These results show that this face recognition algorithm is working.
The next step was to try and recognize a face detected using the camera. The only change to the above
method is that the 7th image added to the list is the input from the camera. The program is executed and
the distances between the input face and the user faces are calculated. Here are samples of the results:
45
Figure 4-21 : Result of face recognition using a camera
Figure 4-22 : Result of face recognition using a camera
In figure 4-3 the distance between the face in the camera and its corresponding user in the database
(vector 0) is less than 0.5 which is very small comparing to the other distances. Also in figure 4-4 the
distance between the input and its corresponding user is very small. These results prove that there is a
correlation between the distance and the likeness of two images. This face recognition program is now
recognizing the face of a user using a camera.
The next step was to set a threshold level. If the distance of a detected face from a vector is under this
threshold, then log him/her on. The average distance of a detected face is 0.4 or less than 10% of the
maximum distance. In this system a face is said to be under the threshold if its distance is smaller than 0.4
and less than 10% of the maximum distance. This rule is included in the programming. A face that meets
46
these requirements with one of the user faces is acknowledged as that user. That user name is then printed
to an output text file. This file is integrated in the authentication program, thus providing a link between
the two systems. Therefore, when the face recognition authentication system identifies a face as a user, it
authenticates that user. The biometric authentication system in this project worked for each user that tried
to log on.
47
Discussion
In this section we will briefly summarise the different phases in this project. For each phase we will
discuss the strategies used, the difficulties encountered, the implemented solutions and the overall results.
The software and tools section of this report includes a basic overview of the hardware and software
packages implemented in this project. This was the initial phase and involved:
Installing Linux
Installing OpenCV
Setting up the USB camera
Fedora Core 5 was the version of Linux used and was installed with the 5 CD set of installation discs
obtained from the technician. Fedora Core 5 provides a graphical installation interface which was easy to
follow. When first installing, one thing that was overlooked was the dual-boot loader option. Fedora had
to be re-installed so the GRUB dual-boot loader option would be provided when starting up the PC.
OpenCV was the next package to be installed after installing Linux. OpenCV required the installation of 3
dependencies. At first the dependencies did not install correctly and were not recognised by OpenCV. The
error occurred because the dependencies were not all installed in the same directory. Therefore it is good
practice and sometimes necessary to install software in the /usr/src/redhat/SOURCES folder. Sample
programs were then run to ensure that the OpenCV libraries were functioning correctly. The last stage was
setting up a USB webcam. This involved installing a webcam driver so the operating system could detect
a webcam. The gspca5xx universal webcam driver for Linux was downloaded and installed to make the
system compatible with webcams. A webcam was then acquired that suited the requirements of this
project. The goal of this phase was the acquisition of resources necessary for project completion; this goal
was reached.
The next phase of the project was the design of a face detection system. The aim of this stage was to
detect a face in an image inputted from a camera. This biometric system was implemented using the
OpenCV library and the sample face detection code. There were no issues with compiling or running the
code but the results were still not satisfactory. There were difficulties with the image captured from the
camera. The image was too bright and blurry. The initial solution was to edit the video configuration file
in the OpenCV libraries but this did not work. The next solution involved changing the image variables
48
(brightness, contrast etc.) using the face detection code. After this was done the face detection program
ran perfectly. The program was tested with a series of face images and non-face images; it operated
correctly for each. At this stage, a fully functional face detection program was up and running.
The Authentication phase involved integrating the biometric system with the PCs authentication. This was
achieved using Pluggable Authentication Modules or PAM. The plan for this section was:
Write and compile a simple PAM module.
Write a PAM module dependant on the biometric system
Integrate the PAM module with the biometric system
The first step was downloading the PAM source code and learning how a PAM module is written. The
source code for a simple PAM module, pam_permt.so, was compiled and run successfully. Writing a PAM
module dependant on the biometric system was the next step. This was accomplished by writing a module
that reads in a text file outputted by the face detection/recognition system. This file would contain user
information that is used in the authentication process. Two variations of PAM module were created:
GNOME login module Reads the user name from the text file and logs that user on. The
module does nothing if the text file is empty.
GNOME screensaver module The module reads in the user name from the file and compares it
to the current user name. If they are equal the user is authenticated. Otherwise the screensaver
doesnt unlock
Testing was performed by manually writing various information into the text file e.g. write a user name to
it and check if it passes authentication. The module works as expected; authenticating only when the
information in the text file is correct. The final part of this section involved getting a working demo of the
face detection authentication system. This system would authenticate a user if any face is detected. The
face detection program had to be changed to output a default user name every time a face is detected. The
face detection was also written to be initialised on start up and to continue running indefinitely. This was
achieved by adding the program to the run level 5 scripts. Initially the program was told to run in the
foreground and therefore froze the system on start up. The run level script was changed to run the
program in the background. The biometric system was now fully integrated with the authentication
49
system. The integrated system was left running for long periods of time to see if it would continually run
without error. The face detection authentication system ran perfectly.
The final phase of the project was to design a face recognition system. This system would try to recognise
a user by reading in a face from the camera and comparing it to the faces of known users. The PCA
algorithm for face recognition was chosen for this task. This algorithm uses eigenvector analysis to
compare the variance in each image, thus breaking down the image into mathematical coefficients. These
coefficients are then used to compare the input face against the faces in the user database. The PCA
functions in OpenCV are used to compute and compare the faces. File IO is used in this program to read
in the images contained in the user database. The face recognition worked correctly and determined which
face in the database, if any, matched the face in front of the camera. Then code was written so the
program could write the recognised users name in to an output file and thus integrating the program with
the authentication system. The program was tested with the faces of known users and other faces. In both
cases the system worked. A face recognition authentication was successfully installed on a PC.
50
Conclusion
The aim of the project was to design and develop a vision-based biometric authentication system. This
involves capturing an image from a camera and detecting the presence of a face. Employing a face
recognition algorithm in order to verify the face as a known user. Finally the integration of the face
recognition with the PCs authentication system.
This has been an interesting and challenging project during which I have learned a lot about image
processing, face detection, face recognition and authentication protocols. I have gained valuable
experience in working with the Linux operating system which is used extensively in the computer
industry. I have also gained further experience with the C programming language.
Implementing a face detection program with a webcam returned great results. The program always
located the face in the image. The selection of the Labtec webcam was a good compromise; its price and
size made up for what it lacked in quality. In the future if the system needed to be improved it would be
recommended to obtain a higher quality camera e.g. network camera. The design of a pluggable
authentication module went very well. Integrating the PAM module with both the face detection program
and face recognition program returned good results. In both cases the appropriate user was logged on.
Principal Component Analysis was selected for face recognition. This algorithm is not the most accurate
of face recognition techniques but definitely is one of the easiest to implement. Due to time constrictions
the PCA method was adopted over more accurate methods. The face recognition program returned good
results with a few discrepancies. The eigenvalues and eigenvectors of a group of images were calculated
correctly. Faces were able to be compared correctly using their decomposition coefficients. Faces were
successfully verified as user faces and then the corresponding user was successfully logged on. The only
problems were with the time it took to recognise a face. Continually loading images from a database took
its toll on the overall recognition time. An improvement would be to store the eigenvectors and
eigenvalues of the training set in a data file so it could be easily retrieved for the program. The add a new
user utility was not fully developed due to time constraints. A basic program that captured a face image
and stored in the user database was developed but this program did not add a new user to the operating
system.
51
Appendix
This entire compilation of source code and results were too big to fit in this report so they have been
included on a CD.
52
References
[1] http://vismod.media.mit.edu/tech-reports/TR-516/node12.html
[2] http://cnx.org/content/m12535/latest/
[3] http://www.bccresearch.com/editors/RG-276.html : GLOBAL MARKET FOR BIOMETRICS TO
REACH NEARLY $3.4 BILLION BY 2007
[4] www.c-vis.com/htd/frt.html
[5] http://www.findbiometrics.com/article/300 : Face Recognition Biometrics Industry Gears Up for
Immense Growth
[6] http://www.autoid.frost.com
[7] http://www.biometricgroup.com/reports/public/market_report.html
[8] http://en.wikipedia.org/wiki/Linux
[9] http://www.winehq.com
[10] http://fedoraproject.org/wiki
[11] http://downlaod.fedora.redhat.com/pub/fedora/linux/core/5/source/SRPMS
[12] http://www.gnome.org
[13] http://en.wikipedia.org/wiki/GCC
[14] http://en.wikipedia.org/wiki/C_(programming_language)
[15] http://sourceforge.net/projects/opencvlibrary/
[16] http://mxhaard.free.fr/spca5xx.html
[17]
[18]http://www.intel.com/technology/itj/2005/volume09issue02/art03_learning_vision/p04_face_detectio
n.html
[19] http://www.comptechdoc.org/os/linux/usersguide/linux/ugpam/.html
[20] http://sourceforge.net/projects/pam-chroot/
[21] K. Kim. Face Recognition using Principal Component Analysis. Department of Computer Science,
University of Maryland, USA.
[22] http://tech.groups.yahoo.com/group/OpenCV/
[23] http://cgi.cs.indiana.edu/~oleykin/website/OpenCVHelp/

FYP Thesis

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

FYP Thesis

Uploaded by

Copyright:

Available Formats

Vision-Based Biometric Authentication

Project Supervisor: Mr. Frank Callaly

Vision-Based Biometric Authentication System

Final Year Project, Dept. of Electronic Engineering, NUI Galway

Vision-Based Biometric Authentication System

Final Year Project, Dept. of Electronic Engineering, NUI Galway

Vision-Based Biometric Authentication System

Final Year Project, Dept. of Electronic Engineering, NUI Galway

Vision-Based Biometric Authentication System

Final Year Project, Dept. of Electronic Engineering, NUI Galway

Vision-Based Biometric Authentication System

Chapter 4 Face recognition........................................................................................................................34

Final Year Project, Dept. of Electronic Engineering, NUI Galway

Vision-Based Biometric Authentication System

Final Year Project, Dept. of Electronic Engineering, NUI Galway

Vision-Based Biometric Authentication System

Final Year Project, Dept. of Electronic Engineering, NUI Galway

Vision-Based Biometric Authentication System

Figure 0-1 : Example of face recognition [4]

Final Year Project, Dept. of Electronic Engineering, NUI Galway

Vision-Based Biometric Authentication System

Figure 0-2 : Biometric Market Technology [7]

Final Year Project, Dept. of Electronic Engineering, NUI Galway

Vision-Based Biometric Authentication System

Less secure Anybody may enter anybody elses password

Retrieves images from a camera in real-time

Detects the presence of a face in the image

Identifies the face against some enrolled images

Final Year Project, Dept. of Electronic Engineering, NUI Galway

Vision-Based Biometric Authentication System

Figure 0-3 : Overall Project Structure

The initial plan for this project was:

Get a face detection program up and running

Get a face recognition program working

Add a utility to add new users to the database of authorized faces

Final Year Project, Dept. of Electronic Engineering, NUI Galway

Vision-Based Biometric Authentication System

Final Year Project, Dept. of Electronic Engineering, NUI Galway

Vision-Based Biometric Authentication System

Figure 1-4 : Linux Logo

Final Year Project, Dept. of Electronic Engineering, NUI Galway

Vision-Based Biometric Authentication System

Create, move and copy files and folders.

Change file/folder permissions

Figure 1-5 : CLI being used from a terminal

Final Year Project, Dept. of Electronic Engineering, NUI Galway

Vision-Based Biometric Authentication System

Mozilla Firefox Internet

VLC player Multimedia

OpenOffice Office suite

The GIMP Graphics

Final Year Project, Dept. of Electronic Engineering, NUI Galway

Vision-Based Biometric Authentication System

Figure 1-6 : Fedora logo

Final Year Project, Dept. of Electronic Engineering, NUI Galway

Vision-Based Biometric Authentication System

Figure 1-7 : GNOME Logo

GNOME also includes a lot of useful applications such as:

Nautilus a file manager

Totem A music player

GNOME Terminal a terminal emulator

gedit a text editor

Final Year Project, Dept. of Electronic Engineering, NUI Galway

Vision-Based Biometric Authentication System