You are on page 1of 29

# Eigenimages

n Unitary transforms
n Karhunen-Love transform and eigenimages
n Sirovich and Kirby method
n Eigenfaces for gender recognition
n Fisher linear discrimant analysis
n Fisherimages and varying illumination
n Fisherfaces vs. eigenfaces

## Digital Image Processing: Bernd Girod, 2013 Stanford University -- Eigenimages 1

Unitary transforms
!
n Sort pixels f [x,y] of an image into column vector f of length N
n Calculate N transform coefficients

c = Af
where A is a matrix of size NxN
n The transform A is unitary, iff

A 1 = *T
A
A

H

Hermitian conjugate

## Digital Image Processing: Bernd Girod, 2013 Stanford University -- Eigenimages 2

Energy conservation with unitary transforms

n For any unitary transform c = Af we obtain

2 H H H 2
c = c c = f A Af = f

## n Interpretation: every unitary transform is simply a rotation of the

coordinate system (and, possibly, sign flips)
n Vector length is conserved.
n Energy (mean squared vector length) is conserved.

## Digital Image Processing: Bernd Girod, 2013 Stanford University -- Eigenimages 3

Energy distribution for unitary transforms
n Energy is conserved, but, in general, unevenly distributed among coefficients.
n Autocorrelation matrix
H H H
Rcc = E cc = E Af f A = AR ff A H

i,i

## Digital Image Processing: Bernd Girod, 2013 Stanford University -- Eigenimages 4

Eigenmatrix of the autocorrelation matrix
Definition: eigenmatrix of autocorrelation matrix Rff
l is unitary
l The columns of form a set of eigenvectors of Rff, i.e.,

\$ 0 '
0
& )
& 1 )
=& )
& ! )
&% 0 N 1 )(

## Digital Image Processing: Bernd Girod, 2013 Stanford University -- Eigenimages 5

Karhunen-Love transform
n Unitary transform with matrix

A= H

## n Transform coefficients are pairwise uncorrelated

Rcc = AR ff A = R ff = =
H H H

## n Columns of are ordered according to decreasing eigenvalues.

n Energy concentration property:
l No other unitary transform packs as much energy into the first J coefficients.
l Mean squared approximation error by keeping only first J coefficients is minimized.
l Holds for any J.

## Digital Image Processing: Bernd Girod, 2013 Stanford University -- Eigenimages 6

Illustration of energy concentration

f2 cos sin c2
A=
sin cos
After KLT:
Strongly correlated
uncorrelated samples,
samples,
equal energies f1 c1 most of the energy in
first coefficient

## Digital Image Processing: Bernd Girod, 2013 Stanford University -- Eigenimages 7

Basis images and eigenimages
n For any transform, the inverse transform
! 1 !
f =A c
can be interpreted in terms of the superposition of columns of A-1 (basis images)
n For the KL transform, the basis images are the eigenvectors of the
autocorrelation matrix Rff and are called eigenimages.
n If energy concentration works well, only a limited number of eigenimages is
needed to approximate a set of images with small error. These eigenimages
span an optimal linear subspace of dimensionality J.

## Digital Image Processing: Bernd Girod, 2013 Stanford University -- Eigenimages 8

Eigenimages for recognition
n To recognize complex patterns (e.g., faces), large portions of an image have to
be considered
n High dimensionality of image space means high computational burden for
many recognition techniques
Example: nearest-neigbor search requires pairwise comparison with every image in a database

n Transform c = Wf
can reduce dimensionality from N to J by representing the
image by J coefficients
n Idea: tailor a KLT to a specific set of training images
representative of the recognition task
to preserve the salient features

## Digital Image Processing: Bernd Girod, 2013 Stanford University -- Eigenimages 9

Eigenimages for recognition

f c
W
f
+
f -
Rejection
Normalization Projection Similarity
1
New Face measure
!T !
Image (e.g., c pk* )

f
p1

Class of most k*

similar pk

!
pK Recognition
Mean Face Result
Database of Similarity
Eigenface Matching
Coefficients

## Digital Image Processing: Bernd Girod, 2013 Stanford University -- Eigenimages 10

Computing eigenimages from a training set
n How to obtain NxN covariance matrix?
! ! !
,
l Use training set 1 2 ,, L+1
(each column vector represents one image)
l Let be the mean image of all L+1 training images
! "! ! "! ! "! ! "!
1 2 (
l Define training set matrix S = , , ,, ,
3 L )
L ! "! ! "! H
and calculate scatter matrix R =
l=1
( )(
l l ) = SS H

## Problem 1: Training set size should be L + 1 >> N

If L < N, scatter matrix R is rank-deficient
Problem 2: Finding eigenvectors of an NxN matrix.

## n Can we find a small set of the most important eigenimages

from a small training set L << N ?

## Digital Image Processing: Bernd Girod, 2013 Stanford University -- Eigenimages 11

Sirovich and Kirby algorithm
n Instead of eigenvectors of SS H , consider the eigenvectors of S H S, i.e.,
H
S Svi = i vi
n Premultiply both sides by S
! !
=
SS H Svi = i Svi

n By inspection, we find that Svi are eigenvectors of SS H

## Sirovich and Kirby Algorithm (for L << N )

l Compute the LxL matrix SHS

l Compute L eigenvectors vi of SHS
l Compute eigenimages corresponding to the L0 L largest eigenvalues
as a linear combination of training images Svi

L. Sirovich and M. Kirby, "Low-dimensional procedure for the characterization of human faces,"
Journal of the Optical Society of America A, 4(3), pp. 519-524, 1987.

## Digital Image Processing: Bernd Girod, 2013 Stanford University -- Eigenimages 12

Example: eigenfaces
n The first 8 eigenfaces obtained from a training set of 100 male and 100 female
training images

Mean Face

## n Can be used to generate faces by adjusting 8 coefficients.

n Can be used for face recognition by nearest-neighbor search in 8-d face space.

## Digital Image Processing: Bernd Girod, 2013 Stanford University -- Eigenimages 13

Gender recognition using eigenfaces
Nearest neighbor search face space

## Digital Image Processing: Bernd Girod, 2013 Stanford University -- Eigenimages 14

Fisher linear discriminant analysis
n Eigenimage method maximizes scatter within the linear subspace over the
entire image set regardless of classification task

## Wopt = arg max det WRW H

W
( ( ))
n Fisher linear discriminant analysis (1936): maximize between-class scatter, while
minimizing within-class scatter
!"
! !" !"! !"
( )( )
c H
RB = N i i i

( )
i=1
det WR W H
B
Wopt = arg max
( )
Samples
Mean in class i
W det WRW W H in class i

( )( )
c H
RW = l i l i
i=1 Class(i)
l

## Digital Image Processing: Bernd Girod, 2013 Stanford University -- Eigenimages 15

Fisher linear discriminant analysis (cont.)
!"
!
n Solution: Generalized eigenvectors wi corresponding to the
J largest eigenvalues {i | i = 1,2,..., J }, i.e.
!"
! !"
!
RB wi = i RW wi , i = 1,2,..., J
!"
! !"
!
n solve eigen-problem on this: R 1
W (
RB wi = i wi , i = 1,2,..., J )
n Problem: within-class scatter matrix Rw at most of rank L-1 (for L images total
in all classes combined), hence usually singular.
n Apply KLT first to reduce dimensionality of feature space to L-1 (or less),
proceed with Fisher LDA in lower-dimensional space

## Digital Image Processing: Bernd Girod, 2013 Stanford University -- Eigenimages 16

Eigenimages vs. Fisherimages
2-d example: f2

## Goal: project samples on

a 1-d subspace, then perform
classification.

f1

## Digital Image Processing: Bernd Girod, 2013 Stanford University -- Eigenimages 17

Eigenimages vs. Fisherimages
2-d example: f2 KLT

## Goal: project samples on

a 1-d subspace, then perform
classification.

## The KLT preserves

maximum energy, but
f1
the 2 classes are no
longer distinguishable.

## Digital Image Processing: Bernd Girod, 2013 Stanford University -- Eigenimages 18

Eigenimages vs. Fisherimages
2-d example: f2 KLT

## Goal: project samples on

a 1-d subspace, then perform
classification.

## The KLT preserves

maximum energy, but
f1
the 2 classes are no
longer distinguishable.

Fisher LDA separates the
classes by choosing
a better 1-d subspace.

Fisher LDA

## Digital Image Processing: Bernd Girod, 2013 Stanford University -- Eigenimages 19

Fisherimages and varying iIllumination
Differences due to varying illumination can be
much larger than differences among faces!

## Digital Image Processing: Bernd Girod, 2013 Stanford University -- Eigenimages 20

Fisherimages and varying iIllumination
n All images of same Lambertian surface with different
illumination (without shadows) lie in a 3d linear subspace
n Single point source at infinity Light source
intensity
surface !
normal n ! !T !
l
light source
( )
f ( x, y ) = a ( x, y ) l n ( x, y ) L
direction

Surface
albedo

## n Superposition of arbitrary number of point sources at infinity

still in same 3d linear subspace, due to linear superposition
of each contribution to image
n Fisherimages can eliminate within-class scatter

## Digital Image Processing: Bernd Girod, 2013 Stanford University -- Eigenimages 21

Side Note: Photometric Stereo!
observed normalized
intensity! lighting direction!
N
L I = L N
albedo! normalized
(constant)! surface normal!

## diffuse (Lambertian) surfaces are viewpoint independent!

[Woodham 1980]!
Side Note: Photometric Stereo!

N Nx
L
I = Lx Ly Lz N y

N
z

## diffuse (Lambertian) surfaces are viewpoint independent!

[Woodham 1980]!
Side Note: Photometric Stereo!

L( 2 ) L(1) N
N
I (1) L (1)
L(1)

L(1) L ( 3) x y z
x
(2) L(2) Ny
I = L(2) L(2)

x y z
I (3)
Lx
(3)
L (3)
L(3)
N z
y z

=! I =! L

## diffuse (Lambertian) surfaces are viewpoint independent!

1
assume albedo is constant, invert matrix! N=L I [Woodham 1980]!
input! Side Note: Photometric Stereo!

## output: recovered normals!

[Woodham 1980]!
Fisherface trained to recognize gender

## Mean image Female mean Male mean

! ! !
1 2
Female face samples

Fisherface

## Digital Image Processing: Bernd Girod, 2013 Stanford University -- Eigenimages 26

Gender recognition using 1st Fisherface

## Digital Image Processing: Bernd Girod, 2013 Stanford University -- Eigenimages 27

Gender recognition using 1st eigenface

## Digital Image Processing: Bernd Girod, 2013 Stanford University -- Eigenimages 28

Person identification with Fisherfaces and eigenfaces

## ATT Database of Faces

40 classes
10 images per class