Singular Value Decomposition and the Face Space

Nicholas Ruof
January 19, 2018

ABSTRACT
The Singular Value Decomposition (SVD) is a powerful tool that decomposes a data matrix
into its principal orthogonal modes and a matrix with singular values on the diagonal. SVD
is used throughout many applications one of which is face recognition. The decomposition
of a face data matrix allows for reconstruction and recognition of new faces not included in
the original data set.

1 Introduction and Overview

In this assignment there will be a theoretical overview of the SVD concept followed by an
application in face recognition. There will be an overview of how the SVD algorithm is
implemented and the results that follow, which will include the analysis of two photo data
sets and a face reconstruction of the famous Seth1 using the proper orthogonal modes from
one of the data sets.

2 Theoretical Background
A matrix is a mathematical object that can transform a vector by changing its direction
and magnitude. For example consider a vector that points in a particular direction in a 2-D
parameter space and is acted upon by a matrix.
    0  0  
a11 a12 v1 v1 v1 a11 v1 + a12 v2
   =  ,   =   (1)
0 0
a21 a22 v2 v2 v2 a21 v1 + a22 v2

Geometrically, matrix multiplication is simply the rotation and stretching of some vector v
to some new vector v0 . Equation 1 can then be re-formated with this interpretation in mind.
Lets say a matrix A is acting on a set of vectors vj and produces a new set of vectors with
stretching factors σj and orthogonal vectors uj .

Avj = σj uj 1≤j≤N (2)

1
Nathan knows who he his

1
Figure 1: Image of a sphere in space S with vector components v1 and v2 acted upon by a
matrix A to produce a new set of vectors that form the basis for an ellipse.

Or in matrix form where A is acting on many vectors

    
σ
 1
 
σ2
   
   
 A  v1 v2 · · · vn  = u1 u2 · · · un   ..

 (3)







 . 
σn

And in compact matrix notation

AV = ÛΣ̂ (4)
Moving V to the other side of the equation gives matrix A in terms of its reduced singular
value decomposition, where A ∈ Cmxn , Û ∈ Cmxn , Σ̂ ∈ Cnxn , and V ∈ Cnxn .

A = ÛΣ̂V† (5)
In the full singular value decomposition the left most matrix in the decomposition is
defined to be unitary and the current Û is not unitary. For the full decomposition m − n
orthogonal columns are added to Û and m − r rows of zeros are added to Σ̂. r is the rank
of A

A = UΣV† (6)
U ∈ Cmxm is unitary
V ∈ Cnxn is unitary
Σ ∈ Rmxn is diagonal

There are two important theorems for SVD that will be used throughout this assignment
and listed below.

2
 Figure 2: Pictoral representation of the reduced SVD

Figure 3: Pictoral representation of the full SVD where the lightly shaded areas are the
m − n added orthogonal columns in U and the m − n rows of zeros in Σ

• Every matrix A ∈ Cmxn has a singular value decomposition where the singular values
are uniquely determined and the singular vectors ui and vi are unique up to complex
signs.

• The singular values in Σ are ordered from largest to smallest going from left to right
and the number of non-zero singular values equals the rank of the matrix A

The definition of the SVD has been established, but so far without any techinques for its
computation. The SVD of any matrix A can be easily computed with some simple matrix
manipulation.

AT A = (UΣV† )T (UΣV† ) (7)

= VΣ2 V†

AAT = (UΣV† )(UΣV† )T (8)

= UΣ2 U†

Multiplying on the right hand side by V and U will give an eigenvalue problem that can

3
be solved
AT AV = VΣ2 (9)
T 2
AA U = UΣ (10)
From this we can deduce interpretations for the matrices involved in the SVD. AT A is
proportional to a covariance matrix where V represents the orthogonal modes for the rows
of A and AAT is proportional to a covariance matrix where U represents the orthogonal
modes for the columns of A. For example if A represents the heights of a 2-D surface then
the vectors in U would represent the orthogonal modes in one dimension, while the vectors
in V would represent the orthogonal modes in the other. The singular values in Σ are the
variance for each corresponding mode.
A particular data set that will be analyzed in this assignment will be an assortment of
face photographs. Each face image will be resized from a matrix to a vector and will be
appended to a matrix A, where an SVD will be computed. In this case the orthogonal
vectors in U will represent the orthogonal modes in face space also known as the eigenfaces
of the data set and Σ will give the variance corresponding to each eigenface. V is odd to
reconcile when illustrating the face space, but it would represent the orthogonal modes of
the rows of A, which are the individual pixels. V forms an orthogonal basis in the pixel
space.

3 Algorithm Implementation and Development

In the MATLAB program there are four categories of implementation
• Reading in the Images
• Computing the SVD
• Reconstruction of a familiar face
• Plotting the Results

3.1 Reading in the images

We will be dealing with two face data sets one where all of the images are cropped tightly
around each face and one where the faces are uncropped. In the uncropped data set each face
has its’ own directory, which is identified and looped through in the MATLAB code. Each
image is read into a 4 dimensional array where the first two indices correspond to the surface
of an image and the last two indices are the directory number and file number respectively
according to the naming convention of the data set (there is no file number 14 and a warning
is flagged when the code runs, but can be ignored). Each image is then reshaped into a
column vector that is appended to a matrix which stores the entire collection of images. The
uncropped images are stored in just one directory so a simple loop of the directory does the
trick. All images are first stored in a 3 dimensional array where the last index corresponds
to different files and the first two indexs are the images themselves represented by a matrix.
There are 2508 images in the cropped data set and 165 in the uncropped data set.

4
3.2 Computing the SVD
Before computing the SVD of the face data set matrix the average face will be subtracted from
each column, which result in our U matrix being the true orthogonal modes of a covariance
matrix. After this step the MATLAB command: [u, s, v] = svd(uncropped data,0 econ0 ) is
performed, noting that the ‘econ’ parameter is being chosen for faster computation and
resembles a reduced SVD. The rank of the face data set matrix is easily computed along
with its singular values.

3.3 Reconstruction of a familiar face

In theory, once the orthogonal modes of the face space are constructed one can expand any
face in terms of the orthogonal modes or eigenfaces.
N
X
f~new = cj ~uj (11)
j=1

The cj coefficients are the overlap of the new face with respect to each eigenface uj .
N
X
f~new = (~uj · f~new )~uj (12)
j=1

The series of coefficients order in one row vector can be expressed from the following trans-
formation.
cj = U† f~new (13)
From this transformation one can reconstruct any face from the eigenfaces and will be shown
later with the familiar face of the famous Seth.

3.4 Plotting the Results

There are a few plots that are produced. The singular values are plotted using the standar
MATLAB plot and semilogy. The eigenmodes are plotted as 2-D images so they have been re-
formatted back into pictures from the command pcolor(flipud(reshape(u(:, 1), num rows, num cols))).

4 Computational Results
Now lets get to the good stuff, the computational results. A few properties of any data set
that is of particular interest is how many non-zero modes or singular values occur in the
data set? Are there any modes that dominate over others? What do the orthogonal modes
look like? In the Cropped Image data set there are four dominant modes followed by a long
tail. The total number of non-zero singular values is given by the rank of the data matrix
which is 2367.
The uncropped data set offers worse results in resolution for the eigenfaces and only two
dominant modes. From the face space we can attempt to reconstruct any new face that is

5
Figure 4: The singular values of the Cropped images data set shown in linear and semi-log
y plots. There are 4 distinct dominate modes followed by a long tall that ends at mode 2367
which is the rank of the data set

Figure 5: The first three dominate modes in the cropped face space

6
Figure 6: The singular values of the Uncropped images data set shown in linear and semi-log
y plots. There are 2 distinct dominate modes followed by a long tall that ends at mode 156
which is the rank of the data set

Figure 7: The first three dominate modes in the uncropped face space. Much less resolution
compared to the modes from the cropped data set.

7
Figure 8: A face reconstruction of Seth. The face becomes recognizable at about 500 modes
and becomes more refined as more modes are added.

not included in the original data set. Take for example the face of the famous Seth himself
and a reconstruction via the cropped face space. The imaged has been cropped to best
resemble the images in the original data set and yes there are some unwanted words layed
over the picture. Amazingly the image is already recognizable after about 500 modes where
the eyes and mouth are easily identifiable while the nose becomes more recognizable after
somewhere between 500 and 1000 modes.

5 Summary and Conclusion

The Singular Value Decomposition is a widely used data analysis technique with many
applications across the board. It has been demonstrated how the SVD can be used for face
recognition and reconstruction. This paper went through the theory behind the technique,
algorithm implementation, and the successful results that followed.

8
6 Appendix A: MATLAB Functions
imresize(A, [num_rows,num_cols]): resizes an image to an image with num_rows rows and num_cols columns
imread(A): reads in an image and converts it into a readable matrix
reshape(A,m,n): reshapes a matrix A into a new matrix with size mxn
[u,s,v] = svd(A): performs a singular value decomposition where A = usv*
rank(A): outputs the rank of a matrix
pcolor(A): checkerboard plot for a matrix A used for plotting images
flipud(A): Flips the A upside down. Using pcolor displays the images upside down so they must be flipped first.

7 Appendix B: MATLAB Code

clear all; close all; clc;

%% Read in Cropped Images

path_cropped = ’/Users/nickruof/Documents/Graduate Courses/AMATH_582/HW/CroppedYale/’;

all_files = dir(path_cropped);
all_dir = all_files([all_files(:).isdir]);
num_dir = numel(all_dir)-2;
imagefiles = ’’;
num_rows = 192;
num_cols = 168;
image_count = 0;
image_matrices = zeros(num_rows,num_cols,1,1);
image_reshape = zeros(num_rows*num_cols,1);

% Cropped Yale has many directories in it. Each directory is added to the path
% and all images in each directory are looped through using the naming
% convention of the image files. Convention:
% image_matrices(:,:,directory_number,file_number)

for i = 1:num_dir
if(i >= 0 && i <= 9)
addpath(strcat(path_cropped,’yaleB0’,num2str(i)));
imagefiles = dir(strcat(path_cropped,’yaleB0’,num2str(i)));
num_images = numel(imagefiles)-2;
for j = 3:num_images+2
image_matrices(:,:,i,j) = imresize(imread(imagefiles(j).name,’pgm’),[num_rows,num_cols]);
end
else
addpath(strcat(path_cropped,’yaleB’,num2str(i)));
imagefiles = dir(strcat(path_cropped,’yaleB’,num2str(i)));
num_images = numel(imagefiles)-2;
for j = 3:num_images+2
image_matrices(:,:,i,j) = imresize(imread(imagefiles(j).name,’pgm’),[num_rows,num_cols]);
end
end
end

for i = 1:length(image_matrices(1,1,:,1))
for j = 1:length(image_matrices(1,1,1,:))
image_count = image_count + 1;
image_reshape(:,image_count) = reshape(image_matrices(:,:,i,j),1,num_rows*num_cols);
end
end

%% SVD computation for Cropped Images

average = zeros(size(image_reshape,1),size(image_reshape,2));
average_face = zeros(size(image_reshape,1),1);
for i = 1:size(image_reshape,2)
average_face = (average_face + image_reshape(:,i))/size(image_reshape,2);

9
end

for i = 1:size(image_reshape,2)
average(:,i) = average_face;
end

image_data = image_reshape - average;

[u,s,v] = svd(image_data,’econ’);
r = rank(image_data);
sig = diag(s);

%% Plotting Singular Values and Modes for Cropped Images

figure(1)
subplot(2,1,1), plot(sig,’ko’,’Linewidth’,[1.5]), title(’Singular Values’)
axis([0,100,0,sig(1)])
xlabel(’Modes’)
ylabel(’A.U. \sigma^2’)
subplot(2,1,2), semilogy(sig,’ko’,’Linewidth’,[1.5]), title(’Singular Values in Semi-Log Scale’)
xlabel(’Modes’)
ylabel(’A.U. \sigma^2’)
axis([0,r,1,sig(1)])

figure(2)
subplot(1,3,1), pcolor(flipud(reshape(u(:,1),num_rows,num_cols))), shading interp, colormap(gray), title(’First Mode’)
subplot(1,3,2), pcolor(flipud(reshape(u(:,2),num_rows,num_cols))), shading interp, colormap(gray), title(’Second Mode’)
subplot(1,3,3), pcolor(flipud(reshape(u(:,3),num_rows,num_cols))), shading interp, colormap(gray), title(’Third Mode’)

%% Seth Face Reconstruction (uncomment end of the for loop to see reconstruction in real time)

seth_original = imresize(imread(’Seth_face’,’png’),[num_rows,num_cols]);
seth_original = im2double(seth_original);
seth_vector = reshape(seth_original(:,:,1),1,num_rows*num_cols)’;
seth_transform = transpose(u)*(seth_vector - average_face);
seth_reconstruct = zeros(length(seth_vector),1);
seth_results = zeros(length(seth_vector),4);

N_modes = 1500;

for i = 1:N_modes
seth_reconstruct = seth_transform(i)*u(:,i) + seth_reconstruct;
if(i == 100)
seth_results(:,1) = seth_reconstruct;
end
if(i == 500)
seth_results(:,2) = seth_reconstruct;
end
if(i == 1000)
seth_results(:,3) = seth_reconstruct;
end
if(i == 1500)
seth_results(:,4) = seth_reconstruct;
end
% pcolor(flipud(reshape(seth_reconstruct,num_rows,num_cols))), shading interp, colormap(gray)
% title(strcat(’N = ’, num2str(i)));
% pause(0.0001)
end

figure(3)
subplot(1,5,1), pcolor(flipud(reshape(seth_results(:,1),num_rows,num_cols))),
shading interp, colormap(gray), title(’Reconstruction: 100 Eigenfaces’)
subplot(1,5,2), pcolor(flipud(reshape(seth_results(:,2),num_rows,num_cols))),

10
shading interp, colormap(gray), title(’Reconstruction: 500 Eigenfaces’)
subplot(1,5,3), pcolor(flipud(reshape(seth_results(:,3),num_rows,num_cols))),
shading interp, colormap(gray), title(’Reconstruction: 1000 Eigenfaces’)
subplot(1,5,4), pcolor(flipud(reshape(seth_results(:,4),num_rows,num_cols))),
shading interp, colormap(gray), title(’Reconstruction: 1500 Eigenfaces’)
subplot(1,5,5), pcolor(flipud(reshape(seth_vector,num_rows,num_cols))),
shading interp, colormap(gray), title(’Orignal Image’)

%% Read in Uncropped Images

path = ’/Users/nickruof/Documents/Graduate Courses/AMATH_582/HW/yalefaces_uncropped/yalefaces/’;

uncropped_files = dir(path);
num_uncropped = length(uncropped_files)-2;
num_rows = 243;
num_cols = 320;
uncropped_matrix = zeros(num_rows, num_cols, 1);
uncropped_reshape = zeros(num_rows*num_cols, 1);

% All uncropped images are stored in one directory and looped over

for i = 1:num_uncropped
uncropped_matrix(:,:,i) = imresize(imread(strcat(path,uncropped_files(i+2).name),’gif’),[num_rows,num_cols]);
uncropped_reshape(:,i) = reshape(uncropped_matrix(:,:,i),1,num_rows*num_cols);
end

%% SVD Computation for Uncropped Images

average = zeros(size(uncropped_reshape,1),size(uncropped_reshape,2));
average_face = zeros(size(uncropped_reshape,1),1);
for i = 1:size(uncropped_reshape,2)
average_face = (average_face + uncropped_reshape(:,i))/size(uncropped_reshape,2);
end

for i = 1:size(uncropped_reshape,2)
average(:,i) = average_face;
end

uncropped_data = uncropped_reshape - average;

[u,s,v] = svd(uncropped_data,’econ’);
r = rank(uncropped_data);
sig = diag(s);

%% Plotting Singular Values and Modes for the Uncropped Images

figure(4)
subplot(2,1,1), plot(sig,’ko’,’Linewidth’,[1.5]), title(’Singular Values’)
axis([0,100,0,sig(1)])
xlabel(’Modes’)
ylabel(’A.U. \sigma^2’)
subplot(2,1,2), semilogy(sig,’ko’,’Linewidth’,[1.5]), title(’Singular Values in Semi-Log Scale’)
xlabel(’Modes’)
ylabel(’A.U. \sigma^2’)
axis([0,r,1,sig(1)])

figure(5)
subplot(1,3,1), pcolor(flipud(reshape(u(:,1),num_rows,num_cols))), shading interp, colormap(gray), title(’First Mode’)
subplot(1,3,2), pcolor(flipud(reshape(u(:,2),num_rows,num_cols))), shading interp, colormap(gray), title(’Second Mode’)
subplot(1,3,3), pcolor(flipud(reshape(u(:,3),num_rows,num_cols))), shading interp, colormap(gray), title(’Third Mode’)

11