HomeWork Assignment 7 ECE661

Muhammad Ahmad Ali PUID: 0022296327 November 23, 2010

1
1.1

Problem Solution
PCA

For doing face recognition through PCA we proceed as follows. First, all the images are vectorized so that i-th image is represented as Xi vector. Then a matrix X is formed as follows, X = [X1 m, X2 m, .., XN m] Where, m= 1 N
N

Xi
i=1

Then, we nd the eigenvector matrix U of XT X. As explained in class, the eigenvectors W of XXT are found as given below, W = XU We then retain p eigenvectors from the matrix W with largest eigenvalues to get the PCA basis Wp . Then the projection of all training images on Wp is obtained, yi = WT p (Xi m) Similarly, each test image is projected on to the PCA basis. yj = WT p (Xj m) For each j -th test image and i-th training image the distance dij is calculated, dij =
Ty yi j

The i for which dij is minimum for j -th test image, gives the index of best matching training image. If the class label of that training image matches the class label of the test image, a correct classication is registered. This is essentially a nearest neighbor classication in the reduced dimensionality in the form of PCA basis. The accuracy of the classication is obtained by, accuracy(p) = No. of images correctly classied Total No. of images

Where, p is the number of eigenvectors used.

1.2

LDA
wT SB w wT SW w

For LDA we need to nd directions w that maximize, J (w) =

Where, SB is between class variance matrix and SW is between class variance matrix given by, SB = 1 Nc 1 Nc
Nc

(mi m)(mi m)T

i=1 Ni

Nc

SW =

i=1

1 Ni

(Xij mi )(Xij mi )T
j =1

To solve this problem Yu and Yang algorithm is used. First of all, we need to nd the eigenvectors of SB . These are found using the same trick as used in PCA. A matrix M is formed as follows, M = [m1 m, m2 m, .., mNc m] Where mi is the mean of the i-th class. Then obviously, SB = MMT (ignoring the scale factor 1/Nc ). So rst the eigenvector matrix V of MT M is obtained and the eigenvectors VB of MMT are found as given below, VB = MV The eigenvectors obtained above are sorted in descending order of their eigenvalues and those with eigenvalues close to zero were discarded. I obtained 29 eigenvectors by this method. Let us call Y the eigenvector matrix left behind after discarding. Then a matrix Z is obtained, Z = YDB
T 1/2

Where, DB = YT SB Y. Finally we obtain the eigenvector matrix U of Z SW Z and they are sorted in ascending order of their eigenvalues. I do not discard the eigenvectors corresponding to largest eigenvalues at this point. So the LDA basis is obtained by, W = ZU Once LDA basis is obtained, the rest of the procedure is same as explained for PCA. We retain rst p eigenvectors from W as a matrix Wp and then project both the training and test images onto the basis and perform nearest neighbor classication in this basis as described in previous section. The results of applying both algorithms to the data are shown in next section. The answer to the question about space dimensionality p is also given in next section.

2
2.1

Results
PCA

Using the training data set we rst compute the PCA basis as described in section 1.1. This gives us a 16384 630 matrix, whose columns contain the desired eigenvectors. If we analyze the eigenvalues we see that the bulk of the energy is concentrated in the initial directions. Specically we found out that the rst 75 vectors span the 95% of the energy. From this we can conclude that 75 eigenvectors are sucient for good result of the PCA method. The plot of the energy as a function of the number of the eigenvectors is given below,
1 0.8 Energy % 0.6 0.4 0.2 0 0 100 200 300 400 500 No. of PCA Eigenvectors 600 700

Figure 1: Cumulative Energy in eigenvectors The plot of the PCA classication accuracy as a function of the number of eigenvectors is given below. We noticed that we achieved 100% accuracy by using 17 eigenvectors:
1 0.8 0.6 0.4 0.2 0

Accuracy of PCA

10

20

30 40 50 No. of Eigenvectors

60

70

Figure 2: PCA accuracy as a function of No. of Eigenvectors

2.2

LDA

The LDA basis was obtained by following the procedure described in section 1.2. This gives a 16384 29 sized matrix W containing 29 basis vectors. Also, as explained in class, the number of basis vectors in LDA cannot be greater than (No. of classes - 1). In our case (No. of classes = 30), so at most 29 basis vectors can be obtained. The plot of the LDA classication accuracy as a function of the number of eigenvectors is given below. We noticed that we achieved 100% accuracy by using 10 eigenvectors:

1 0.8 0.6 0.4 0.2 0

Accuracy of LDA

10 15 20 No. of Eigenvectors

25

Figure 3: LDA accuracy as a function of No. of Eigenvectors

2.3

Comparison of PCA and LDA

The comparison of the two methods is shown in the gure below. We can see that PCA achieves 100% accuracy at 17 eigenvectors while LDA achieves 100% accuracy for just 10 eigenvectors.

1 0.99 0.98 Accuracy 0.97 0.96 0.95 0.94 0.93 0.92 5 10 15 20 No. of Eigenvectors 25 PCA LDA

Figure 4: Comparison of PCA and LDA accuracy as a function of No. of Eigenvectors

Matlab Code

The code is given below, %--------------------------------------------------------% readData.m %--------------------------------------------------------function [X,labelVector] = readData srcDir = D:\CourseWork\ECE661\HW7\pca-lda\ECE661_hw7_images\hw7_images\train; d = dir([srcDir,/*.png]); % N = # of examples N = length(d); im = imread([srcDir , / , d(1).name ]); [h,w,k] = size( im ); % M = feature dimensionality M = h*w; X = zeros( M , N ); for( i = 1 : N ) dispStr = [ >>>>>> Reading file : , num2str(i) , of , num2str(N) ]; disp( dispStr ); fileName = d(i).name; im = imread( [srcDir , / , d(i).name] ); [h,w,k] = size( im ); if( k == 3 ) im = rgb2gray( im ); end X( : , i ) = double( im(:) ); labelVector(i) = getSubjectID( fileName ); end function sub_id = getSubjectID( fileName ) usIndices = find( fileName == _ ); sub_id = str2num( fileName( 1 : usIndices(1)-1) );

%--------------------------------------------------------% pca.m %--------------------------------------------------------function [W,mn] = pca(X) [M,N] = size(X); % subtract off the mean for each dimension mn = mean(X,2); X = X - repmat(mn,1,N); G = X*X; % find the eigenvectors and eigenvalues [EV, V] = eig(G); size(V) [ V , idx ] = sort( diag(V) );

idx = idx( end : -1 : 1 ); V = V( idx ); EV = EV( : , idx ); energy = cumsum(V)./ sum(V); save(Eigen-Value-Energy.mat,energy); cutoff = find( energy >= 1 ); cutoff = cutoff(1); W = X * EV; W = normalizeVectors(W);

%--------------------------------------------------------% runPCAClassification.m %--------------------------------------------------------function accuracy = runPCAClassification load Train-Data; X_train = X; labels_train = labelVector; load Test-Data; 8

X_test = X; labels_test = labelVector; V_pca = pca(X_train); nBasis = size( V_pca , 2 ); for( i = 1 : nBasis ) dispStr = [>>>>> Testing using the first , num2str(i) , eigen vectors ]; disp( dispStr ); V_subset = V_pca( : , 1 : i ); X_train_sub = V_subset * X_train; X_test_sub = V_subset * X_test; accuracy(i) = runNearestNeighborClassification( X_train_sub , labels_train , X_test_sub , labels_test ); end

%--------------------------------------------------------% lda.m %--------------------------------------------------------function W = lda( X , labelVector ) [ featureVectorSize , dataSetSize ] = size(X); uniqueLabels = unique( labelVector ); nClasses = length( uniqueLabels ); globalMean = mean( X , 2 ); phi_b = zeros( featureVectorSize , nClasses ); phi_w = zeros( featureVectorSize, dataSetSize); for( i = 1 : nClasses ) thisClassIndices = find( labelVector == uniqueLabels( i ) ); thisClassData = X( :, thisClassIndices ); thisClassMean = mean( thisClassData , 2 ); M = thisClassMean - globalMean; phi_b(:,i) = M; thisClassCount = length( thisClassIndices ); Y = thisClassData - repmat( thisClassMean , 1 , thisClassCount ); 9

phi_w( :, thisClassIndices) = Y; end Sb_trick = phi_b * phi_b; [V,D] = eigs( Sb_trick, nClasses ); V = phi_b * V; retained = length( find( diag(D)>0.05 ) ); Y = V(:,1:retained); Db = D(1:retained,1:retained); Z = Y * diag( (diag(Db)).^(-0.5)); phi_w_Z = phi_w*Z; Z_Sw_Z = phi_w_Z*phi_w_Z; [U,Dw] = eigs( Z_Sw_Z, size(Z_Sw_Z, 1) ); U = U(:,end:-1:1); W = Z*U; W = normalizeVectors(W);

%--------------------------------------------------------% runLDAClassification.m %--------------------------------------------------------function accuracy = runLDAClassification load Train-Data; X_train = X; labels_train = labelVector; load Test-Data ; X_test = X; labels_test = labelVector; V_lda = lda( X_train , labels_train ) ; nBasis = size( V_lda , 2 ) ; for( i = 1 : nBasis ) dispStr = [>>>>> Testing using the first , num2str(i) , of , num2str(nBasis) , eigen vectors ]; disp( dispStr ) ; V_subset = V_lda( : , 1 : i ) ; X_train_sub = V_subset * X_train ; X_test_sub = V_subset * X_test ;

10

accuracy(i) = runNearestNeighborClassification( X_train_sub , labels_train , X_test_sub , labels_test ); end

%--------------------------------------------------------% runNearestNeighborClassification.m %--------------------------------------------------------function p = runNearestNeighborClassification( X_train , labels_train, X_test , labels_test ) testCount = size( X_test , 2 ); trainCount = size( X_train, 2 ); correct = 0; for( i = 1 : testCount ) thisExample = X_test( : , i ); for( j = 1 : trainCount ) ssd(j) = sum( (thisExample(:) - X_train(:,j)).^2 ); end [minDiff , minIdx ] = min( ssd ); predicted_label( i ) = labels_train( minIdx ); if( predicted_label(i) == labels_test(i) ) correct = correct + 1; end end p = correct / testCount;

%--------------------------------------------------------% normalizeVectors.m %--------------------------------------------------------function K = normalizeVectors(V) sizeV = size(V); K = V; numberOfVectors = sizeV(2); V2 = V.^2; for i = 1 : numberOfVectors denom = sum(V2(:,i)); denom = sqrt(denom); 11

K(:,i) = V(:,i)/denom; end

%---------------------------------------------------------

12