LECTURE 3: Review of Linear Algebra and MATLAB®

g Vector and matrix notation

g Vectors

g Matrices

g Vector spaces

g Linear transformations

g Eigenvalues and eigenvectors

g MATLAB
® primer

Introduction to Pattern Analysis 1

Ricardo Gutierrez-Osuna
Texas A&M University
Vector and matrix notation
g A d-dimensional (column) vector x and its transpose are written as:
 x1 
x 
x =  2  and x T = [x1 x1 L x d ]
M
 
x d 

g An n×d (rectangular) matrix and its transpose are written as:

 a11 a 21 L an1 
a an 2 
a11 a12 a13 L a1d   12 a 22 L
a a 22 a 23 L 
a 2d  a a 23 L an3 
A =  21 and A T =  13 
 M M M O M   M M O M 
   
an1 an2 an3 and 
 
 a1d a2d and 

g The product of two matrices is

 b11 b12 L b1n   c 11 c 12 c 13 L c1n 
 a11 a12 a13 L a1d  
a b 21 b 22 L b 2n   c 21 c 22 c 23 L c 2n 
a 22 a 23 L a 2d   d
AB =  21 b b 32 L b 3n  =  c 31 c 32 c 33 L c 3n  where c ij = ∑ aikbkj
 M M M O   31    k =1
  M M O   M M M O
 am1 am2 am3 amd   
b d1 b d2 b dn   c m1 c m2 c m3 c mn 

Introduction to Pattern Analysis 2

Ricardo Gutierrez-Osuna
Texas A&M University
Vectors
g The inner product (a.k.a. dot product or scalar product) of two vectors is defined by
d
x, y = x y = y x = ∑ x k y k
T T

k =1

g The magnitude of a vector is

1/ 2
d 
x = x x = ∑ x k x k 
T

 k =1  y
x
g The orthogonal projection of vector y onto vector x is
θ
T
y ux ux ux
n where vector ux has unit magnitude and the same direction as x yTux
g The angle between vectors x and y is
x, y |x|
cos θ =
x⋅y

g Two vectors x and y are said to be

n orthogonal if xTy=0
n orthonormal if xTy=0 and |x|=|y|=1
g A set of vectors x1, x2, …, xn are said to be linearly dependent if there exists a set of
coefficients a1, a2, …, an (at least one different than zero) such that
a1x1 + a 2 x 2 + L an x n = 0
g Alternatively, a set of vectors x1, x2, …, xn are said to be linearly independent if
a1x1 + a 2 x 2 + L an x n = 0 ⇒ ak = 0 ∀k
Introduction to Pattern Analysis 3
Ricardo Gutierrez-Osuna
Texas A&M University
Matrices
g The determinant of a square matrix Ad×d is
d
A = ∑ aik A ik ( −1)k +i
k =1

n where Aik is the minor matrix formed by removing the ith row and the kth column of A
n NOTE: the determinant of a square matrix and its transpose is the same: |A|=|AT|
g The trace of a square matrix Ad×d is the sum of its diagonal elements
d
tr ( A ) = ∑ akk
k =1

g The rank of a matrix is the number of linearly independent rows (or columns)
g A square matrix is said to be non-singular if and only if its rank equals the number of rows
(or columns)
n A non-singular matrix has a non-zero determinant
g A square matrix is said to be orthonormal if AAT=ATA=I (more on this later)
g For a square matrix A
n if xTAx>0 for all x≠0, then A is said to be positive-definite (i.e., the covariance matrix)
n if xTAx≥0 for all x≠0, then A is said to be positive-semidefinite
g The inverse of a square matrix A is denoted by A-1 and is such that AA-1= A-1A=I
n The inverse A-1 of a matrix A exists if and only if A is non-singular
g The pseudo-inverse matrix A† is typically used whenever A-1 does not exist (because A is
not square or A is singular):
[
A† = ATA ]−1
A T with A † A = I (assuming A T A is non - singular, note that AA † ≠ I in general )

Introduction to Pattern Analysis 4

Ricardo Gutierrez-Osuna
Texas A&M University
Vector spaces
g The n-dimensional space in which all the n-dimensional vectors u3
reside is called a vector space
g A set of vectors {u1, u2, ... un} is said to form a basis for a vector
a3
space if any arbitrary vector x can be represented by a linear
combination of the {ui}
x = a1u1 + a 2u2 + L anun a

n The coefficients {a1, a2, ... an} are called the components of vector x with respect a1
u1
to the basis {ui}
n In order to form a basis, it is necessary and sufficient that the {ui} vectors be a2
linearly independent
T ≠ 0 i= j u2
g A basis {ui} is said to be orthogonal if ui u j 
= 0 i≠ j
T 1 i= j
g A basis {ui} is said to be orthonormal if ui u j =  u3
0 i≠ j
dE(x,y)
g As an example, the Cartesian coordinate base is an orthonormal base
g Given n linearly independent vectors {x1, x2, ... xn}, we can construct x-y
an orthonormal base {φ1, φ 2, ... φ n} for the vector space spanned by x

{xi} with the Gram-Schmidt Orthonormalization Procedure y

g The distance between two points in a vector space is defined as the u1
magnitude of the vector difference between the points
1/ 2
d 2
dE ( x, y ) = x − y = ∑ (x k − y k )  u2
 k =1 
n This is also called the Euclidean distance

Introduction to Pattern Analysis 5

Ricardo Gutierrez-Osuna
Texas A&M University
Linear transformations
g A linear transformation is a mapping from a vector space XN onto a vector
space YM, and is represented by a matrix
n Given vector x∈XN, the corresponding vector y on YM is computed as
x 
 y1   a11 a12 a13 L a1N   1 
y   a x2
 2  =  21 a 22 a 23 L a 2N   
M
 M   M M M O  
   
 y M   aM1 aM 2 aM3 aMN   
 x N 
n Notice that the dimensionality of the two spaces does not need to be the same.
n For pattern recognition we typically have M<N (project onto a lower-dimensionality space)
g A linear transformation represented by a square matrix A is said to be
orthonormal when AAT=ATA=I
n This implies that AT=A-1
n An orthonormal transformation has the property of preserving the magnitude of the vectors:
y = y T y = ( Ax )T ( Ax ) = x T A T Ax = x T x = x

n An orthonormal matrix can be thought of as a rotation of the reference frame

n The row vectors of an orthonormal transformation form a set of orthonormal basis vectors
← a1 →
← a → 0 i ≠ j
y M×1 =  2
x N×1 with aiT a j = 
  1 i = j
 
← a N →
Introduction to Pattern Analysis 6
Ricardo Gutierrez-Osuna
Texas A&M University
Eigenvectors and eigenvalues
g Given a matrix AN×N, we say that v is an eigenvector* if there exists a scalar λ
(the eigenvalue) such that
v is an eigenvecto r
Av = λv ⇔ 
λ is the corresponding eigenvalue
g Computation of the eigenvalues
v = 0 trivial solution
Av = λv ⇒ Av − λv = 0 ⇒ (A − λI)v = 0 ⇒ 
(A − λI) = 0 non − trivial solution
(A − λI) = 0 ⇒ A − λI = 0 ⇒ λ1N 4
+ a1λN−1 + L aN−1λ + a0 = 0
44442444443
Characteristic Equation
g The matrix formed by the column eigenvectors is called the modal matrix M.
n Matrix Λ is the canonical form of A: a diagonal matrix with eigenvalues on the main diagonal
   λ1 
   
↑ ↑ ↑ ↑  λ2 
M =  v1 v2 v3 L vN  Λ =  O 
   
↓ ↓ ↓ ↓  
g Properties 



 λN 
n If A is non-singular
g All eigenvalues are non-zero
n If A is real and symmetric
g All eigenvalues are real
g The eigenvectors associated with distinct eigenvalues are orthogonal
n If A is positive definite
g All eigenvalues are positive
Introduction to Pattern Analysis 7
*The "eigen-" in "eigenvector" is normally translated as "characteristic"
Ricardo Gutierrez-Osuna
Texas A&M University
Interpretation of eigenvectors and eigenvalues (1)
g If we view matrix A as a linear transformation, an eigenvector represents an
invariant direction in the vector space
n When transformed by A, any point lying on the direction defined by v will remain on that
direction, and its magnitude will be multiplied by the corresponding eigenvalue λ

P’
y2

x2 P P
y=Ax
v v d’=λd
d

x1 y1

n For example, the transformation which rotates 3-d vectors about the Z axis has vector [0 0 1]
as its only eigenvector and 1 as the corresponding eigenvalue

 cos β − sin β 0 
A =  sin β cos β 0 

v = [0 0 1]
T
 0 0 1 
x

Introduction to Pattern Analysis 8

Ricardo Gutierrez-Osuna
Texas A&M University
 Interpretation of eigenvectors and eigenvalues (2)
g Given the covariance matrix Σ of a Gaussian distribution
n The eigenvectors of Σ are the principal directions of the distribution
n The eigenvalues are the variances of the corresponding principal directions
g The linear transformation defined by the eigenvectors of Σ leads to vectors that
are uncorrelated regardless of the form of the distribution
n If the distribution happens to be Gaussian, then the transformed vectors will be statistically
independent
↑ ↑ ↑ λ1 
 
ΣM = MΛ with M =  v1 v2 L v N  and Λ =  

↓ ↓ ↓   λN 


1  1  N
1  (y i − µyi )2 
f X (x) = 1/2
exp − (X − µ)T ∑ −1(X − µ) fY (y) = ∏ exp − 
(2 π)N/2 ∑  2  i=1 2 π λi  2λ i 

x2 y2
y=MTx

µx2

µy2
v2 v1

µx1 x1 µy1 y1

Introduction to Pattern Analysis 9

Ricardo Gutierrez-Osuna
Texas A&M University
MATLAB primer
g The MATLAB environment g I/O (help iofun)
n Starting and exiting MATLAB n Console I/O
n Directory path g The fprintf and sprintf commands
n The startup.m file g the input command
n The help command n File I/O
n The toolboxes g load and save commands
g Basic features (help general) g The fopen, fclose, fprintf and fscanf commands
n Variables g 2D Graphics (help graph2d)
n Special variables (i, NaN, eps, realmax, realmin, pi, …) n The plot command
n Arithmetic, relational and logic operations n Customizing plots
n Comments and punctuation (the semicolon shorthand) g Line styles, markers and colors
n Math functions (help elfun) g Grids, axes and labels
g Arrays and matrices n Multiple plots and subplots
n Array construction n Scatter-plots
g Manual construction n The legend and zoom commands
g The 1:n shorthand g 3D Graphics (help graph3d)
g The linspace command n Line plots
n Matrix construction n Mesh plots
g Manual construction n image and imagesc commands
g Concatenating arrays and matrices n 3D scatter plots
n Array and Matrix indexing (the colon shorthand) n the rotate3d command
n Array and matrix operations g Linear Algebra (help matfun)
g Matrix and element-by-element operations n Sets of linear equations
n Standard arrays and matrices (eye, ones and zeros) n The least-squares solution (x = A\b)
n Array and matrix size (size and length) n Eigenvalue problems
n Character strings (help strfun) g Statistics and Probability
g String generation n Generation
g The str2mat function g Random variables
g M-files n Gaussian distribution: N(0,1) and N(µ,σ)
n Script files n Uniform distribution
n Function files g Random vectors
g Flow control n correlated and uncorrelated variables
n if..else..end construct n Analysis
n for construct g Max, min and mean
n while construct g Variance and Covariance
n switch..case construct g Histograms

Introduction to Pattern Analysis 10

Ricardo Gutierrez-Osuna
Texas A&M University