Outline

Review of Linear Algebra

vectors and matrices products and norms vector spaces and linear transformations eigenvalues and eigenvectors

Introduction to Matlab

Why Linear Algebra?

For each data point, we will represent a set of features as feature vector
[length, weight, color,]

Collected data will be represented as collection of (feature) vectors

[l1, w 1 , c 1 , ] [l2, w2 , c2 ,] [l 3 , w3 , c3 ,]

Linear models are simple and computationally feasible

Vectors
X2

[x 1 , x 2 ]
X1

z y-z

n-dimensional row vector x = [x1 x2

x1 x Transpose of row vector is column vector xT = 2

Vector product (or inner or dot product)

xn ]

xn
i =1 k

x, y = x y = xT y = x1y1 + x2y2 + + xnyn =

xi yi

More on Vectors
Euclidian norm or length x =

x, x =

x
i =1 n

2 i

If |x|=1 we say x is normalized or unit length xT y Angle between vectors x and y cos = x y y x x y x
cos = 0 xT y = 0 xy
cos = 1 xT y =| x || y |> 0

y
cos = 1 xT y = | x || y |< 0

x orthogonal to y

Thus inner product captures direction relationship between x and y

More on Vectors
Vectors x and y are orthonormal if they are orthogonal and |x|=|y|=1 Euclidian distance between vectors x and y
xy =
x
i =1 n

( xi y i )
x-y y

Linear Dependence and Independence

Vectors x1, x2 , , xn are linearly dependent if there exist constants 1, 2 , , n s.t. 1. 1 x 1 + 2 x 2 + + n x n = 0 2. at least one i 0 Vectors x1, x2 , , xn are linearly independent if 1 x1 + 2 x2 + + n xn = 0 1 = = n = 0

Vector Spaces and Basis

The set of all n-dimensional vectors is called a vector space V A set of vectors u 1,u 2 , ,u n are called a basis for vector space if any v in V can be written as v = 1u 1 + 2u 2 + + nu n
u 1,u 2 , ,u n are independent implies they form a basis, and vice versa u 1,u 2 , ,u n give an orthonormal basis if 1. ui = 1 i 2. ui u j i j

Matrices
n by m matrix A and its m by n transpose A
T

x11 x12 A = x21 x22 xn1 xn2

x1m x2m xnm

x11 x12 x x T 12 22 A = x1m x2 m

xn1 x n2 xnm

Matrix Product
AB = a11 a12 a13 an1 an2 an3 a1d and b11 b21 b31 bd1 b1m b2m b3m = bdm cij =C

cij = a i , b j
a i is row i of A b j is column j of B

# of columns of A = # of rows of B even if defined, in general AB BA

Matrices
Rank of a matrix is the number of linearly independent rows (or equivalently columns) A square matrix is non-singular if its rank equal to the number of rows. If its rank is less than number of rows it is singular. 10 0 0 1 0 Identity matrix I = 00 AI=IA=A 00 1 1 2 9 5 T 2748 Matrix A is symmetric if A=A 9436 5864

Matrices
Matrix A is positive definite if xT Ax = Ai , j xi x j > 0
i,j

Matrix A is positive semi-definite if T x Ax = Ai , j xi x j 0

i,j

Trace of a square matrix A is sum on the elements on the diagonal

tr[A] =
n i =1

aii

Matrices
Inverse of a square matrix A is matrix A -1 s.t. AA = I
-1

If A is singular or not square, inverse does not exist. Pseudo-inverse A is defined T whenever A A is not singular (it is square) T -1 T A = (A A ) A -1 T T AA =(A A) AA=I

Matrices
Determinant of n by n matrix A is
det( A) =
n k =1

( 1)

k +i

aik det(Aik )

Where Aik obtained from A by removing the ith row and kth column Absolute value of determinant gives the volume of parallelepiped spanned by the matrix rows
1 2 n } a + a + ... + a 1 2 n

i [0 ,1] i

Linear Transformations
A linear transformation from vector space V to vector space U is a mapping which can be represented by a matrix M:
u = Mv

V U

M
Mv

If U and V have the same dimension, M is a square matrix In pattern recognition, often U has smaller dimensionality than V, i.e. transformation M is used to reduce the number of features. M v= u

Eigenvectors and Eigenvalues

Note: A0=0 for any , not interesting

Given n by n matrix A, and nonzero vector x. Suppose there is which satisfies Ax= x
x is called an eigenvector of A is called an eigenvalue of A Linear transformation A maps an eigenvector v in a simple way. Magnitude changes by , direction If > 0 If < 0
Av v v Av

Eigenvectors and Eigenvalues

If A is real and symmetric, then all eigenvalues are real (not complex) If A is non singular, all eigenvalues are non zero If A is positive definite, all eigenvalues are positive

MATLAB

Starting matlab
xterm -fn 12X24 matlab

Basic Navigation
quit more help general

Scalars, variables, basic arithmetic

Clear + - */ ^ help arith

Relational operators
==,&,|,~,xor help relop

Lists, vectors, matrices

A=[2 3;4 5] A

Matrix and vector operations

find(A>3), colon operator * / ^ .* ./ .^ eye(n),norm(A),det(A),eig(A) max,min,std help matfun

Elementary functions help elfun Data types double Char Programming in Matlab .m files scripts function y=square(x) help lang Flow control if i== 1else end, if else if end for i=1:0.5:2 end while i == 1 end Return help lang Graphics help graphics help graph3d File I/O load,save fopen, fclose, fprintf, fscanf