Objectives_template

Module 6: Dimensionality Reduction

Lecture 29: Singular Value Decomposition (SVD)

The Lecture Contains:

Singular Value Decomposition (SVD)
SVD of real symmetric matrix
Transformation using SVD
Example
Example: compact form

Dimensionality reduction using SVD

Example of dimensionality reduction (k = 1)

Compact way of dimensionality reduction (k = 1)

How many dimensions to retain?

file:///C|/Documents%20and%20Settings/iitkrana1/My%20Documents/Google%20Talk%20Received%20Files/ist_data/lecture29/29_1.htm[6/14/2012 3:51:38 PM]

Objectives_template

Module 6: Dimensionality Reduction

Lecture 29: Singular Value Decomposition (SVD)

Singular Value Decomposition (SVD)

Factorization of a matrix

If A is of size

Columns of

, then

is

is

and

is

matrix

are eigenvectors of

Left singular vectors

(orthonormal)

Columns of

are eigenvectors of

Right singular vectors

(orthonormal)

are the singular values

is diagonal
Singular values are positive squareroots of eigenvalues of
(assuming

or

singular values)

file:///C|/Documents%20and%20Settings/iitkrana1/My%20Documents/Google%20Talk%20Received%20Files/ist_data/lecture29/29_2.htm[6/14/2012 3:51:39 PM]

Objectives_template

Module 6: Dimensionality Reduction

Lecture 29: Singular Value Decomposition (SVD)

SVD of real symmetric matrix

A is real symmetric of size

since

is of size

and contains eigenvectors of

This is called spectral decomposition of

contains
Eigenvectors of

singular values
= eigenvectors of

Eigenvalues of

= squareroot of eigenvalues of

Eigenvalues of

= singular values of

file:///C|/Documents%20and%20Settings/iitkrana1/My%20Documents/Google%20Talk%20Received%20Files/ist_data/lecture29/29_3.htm[6/14/2012 3:51:39 PM]

Objectives_template

Module 6: Dimensionality Reduction

Lecture 29: Singular Value Decomposition (SVD)

Transformation using SVD

Transformed data

is called SVD transform matrix

Essentially,

is just a rotation of

Dimensionality of

is

different basis vectors than the original space

Columns of

give the basis vectors in rotated space

file:///C|/Documents%20and%20Settings/iitkrana1/My%20Documents/Google%20Talk%20Received%20Files/ist_data/lecture29/29_4.htm[6/14/2012 3:51:39 PM]

Objectives_template

Module 6: Dimensionality Reduction

Lecture 29: Singular Value Decomposition (SVD)

Transformation using SVD

Transformed data

is called SVD transform matrix

Essentially,

is just a rotation of

Dimensionality of

is

different basis vectors than the original space

Columns of

give the basis vectors in rotated space

shows how each dimension can be represented as a linear combination of other

dimensions
Columns are input basis vectors
shows how each object can be represented as a linear combination of other objects
Columns are output basis vectors

file:///C|/Documents%20and%20Settings/iitkrana1/My%20Documents/Google%20Talk%20Received%20Files/ist_data/lecture29/29_5.htm[6/14/2012 3:51:39 PM]

Objectives_template

Module 6: Dimensionality Reduction

Lecture 29: Singular Value Decomposition (SVD)

Transformation using SVD

Transformed data

is called SVD transform matrix

Essentially,

is just a rotation of

Dimensionality of

is

different basis vectors than the original space

Columns of

give the basis vectors in rotated space

shows how each dimension can be represented as a linear combination of other

dimensions
Columns are input basis vectors
shows how each object can be represented as a linear combination of other objects
Columns are output basis vectors
Lengths of vectors are preserved

file:///C|/Documents%20and%20Settings/iitkrana1/My%20Documents/Google%20Talk%20Received%20Files/ist_data/lecture29/29_6.htm[6/14/2012 3:51:39 PM]

Objectives_template

Module 6: Dimensionality Reduction

Lecture 29: Singular Value Decomposition (SVD)

Example

file:///C|/Documents%20and%20Settings/iitkrana1/My%20Documents/Google%20Talk%20Received%20Files/ist_data/lecture29/29_7.htm[6/14/2012 3:51:40 PM]

Objectives_template

Module 6: Dimensionality Reduction

Lecture 29: Singular Value Decomposition (SVD)

Example: compact form

If

is of size

, then

Works because there at most

is

is

and

is

matrix

non-zero singular values in

file:///C|/Documents%20and%20Settings/iitkrana1/My%20Documents/Google%20Talk%20Received%20Files/ist_data/lecture29/29_8.htm[6/14/2012 3:51:40 PM]

Objectives_template

Module 6: Dimensionality Reduction

Lecture 29: Singular Value Decomposition (SVD)

Dimensionality reduction using SVD

Use only
Retain first
First

dimensions
columns for

columns of

Best rank

and

and first

values for

give the basis vectors in reduced space

approximation in terms of sum squared error

file:///C|/Documents%20and%20Settings/iitkrana1/My%20Documents/Google%20Talk%20Received%20Files/ist_data/lecture29/29_9.htm[6/14/2012 3:51:40 PM]

Objectives_template

Module 6: Dimensionality Reduction

Lecture 29: Singular Value Decomposition (SVD)

Example of dimensionality reduction (

= 1)

file:///C|/Documents%20and%20Settings/iitkrana1/My%20Documents/Google%20Talk%20Received%20Files/ist_data/lecture29/29_10.htm[6/14/2012 3:51:40 PM]

Objectives_template

Module 6: Dimensionality Reduction

Lecture 29: Singular Value Decomposition (SVD)

Compact way of dimensionality reduction (

= 1)

file:///C|/Documents%20and%20Settings/iitkrana1/My%20Documents/Google%20Talk%20Received%20Files/ist_data/lecture29/29_11.htm[6/14/2012 3:51:40 PM]

Objectives_template

Module 6: Dimensionality Reduction

Lecture 29: Singular Value Decomposition (SVD)

How many dimensions to retain?

file:///C|/Documents%20and%20Settings/iitkrana1/My%20Documents/Google%20Talk%20Received%20Files/ist_data/lecture29/29_12.htm[6/14/2012 3:51:41 PM]

Objectives_template

Module 6: Dimensionality Reduction

Lecture 29: Singular Value Decomposition (SVD)

How many dimensions to retain?

There is no easy answer

file:///C|/Documents%20and%20Settings/iitkrana1/My%20Documents/Google%20Talk%20Received%20Files/ist_data/lecture29/29_13.htm[6/14/2012 3:51:41 PM]

Objectives_template

Module 6: Dimensionality Reduction

Lecture 29: Singular Value Decomposition (SVD)

How many dimensions to retain?

There is no easy answer
Concept of energy of a dataset
Total energy is sum of squares of singular values (aka spread or variance)

Retain

dimensions such that

Generally,

is between

of the energy is retained

to

file:///C|/Documents%20and%20Settings/iitkrana1/My%20Documents/Google%20Talk%20Received%20Files/ist_data/lecture29/29_14.htm[6/14/2012 3:51:41 PM]

Objectives_template

Module 6: Dimensionality Reduction

Lecture 29: Singular Value Decomposition (SVD)

How many dimensions to retain?

There is no easy answer
Concept of energy of a dataset
Total energy is sum of squares of singular values (aka spread or variance)

Retain

dimensions such that

Generally,

is between

In the above example,

of the energy is retained

to

= 1 retains

of the energy

file:///C|/Documents%20and%20Settings/iitkrana1/My%20Documents/Google%20Talk%20Received%20Files/ist_data/lecture29/29_15.htm[6/14/2012 3:51:41 PM]

Objectives_template

Module 6: Dimensionality Reduction

Lecture 29: Singular Value Decomposition (SVD)

How many dimensions to retain?

There is no easy answer
Concept of energy of a dataset
Total energy is sum of squares of singular values (aka spread or variance)

Retain

dimensions such that

Generally,

is between

of the energy is retained

to

In the above example,

= 1 retains

Running time:

for

of size

of the energy

and rank

file:///C|/Documents%20and%20Settings/iitkrana1/My%20Documents/Google%20Talk%20Received%20Files/ist_data/lecture29/29_16.htm[6/14/2012 3:51:41 PM]