Finding Eigenvalues and

Eigenvectors

What is really
important?

Approaches

Find the characteristic polynomial

Find the largest or smallest eigenvalue

Power Method
Inverse Power Method

Find all the eigenvalues

Leverriers Method

Jacobis Method
Householders Method
QR Method
Danislevskys Method
02/14/15

DRAFT Copyright, Gene A

Tagliarini, PhD

Finding the Characteristic Polynomial

Reduces

to finding the coefficients of the

polynomial for the matrix A
Recall |I-A| = n+ann-1+an-1n-2++a21+a1
Leverriers

Method

Set Bn = A and an = -trace(Bn)

For k = (n-1) down to 1 compute

Bk

= A (Bk+1 + ak+1I)

ak

= - trace(Bk)/(n k + 1)
02/14/15

DRAFT Copyright, Gene A

Tagliarini, PhD

Vectors that Span a Space and Linear

Combinations of Vectors
Given

a set of vectors v1, v2,, vn

The vectors are said span a space V, if given
any vector x V, there exist constants
c1, c2,, cn so that
c1v1 + c2v2 ++ cnvn = x and x is called a
linear combination of the vi

02/14/15

DRAFT Copyright, Gene A

Tagliarini, PhD

Linear Independence and a Basis

Given

a set of vectors v1, v2,, vn and

constants c1, c2,, cn

The

vectors are linearly independent if the

only solution to c1v1 + c2v2 ++ cnvn = 0 (the
zero vector) is c1= c2==cn = 0

A linearly

independent, spanning set is called

a basis
5

02/14/15

DRAFT Copyright, Gene A

Tagliarini, PhD

Example 1: The Standard Basis

Consider the vectors v1 = <1, 0, 0>, v2 = <0, 1, 0>,

and v3 = <0, 0, 1>
Clearly, c1v1 + c2v2 + c3v3 = 0 c1= c2= c3 = 0
Any vector <x, y, z> can be written as a linear
combination of v1, v2, and v3 as
<x, y, z> = x v1 + y v2 + z v3
The collection {v1, v2, v3} is a basis for R3; indeed, it
is the standard basis and is usually denoted with
vector names i, j, and k, respectively.
02/14/15

DRAFT Copyright, Gene A

Tagliarini, PhD

Another Definition and Some Notation

Assume

that the eigenvalues for an n x n

matrix A can be ordered such that
|1| > |2| |3| |n-2| |n-1| > |n|

1 is the dominant eigenvalue and |1| is

the spectral radius of A, denoted (A)
The ith eigenvector will be denoted using
superscripts as xi, subscripts being reserved
for the components of x
Then

02/14/15

DRAFT Copyright, Gene A

Tagliarini, PhD

Power Methods: The Direct Method

Assume

an n x n matrix A has n linearly

independent eigenvectors e1, e2,, en
ordered by decreasing eigenvalues
|1| > |2| |3| |n-2| |n-1| > |n|

Given

any vector y0 0, there exist constants

ci, i = 1,,n, such that
y0 = c1e1 + c2e2 ++ cnen

02/14/15

DRAFT Copyright, Gene A

Tagliarini, PhD

The Direct Method (continued)

If

y0 is not orthogonal to e1, i.e., (y0)Te1 0,

y1 = Ay0 = A(c1e1 + c2e2 ++ cnen)

= Ac1e1 + Ac2e2 ++ Acnen

= c1Ae1 + c2Ae2 ++ cnAen

Can

you simplify the previous line?

02/14/15

DRAFT Copyright, Gene A

Tagliarini, PhD

The Direct Method (continued)

If

y0 is not orthogonal to e1, i.e., (y0)Te1 0,

y1 = Ay0 = A(c1e1 + c2e2 ++ cnen)

= Ac1e1 + Ac2e2 ++ Acnen

= c1Ae1 + c2Ae2 ++ cnAen

y1 =

c11e1 + c22e2 ++ cnnen

What

10

is y2 = Ay1?

02/14/15

DRAFT Copyright, Gene A

Tagliarini, PhD

The Direct Method (continued)

n

y c1 e c2 e ... cn e ck 2k e k
2

2 1
1

2
2

2
n

k 1

in general,
n

y i c11i e1 c2 i2e 2 ... cn in e n ck ik e k

k 1

or

e
y c1e ck

1
k 2

11

i
1

02/14/15

DRAFT Copyright, Gene A

Tagliarini, PhD

The Direct Method (continued)

y i 1i c1e1 ck k ek

1
k 2

So what?
n

k
k
0 as i
Recall that
1, so
1
1
12

02/14/15

DRAFT Copyright, Gene A

Tagliarini, PhD

The Direct Method (continued)

Since

y i 1i c1e1 ck k e k

1
k 2

then
n

y i 1i c1e1 as i
13

02/14/15

DRAFT Copyright, Gene A

Tagliarini, PhD

The Direct Method (continued)

Note:

any nonzero multiple of an eigenvector

is also an eigenvector
Why?
Suppose e is an eigenvector of A, i.e., Ae=e
and c0 is a scalar such that x = ce
Ax = A(ce) = c (Ae) = c (e) = (ce) = x

14

02/14/15

DRAFT Copyright, Gene A

Tagliarini, PhD

The Direct Method (continued)

Since y i 1i c1e1 and e1 is an eigenvector
i

y will become arbitrarily close to an eigenvector

and we can prevent y i from growing inordinately
by normalizing
i 1
Ay
i
y
Ay i 1

15

02/14/15

DRAFT Copyright, Gene A

Tagliarini, PhD

Direct Method (continued)

Given

an eigenvector e for the matrix A

We have Ae = e and e0, so eTe 0 (a
scalar)
Thus, eTAe = eTe = eTe 0
So = (eTAe) / (eTe)

16

02/14/15

DRAFT Copyright, Gene A

Tagliarini, PhD

Direct Method (completed)

y Ay

y y
i T

i T

approximates the dominant eigenvalue

r i i I A y i is the residual error vector

and
r i 0 when i is an eigenvalue with eigenvector y i
17

02/14/15

DRAFT Copyright, Gene A

Tagliarini, PhD

Direct Method Algorithm

1. Choose 0, m 0, and y 0.
Set i 0 and compute x Ay and .
2. Do
y

x
x

x Ay
yT x
T
y y
r y - x
i i 1
3. While ( r ) and (i m)

18

02/14/15

DRAFT Copyright, Gene A

Tagliarini, PhD

Jacobis Method
Requires

a symmetric matrix
May take numerous iterations to converge
Also requires repeated evaluation of the
arctan function
Isnt

there a better way?

Yes, but we need to build some tools.
19

02/14/15

DRAFT Copyright, Gene A

Tagliarini, PhD

What Householders Method Does

Preprocesses

a matrix A to produce an
upper-Hessenberg form B
The eigenvalues of B are related to the
eigenvalues of A by a linear transformation
Typically, the eigenvalues of B are easier to
obtain because the transformation simplifies
computation
20

02/14/15

DRAFT Copyright, Gene A

Tagliarini, PhD

Definition: Upper-Hessenberg Form

A matrix

B is said to be in upper-Hessenberg
form if it has the following structure:
b1,1 b1,2 b1,3 b1,n -1
b b b b
2, n -1
2,1 2,2 2,3

0 b3,2 b 3,3
B
0 0 b 4,3

0 0

21

b3,n -1

b n -1,n -1

b1,n
b 2,n
b3,n

b n -1,n

b n,n

b n,n -1

02/14/15

DRAFT Copyright, Gene A

Tagliarini, PhD

A Useful Matrix Construction

an n x 1 vector u 0
Consider the matrix P(u) defined by
P(u) = I 2(uuT)/(uTu)
Where
Assume

22

I is the n x n identity matrix

(uuT) is an n x n matrix, the outer product of u with
its transpose
(uTu) here denotes the trace of a 1 x 1 matrix and
is the inner or dot product
02/14/15

DRAFT Copyright, Gene A

Tagliarini, PhD

Properties of P(u)

P2(u) = I

P-1(u) = P(u)

23

P(u) is its own inverse

PT(u) = P(u)

The notation here P2(u) = P(u) * P(u)

Can you show that P2(u) = I?

P(u) is its own transpose

Why?

P(u) is an orthogonal matrix

02/14/15

DRAFT Copyright, Gene A

Tagliarini, PhD

Householders Algorithm
Set

Q = I, where I is an n x n identity matrix

For k = 1 to n-2

= sgn(Ak+1,k)sqrt((Ak+1,k)2+ (Ak+2,k)2++ (An,k)2)

uT = [0, 0, , Ak+1,k+ , Ak+2,k,, An,k]
P = I 2(uuT)/(uTu)
Q = QP
A = PAP

Set

24

B=A
02/14/15

DRAFT Copyright, Gene A

Tagliarini, PhD

Example
1 2 3
Let A 3 5 6 .
4 8 9 3 x 3
Clearly, n 3 and since n - 2 1, k takes only the value k 1.
2
2
Then sgn(a 21 ) a21
a31
sgn(3) 32 4 2 1 5 5

25

02/14/15

DRAFT Copyright, Gene A

Tagliarini, PhD

Example
uT [0,...,0, a21 , a31 ,..., an1 ] [0, a21 , a31 ]
[0 ,3 5,4] [0,8,4]

1 0 0
uu
P I 2 T 0 1 0
u u
0 0 1
T

26

02/14/15

0
8 0 8 4

4
2
?
0
0 8 4 8
4

DRAFT Copyright, Gene A

Tagliarini, PhD

Example
1 0 0
uu
P I 2 T 0 1 0
u u
0 0 1
T

0
8 0 8 4

4
2
0
0 8 4 8
4

1 0 0
0 0 0 1

0 1 0 0 64 32 0
80
0 0 1
0 32 16
0

27

02/14/15

0
3
5
4
5

0
4
2
. Find P ?
5
3
5
DRAFT Copyright, Gene A
Tagliarini, PhD

Example
Initially, Q I, so

1 0 0 1

Q QP 0 1 0 0
0 0 1
0

28

02/14/15

0
3
5
4
5

0 1
4
0
5
3
0
5

0
3
5
4
5

0
4

5
3
5

DRAFT Copyright, Gene A

Tagliarini, PhD

Example
Next, A PAP , so

A 0

29

0
3
5
4
5

0
4

5
3
5

1 2 3 1
3 5 6 0

4 8 9
0

02/14/15

0
3
5
4
5

0
4
?
5
3
5

DRAFT Copyright, Gene A

Tagliarini, PhD

Example
Hence,

3
A 0
5

0 4

5
- 18

357
-5
25

0 - 24

25

30

0 1 2 3 1

3 5 6 0
5

3 4 8 9
0

1
5
26

25
-7
25

0
3
5
4
5

02/14/15

0 1
4
5
5
3
0

2
47
5
4
5

3
54

5
3
5

0
3
5
4
5

DRAFT Copyright, Gene A

Tagliarini, PhD

0
4

5
3
5

Example
Finally, since the loop only executes once

- 18
1
5

357
B A -5
25

0 - 24

25
So what?

31

02/14/15

1
5
26
.
25
-7
25
DRAFT Copyright, Gene A
Tagliarini, PhD

How Does It Work?

Householders algorithm uses a sequence of

similarity transformations
B = P(uk) A P(uk)
to create zeros below the first sub-diagonal
uk=[0, 0, , Ak+1,k+ , Ak+2,k,, An,k]T
= sgn(Ak+1,k)sqrt((Ak+1,k)2+ (Ak+2,k)2++ (An,k)2)
By definition,

32

sgn(x) = 1, if x0 and
sgn(x) = -1, if x<0
02/14/15

DRAFT Copyright, Gene A

Tagliarini, PhD

How Does It Work? (continued)

The

33

matrix Q is orthogonal

the matrices P are orthogonal

Q is a product of the matrices P
The product of orthogonal matrices is an
orthogonal matrix

= QT A Q hence Q B = Q QT A Q = A Q
Q QT = I (by the orthogonality of Q)

02/14/15

DRAFT Copyright, Gene A

Tagliarini, PhD

How Does It Work? (continued)

If

ek is an eigenvector of B with eigenvalue

k, then B ek = k ek

Since

Q B = A Q,
A (Q ek) = Q (B ek) = Q (k ek) = k (Q ek)

Note

34

from this:

k is an eigenvalue of A
Q ek is the corresponding eigenvector of A

02/14/15

DRAFT Copyright, Gene A

Tagliarini, PhD

The QR Method: Start-up

Given

a matrix A
Apply Householders Algorithm to obtain a
matrix B in upper-Hessenberg form
Select

35

>0 and m>0

is a acceptable proximity to zero for subdiagonal elements

m is an iteration limit

02/14/15

DRAFT Copyright, Gene A

Tagliarini, PhD

The QR Method: Main Loop

Do {
Set Q T I
For k 1 to n - 1{
c

Bk ,k
2
k ,k

2
k 1, k

; s

Bk 1, k
2
k ,k

2
k 1, k

Set P I; Pk,k Pk 1,k 1 c; Pk 1, k Pk,k 1 s;

B PB;
Q T PQ T ;
}
B BQ;
i ;
} While (B is not upper block triangular) and (i m)

36

02/14/15

DRAFT Copyright, Gene A

Tagliarini, PhD

The QR Method: Finding The s

Since B is upper block triangular, one may compute k from
the diagonal blocks of B. Specifically, the eigvalues of B are
the eigenvalues of its diagonal blocks B k .

If a diagonal block B k is 1x1, i.e., B k a , then k a.

a b
If a diagonal block B k is 2x2, i.e., B k
,

c d
trace(B k ) trace 2 (B k ) 4 det(B k )
then k ,k 1
.
2

37

02/14/15

DRAFT Copyright, Gene A

Tagliarini, PhD

Details Of The Eigenvalue Formulae

a b
Suppose B k
.

c d
a b
I B k

c d

I B k ?
38

02/14/15

DRAFT Copyright, Gene A

Tagliarini, PhD

Details Of The Eigenvalue Formulae

a b
Given I B k

c d

I B k ( a )( d ) bc
(a d ) ad bc
2

trace(B k ) det(B k )
2

39

02/14/15

DRAFT Copyright, Gene A

Tagliarini, PhD

Finding Roots of Polynomials

Every

n x n matrix has a characteristic

polynomial
Every polynomial has a corresponding n x n
matrix for which it is the characteristic
polynomial
Thus, polynomial root finding is equivalent to
finding eigenvalues
40

02/14/15

DRAFT Copyright, Gene A

Tagliarini, PhD

Example Please!?!?!?
Consider

the monic polynomial of degree n

f(x) = a1 +a2x+a3x2++anxn-1 +xn
and the companion matrix
an an1 a2 a1
1

0
0

A 0
1 0
0

0 0
0
0
0
0
0 1
0

41

02/14/15

DRAFT Copyright, Gene A

Tagliarini, PhD

Find The Eigenvalues of the

Companion Matrix

an an1 a2 a1
1

0
0

I A I 0
1 0
0 ?

0 0
0
0
0
0
0 1
0
42

02/14/15

DRAFT Copyright, Gene A

Tagliarini, PhD

Find The Eigenvalues of the

Companion Matrix
an an1 a2
1
0
I A 0
1 0
0
0
0
0 0 1

a1
0
0
0

an1 a2 a1
( an )n1 (1)

43

1 0 0
0 0
0 0 1
02/14/15

DRAFT Copyright, Gene A

Tagliarini, PhD