Matrix algorithms

(part 1)

Computational Techniques II

Contents
1 System of linear equations

2 Gaussian elimination

3 Gauss-Jordan elimination

4 Pivoting strategies
4.1 Partial pivoting . . . .
4.2 Complete pivoting . .
4.3 Keeping track of swaps
4.3.1 Method 1 . . .
4.3.2 Method 2 . . .

9
9
11
12
12
13

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

5 Some theory behind Gauss-Jordan elimination

14

6 Inversion using Gauss-Jordan elimination

15
6.1 Ecient implementation . . . . . . . . . . . . . . . . . . . . . . . 16
7 Determinant using Gaussian elimination

18

8 LU
8.1
8.2
8.3
8.4

18
19
19
20
20

decompositions
Application . . . . . . .
Algorithm . . . . . . . .
Ecient implementation
Some theory . . . . . . .

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

9 QR
9.1
9.2
9.3
9.4

decomposition
Householders transformation
Using Householder for QR . .
Ecient implementation . . .
Application to least squares .

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

21
22
22
23
27

System of linear equations

Example: Consider the following three equations:

2x 3y + z

3x + y + 2z
2x + 3z

=
=

15
2

This is an example of a system of linear equations. Here system means a

collection, and the equations are called linear because they do not involve
any squares or higher powers of the unknowns like x2 , z 3 , or product terms like
xy, xy 3 z etc.
In the above example we have three unknowns: x, y and z.
We can write a system of m linear equations in n unknowns using matrix notation:
Ax = b,
where Amn is called the coecient matrix, xm1 is called the vector of unknowns, and bn1 is called the rhs vector.
Example: The system in the last example can be written as

2
3
2

3 1
x
5
1 2 y = 15 .
0 3
z
2

Thus, here

2 3 1
A = 3 1 2 ,
2 0 3

x
x = y ,
z

5
b = 15 .
2

If the rhs vector b is 0, then we call the system homogeneous, else we call it
non-homogeneous.
Any value of x for which Ax = b, is called a solution of the system. A system
of linear equations has either exactly one solution, or innitely many solutions
or no solution at all. In the last case we call the system inconsistent, otherwise
it is called consistent.

If A is a nonsingular matrix then for any b, the system

Ax = b
is consistent, and has exactly one solution, viz., x = A1 b.

Gaussian elimination

This is the matrix version of a method for solving a system of linear equations
that we learn at high school.
Example: Consider the following system of three linear equations, which we
call 1 , 1 and 1 .
1 :
1 :
1 :

x y
2x +5y
x +2y

+z
z
3z

=
2
=
9
= 4

In high school we used to solve this by eliminating the unknowns one by one
until only one remained. Then we could solve for that one. Once we know one
of the unknowns we substitute this value in the equations to get a system with
only two unknowns. Then we eliminated one of these, and so on.
Example: Let us rst eliminate x from the last two equations by subtracting
multiples of the rst equation from them. Here are the resulting 3 equations,
which we call 2 , 2 and 2 .
2 =
2 =
2 =

1 :
1 21 :
1 1 :

y
7y
3y

+z
3z
4z

=
=
=

+z
3z
19 z
7

=
=
=

2
5
6

Now let us eliminate y from the last equation

3 =
2 :
3 =
2 :
3
3 = 2 7 2 :

y
7y

2
5
57
7

Now the third equation involves only one unknown, z. So we can solve for z.
The second equation involves both z and y. So once we have solved for z from
the third equation we can use the second equation to solve for y. The rst

equation (which involves x as well as y and z) can then be used for solving x.
This approach is called back substitution. From the third equation:
z = 3,
then from the second equation
y=

1
(5 + 3z) = 2,
7

and nally, from the rst equation,

x = 2 + y z = 1.

This is what is called Gaussian elimination (with back substitution) in matrix

theory. While doing Gaussian elimination it is customary to write the system
at each step in the augmented matrix form. This is done in the example below.
Example: The augmented matrix form of the given system is as follows.

1
2
1

1
1
2
9
5 1
2 3 4

It is obtained by appending the rhs after the matrix. We draw a vertical line to
keep the rhs separate.
While doing Gaussian elimination we work with only the augmented matrices.
To move from one augmented matrix to the next we need certain kind of operation that we call an F -move. Before each F -move we have to choose some
nonzero entry in the matrix. This is called the pivot for that step. The position
of the pivot may change from step to step. The row (or column) that contains
the pivot is called the pivotal row (or column).
Forward elimination (F): In this move we add multiples of the pivotal
row to the rows below it so that the entries in the pivotal column below
the pivot become zero. In the example below, we need to add 5 times the
rst row to the second row, and 2 times the rst row to the third row.

1
2
1

-1
5
2

1
2
9
1
3 4

1
7
3

-1
0
0

1 2
4 19
1 0

This operation only aects the rows below the pivotal row. The pivotal
row itself, as well as the rows above it remain unchanged.
5

An F -move may be computed easily if you remember the following diagram.

Here p is the pivot, and we want to know how a will change by a F - or B-move.
First locate the entry b in the pivotal row which is in the same column as a.
Similarly, let c be the entry in the pivotal column having the same row as a.
Then a F -move will change a to a bc/p.
pivotal column

pivotal
row

a could also be to the left of the pivotal column.

Example: Let us use this method see how the bold entry changes when we
apply F -move to the following augmented matrix:

2 2 3
7
4 5 6 15
1 1 1
3
The entry in the pivotal row sharing same column with 15 is 7. The entry in
the pivotal column sharing the same row with 15 is 4. The pivot is 2. So 15
will change to
74
= 1.
15
2

In Gaussian elimination of an n n system (i.e., where the augmented matrix

is n (n + 1)) we start with the (1, 1)-th entry as the pivot. Then we perform
the following operations.
F, , , F, .
n1

times

The symbol means shift the pivot diagonally downward by one place.
If the pivot turns out to be zero at any step, then stop and output Failure.
Otherwise, you will end up with an augmented system

A b

where A will be an upper triangular matrix with nonzero diagonal entries. Now
you have to apply back substitution to solve for xn , ..., x1 in this order as follows.

n
1
bk
akj xj ,

xk =
akk

j=k+1

for k = n, n 1, ..., 2, 1.
It might help to write down the back substitution part explicitly for the 3 3
case where the unknowns are x, y and z.
= 3 /33
b a
y = (2 a23 z)/22
b

a
1 a13 z a12 y)/11
x = (b

a
z

Example: Here is the algorithm in action. We have applied it to the system

from the last example.

1 1
1
2
2
5 1
9
1
2 3 4

1 1
0
7

0
0

1
3
-19/7

2
5

57/7

1
0
0

2
1
1
7 3
5
3 4 6

Now we have an upper triangular system, i.e., a system where the coecient
matrix (the part to the left of the vertical divider in the augmented matrix) is
upper triangular. We shall now apply back substitution to solve for z, then y,
and then x as follows.
57/7
z=
= 3,
19/7
7

and
y=

5+33
1
(5 + 3z) =
= 2,
7
7

and nally,
x = 2 + y z = 2 + 2 3 = 1.

The following diagram shows the order of the Gaussian elimination operations
for a 4 4 matrix. The gray regions hold the nonzero entries. The white regions
are all zeroes. The little triangles to the left mark the rows that are going to be
aected by the next move. The position of the pivot is shown as a circle.
F

Project 1:Implement the Gaussian elimination algorithm discussed above.

Gauss-Jordan elimination

Gauss-Jordan elimination is basically the same as Gaussian elimination except

that here we have two new moves: the S-move and the B-move.
Scaling (S): This operation involves dividing the entire pivotal row by
the pivot. So in place of the pivot we shall have 1. This operation aects
only the pivotal row, and leaves the rest of the matrix unaltered. Here is
an example. The pivot is shown in a box.

1
2
1

2
1
1
5 1
9
2 3 4

2
1 1
1
2/5 1 1/5 9/5
1
2
3 4

Back substitution (B): This is same as a F-move, except that now we

work on the rows above the pivotal row. This operation leaves the pivotal
row and the ones below it unaltered.

1 1
1
2
4/3 1/3
0 2/3
B
2
9 5/3 13/3
5 1
0 31/3
1
2 -3 4
1
2 -3
4
8

Here we have subtracted 1 times the last row from the second row, and added
3
1
3 times the last row to the rst row.
In Gauss-Jordan elimination of a n n system we start with the pivot at the
(1, 1)-th position. Then we perform the following operations.
S, F, B, , , S, F, B, , S, B.
n1

times

As before the symbol means shift the pivot diagonally downward by one
place.

Example: Let us perform Gauss-Jordan elimination for the same system. We

start with the original system in augmented matrix form.

2
1 1
1
1 1
1
2
(S,F )
2
0 7 3
5 1
9
5
1
2 3 4
0
3 4 6
(S, F, B)

19/7
1 0
4/7
1 0 0 1
(S,B)

5/7
3/7
0 1 0 2
0 1

0 0 1 3
0 0 -19/7 57/7

Pivoting strategies

We have already mentioned the three basic steps: B,F, and S. All these steps
involve division by the pivot. So the pivot needs to be nonzero. However, it may
happen that while performing the elimination the pivot turns out to be zero.
Then we need to swap rows and/or columns to bring a nonzero number in the
position of the pivot. This is called pivoting. Pivoting is useful even when the
pivot is nonzero, but is very small in absolute value. This is because, division
by numbers new zero introduces large errors in the computer output. We shall
discuss two pivoting strategies: partial pivoting, and complete pivoting.

4.1

Partial pivoting

Partial pivoting involves swapping rows. Suppose that at some step in the
Gaussian or Gauss-Jordan algorithm you have the following augmented matrix,
9

where the pivot is shown in a box.

1
2 3
6
2 9 4 3
5
1 2 2
To perform partial pivoting look at the pivot as well as all the entries below the
pivot in the pivotal column. These are shown bold. Locate among these the
entry with the maximum absolute value. Here it is 5. Swap the pivotal row
with with the row containing this entry. In this example we swap row 1 with
row 3 to get

-5
1 2 2

2 9 4 3
6
1
2 3
Note that the position of the pivot (shown by the box) has remained unchanged,
though the pivot itself has changed from 1 to 5.
In partial pivoting you look for a candidate pivot from among the current pivot
and the entries below it:

Partial pivoting

Example: Here are some more examples of partial pivoting.

2
0
0

10 1
1 2
3 1

10
3 4
10 1 3
2 13 8

2
1
Pp
2 0
5
0

10 1
3 1
1 2

1
5
2

3
10
3 4
Pp
5 10 1 3
9
2 13 8

3
5
9

We have called the partial pivoting step as Pp . So now we have four dierent
kinds of steps: S, F, B and Pp .

10

In Gaussian elimination with partial pivoting we perform the following steps:

Pp , F, , , Pp , F, ,
n1

times

and then we perform back substitution as before. If the pivot turns out to be
zero at any step (i.e., all the entries at and below the pivotal position are zero
then the system is singular, and the pivotal variable can be set to any value.
For Gauss-Jordan elimination we do
Pp , S, F, B, , , Pp , S, F, B, , S, B.
n1

4.2

times

Complete pivoting

In complete pivoting we swap rows and/or columns. Suppose the pivot is at the
(i, i)-th entry. Then we look at all the (r, s)-th entries where r, s i.

Complete pivoting

Notice that we do not cross the vertical divider.

1
2
3
6
3 2
1 5
4
1 4 10
Locate the entry with the maximum absolute value among the bold entries. If
this is the (r, s)-th entry then swap i-th row of the augmented matrix with r-th
row, and the i-column with the s-th column. Note that the order of row and
column swaps does not matter. In this example we swap row 2 with row 3, and
column 2 with column 3 to get

6
1
2 3
4 -4 1 10
3
1 2 5
Recall that the columns of the matrix correspond to the variables of the equations. So swapping the columns also involves reordering the variables. A simple
way to keep track of the order of the variables is to write the variables above
the columns. If we call the variables as x, y, z in the last example then we shall
write:
11

x
1

3
4

y
2
2
1

z
3
6

1 5
4 10

x
1

4
3

z
2
-4
1

y
3
6

1 10
2 5

Here Pc is our symbol for complete pivoting.

In Gaussian elimination with complete pivoting we perform the following steps:

Pc , F, , , Pc , F, ,
n1

times

followed by back substitution as before. If we cannot avoid having a zero pivot

at some step (i.e., all the entries at and to the south-east of the pivotal position
are zeros) then the system is singular, and the remaining variables (including
the pivotal variable) can be set to any arbitrary value if the system is consistent.
For Gauss-Jordan elimination we do
Pc , S, F, B, , , Pc , S, F, B, , S, B.
n1

4.3
4.3.1

times

Keeping track of swaps

Method 1

There are two ways to keep track of the column swaps. Both uses an extra integer array. In the rst version we keep an integer array colPerm[0],...,colPerm[n-1],
say. Initially colPerm[i] stores i for all i. This means all the columns are in
their right places to start with. Every time we swap two columns we also swap
the corresponding entries of the colPerm array.
Thus, at the end colPerm[i] stores the index of the variable at the i-th position
now.
We need to nd its inverse permutation, to know where a particular variable is.
Remember that if i a permutation, then its inverse permutation is 1 where
i 1 ((i)) = i.
So we can dene a new array called, say, invColPerm[0,...,n-1] and ll it up
as
for(i=0;in;i++) invColPerm[colPerm[i]] = i;
Now the solution for the i-th variable can be found in a[invColPerm[i]][n].

12

4.3.2

Method 2

This is a conceptually simpler, though computationally more demanding solution. Here we keep an integer array colIndex[0,...,n-1]. Instead of referring
to the (i, j)-th entry of A as a[i][j] we shall use a[i][colIndex[j]]. Initially,
colIndex[i] will store i for all i = 0, ..., n 1. So a[i][colIndex[j]] will be
the same as a[i][j]. When we need to swap two columns we shall instead
only swap the corresponding elements of colIndex. The following example will
make things clear.
Example: Let n = 3. After we swap columns 0 and 1, we have
colIndex[0] = 1, colIndex[1] = 0, colIndex[2] = 2.
So the (0, 1)-th entry of the new matrix is now a[0][colIndex[1]], which is
a[0][0], as it should be.
If we perform the entire Gauss-Jordan algorithm like this, then there is no need
to change anything at the end. The last c9olumn of the nal augmented matrix
will automatically store the required solution in the correct order!

Exercise 1:Show that the operations F, S and B all correspond to premultiplication

by special matrices. These matrices are called elementary matrices. Write down
the elementary matrices for performing these three basic moves for an n n matrix
assuming that the pivotal position is (i, i). What similar statement can you make about
Pp and Pc ? Compute the product of the elementary matrices for S, F and B moves.

Project 2:Implement Gauss-Jordan elimination with complete pivoting.

Exercise 2:Suppose that we want to solve multiple systems with a common coecient
matrix but dierent r.h.s. vectors and unknown vectors.
Ax1 = b1
Ax2 = b2
.
.
.
Axk = bk .
This is the same as solving the matrix equation:
AX = B,

13

where the bi s are the columns of B, and the xi s are the columns of X. Here we can
apply Gauss/Gauss-Jordan elimination with the augmented matrix:
[A | B].
Write a program to implement Gauss-Jordan elimination (with complete pivoting) in
this set up.

Some theory behind Gauss-Jordan elimination

Notice that we are using four basic actions: dividing a row by a number, subtracting multiple of one row from another, swapping two rows, and swapping
two columns. Of these the rst three are row operations and can be done by
left multiplication with elementary matrices. The last operation is a column
operation, and hence done by right multiplication. If we accumulate all the left
and right marices then at the end of Gauss-Jordan elimination we are left with
LAR = I.
So A = L1 R1 , or
A1 = RL.
Convince yourself that left multiplying by R is same as applying the column
swaps to the rows in the reverse order.
From now on we shall assume that R = I for ease of exposition. The results
should be easily modiable for general R.
Here is another interesting point. After p steps the matrix is of the form
Ip
O

P b1
Q b2

When does total pivoting fail? When Q = O. Thus the system is inconsistent
if b2 = 0. If b2 = 0 then the general solution is y arbitrary and x = b1 P y.
Another interesting point is
Q = A22 A21 A1 A12 .
11
This is because if all the accumulated left multiplier matrix is L then it is of
the form
Upp V
L=
.
W
I
Do you see why the lower right hand block is I? Now we have
LA =

Upp
W

V
I

A11
A21
14

A12
A22

Ip
O

P
Q

So
Q = W A12 + A22 .
To nd W we note that W A11 + A21 = O, or
W = A21 A1 .
11
So combining everything we have
Q = A22 A21 A1 A12 .
11
an easy proof by just performing the block matrix multiplication. This result
is useful because we often have to compute the rhs in statistics. It is called the
conditional covariance matrix.

Inversion using Gauss-Jordan elimination

Gauss-Jordan elimination (with pivoting) may be used to nd inverse of a given

nonsingular square matrix, since nding inverse is the same as solving the system
AX = I.

1
Example: Suppose that we want to nd the inverse of the matrix 2
1
Then we append an identity matrix of the same
to get the augmented matrix

1 1
1 1 0
2
5 1 0 1
1
2 3 0 0

1 1
5 1 .
2 3

size to the right of this matrix

0
0 .
1

Now go on applying Gauss-Jordan elimination until the left hand matrix is

reduced to identity. The right hand at the nal step will give the required
inverse. Each step combines the S,F and B-moves.

1 1
1 1 0 0
1 1
1
1 0 0
(S,F )
2
5 1 0 1 0 0
7 3 2 1 0
1
2 3 0 0 1
0
3 4 1 0 1
After a set of S, F and B steps we arrive at

5/7
4/7
1 0
-3/7 -2/7
0 1
0 0 -19/7 -1/7
15

1/7
1/7
-3/7

0
0 ,
1

which after the nal S, B steps yields

1 0
0 1
0 0
Thus, the required inverse is

6.1

0
0
1

91/133

7/133

4/19

-35/133

28/133

-3/19

1/19

3/19

-7/19

91/133

7/133

4/19

-35/133
1/19

28/133
3/19

-3/19
-7/19

Ecient implementation

First consider the case where we

Then we start out like this:

do not need to perform any row/column swap.

1 0
0 1
0 0

Here a denotes an entry that depends

sweep we have

1
0
0
Notice that the rst column is now
have

1 0
0 1
0 0
After the third step we have

1
0
0

0
0 .
1

on the matrix being inverted. After one

0
1
0

0
0 .
1

forced to be (1, 0, 0) . After another step we

0
0 .
1

0 0
1 0
0 1

The s in the right hand half now constitute the inverse matrix. Notice that
at each step there are exactly as many s as there are entries in the original
matrix. Clearly, all that we really need to store are the s, because the othr
entries are anyway constants irrespective of the matrix being being inverted.
It should not be dicult to check that the following algorithm computes the s:

16

For p = 0 to n 1 you compute

aij

aip

aip apj
i, j = p
app
aip /app i = p

apj
app

=
=

apj /app j = p
1/app .

aij

Then in the above algorithm the i-th column of A is needed only to compute the
i-column of A1 . So we can overwrite the i-th column of A with the i-th column
of A1 to save space. This is an example of an in place algorithm, i.e., where
the output gradually overwrites the input. If we need to perform row/column
swaps, then the situation is very similar. To see this consider the same example
once again. We start out as

1 0 0
0 1 0 .
0 0 1
After one sweep we have

1
0
0

Now suppose we swap the second and

1
0
0
The next sweep will produce

1
0
0

0
1
0

0
1
0

0
0 .
1

third rows:

0 0
0 1 .
1 0
0
0
1

Notice that we still have exactly the same

more side by side.
The third sweep will yield

1 0 0
0 1 0
0 0 1

number of s, but now they are no

Here the right hand half stores the inverse of the row-swapped version of the
original matrix, i.e., (LA)1 , where L is the row-swap matrix. Now (LA)1 =
A1 L1 . Hence
A1 = (LA)1 L.
17

When L is multiplied from the right, it swaps the columns in the reverse order.
Thus we need to keep track of the swaps and apply them in the reverse order
to the inverse. Remember that a row swap becomes a column swap and vice
versa.
Project 3:Implement the in-place version of the Gauss-Jordan inversion algorithm using complete pivoting.

Determinant using Gaussian elimination

One may use the Gaussian elimination or Gauss-Jordan elimination algorithm

to compute the determinant.

Start by applying the n 1 moves of type (Pc , F ), as in Gaussian elimination.

Then multiply the diagonal entries of the resulting matrix. If you have done
k row/col swaps, then multiply with (1)k . A row swap is counted separately
from a column swap.

Project 4:Implement the above algorithm to compute the determinant of a

given square matrix.

LU decompositions

We say that a nonsingular matrix has LU decomposition if it can be written as

A = LU,
where L is a lower triangular and U is an upper triangular matrix.
Exercise 3:Show that such a factorization need not be unique even if one exists.
If L has 1s on its diagonals then it is called Doolittle decomposition and if U
has 1s on its diagonals, it is called Crouts factorization. If L = U then we call
it Cholesky decomposition (read Cholesky as Kolesky.)
We shall work with Crouts decomposition as a representative LU decomposition. The others are similar.

18

8.1

Application

LU decomposition is mainly used as a substitute for matrix inversion. If A = LU

then you can solve Ax = b as follows.
First write the system as two triangular systems
Ly = b, where y = U x.
Being triangular, the systems can be solved by forward or backward substitution.
Apply forward substitution to solve for y from the rst eqation, and then apply
backward substitution to solve for x from the second equation.
Notice that, unlike Gausian/Gauss-Jordan elimination, we do not need to know
b while computing the LU decomposition. This is useful when we want to solve
many systems of the form Ax = b where A is always the same, but b will
change later depending on the situation. Then the LU decomposition needs to
be computed once and for all. Only the two substitutions are to be done afresh
for each new b.

8.2

Algorithm

From denition of matrix multiplication we have

n

aij =

lik ukj .
k=1

Now, lik = 0 if k > i, and ukj = 0 if k < j. So the above sum is eectively
min{i,j}

aij =

lik ukj .
k=1

You can compute li1 s for i 1 by considering

ai1 = li1 u11 = li1 ,
since diagonal entries of U are 1. Once l11 has been computed you can compute
u1i s for i 2 by considering
u1i = a1i /l11 .
Next you will compute li2 s and after that u2i s, and so on. The order in which
you compute the lij s and uij s is shown in the diagram below.
1

2
3

4
5

L=

U=

= one
8

10
11

= zero
12

13

LU decomposition computation order

19

The general formulas to compute lij and uij are

lij
uij

= aij
=

1
lii

aij

j1
k=1 lik ukj
i1
k=1 lik ukj

(i j)
(i < j).

Exercise 4:What should we do if for some i we have lii = 0? Does this necessarily
mean that LU decomposition does not exist in this case?

8.3

Ecient implementation

Notice that L and U have nonzero elements at dierent locations. The only
place where both has nonzero elements is the diagonal, where U has only 1s.
So we do not need to explicitly store the diagonal entries of U. This lets us store
L and U in a single n n matrix.
Also, observe that aij for i < j is required to compute only uij . Similarly aij for
i j is required to compute only lij . Thus, once uij is computed (for i < j) we
can throw away aij . Similarly, for the case i j. This suggests that we overwrite
A with L and R. Here is how the algorithm overwrites A :
U
L
A
Snapshot of A during Crouts decomposition

Project 5:Implement the ecient version of Crouts decomposition discussed

above. Your software should also be able to solve a system Ax = b by forward
and backward substitution.

8.4

Some theory

Exercise 5:Show that if all the leading principal minors are nonzero then all the lii s
will be nonzero. In fact, if i is the smallest number such that the i-th leading principal
minor is zero, then i is also the smallest number with lii = 0. [Hint: If A = LU and
you partition A as

Aii B
,
A=
C
D
20

where Aii is i i, then what is the LU decomposition of Aii ? Now apply the formula
for determinant of partitioned matrix to show that
lii = |Aii |/|Ai1,i1 |.

Exercise 6:Use the above exercises to characterize all square matrices having LU
decomposition.

QR decomposition

Theorem
For any n by p matrix A with n p we have an n by n orthogonal matrix Q
and an n by p upper triangular matrix R such that
A = QR.
This decomposition is the matrix version of Gram-Schmidt Orthogonalization
(GSO). To see this we rst consider the case where A is full column rank (i.e.,
the columns are independent.) Call the columns
a1 , ..., ap .
Apply GSO to get an orthonormal set of vectors
q1 , ..., qp
given by
i1

qj ai )

qi = unit(ai
j=1

computed in the order i = 1, ..., p. Here unit(v) denotes the unit vector
unit(v) = v/ v ,
for any nonzero v. If v = 0, then unit(v) is not dened.
Notice that ai is a linear combination of q1 , ..., qi only and not of qi+1 , ..., qp . So
we can write
qi = r1i a1 + r2i a2 + + rii ai .
Dene the matrix R using the rij s.

21

i1

If A is not full column rank then some (ai j=1 qj ai ) will be zero, hence we
cannot apply unit to it. But then we can take qi equal to any unit norm vector
orthogonal to q1 , ..., qi1 and set rii = 0.
However, GSO is not the best way to compute QR decomposition of a matrix. This is because in the unit steps you have to divide by the norms which
may be too small. The standard way to implement it is using Householders
transformation.

9.1

Householders transformation

If A is any orthogonal matrix then

Ax = x .
In other words an orthogonal matrix does not change shape or size. It can only
rotate and reect objects. We want to ask if the reverse is true: if x and y are
two vectors of same length then does there exist an orthogonal A that takes
x to y and vice versa? That is, we are looking for an orthogonal A (possibly
depending on x, y) such that
Ax = y and Ay = x?
The answer is Yes. In fact, there may be many. Householders transform is
one such:
A = I 2uu , where u = unit(x y).

Exercise 7:Show that this A is orthogonal and it sends x to y and vice versa.

Exercise 8:Show that in 2 dimensions Householders transform is the only such

transform. Show that this uniqueness does not hold for higher dimensions.

Exercise 9:In general you need n2 scalar multiplications to multiply an n n matrix

with an n 1 vector. However, show that if the matrix is a Householder matrix then
only 2n scalar multiplications are needed.

9.2

Using Householder for QR

The idea is to shave the columns of X one by one by multiplying with Householder matrices. For any non zero vector u dene Hu as the Householder matrix
22

that reduces u to

u
0
.
.
.

v=

Exercise 10:Let us partition a vector u as

un1 =

u1
u2

where both u1 , u2 are vectors (u2 being k 1). Consider the n n matrix

I
0
.
H=
0 Hu2
Show that H is orthogonal and compute Hu. We shall say that H shaves the last k 1
elements of u.

Let the rst column of X be a. Let H1 shave its lower n 1 entries. Consider
the second column b of H1 A. Let H2 shave o its lower n 2 entries. Next let c
denote the third column of H2 H1 X, and so on. Proceeding in this way, we get
H1 , H2 , ..., Hp all of which are orthogonal Householder matrices. Dene
Q = (H1 H2 Hp )
and R = Q X to get a QR decomposition.
Exercise 11:Carry this out for the following 5 4 case.
2
6
6
6
6
4

9.3

1
1
1
7
2

2
3
2
2
8

3
2
5
1
5

4
4
0
3
4

3
7
7
7
7
5

Ecient implementation

Notice that though the Householder matrix

I 2uu
is an n n matrix, it is actually determined by only n numbers. Thus, we can
eectively store the matrix in linear space. In particular, the matrix H1 needs
only n spaces, H2 needs only n 1 spaces and so on.
23

So we shall try to store these in the shaved parts of X. Let H1 = 1 2u1 u

1
and H1 X be partitioned as
v
.
0 X1
Then we shall try to store u1 in place of the 0s. But u1 is an n 1 vector,
while we have only n 1 zeroes. So the standard practice is to store (which is
the squared norm of the rst column) in a separate array, and store u1 in place
of the rst column of A.
The nal output will be a n p matrix and a p-dimensional vector (which is like
an extra row in A). The matrix is packed with the us and the strictly upper
triangular part of R :
strict upper triangular
part of R

u1

u2

u6

u4
u3

u5

diagonal entries
of R
Output of ecient QR decomposition

The extra row stores the diagonal entries of R. It is possible to unpack Q

from the us. However, if we need Q only to multiply some x to get Qx, then
even this unpacking is not necessary.
Exercise 12:Write a program that performs the above multiplication without explicitly computing Q.

Project 6:Write a program to implement the above ecient version of QR

decomposition of a full column rank matrix A. Your program should be able to
detect if A is not full column rank, in which case it should stop with an error
message.
This is not entirely easy to do. So lets proceed step by step. First we chalk out
the basic owchart.
24

n, p, A

i = 0, ..., p 1

i
A.

The main algo

The easiest part to expand here is the input part. The only subtlety is to
remember to allocate an extra row in A.

n, p

An+1p
A
n p

The input part

Now comes the tricky step: the shaving. There are two parts here. First, we
have to compute the u vector, and save it in the A matrix. Second, we have to
multiply the later columns by Hu .

25

 i.
j = i + 1, ..., p 1
j

How to shave

The owchart below is quite technical in nature. It tells you how to compute
the u vector, and save it in the A matrix. It also saves the diagonal element of
R in the extra row of A.
n1
2
j=i+1 aji

normsq1 =

normsq1 + a2
ii

norm =

aii = aii norm

normDif f =

normsq1 + a2
ii

aii, ..., an1,i

norm

normDif f
ani

Processing the shaved column (i)

Updating the subsequent columns is basically multiplying by a Householder

matrix.

26

mult = 2

n1
k=i aki akj

k = i, ..., n 1

akj = akj mult aki

Updating column j

9.4

Application to least squares

An important use is in solving least squares problems. Here we are given a

(possibly inconsistent) system
Ax = b,
where A is full column rank (need not be square.) Then we have the following
theorem.
Theorem
The above system has unique least square solution x given by
x = A(A A)1 A b.
Note that the full column rankness of A guarantees the existence of the inverse.
However, computing the inverse directly and then performing matrix multiplication is not an ecient algorithm. A better way (which is used in standard
software packages) is to rst form a QR decomposition of A as A = QR. The
given system now looks like
QRx = b.
The lower part of R is made of zeros:
R1
O

Q
or

R1
O

x = b,

x = Q b,

Partition Q b appropriately to get

R1
O

x=

c1
c2

where c1 is p 1. This system is made of two systems:

R1 x = c1
27

and
Ox = c2 .
The rst system is always consistent and can be solved by back-substitution.
The second system is trivial and inconsistent unless c2 = 0.
1
Exercise 13:Show that x = R1 c1 is the least square solution of the original system.

Notice that you use back-substituting to nd this x and not direct inversion of R1 .

Exercise 14:Find a use of c2 to compute the residual sum of squares:

b Ax 2 .

28