333

1
\Politehni a" University of Bu harest

Department of Engineer S ien es
Ele tri al Engineering and Computers Se tion
NUMERICAL METHODS
Boris Jora
Bogdan Dumitres u
Cristian Oara
1995
2
Contents
1 Main features of numeri al omputation 5
1.1 The oating point representation . . . . . . . . . . . . . . . . . . . . 6
1.2 Computer arithmeti . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
1.3 Numeri al ondition and numeri al stability . . . . . . . . . . . . . . 12
2 Dire t methods for solving linear systems 17
2.1 Types of linear systems . . . . . . . . . . . . . . . . . . . . . . . . . 17
2.2 Triangular systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
2.3 Gaussian elimination . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
2.4 Pivoting strategies . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
2.4.1 Partial pivoting . . . . . . . . . . . . . . . . . . . . . . . . . . 29
2.4.2 Complete pivoting . . . . . . . . . . . . . . . . . . . . . . . . 31
2.5 Solving determined linear systems . . . . . . . . . . . . . . . . . . . 33
2.6 Matrix inversion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
2.6.1 Triangular matrix inversion . . . . . . . . . . . . . . . . . . . 36
2.6.2 Matrix inversion using GP P and GCP . . . . . . . . . . . . 38
2.7 Determinant al ulus . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
2.8 LDU and LU matrix fa torizations . . . . . . . . . . . . . . . . . . . 42
2.9 Cholesky fa torization . . . . . . . . . . . . . . . . . . . . . . . . . . 47
2.10 Appli ations of the LU fa torizations . . . . . . . . . . . . . . . . . . 51
2.10.1 Linear system solving . . . . . . . . . . . . . . . . . . . . . . 51
2.10.2 Matrix inversion . . . . . . . . . . . . . . . . . . . . . . . . . 52
2.10.3 Determinant al ulus . . . . . . . . . . . . . . . . . . . . . . . 52
2.11 Numeri al ondition of linear systems . . . . . . . . . . . . . . . . . 52
2.11.1 Matrix norms . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
2.11.2 Numeri al ondition of linear systems . . . . . . . . . . . . . 54
2.11.3 Numeri al stability of dire t algorithms for linear systems . . 56
3
4 CONTENTS
3 The linear least squares problem 59

3.1 The linear least squares problem . . . . . . . . . . . . . . .. . .. . 59
3.2 Orthogonality. Householder and Givens transformations . .. . .. . 60
3.2.1 Householder transformations . . . . . . . . . . . . .. . .. . 63
3.2.2 Givens transformations . . . . . . . . . . . . . . . .. . .. . 66
3.3 Orthogonal triangularization. QR fa torization . . . . . . .. . .. . 67
3.4 Solving the linear least squares problem . . . . . . . . . . .. . .. . 71
3.5 Solving underdetermined linear systems . . . . . . . . . . .. . .. . 75
4 Iterative te hniques in matrix algebra 79
4.1 Ve tor and matrix sequen es. Convergen e . .. . .. . . .. . .. . 79
4.2 Iterative te hniques for linear system solving .. . .. . . .. . .. . 81
4.2.1 Ja obi iterative method . . . . . . . . .. . .. . . .. . .. . 84
4.2.2 Gauss-Seidel iterative method . . . . . .. . .. . . .. . .. . 86
4.2.3 Relaxation methods . . . . . . . . . . .. . .. . . .. . .. . 88
4.3 Iterative renement of the omputed solution .. . .. . . .. . .. . 90
5 Eigenvalues and eigenve tors 93
5.1 The spa e Cn . . . . . . . . ............ . .. . . .. . .. . 93
5.2 Eigenvalues and eigenve tors . . . . . . . . . . . . .. . . .. . .. . 95
5.3 The real S hur form. De ation te hniques . . . . . .. . . .. . .. . 98
5.4 Power and inverse power methods . . . . . . . . . .. . . .. . .. . 100
5.4.1 The power method . . . . . . . . . . . . . . .. . . .. . .. . 101
5.4.2 The inverse power method . . . . . . . . . . .. . . .. . .. . 102
5.5 The QR algorithm . . . . . . . . . . . . . . . . . . .. . . .. . .. . 104
5.5.1 Redu tion to Hessenberg form . . . . . . . .. . . .. . .. . 104
5.5.2 The iterative stage of the QR algorithm . . .. . . .. . .. . 106
6 The singular value de omposition 113
6.1 Denitions. Properties . . . . . . . . . . . . . . . . .. . . .. . .. . 113
6.2 The SVD Algorithm . . . . . . . . . . . . . . . . . .. . . .. . .. . 123
6.3 Appli ations of the SVD . . . . . . . . . . . . . . . .. . . .. . .. . 139
6.3.1 On omputing the rank of a matrix . . . . . .. . . .. . .. . 139
6.3.2 The general least square problem . . . . . . .. . . .. . .. . 140
6.3.3 Orthonormal basis for linear subspa es . . . .. . . .. . .. . 141
6.3.4 The pseudoinverse omputation . . . . . . . .. . . .. . .. . 143
Solutions to the problems 147
Chapter 1
Main features of numeri al

omputation
This hapter is devoted to a dis ussion of the pra ti al onsiderations that make
the dieren e between an inee tive and an ee tive algorithm. Many existen e
proofs in mathemati s spe ify pro edures by whi h the obje t whose existen e is to
be established may a tually be onstru ted. However, there is a wide gap between
a mathemati al algorithm and a reliable omputer algorithm for solving a given
problem.
A orre t mathemati al algorithm an be inadequate for pra ti al purposes be-
ause of possibly desastrous ee ts of rounding errors or be ause its large omplexity
( on erning memory and exe ution time requirements). The memory requirement
an be evaluated by the amount of input and auxiliary data. The exe ution time is
usually evaluated by the number of arithmeti (and sometimes logi al) operations
needed to ompute the solution from the input data. A more diÆ ult task is to
estimate the ee ts of rounding errors on the omputed solution a ura y.
To approa h this problem, let us introdu e some denitions. Let x and x be real
numbers, where x is interpreted as an approximation of x. Two natural measures
of the approximation a ura y are given by:
Denition 1.1 Let x; x 2 R, with x regarded as an approximation to x. The
absolute error in x is the number
"a = jx xj:
If x 6= 0, the relative error is the number
x

x
"r =
x = j"xaj :
5
6 CHAPTER 1. MAIN FEATURES OF NUMERICAL COMPUTATION
Example 1.2 Let x = 1:0, and x = 0:999 its approximation. Then, "a = 10 3 and
"r = 10 3 . If y = 0:009 is an approximation to x = 0:01, then the absolute error is
the same, "a = 10 3 , but the relative error is a hundred times greater: "r = 10 1 .
This example indi ates that the relative error is mu h more appropriate to estimate
approximation a ura y.
1.1 The oating point representation

The rounding errors appear be ause of omputer representation of real numbers by
a nite sequen e of symbols (binary digits). To give an estimate to su h errors, let
remember the well known positional representation of real numbers. Let introdu e:
2 N, 2, the numeration base;
C = fd0 ; d1 ; : : : ; d 1 g, a set of symbols alled digits, for the rst natural
numbers;
S = f+; ; : g, an additional set of symbols.
Theorem 1.3 For every x 2 R, there is a unique innite sequen e of symbols from
C and S , of form:
x = san 1 an 2 : : : a0 :a 1 a 2 a 3 : : : ;
whi h does not end by an innite sequen e of d 1 , where s 2 f+; g and ai 2 C ,
so that the value of x is
!
Xn X1
x=s ai + a i :
i i
i=0 i=1
In this formula, ai are rather regarded as values of the natural numbers symbolised
by the symbol ai , and s is regarded as 1 or -1.
Example 1.4 In ten based numeration system the two sequen es

0:99999 : : :
1:00000 : : :
represent the same real number. The binary representations of the same numbers
are
0:11111 : : :
1:00000 : : :
The a epted representation is, in both situations, 1:0000 : : :.
1.1. THE FLOATING POINT REPRESENTATION 7
Obviously, the omputer representation of a real number may have only a nite
number of symbols, and so it is, inherently, approximate. A "good" representation by
a nite sequen e of xed number of symbols must satisfy the following requirements:
a suÆ iently large range of represented numbers;
a suÆ iently small relative representation errors;
an uniform error distribution.
A good ompromise between these requirements is given by the so alled Floating
Point Format (FPF).
Denition 1.5 A Floating Point Format (FPF) is dened by three integers, (; t; s),
having the following signi an e:
{ the numeration base ( 2);
t { the pre ision, i.e. the number of signi ant digits (the mantissa "length");
s { the number of exponent digits.
A oating point number is a pair (f; e), where
f = 0:f1 f2 : : : ft ; f1 6= d0 ; fi 2 C
is a signed fra tional, normalized (f1 6= "0"), real number with t digits, alled
mantissa and
e = e1 e2 : : : es
is a signed integer with s digits alled exponent. The exa t value asso iated with
the oating point number (f; e) is
x = f e:
Let L be the minimum value of the exponent (L = d 1 : : : d 1 ) and U = L
the maximum. Noti e that all oating point numbers x satisfy:
L 1 = m jxj M = U (1 t )
i.e. the range of oating point representation is R = [ M; M ℄. Obviously, in all
omputer representation, = 2.
Example 1.6 Let onsider the FPF with (; t; s) = (2; 2; 1). Then L = 1, U = 1,
m = 2 2 = 0:2510 , M = 2(1 2 2 ) = 1:510 . The oating point numbers are:
x1 = ( 0:11; +1) = 1:510 x7 = (+0:10; 1) = 0:2510
x2 = ( 0:10; +1) = 1:010 x8 = (+0:11; 1) = 0:37510
x3 = ( 0:11; 0) = 0:7510 x9 = (+0:10; 0) = 0:510
x4 = ( 0:10; 0) = 0:510 x10 = (+0:11; 0) = 0:7510
x5 = ( 0:11; 1) = 0:37510 x11 = (+0:10; +1) = 1:010
x6 = ( 0:10; 1) = 0:2510 x12 = (+0:11; +1) = 1:510
-1.5 -1 -0.5 0 0.5 1 1.5 -

x1 x2 x3 x4 x5 x6 x7 x8 x9 x10 x11 x12
Figure 1.1: The representation of all oating point numbers of example 1.6.
01001100 01101001 10111000 00001011

67 bits exponent- 6 (23+1) bit mantissa -
exponent sign number (mantissa) sign
Figure 1.2: Floating point number stru ture for (; t; s) = (2; 24; 7).
The geometri al representation, on the real axis, of all the numbers above is given
in gure 1.1.
The oating point numbers are uniformly distributed, relatively; i.e., jxi
xi 1 j=jxi j is approximately the same for all i.
Example 1.7 A standard 32-bit FPF may be the following: (; t; s) = (2; 24; 7).
Two bits are assigned to mantissa and exponent signs, but only the latest must be
physi ally stored; the former takes the pla e of the rst bit of the mantissa whi h
always have the value "1" and thus need not to be stored. An example of oating
point number in su h format is presented in gure 1.2.
For this FPF: L = 127, U = 127, m = 2 127 = (2 10 )13 23 10 38 , M 1038
(we an approximate 210 103 ). This indi ates that FPF provides a wide range of
numbers, represented with a relatively small number of digits. This example is very
lose to the a tual standard IEEE format for single pre ision oating point numbers.
In order to estimate the pre ision of a oating point representation, let be given
a FPF, i.e. (; t; s), and denote
F = fx 2 R j x has an exa t FP representationg [ f0g: (1.1)
Of ourse, F is a nite set of rational numbers.
To represent a real number x 2 [ M; M ℄ in FPF means to approximate it by
a number x 2 F . This approximation an be expressed by a so alled rounding
fun tion.
Denition 1.8 If (; t; s) is a FPF and F the set dened in (1.1), then a fun tion
fl : [ M; M ℄ ! F
1.1. THE FLOATING POINT REPRESENTATION 9
whi h asso iates with every x 2 [ M; M ℄ a unique FP representation
x = fl(x)
is alled rounding fun tion. The approximation relative error
jx fl(x)j
jxj
dened for all nonzero x 2 [ M; M ℄ is alled roundo error.
As the interval [ M; M ℄ is an innite set of real numbers, ea h x 2 F repre-

sents an innite set of numbers from [ M; M ℄; so, we are interested to establish
an upper bound of roundo errors for a given rounding fun tion. There are several
rounding fun tions. We will present only the one alled hopped rounding. In order
to introdu e it, let write the number x 2 [ M; M ℄ in the form:
x = f e = 0:f1 f2 : : : ft ft+1 : : : e
= 0:f1 f2 : : : ft e 0:ft+1 ft+2 : : : e t
= f e + g e t;
where fi 2 C , f1 6= d0, f = 0:f1 f2 : : : ft, g = 0:ft+1ft+2 : : :.
Obviously:
1= jf j < 1; 1= jfj < 1; 0 jgj < 1: (1.2)
Denition 1.9 The hopped rounding fun tion
fl1 : [ M; M ℄ ! F
is dened by:
f e ; for x 6= 0
(
x = fl1 (x) = 0; for x = 0:
Using (1.2) it is easy now to establish an upper bound for roundo errors intro-
du ed by fl1. Indeed, for every x 2 [ M; M ℄ f0g:
jx fl1(x)j = jf e f ej jgj e t t
"r = = < = t+1 :
jxj jf j e jf j e 1
This formula indi ates that the roundo error level is ex lusively determined by
the number of mantissa digits and this is the reason to all t the oating point
pre ision. On most omputers, the oating point numbers have a xed pre ision.
Many omputers have the ability to manipulate oating point numbers with about 2t
mantissa bits. Su h numbers are alled double pre ision and omputations involving
them are said to be done in double pre ision. For instan e, in the IEEE standard,
the double pre ision numbers are stored on 64 bits, with t = 53 and s = 10 (an extra
bit is for the exponent sign).
For all used rounding fun tions, the relative error bound has the form
"r t ; (1.3)
where is a number of order unity.
The t mantissa digits are known as the signi ant digits of the real number in-
volved. In the de imal numeration basis, the IEEE single pre ision numbers (t = 24)
have 7 signi ant digits (we ount 3 de imal digits for 10 binary digits) and double
pre ision numbers (t = 53) have 16 signi ant digits; onsequently, the roundo er-
ror bound is about 10 7 and 10 16 , respe tively; this quantity is also alled ma hine
epsilon.
From (1.3) the existen e of a number results, su h that
x = fl(x) = x(1 + ); jj t (1.4)
whi h is another way to express the roundo error.
1.2 Computer arithmeti

Computers with oating point hardware are provided with a set of instru tions for
manipulating oating point numbers, su h as addition, subtra tion, multipli ation
or division. We must stress that the operations mentioned above dier from the
orresponding mathemati al ones be ause the result ne essarily belongs to the nite
set F of oating point numbers. Hen e, the arithmeti operations annot be per-
formed exa tly. The error introdu ed by the oating point arithmeti operations is
alled rounding error. The onsequen es an be very important; there is a real pos-
sibility that rounding error a umulate and, in extensive omputation, ontaminate
the answers to make them unusefull. Therefore, it is desirable that every algorithm
is analyzed to show that rounding errors will not ae t the results unduly. This is,
in general, a very diÆ ult task.
In order to make a rounding error analysis we need to know bounds for errors in
oating point operations. For most omputers, the following model of oating point
operation errors, based on the evaluation (1.4), is available: if we denote by fl(xy),
fl(x=y) and fl(x + y) the omputed produ t, quotient and sum of the oating point
numbers x and y, then, in a t-digit FPF:
fl(x op y) = (x op y)(1 + ); jj t ; (1.5)
1.2. COMPUTER ARITHMETIC 11
where op an be +, or /; is of order unity.
This relation guarantees that these operations introdu e only small relative errors
into the omputation. For addition, the relation an be written as fl(x + y) =
x(1 + ) + y(1 + ), whi h says that the omputed sum of x and y is the exa t sum
of two numbers that are very near to x and y. In numeri al analysis terminology, it
means that the addition algorithm is stable. For most purposes, this result is good
enough; however, when x and y are of opposite sign and of nearly equal module, the
rounding error may be important.
Example 1.10 If x and y oating point representations have the same exponent
and the same rst k mantissa digits, i.e.:
x = fl(x) = 0:d1 d2 : : : dk dk+1 : : : dt e
y = fl(y) = 0:d1 d2 : : : dk k+1 : : : t e
then the oating point form of x y will be:
fl(x y) = 0:f1 : : : ft k gt k+1 : : : gt e k
where the rst t k digits are orre t, but the others, denoted by g, have no signi-
an e (they even might be randomly assigned). In any further omputation involving
x y we will an rely only on at most t k orre t signi ant digits, sin e any hain
of al ulations would not be expe ted to be more a urate than its weakest point
(however, there are situations when the results are more a urate; for instan e, if we
ompute (x y) + y, the result will have at least t 1 orre t digits; but, usually,
the analysis is done for the worst ase). This loss of exa t signi ant digits is due
to the data involved, and not to the algorithm. It an often be avoided by hanging
the algorithm whi h implies su h subtra tions, as in the example below.
Example 1.11 The well known formulas for root al ulation of ax2 + bx + = 0,
a 6= 0: p2 p
x1 =
b b 4a ; x2 =
b + b2 4a
2a 2a
are not the best ones, from the numeri al point of view, espe ially when b2 4a .
The formulas:
p
b sign(b) b2 4a 2p
x1 = ; x2 =
2a b + sign(b) b2 4a
are better, by avoiding subtra tions of nearby numbers (the se ond relation is de-
du ed from x1x2 = =a).
We do not present how the basi arithmeti oating point operations are per-
formed. We must only retain the evaluation (1.5) of the omputed numbers and
its possible onsequen es. Finally, we mention that when a oating point operation
produ es a number with too large absolute value (> M ), a oating point over ow
o urs. When the result has a too small absolute value (< m, but nonzero), an
under ow o urs. Obviously, every well onstru ted algorithm must spe ify what to
do in the event of an over ow or an under ow. Most omputers, automati ally, set
the resulting number to zero in the ase of an under ow, and stop omputation, or,
at least, write an warning message in the ase of an over ow. In the IEEE oating
point standard, there is a spe ial value alled innity whi h is assigned to the result
at over ow; the omputation ontinues with this value; the results follow the known
rules for operations with innity. Another spe ial value { Not a Number (NaN) {
is assigned to an indenite result, su h for 0 1; an operation involving a NaN has
always NaN as result.
1.3 Numeri al ondition and numeri al stability

Any numeri al problem requires the omputation of some numeri al results starting
with some numeri al initial data, su h that the results an be interpreted as an
approximation to the solution of a mathemati ally expressed problem at the initial
data point. For su h a problem to be well dened, the existen e and uniqueness of
the solution must be assured. Thus, a numeri al problem an always be des ribed
as an evaluation of a fun tion:
f : X Rm ! Y Rn (1.6)
in a given point x 2 X . The m omponents of the argument are the entry data and
the n omponents of f (x) are the answers or the output data.
Example 1.12 a) Given a set X of three real numbers and
2 3
a
x = 64 b 7
5 2 X = fx 2 Rn j a 6= 0; b2 4a 0g R3

to solve the quadrati equation
ay2 + by + = 0
is a well dened numeri al problem sin e for any x 2 X there exists a unique ve tor
" # 2 p 3
b sign(b) b2 4a
y = f (x) = y1 = 4 22a 5 2 R2 = Y
y2 p2
b+sign(b) b 4a
1.3. NUMERICAL CONDITION AND NUMERICAL STABILITY 13
whi h is the problem solution.
b) Compute Z b
a
e 2
t dt;
where a; b 2 R are two given numbers

" # is also a well dened numeri al problem.
Indeed, for the entry data x = ab 2 X = R2 , there is a unique integral value
y 2 Y = R, although the indenite integral e t dt has no expressed primitive.
R 2
The ina ura y of the omputed solution for a given numeri al problem may
have two essential dierent sour es:
1. The problem's sensibility to initial data perturbations. In pra ti e, only an
approximation x to the entry data x is known and even at best we an al ulate
only f (x) instead of f (x). If the fun tion f dening our problem is too sensitive at
argument variations, f (x) and f (x) may dier greatly even for a small relative error
kx xk=kxk of the entry data (k k is a ve tor norm). Su h a problem is alled ill
onditioned.
If we attempt to solve an ill onditioned problem starting with inexa t data, the
solution will be inexa t no matter how it is omputed !
A numeri al ondition of a problem f : X Rm ! Y Rn at a point x 2 X
an be expressed by the relative error quotient:
(x) =
kf (x) f (x)k=kf (x)k ; (1.7)
kx xk=kxk
for x 6= 0 and f (x) 6= 0. When (x) is small (say, of order unity) the problem is said
to be well onditioned in x. Otherwise, it is ill onditioned.
2. The algorithm numeri al quality. In order to solve a given numeri al problem
dened by the fun tion (1.6), a omputer exe utes a denite sequen e of arithmeti
and logi al operations alled algorithm. In general, for a problem there may exist
several algorithms. An algorithm gives a uniquely determined answer for given entry
data and so it an be also mathemati ally expressed by a fun tion
f : X Rm ! Y Rn :
We do not expe t f to solve ill onditioned problems more a urately than the data
warrants. However, we want that f do not introdu e larger inna ura ies by itself.
This suggests that we require the following about f: "For any x 2 X , there
is a nearby x 2 X su h that f (x) is equal to f(x)", i.e. the omputed solution by
the given algorithm with exa t entry data is the exa t solution for some slightly
perturbed entry data.
Algorithm with this property are alled stable. It is easy to show that a well
onditioned problem is solved by a stable algorithm with a high a ura y. Indeed,
from the numeri al stability of the algorithm f for exa t initial data x, there exists
x su h that the relative error is of roundo error order:
kx xk t
kxk
and
f (x) = f(x): (1.8)
The problem being well onditioned, from (1.7) it results that:
kf (x) f (x)k = kx xk t ; (1.9)
kf (x)k kxk
where , are of order unity.
Hen e, from (1.8) and (1.9):
kf (x) f(x)k t;
kf (x)k
i.e., for a well onditioned problem, a stable algorithm do not introdu e signi ant
errors by itself.
Similarly, it an be seen that in all other situations (i.e. for ill onditioned
problems or unstable algorithms), the resulting error annot be under ontrol. When
a stable algorithm is used to solve an ill onditioned problem, there is no guarantee
that f (x) and f (x) are near one another and although f (x) is equal to f(x), the
dieren e between f (x) and f(x) may be important. When an unstable algorithm is
used to solve a well onditioned problem, the omputed solution with exa t entries
f(x) is an exa t solution to the problem with x far to x, and so it is f (x) to f (x).
At last, there is no hope in small errors when an unstable algorithm is applied to
an ill onditioned problem.
Example 1.13 Let onsider the problem of omputing the sum of two numbers,
dened by:
f : R2 ! R; y = f (x) = x1 + x2 ;
" #
where x = x1
2 R2 is the entry data and y = f (x) 2 R the output data.
x2
The problem may or may not be ill onditioned, depending on the input data (see
example 1.10).
1.3. NUMERICAL CONDITION AND NUMERICAL STABILITY 15
Suppose now that the sum is omputed in oating point arithmeti by an algo-
rithm f : R2 ! R:
y = f(x) = fl(fl(x1) + fl(x2 ));
where fl is a rounding fun tion. From (1.4) it results that
fl(x1 ) = x1 (1 + 1 ); fl(x2 ) = x2 (1 + 2 ); j1 j; j2 j t :
Using now (1.5), it an be easily proved that:
y = x1 (1 + 1 ) + x2 (1 + 2 );
where j1 j; j2 j 2 t (1 = + 1 + 1 ).

Thus, given any ve tor x 2 R2 , there exists a nearby ve tor
" #
x = x1 (1 + 1 )
x2 (1 + 2 )
su h that
f(x) = f (x):
Hen e, the oating point addition whi h satises (1.5) is a stable way of omputing
the sum of two numbers.
The above example is a very simple illustration of the so alled ba kward round-
ing error analysis of an algorithm. In this te hnique, one uses the error bounds (1.5)
for oating point arithmeti to prove that the omputed solution of a given prob-
lem is the exa t solution of the problem with slightly perturbed input data. This
proof guarantees that the algorithm is numeri ally stable and, for well onditioned
problems, small relative errors in the omputed data are assured.
For nontrivial problems, to show that a proposed algorithm is stable may be a
very diÆ ult task. Now, there are a relatively small number of algorithms for whi h
a omplete proof of numeri al stability exists. Nevertheless, a lot of important
algorithms have a pra ti al validity based on a numeri al experien e over de ades.
As Wilkinson says: "the main obje t of an error analysis is not to establish pre ise
bounds, but to expose the potential instabilities, if any, of an algorithm, and thus
to nd a way to improve it".
In our approa h, the main interest will be rather the presentation of the best
existing algorithms that solve a ertain problem, than a proof of their numeri al
stability.
Problems
P 1.1 The following example indi ates that oating point addition is not asso iative. Let
onsider x = 0:001, x = 1, x = 1; the FPF is hara terized by = 10, t = 3. How big
1 2 3
is the relative error when omputing y = x + x + x ? 1 2 3
P 1.2 Give a bound of the relative error for the oating point omputation of y = x + 1
x + x ; suppose that x , x , x are oating point numbers.
2 3 1 2 3
P 1.3 Consider the problem of omputing the solution of the linear equation ax + b = 0,
with a; b 2 , a 6= 0. For whi h values of the input data a, b is this problem ill onditioned ?
R
An "algorithm" to ompute the solution is x = b=a; is this algorithm numeri ally stable ?
P 1.4 The same question for the equation (a + a )x + (b + b ) = 0, where the input data
1 2 1 2
is a ; a ; b ; b 2 .
1 2 1 2 R
P 1.5 Give an expression for the distan e between 1 and the next greater oating point
number in the FPF dened by (; t; s).
P 1.6 Give the relative error bound for the rounding fun tion whi h asso iates with ea h
x 2 [ M; M ℄ the nearest oating point number.
Chapter 2
Dire t methods for solving

linear systems
A set of m linear equations:
8
>
> a11 x1 + a12 x2 + : : : + a1n xn = b1
a21 x1 + a22 x2 + : : : + a2n xn = b2 ;
>
<
> ::: (2.1)
>
am1 x1 + am2 x2 + : : : + amn xn = bm
>
:
where aij 2 R, bi 2 R, for i = 1 : m, j = 1 : n, are given and xj , j = 1 : n, are the

unknowns, is alled a linear system. In matrix form, it an be written as:
Ax = b; (2.2)
where A = (aij )i=1:m;j=1:n 2 Rmn, b = (bi )i=1:m 2 Rm and x = (xj )j=1:n 2 Rn .
2.1 Types of linear systems

The rst main hara teristi of a linear system is the relation between m and n:
when the number of equations is greater than the number of unknowns (m >
n), the system (2.2) is alled overdetermined.
when, onversely, the number of unknowns is greater, the system is said to be
underdetermined.
in the ase of a square matrix A (i.e. m = n), the system (2.2) is determined.
if the right hand side is a zero ve tor (b = 0), the system is alled homogenous.
17
18 CHAPTER 2. DIRECT METHODS FOR SOLVING LINEAR SYSTEMS
Finding a solution to the linear system (2.2) means to ompute a ve tor x 2 Rn

su h that the system is satised. Of ourse, we an do this only if the system has
really (at least) a solution. If it has not, we must hange the meaning of "solution",
in su h a manner su h the system has one. When the solution is not unique, we
must sele t, in some way, a single one and then ompute it.
The onditions under whi h the system (2.2) has a solution and when it is unique
are well known from Linear Algebra. So we will present them without proofs.
Given a matrix A 2 Rmn , let rst introdu e the following linear subspa es of a
real linear spa e:
ImA = fy 2 Rm j 9x 2 Rn su h that y = Axg Rm
KerA = fz 2 Rn j Az = 0 2 Rm g Rn :
The existen e onditions for a solution of Ax = b are given by:
Theorem 2.1 The linear system (2.2) has a solution if and only if one of the fol-
lowing equivalent onditions holds:
a) rank[A b℄ = rank(A)
b) b 2 ImA.
Corollary 2.2 If rank(A) = m, then the system (2.2) always has a solution.
2 3
1 07
Example 2.3 If A = 4 1 1 5, then ImA is the plane y2 = y1 + y3 . If b = [b1 b2 b3 ℄T
6
0 1
has b2 = b1 + b3, then the system Ax = b has a solution sin e both onditions in
Theorem 2.1 hold. Otherwise, it has none (see gure 2.1).
When the solutions exist, the uniqueness is derived from:
Theorem 2.4 If x 2 Rn is a solution to the linear system (2.2), then the set of all
solutions of (2.2) is the aÆne subspa e
x + KerA = fy = x + z j z 2 KerAg:
The solution x is unique if and only if
KerA = f0g
i.e.the matrix A is moni (A has linearly independent olumns).
2.1. TYPES OF LINEAR SYSTEMS 19
6y3
HbH2HImA
Y b 62 ImA
1
HH y1
H -

y2

ImA
Figure 2.1: ImA and possible b for example 2.3.
2 3
" # " # 1
Example 2.5 If A = 10 11 01 and b = 21 , then x = 64 1 75 is a solution to
0
Ax = b. On the other hand, KerA is the line des ribed by:
(
x1 + x2 = 0
x2 + x3 = 0
whi h passes through the points O(0; 0; 0) and P (1; 1; 1). So that, all the ve tors
from KerA have the form: 2 3
1
KerA 3 z = 64 1 75 ; 2 R
1
and, hen e (see gure 2.2),
8 2 3 9
>
< 1 + >
=
x + KerA = >y = 4 1 5 j 2 R> :
6 7
: ;
Remark 2.6 When the system (2.2) has more than one solution, a possible way
to sele t a single one is to ask it to be of minimum "length", where the length is
dened by a suitable ve tor norm.
When the system (2.2) has no solution it is onvenient to dene a new kind
of (pseudo)solution, like the ve tor x whi h minimizes a suitable ve tor norm of
residual r = b Ax. It is obvious that if b 2 ImA, then su h a pseudosolution
be omes a true solution of the system.
6x3
P
y :
x-1
x
O
9
x2 R

KerA x + KerA
Figure 2.2: Solutions (x { parti ular solution, y { general solution) for the system
in example 2.5.
Theorem 2.7 When the matrix A is square (A 2 Rnn ), the system (2.2) admits
a unique solution for all b 2 Rn if and only if A is nonsingular, i.e.
det(A) 6= 0:
If A is nonsingular, there exists a unique matrix A 1 2 Rnn , alled the inverse of
A, su h that
A 1 A = AA 1 = In
and the unique solution of system (2.2) an be written as
x = A 1 b: (2.3)
We must stress that (2.3) is not a good formula for the omputation of the
solution of the system Ax = b and so are other known methods, like the Cramer's
rule, et .
In the sequel, we will see how to solve determined nonsingular systems in a
reliable and eÆ ient manner.
There are two main ategories of methods for nding a reliable solution to non-
singular linear systems:
Dire t methods, based on the redu tion, by a nite sequen e of transformations,
to one or two triangular systems, whi h are solved by a simple substitution.
To this ategory belong methods like the Gaussian elimination and the LU
fa torizations, whi h are re ommended for systems of medium size (say, n <
200, but this limit depends on the omputer power).
2.2. TRIANGULAR SYSTEMS 21
0
0
L U
Figure 2.3: Lower and upper triangular matri es.
Iterative methods, based on the re ursive building of a ve tor sequen e onver-

gent to the system solution; these methods are re ommended for huge n, or
when A has some spe ial stru ture.
This hapter is devoted to a presentation of dire t methods.
2.2 Triangular systems

Denition 2.8 A matrix L 2 Rnn , whi h has lij = 0 for all i < j is alled lower
triangular. A matrix U 2 Rnn , with uij = 0 for all i > j is alled upper triangular.
The system Ax = b is alled lower (upper) triangular if the matrix A is lower (upper)
triangular. See gure 2.3.
The following result is obvious:
Proposition 2.9 A triangular matrix is nonsingular if and only if all its diagonal
elements are nonzero.
The algorithms for triangular system solving are very simple, be ause the un-
knowns an be omputed, in a denite order, by numeri al substitution. Consider,
rst, a lower triangular system:
Lx = b;
where L 2 Rnn, with lij = 0 for i < j , lii 6= 0, i = 1 : n, and b 2 Rn . The rst
equation is
l11 x1 = b1 ;
from whi h
x1 = b1 =l11 : (2.4)
In general, if we know x1, x2, .. ., xi 1, we an solve the i-th equation:

i 1
X
lik xk + lii xi = bi
k=1
to obtain i 1 !
X
xi = bi lik xk =lii : (2.5)
k=1
Formulas (2.4) and (2.5) dene an algorithm for the omputation of the solution
of Lx = b, known as the method of forward substitution.
Algorithm 2.10 (LTRIS) (Given a nonsingular lower triangular
n n real matrix L and a ve tor b 2 Rn , this algorithm omputes
the solution x of the system Lx = b.)
1. for i = 1 : n
1. s bi
2. if i > 1 then
1. for k = 1 : i 1
1. s s lik xk
3. xi s=lii
This algorithm requires NLT RIS = Pni=1 (2(i 1) + 1) = n(n 1) + n = n2
oating point operations ( ops) and MLT RIS = n(n +1)=2+2n +1 n2=2 memory
oat lo ations. If the ve tor b is no further needed, the solution x an be stored in
the same memory spa e as b.
In order to solve an upper triangular nonsingular linear system:
Ux = b;
where U is an upper triangular matrix of order n, with uii 6= 0, i = 1 : n, and
b 2 Rn , let observe that the last equation is
unn xn = bn ;
from whi h
xn = bn =unn : (2.6)
Moreover, if xn, xn 1, . .. , xi+1, are omputed, then, from the i-th equation:
n
X
uii xi + uik xk = bi
k=i+1
2.3. GAUSSIAN ELIMINATION 23
we obtain 0
n
1
X
xi = bi uik xk A =uii : (2.7)
k=i+1
From formulas (2.6) and (2.7), whi h dene the so alled ba k substitution, we
obtain the following:
Algorithm 2.11 (UTRIS) (Given a nonsingular upper triangular
n n real matrix U and a ve tor b 2 Rn , this algorithm omputes the
solution x of the system Ux = b.)
1. for i = n : 1 : 1
1. s bi
2. if i < n then
1. for k = i + 1 : n
1. s s uik xk
3. xi s=uii
As in the lower triangular ase, the omputational eort is given by NUT RIS = n2
ops and the memory required is MUT RIS n2=2 (s alar) oat lo ations.
2.3 Gaussian elimination
The Gaussian elimination is a tehnique for system redu tion to triangular form. To
des ribe it in terms of matrix operations, let us introdu e the notion of elementary
triangular matrix.
Denition 2.12 An elementary lower triangular matrix of order n and index s is
a matrix of the form:
Ms = In ms eTs ; (2.8)
where
ms = [0 0 : : : 0 s+1;s : : : ns ℄T (2.9)
es = [0 0 : : : 1 0 : : : 0 ℄T :
The stru ture of an elementary lower triangular (ELT) matrix of order n and
index s is: 2 3
1 0 ::: 0 ::: 0
6 0 1 ::: 0 : : : 0 777
6
6
6
6
::: ::: 7
7
Ms = 6
60 0 ::: 1 ::: 0 7:
7
6
60 0 ::: s+1;s ::: 0 7
7
6
4 ::: ::: ::: 0 7
5
0 0 ::: ns ::: 1
The main properties of su h a matrix are given by:
Proposition 2.13 a) An ELT matrix is nonsingular and

Ms 1 = In + mseTs : (2.10)
b) Let x 2 Rn be a ve tor with xs = eTs x 6= 0. Then, there exists an ELT matrix
su h that
for i = 1 : s
(Msx)i = 0x;i; for i = s + 1 : n. (2.11)
If xs = 0, then for all Ms:
Ms x = x: (2.12)
Proof. a) Obviously, det(Ms ) = 1, i.e. Ms is nonsingular. Moreover,
Ms (In + ms eTs ) = (In mseTs )(In + ms eTs ) = In ms eTs + ms eTs ms eTs ms eTs
= In ms(eTs ms)eTs = In
be ause, from (2.9), eTs ms = 0. Hen e, (2.10) is true.
b) Using (2.8), we obtain:
(Ms x)i = ((In mseTs )x)i = (x mseTs x)i = xi isxs (2.13)
and, hen e, from (2.9), (Msx)i = xi, for i = 1 : s. Now hoosing
is = xi =xs ; for i = s + 1 : n (2.14)
from (2.13) it results that (Msx)i = 0, for i = s + 1 : n. If xs = 0, then (2.13) leads
immediately to (2.12). Note that, if xs 6= 0, then the ELT matrix dened by (2.14)
is the unique ELT matrix whi h repla e the last n s omponents of x by zeros. 2
The properties stated by (2.11) and (2.12) are ru ial in a matrix redu tion
to triangular form. To obtain this goal, in the method of Gaussian elimination,
the given matrix is premultiplied by a sequen e of ELT matri es, ea h hosen to
introdu e a olumn with zeros below the diagonal. To show, formally, how to do
this, we need a denition and two propositions.
Denition 2.14 If A 2 Rnn , then the leading prin ipal submatrix of order s of
matrix A is the matrix:
A[s℄ = (aij )i=1:s;j =1:s = A(1 : s; 1 : s)
i.e. is the s s submatrix from the left upper orner of A.
Proposition 2.15 The produ t of two (unit, i.e. with all diagonal elements equal
to 1) lower (upper) triangular matri es is a (unit) lower (upper) triangular matrix
and so is a produ t of any nite sequen e of su h matri es.
The proof is left as exer ise.
Proposition 2.16 Let A 2 Rnn and L 2 Rnn a lower triangular matrix. Then,
(LA)[s℄ = L[s℄A[s℄:
If Li 2 Rnn, i = 1 : k, are all lower triangular matri es, then
(L1 L2 : : : Lk A)[s℄ = L[1s℄L[2s℄ : : : Lk[s℄A[s℄:
The proof is left as exer ise.
The following theorem and its proof des ribe the Gaussian elimination method
for redu ing a matrix to upper triangular form.
Theorem 2.17 If the matrix A 2 Rnn has all the leading prin ipal submatri es
A[s℄ nonsingular, for s = 1 : n 1, then there exists a unit lower triangular matrix
M 2 Rnn su h that the matrix
MA = U
is upper triangular. If A is nonsingular, then U is nonsingular.
The proof is onstru tive, i.e. it will des ribe a pro edure for an ee tive
Proof.
redu tion of matrix A to the triangular form U . The pro edure onsists of n 1
steps.
Step 1. Let A1 = A and a1 = Ae1 its rst olumn. By hypothesis, A[1℄ 1 =A =
[1℄
a11 6= 0. From proposition 2.13, there exists an ELT M1 su h that
(M1a1 )i = a0;11 ; for i=1

for i = 2 : n.
i.e. the matrix A2 = M1A1 has all subdiagonal elements of its rst olumn equal to
zero: 2
(2) (2) (2) 3
6
a11 a12 : : : a1n 7
6
6 0 a(2)
22 : : : a(2)
2n
7
7
A2 = 6
6
6 0 a(2)
32 : : : a(2)
3n
7
7:
7
6
4 ::: 7
5
(2)
0 an2 : : : a(2)
nn
Step s. Let suppose that the matrix

As = Ms 1 : : : M2 M1 A = Ms 1 As 1
has all subdiagonal elements, in the rst s 1 olumns, equal to zero. Let us onsider
a olumn partition of the matrix As:
As = [ a(1s) a(2s) : : : a(ss) : : : a(ns) ℄:
Applying the proposition 2.16, we have
A[ss℄ = Ms[s℄ 1 : : : M1[s℄A[s℄:
Sin e the matri es Mi[s℄ are all unit lower triangular, then detMi[s℄ = 1, i = 1 : s 1.
Hen e: Ys
detAs = a(iis) = detA[s℄ 6= 0
[ s ℄
i=1
a(s) ,
and the element ss alled pivot, is nonzero. Now, we an use proposition 2.13 to
laim the existen e of the ELT matrix Ms dened by:
is = a(iss) =a(sss) ; i = s + 1 : n (2.15)
so that (Msas(s) )i = 0, for i = s +1 : n. Moreover, the premultipli ation of As by Ms
do not alter the zeros introdu ed in previous steps, be ause a(sjs) = 0, for j = 1 : s 1.
Thus, the zeroing pro edure started at step 1 an be ontinued up to step n 1
(in luded), to make the matrix:
U = An = Mn 1 Mn 2 : : : M1 A
upper triangular. Obviously, from proposition 2.15, the matrix
M = Mn 1 Mn 2 : : : M1
is unit lower triangular and, if A is nonsingular, U is also nonsingular, as a produ t
of two nonsingular matri es. The proof is omplete. 2
The proof above gives a pro edure for the triangularization of a matrix, known
as the method of Gaussian elimination. This pro edure is based on the following
omputation s heme:
1. for s = 1 : n 1
1. Compute the ELT matrix Ms (i.e. is , i = s + 1 : n), so that
(MsA)i = 0, for i = s + 1 : n
2. A MsA
The Gaussian multipliers is are omputed with (2.15), and we an store them
in the pla e of zeroed elements. By the mean of instru tion 1.2 of the s heme above,
2
u11 u12 : : : u1s u1;s+1 : : : u1n 3 2
u11 u12 : : : u1s : : : u1n
3
6
6 21 u22 : : : u2s u2;s+1 : : : u2n 77 6
6 21 u22 : : : u2s : : : u2n 777
6
6 ::: ::: 7
7
6
6 ::: ::: 7
6
6 s1
6
s2 : : : uss us;s+1 : : : usn 777 6
6 s1 s2 : : : uss
6 : : : usn 77
7
6
6 s+1;1 s+1;2 : : : s+1;s a(ss+1
+1) : : : a(s+1) 7
;s+1 s+1;n 7
6
6 ::: ::: 7
7
6 7 6 7
4 ::: ::: 5 4 ::: ::: 5
n1 n2 : : : ns (s+1) (
an;s+1 : : : anns+1) n1 n2 : : : ns : : : unn
After step s Final ontents

Figure 2.4: The ontents of matrix A after step s of the Gaussian elimination and
after the ompletion of the algorithm.
all intermediate and nal results are stored in the memory spa e o upied by the
matrix A; see gure 2.4.
The matrix multipli ation A MsA an be eÆ iently a omplished, taking
into a ount that only the submatrix A(s + 1 : n; s : n) is altered. The olumn s
is modied by zeroing the subdiagonal elements. For the rest of the submatrix, if
aj = Aej , then
(Msaj )i = ((In mseTs )aj )i = (aj msasj )i = aij isasj ; j = s+1 : n; i = s+1 : n:
(2.16)
Relations (2.15) and (2.16), and the omputation s heme presented above dene
the algorithm G of Gaussian elimination des ribed (below.
s)
The algorithm an be
arried out to ompletion only if all the numbers ass , s = 1 : n 1, alled pivot
elements, are nonzero, i.e. all the leading prin ipal submatri es A[s℄, s = 1 : n 1,
are nonsingular.
Algorithm 2.18 (G) (Given a n n real matrix A with all leading
prin ipal submatri es A[s℄, s = 1 : n 1, nonsingular, this algorithm
overwrites the upper triangle of A, in luding the diagonal, with the upper
triangular matrix U = Mn 1Mn 2 : : : M1A. The lower triangle of A is
overwritten by the Gaussian multipliers is dening the ELT matri es
Ms , s = 1 : n 1.)
1. for s = 1 : n 1
1. for i = s + 1 : n
1. ais is = ais=ass
2. for i = s + 1 : n
1. for j = s + 1 : n
1. aij aij isasj
The required number of ops is:

nX1
n(n 1) + 2 n(n 1)(2n 1) 2n3 ;
NG = (n s + 2(n s)2 ) =
s=1 2 6 3
while the size of ne essary memory is MG = n2 oat lo ations.
The algorithm G an be used to solve the linear system Ax = b when the matrix A
is nonsingular and all its leading prin ipal submatri es are also nonsingular. Indeed,
the system Ax = b has the same solution as the upper triangular system:
Ux = Mn 1 Mn 2 : : : M1 b;
where the matrix U is given by algorithm G. This triangular system an be solved
by algorithm UT RIS .
However, the nonsingularity of the leading prin ipal submatri es of A is not a
ne essary ondition for the existen e and uniqueness of a system solution. Thus, we
have to modify the algorithm G in order to handle the situations when some leading
prin ipal submatri es are singular, but A is still nonsingular.
2.4 Pivoting strategies

The singularity of a leading prin ipal submatrix A[s℄ implies that, at step s of Gaus-
sian elimination, the pivot a(sss) is equal to zero; thus, there exists no Ms to zero, by
premultipli ation, the subdiagonal elements in olumn s of As. The modi ation of
algorithm G onsists of a row (or/and olumn) inter hange, whi h brings in the pivot
position a nonzero element. Another reason for the row and olumn permutations is
the numeri al improvement of the omputational pro ess. An intuitive explanation
is that when a mathemati al pro ess fails to be dened for a parti ular value of a
parameter, there is a good han e that the orresponding numeri al pro ess will be
unstable when the parameter is near the oending value. In our ase, we must avoid
small pivot elements in order to improve the numeri al stability of the algorithm.
Let us introdu e now the notion of permutation matrix, whi h allow the des rip-
tion of row and olumn inter hanges in terms of matrix operations.
Denition 2.19 A matrix Pij 2 Rnn obtained from the n n unity matrix by
2.4. PIVOTING STRATEGIES 29
inter hanging the i and j olumns (or rows), i.e. a matrix of the form (here i < j ):
2 3
Ii 1
6
6 0 1 7
7
Pij = [e1 e2 : : : ei 1 ej ei+1 : : : ej 1 ei ej +1 : : : en ℄ = 6
6 Ij i 1
7
7
6
4 1 0 7
5
In j
where all the missing elements are zero, is alled elementary permutation matrix
(EPM).
A produ t of elementary permutation matri es is a permutation matrix.
The properties of elementary permutation matri es are stated in the folowing
proposition, whi h proof is obvious.
Proposition 2.20 a) An EPM is nonsingular and Pij 1 = Pij .
b) The postmultipli ation of a matrix A by the EPM Pij inter hanges the olumn
i with olumn j , i.e.:
< Aek ; for k 6= i; j
8
(APij )ek = : Aej ; for k = i

Aei ; for k = j .
) The premultipli ation of a matrix A by the EPM Pij inter hanges the row i
with row j , i.e.: 8
< ek A; for k 6= i; j
> T
ek (Pij A) = > eTj A; for k = i
T
ei A; for k = j .
: T
2.4.1 Partial pivoting
We are able now to present a rst pivoting strategy. Let modify the algorithm G
as follows. If at the s-th step of the redu tion to triangular form (see the proof of
theorem 2.17), the pivot element a(sss) is unsatisfa torily small, we may hoose some
other element, say a(iss) 6= 0, the greatest in module on olumn s. The inter hanging
s
of rows s and is moves a(iss) into the (s; s) position so that it be omes the new pivot.
In order to avoid the alteration of the matrix stru ture obtained up to step s, we
s
must hoose is s. See gure 2.5.

For simpli ity, we shall denote the EPM Pi s by Ps . Thus, the s-th step of
algorithm G is modied into:
s
1. Determine is so that jai sj = maxi=s:n jaisj.

s
2. Inter hange the is-th and s-th rows, i.e. A PsA.

2 3 2 3
u11 : : : u1s : : : u1n u11 : : : u1s : : : u1n
6
6 0
6 ... : : : 7
7
7
6
6 0
6 . . . : : : 777
6 s) 7 6 7
6
6 a(sss) : : : (
asn 77 6
6 a(iss) : : : a(isn) 77
As = 6 0 Psi As = 66 0
s s
6 ::: 7
7 s ::: 7
7
6
6 ( s )
ai s : : : a(issn) 77
7 6
6 a (s) : : : a(s) 7
6 6 ss sn 7 7
0 0
s
6
4 ::: 7
5
6
4 ::: 7
5
( s)
ans : : : a(nn
s) (s)
ans : : : ann (s)
Figure 2.5: Step s of Gaussian

(s)
elimination, with row inter hange; a(sss) is too small;
in Psi , the new pivot is ai s and have a "good" (big enough) value.
s s
3. Determine the orresponding ELT matrix Ms.

4. Compute A MsA.
whi h provides a new matrix given by:
A As+1 = Ms Ps As :
The overall pro ess, known as Gaussian elimination with row partial pivoting,
leads nally to the upper triangular form:
U = An = Mn 1 Pn 1 Mn 2 Pn 2 : : : M1 P1 A: (2.17)
The orresponding algorithm is given below.
Algorithm 2.21 (GPP { Gaussian elimination with partial pivot-
ing) (Given a matrix A 2 Rnn , this algorithm overwrites the upper
triangle of A, in luding the diagonal, with the upper triangular matrix
U from (2.17). The lower triangle of A is overwritten by the Gaussian
multipliers is dening the ELT matri es Ms, s = 1 : n 1. The inte-
gers is, dening the EPM-s Ps, are stored in a n 1 ve tor p, so that
p(s) = is , for s = 1 : n 1.)
1. for s = 1 : n 1
1. Find the rst is 2 s : n su h that jai sj = maxi=s:n jaisj.
2. p(s) is
s
3. for j = s : n finter hange rows s and isg

1. asj $ ai j s
2.4. PIVOTING STRATEGIES 31
4. for i = s + 1 : n
1. ais is = ais=ass
5. for i = s + 1 : n
1. for j = s + 1 : n
1. aij aij isasj
The extra operations required by algorithm GP P , ompared P tonalgorithm G, ap-
pear only in the sear h of the pivot; the number of omparisons is s=1 (n s + 1)
1
n2 =2 2n3 =3, so that the partial pivoting does not introdu e a signi ant overhead.
The memory requirements is MGP P = n2.
An important result related to algorithm GP P is given by:
Theorem 2.22 If A 2 Rnn is nonsingular, then algorithm GP P an be arried
out to ompletion and the resulting upper triangular matrix U is nonsingular.
Proof. Algorithm GP P annot be arried out if, at some step s, the pivot element
ai s = 0. In su h a ase, sin e the pivot has maximal module, all the other elements
ais , for i = s : n are zero and, hen e, the matrix As is singular (see the stru ture of
s
matrix As).
The matri es Mk and Pk being nonsingular, the singularity of
As = Ms 1 Ps 1 : : : M1 P1 A
implies that A is singular; ontradi tion. Hen e, all ai s 6= 0 and the algorithm
an be arried out to ompletion. The nonsingularity of matrix U derives from the
s
nonsingularity of all matri es in the right hand side of (2.17). 2

2.4.2 Complete pivoting
Better numeri al properties an be a hieved if we look for the pivot among aij , with
i = s : n, j = s : n, i.e in the whole right lower orner of matrix As . The greatest
element in module, say ai j , is moved into the pivot position (s; s), by the mean of
a row and a olumn inter hange; see gure 2.6.
s s
Denoting the EPM Psi by Ps, and Qsj by Qs, the s-th step of algorithm G
be omes:
s s
1. Find is and js so that jai j j = maxi=s:n;j=s:n jaij j.

s s
2. Inter hange the is-th and s-th rows, i.e. A PsA.

3. Inter hange the js-th and s-th olumns, i.e. A AQs.
4. Determine the orresponding ELT matrix Ms.
2 3 2 3
u11 : : : u1s : : : u1j : : : u1n u11 : : : u1j : : : u1s : : : u1n
6 0 ... 6 0 ...
s s
6 7 6 7
6 ::: ::: 7
7 6 ::: ::: 7
7
6
6 ( s)
ass : : : (s)
asj : : : ( s) 7
asn 77
6
6 ai j : : : a(iss)
( s) (s) 7
: : : ai n 77
As = 66 = 66 0
6 6
0
s s s s s
::: 7
7 Psi As Qsj
s s ::: 7
7
6
6 (s )
ai s : : : a(isj) : : : a(issn) 77
7 6
6 asj : : : a(sss)
(s ) ::: a(sns) 77
7
6 6
0 0
s s s s
6
4 ::: 7
5
6
4 ::: 7
5
( s)
ans : : : a(njs) : : : a(nn
s)
anj : : : a(nss)
( s)
: : : a(nn
s)
s s
Figure 2.6: Step s of Gaussian elimination, with row and olumn inter hange.
5. Compute A MsA.
Thus, the step s provides the following matrix transformations:
A As+1 = Ms Ps As Qs :
and the overall omputational pro ess leads to the upper triangular matrix:
A U = Mn 1 Pn 1 : : : M1 P1 AQ1 : : : Qn 1 : (2.18)
The orresponding algorithm, known as the method of Gaussian elimination with
omplete pivoting, is given below:
Algorithm 2.23 (GCP { Gaussian elimination with omplete piv-
oting) (Given a matrix A 2 Rnn, this algorithm overwrites the upper
triangle of A, in luding the diagonal, with the upper triangular matrix
U from (2.18). The lower triangle of A is overwritten by the Gaussian
multipliers is dening the ELT matri es Ms, s = 1 : n 1. The integers
is and js , dening the EPM-s Ps and Qs , are stored in two n 1 ve tors
p and q, so that p(s) = is and q(s) = js , for s = 1 : n 1.)
1. for s = 1 : n 1
1. Find is 2 s : n and js 2 s : n su h that jai j j = maxi=s:n;j=s:n jaij j.
2. p(s) is
s s
3. q(s) js
4. for j = s : n finter hange rows s and isg
1. asj $ ai j
5. for i = 1 : n finter hange olumns s and jsg
s
1. ais $ aij s
2.5. SOLVING DETERMINED LINEAR SYSTEMS 33
6. for i = s + 1 : n
1. ais is = ais=ass
7. for i = s + 1 : n
1. for j = s + 1 : n
1. aij aij isasj
The extra operations required by algorithm GCP , ompared to algorithm G, are
again only in the sear h of the pivot; the number of omparisons is now
nX1 Xn
(n s + 1)2 = s2 n3 =3;
s=1 s=2
unlike the partial pivoting, the omplete pivoting introdu es a number of omparison
whi h is omparable to the number of other operations; this overhead is signi ant,
so that the omplete pivoting must be used only when a great a ura y of the results
is required. The memory requirements is MGCP = n2.
The orrespondent of theorem 2.22, with a similar proof, is:
Theorem 2.24 If A 2 Rnn is nonsingular, then algorithm GCP an be arried out
to ompletion and the resulting upper triangular matrix U from (2.18) is nonsingular.
Remark 2.25 If we want to make the GPP and GCP algorithms to triangularize
singular matri es, there are some slight modi ations to be done.
In GPP, the following test must be added: if ai s = 0, then all ais, i = s : n, are
zero and, for su h s, the instru tions 1.2 { 1.5 must not be exe uted.
s
In GCP, if ai j = 0, then all aij , i = s : n, j = s : n, are zero and the algorithm

terminates at step s be ause the matrix As is already upper triangular.
s s
2.5 Solving determined linear systems

Theorems 2.22 and 2.24 show that the GP P and GCP algorithms provide a good
base to solve the nonsingular linear system Ax = b. Although, from numeri al point
of view, the GCP algorithm is better, GP P is good enough for pra ti al uses, so
that it is widely used.
Let us onsider the following informal syntax for the use of the GP P and GCP
algorithms:
[M; U; p℄ = GP P (A)
[M; U; p; q℄ = GCP (A)
where M is the set of Gaussian multipliers is, s = 1 : n 1, i = s + 1 : n, U is the
resulting upper triangular matrix and p and q are integer ve tors dening the row
and olumn permutations. Although the matri es M and U overwrite the matrix
A, we will use the syntax above for larity.
Let now onsider the linear system Ax = b, with nonsingular A. Be ause all the
ELT matri es Ms and EPM-s Ps are nonsingular, the system Ax = b is equivalent
(i.e. has the same solution) to
Mn 1 Pn 1 : : : M1 P1 Ax = Mn 1 Pn 1 : : : M1 P1 b; (2.19)
where Ms, Ps are those from 2.17. Hen e, (2.19) des ribes an upper triangular
system whi h an be solved by the UT RIS algorithm. We onsider the following
informal syntax for the use of this algorithm:
x = UT RIS (U; b):
Thus, to solve Ax = b, we must ompute the ve tor:
b d = Mn 1 Pn 1 : : : M1 P1 b; (2.20)
using the following s heme:
1. for s = 1 : n 1
1. b Psb
2. b Msb
and the GP P and UT RIS algorithms.
The orresponding algorithm is given below:
Algorithm 2.26 (LSS GPP { linear system solver using GPP)
(Given a nonsingular matrix A 2 Rnn and a ve tor b 2 Rn, this
algorithm omputes the solution x 2 Rn of the linear system Ax = b,
using Gaussian elimination with partial pivoting.)
1. [M; U; p℄ = GP P (A)
2. for s = 1 : n 1
1. bs $ bp(s)
2. for i = s + 1 : n
1. bi bi isbs
3. x = UT RIS (U; b)
The omputational eort is given by:
NLSS GP P = NGP P +
nX1
2(n s) + NUT RIS
2n3 + n2 + n2 2n3 NGP P ;
s=1 3 3
2.5. SOLVING DETERMINED LINEAR SYSTEMS 35
i.e. the main amount of work is in the triangularization of A. Obviously, the memory
needed is MLSS GP P n2.
GCP an be used in a similar way; it is easy to see that the system Ax = b is
equivalent to
Mn 1 Pn 1 : : : M1 P1 AQ1 : : : Qn 1 Qn 1 : : : Q1 x = Mn 1 Pn 1 : : : M1 P1 b: (2.21)
Denoting
Qn 1 : : : Q1 x = y; (2.22)
it results from (2.18) that y an be omputed as the solution of the upper triangular
system
Uy = d; (2.23)
where d is the ve tor given by (2.20). From (2.22), and knowing that Qs = Qs, we 1
obtain:
x = Q1 Q2 : : : Qn 1 y (2.24)
whi h an be omputed by the s heme:
1. x y
1. for s = n 1 : 1 : 1
1. x Qsx
From relations (2.20){(2.24), it results the following
Algorithm 2.27 (LSS GCP { linear system solver using GCP)
(Given a nonsingular matrix A 2 Rnn and a ve tor b 2 Rn, this
algorithm omputes the solution x 2 Rn of the linear system Ax = b,
using Gaussian elimination with omplete pivoting.)
1. [M; U; p; q℄ = GCP (A)
2. for s = 1 : n 1
1. bs $ bp(s)
2. for i = s + 1 : n
1. bi bi isbs
3. x = UT RIS (U; b)
4. for s = n 1 : 1 : 1
1. xs $ xq(s)
The omplexity of this algorithm is asimptoti ally the same as that of GCP .
Remark 2.28 The Gaussian elimination, presented here in a systemati al manner,
is the well known manual omputation method of redu tion and substitution. The
row pivoting orresponds to a reordering of equations in (2.1) and the olumn piv-
oting orresponds to a reordering of the omponents of x. The numeri al stability
of these algorithms will be dis ussed later.
2.6 Matrix inversion

The Gaussian elimination with pivoting is an eÆ ient way to ompute the inverse
of a nonsingular square matrix. To use it to this goal, we need to know rst how to
inverse a triangular nonsingular matrix.
2.6.1 Triangular matrix inversion
Let L 2 Rnn be a lower triangular matrix. Assume that L is nonsingular, i.e.

lii 6= 0, i = 1 : n.
In these onditions, it is well known that there exists a unique matrix X 2 Rnn
su h that:
LX = XL = In ;
alled the inverse of L. In order to ompute it, let onsider the olumn partition of
the matrix equation LX = In:
Lxj = ej ; j = 1 : n; (2.25)
where xj = Xej is the j -th olumn of matrix X . The lower triangular system (2.25)
may be solved in an eÆ ient manner taking into a ount the parti ular form of the
right hand term. Indeed, the system (2.25) an be partitioned as follows:
" #" # " #
L(11j ) 0 x0j
= 0 ; (2.26)
L(21j ) L(22j ) x00j e00j
where L(11j) is the (j 1) (j 1) upper left submatrix of L and

e00j = [1 0 : : : 0℄T 2 Rn j +1 :
From (2.26) it results that:
( (j )
L11 x0j = 0
L(21j ) x0j + L(22j ) x00j = e00j :
(2.27)
Sin e L is nonsingular and, hen e, so are L(11j) and L(22j), and the relation (2.27)
be omes: (
x0j = 0
L(22j ) x00j = e00j :
(2.28)
This relation states an important fa t, given by:
Proposition 2.29 The inverse of a nonsingular lower triangular matrix is lower
triangular.
2.6. MATRIX INVERSION 37
An algorithm whi h omputes the inverse of a nonsingular lower triangular ma-
trix may be derived from (2.28) and has the following s heme:
1. for j = 1 : n
1. if j > 1 then x0j = 0
2. x00j = LT RIS (L(22j); e00j )
It an be seen that the omputed inverse may overwrite the given matrix. Writing
expli itly the lower triangular system solving, we obtain:
Algorithm 2.30 (LINV) (Given a nonsingular lower triangular ma-
trix L 2 Rnn, this algorithm overwrites L by its inverse.)
1. for j = 1 : n
1. ljj xjj = 1=ljj
2. if j = n then stop
3. for i = j + 1 : n P
1. lij xij = i
k=j +1 ik kj =lii
l x
The number of ops required is (exer ise: prove this result):

nX1 X
n
n3
NLINV = n+ 2(i j)
3:
j =1 i=j +1
The ne essary memory is MLINV n2=2 oat lo ations.

In order to ompute the inverse of an upper triangular nonsingular matrix U 2
Rnn , we must solve the matrix equation:
UX = In ;
or, equivalently, the n linear equations:
Uxj = ej ; j = 1 : n; (2.29)
where xj is the j -th olumn of matrix X , the inverse of U . Using the partition:
" #" # " #
(j ) U (j )
U11 x0j e0j ;
12
0 U22(j) x00j = 0 (2.30)
where U11(j) is the j j upper left submatrix of U and e0j = [0 0 : : : 0 1℄T 2 Rj , the
system (2.29) be omes: ( (j )
U11 x0j + U12 (j ) x00 = e0
(j ) x00 = 0
U22
j j (2.31)
j
Sin e U11(j) and U22(j) are nonsingular, the relation (2.31) be omes:
(
(j ) x0 = e0
U11
xj = 0
00
j j (2.32)
and have the following signi an e:
Proposition 2.31 The inverse of a nonsingular upper triangular matrix is upper
triangular.
The omputational s heme resulted from (2.32) is:
1. for j = 1 : n
1. x0j = UT RIS (U11(j) ; e0j )
2. if j < 1 then x00j = 0
If the systems (2.32) are solved in reverse order (i.e., for j = n; n 1; : : : ; 1), it
an be easily seen that the inverse matrix omponents may overwrite those of the
original matrix. Hen e, we have the following:
Algorithm 2.32 (UINV) (Given a nonsingular upper triangular ma-
trix U 2 Rnn, this algorithm overwrites U by its inverse.)
1. for j = n : 1 : 1
1. ujj xjj = 1=ujj
2. if j = 1 then stop
3. for i = j 1 : 1 :1P
1. uij xij = j
k=i+1 ik kj =uii
u x
As for the lower triangular ase, NUINV n3=3 and MUINV n2=2.
2.6.2 Matrix inversion using GPP and GCP
If A is a nonsingular n n real matrix, then the Gaussian elimination with partial
pivoting arries out to ompletion and provides a nonsingular upper matrix U :
Mn 1 Pn 1 : : : M1 P1 A = U; (2.33)
where Ms are (nonsingular) elementary triangular matri es and Ps are elementary
permutation matri es. From (2.33):
X A 1 = U 1 Mn 1 Pn 1 : : : M1 P1
and, hen e, the inverse of a matrix A an be omputed as follows
2.6. MATRIX INVERSION 39
U U 1
A ) A
M M
Figure 2.7: Data storage in matrix inversion.
1. [U; M; p℄ = GP P (A)
2. X = UINV (U )
3. for s = n 1 : 1 : 1
1. X XMs
2. X XPs
Remember that in the GP P algorithm, the matrix U and the set of Gaussian
multipliers is, s = 1 : n 1, i = s + 1 : n, denoted by M in the s heme above,
overwrite the matrix A as in the left side of gure 2.7. In algorithm UINV , we have
seen that U 1 an overwrite U , so that, after the instru tion 2 in the s heme above,
the data storage is like in the right side of gure 2.7.
We shall des ribe an algorithm, based on the presented s heme, whi h omputes
the inverse of A with a minimum amount of used memory. To do this, observe that
the rst two instru tions need additional memory only for the ve tor p 2 Rn 1 ,
whi h stores the row representation of GP P . Let now analyze the for loop in
instru tion 3. The rst assignment, X XMs, an be detailed by a row partitioning
of X :
xTi Ms = xTi (In mseTs ) = xTi (xTi ms)eTs
= [xi1 xi2 : : : xi;s 1 xis xTi ms xi;s+1 : : : xin℄; i = 1 : n: (2.34)
It an be seen that only the elements on the s-th olumn of X are modied.
The se ond instru tion of the for loop, X XPs, inter hange the s-th and is-th
olumns (is > s).
Taking into a ount the above onsiderations and the reverse order of olumn
al ulation, the step s of the for loop 3 in the s heme above, i.e. X XMs Ps, an
be des ribed like in gure 2.8, where U 1 and M denote the elements of U 1 and the
Gaussian multipliers whi h were still not used (and not ae ted) in the exe ution of
the loop until the urrent step; ae ted elements (part of X ) are denoted by .
Thus, the only information that must be saved onsists of the Gaussian multi-
pliers is, i = s + 1 : n, required to perform (2.34).
( 1) : : : :::
U 1 u1s. U 1 ...
.. : : : :::
uss 1) : : : X
( XMs-
Ps :::
M
s+1;s : : :
M
:::
... ...
ns ::: :::
s s
Figure 2.8: Data storage in step s of matrix inversion.
In this way, a matrix inversion an be done with only two additional (n 1)

ve tors: p for storing the permutation and, say, m for temporary saving the urrent
Gaussian multipliers.
The algorithm that implements the above onsiderations is:
Algorithm 2.33 (INV GPP) (Given a nonsingular matrix A 2 Rnn ,
this algorithm overwrites A by its inverse, A 1 . The algorithm uses GP P
for Gaussian triangularization and UINV to invert an upper triangular
matrix.)
1. [U; M; p℄ = GP P (A) fU and M are stored over Ag
2. A X = UINV (U ) fonly the upper left triangle is modiedg
3. for s = n 1 : 1 : 1
1. for i = s + 1 : n
1. mi is fsave the multipliersg
2. for i = 1 : s Pn
1. ais ais s+1 aik mk frelation (2.34), rst s rowsg
3. for i = s + 1P: nn
1. ais k=s+1 aik mk frelation (2.34), other rowsg
4. if p(s) 6= s then
1. for i = 1 : n finter hange olumns s and p(s)g
1. ais $ ai;p(s)
The number of oating point operations is given by:
NINV GP P = NGP P + NUT RIS +
nX1
2n(n s)
2n3 + n3 + n3 = 2n3
s=1 3 3
and the memory needed is, of ourse, MINV GP P = n2.
It is remarkable that the matrix inversion is not more diÆ ult (as number of
operations) than the matrix multipli ation.
2.7. DETERMINANT CALCULUS 41
A better pre ision an be obtained by using GCP instead of GP P . In this ase:
Mn 1 Pn 1 : : : M1 P1 AQ1 Q2 : : : Qn 1 = U:
Then:
X A 1 = Q1 Q2 : : : Qn 1 U 1 Mn 1 Pn 1 : : : M1 P1
and the resulting algorithm is obtained by using GCP and adding to algorithm
INV GP P the row inter hanges whi h orrespond, in reverse order, to the olumn
permutations from GCP .
In pra ti e, the pre ision obtained by INV GP P is good enough for most ap-
pli ations.
Remark 2.34 The omputational eort needed to invert a nonsingular matrix is
about three times greater then the eort for solving a linear system of the same
dimension. So, whenever possible, matrix inversion has to be avoided and repla ed
by linear system solving. For example, for omputing the real s alar:
= bT A 1 ;
where A 2 Rnn, b; 2 Rn, the re ommended s heme is:
1. Solve Ax = to ompute x = A 1
2. bT x
2.7 Determinant al ulus

The GP P and GCP algorithms are good tools to ompute the determinant of a
square matrix. The following well known or easy to verify fa ts give the idea of this
omputation.
Proposition 2.35 If A; B 2 Rnn are two square matri es, then
det(A B ) = det(A) det(B ):
The property holds for any nite number of square matri es.
Proposition 2.36 If A 2 Rnn is a triangular matrix, then:
n
Y
det(A) = aii :
i=1
Proposition 2.37 The inter hange of two rows or olumns hanges the sign of the
matrix determinant, or, equivalently, if P is an elementary permutation matrix,
then:
det(P ) = 1:
The elementary lower triangular matri es Ms, s = 1 : n 1, in (2.33), have unit
diagonal elements. Then,
det(Ms ) = 1
and, hen e, n
Y
det(A) = ( 1)k det(U ) = ( 1)k uii;
i=1
where k n 1 is the number of proper elementary permutation matri es (i.e.
P 6= In ) in (2.33). The orresponding algorithm is:
Algorithm 2.38 (DET) (Given a square matrix A 2 Rnn , this
algorithm omputes the matrix determinant using Gaussian redu tion
to triangular form, with row pivoting.)
1. [U; M; p℄ = GP P (A)
2. det 1
3. for s = 1 : n
1. det det uss
4. for s = 1 : n 1
1. if p(s) 6= s then
1. det det
The main omputational eort in the algorithm above is due to GP P exe ution.
Obviously, GCP an be used instead of GP P , with slightly better pre ision of the
answer.
2.8 LDU and LU matrix fa torizations

In many situations, it is onvenient to express a given matrix as a produ t of two
triangular matri es. This form an be used to simplify al ulations involving the
given matrix.
Denition 2.39 Let A 2 Rnn be given. If there exist a lower triangular matrix
L 2 Rnn and an upper triangular matrix U 2 Rnn su h that
A = LU (2.35)
then the above expression is alled LU matrix fa torization.
2.8. LDU AND LU MATRIX FACTORIZATIONS 43
Denition 2.40 A matrix D 2 Rnn simultaneously lower and upper triangular
(i.e. dij = 0 if i 6= j ) is alled diagonal.
It is an easy exer ise to the reader to prove the following:
Proposition 2.41 The produ t of any number of (unit) lower (upper) triangular
matri es is a (unit) lower (upper) triangular matrix.
Let now A 2 Rnn be a LU -de omposable matrix and D a nonsingular diagonal
matrix. Then, from propositions 2.29 and 2.31, D 1 is also diagonal, and, hen e,
(2.35) an be written in the form:
A = LU = LDD 1U = L0 U 0 ; (2.36)
where L0 = LD is lower triangular and U 0 = D 1U is upper triangular. If D 6= I ,
then the fa torizations (2.35) and (2.36) dier and so that, the LU fa torization, if
there exists one, is not unique. Thus, it is useful to introdu e:
Denition 2.42 If A 2 Rnn, then the expression
A = LDU;
where L 2 Rnn is unit lower triangular, U 2 Rnn is unit upper triangular and
D 2 Rnn is a diagonal matrix, is alled the LDU fa torization of matrix A.
The following important fa t shows under whi h onditions a LDU fa torizations

exists and is unique.
Theorem 2.43 A matrix A 2 Rnn has a unique LDU fa torization if and only if
the leading prin ipal submatri es A[s℄, s = 1 : n 1, are all nonsingular.
Proof. We give a proof for the ase when the matrix A itself is nonsingular. The
theorem's proof for singular A is a not trivial exer ise left to the reader.
Let show rst that if a LDU de omposition exists, it is unique. Suppose that
there exist two dierent LDU de ompositions:
A = LDU = L0 D0 U 0 : (2.37)
L, L0 , U , U 0 are unit triangular matri es and, so that, nonsingular; moreover, L 1 ,
(L0 ) 1, U 1 , (U 0) 1 have the same stru ture. A is nonsingular, hen e so are D and
D0 , and the matri es D 1 and (D0 ) 1 are diagonal. Then, from (2.37):
(L0) 1 L = D0U 0U 1 D 1
where the left hand side is a unit lower triangular matrix and the right hand side is
an upper triangular matrix. Hen e
(L0 ) 1 L = In
i.e.
L = L0 :
Now, it results that:
U 0 U 1 = (D0 ) 1 D:
The left hand side is a unit upper triangular matrix and the right hand side a
diagonal one. This is possible if and only if both are unit matri es In. Hen e:
U = U 0; D = D0
whi h ompletes the proof of the LDU de omposition uniqueness.
Also, if the LDU de omposition of the nonsingular matrix A exists, then D is
nonsingular and it is easy to see that:
A[s℄ = L[s℄D[s℄U [s℄; s = 1 : n;
where, obviously, L[s℄, D[s℄ and U [s℄ are all nonsingular. Hen e, A[s℄, s = 1 : n, are
all nonsingular.
It remains to show the existen e of the LDU de omposition when A[s℄ are non-
singular for s = 1 : n 1. From theorem 2.17, whi h states the theoreti al base
for Gaussian elimination, if A[s℄, s = 1 : n 1, are nonsingular, then there exists
elementary lower triangular matri es Ms, s = 1 : s 1, su h that:
Mn 1 : : : M2 M1 A = R; (2.38)
where R is a nonsingular upper triangular matrix. Denoting:
L = (Mn 1 : : : M2 M1 ) 1
D = diag(r11 ; r22 ; : : : ; rnn )
U = D 1 R;
then (2.38) an be written as A = LDU , with L, D, U satisfying the onditions for
a LDU de omposition. 2
Pra ti al LU fa torizations asso iate the diagonal matrix D from the LDU fa -
torization:
with the matrix U , dening the so alled Doolittle fa torization A = LU , where
L is unit lower triangular and U is upper triangular,
2.8. LDU AND LU MATRIX FACTORIZATIONS 45
or with the matrix L, dening the Crout fa torization A = LU , where L is
lower triangular and U unit upper triangular.
From theorem 2.43 it results that both Doolittle and Crout fa torizations exist
and are unique if and only if the leading prin ipal submatri es A[s℄, s = 1 : n 1,
are all nonsingular.
An algorithm to ompute the Doolittle fa torization is provided by the algo-
rithm G for Gaussian triangularization. Indeed, the algorithm G produ es the
ELT matri es Ms so that from (2.38) it result the LU fa torization A = LU , with
L = (Mn 1 : : : M2 M1 ) 1 and U = R. Hen e, to ompute the Doolittle fa torization,
it is suÆ ient to apply the algorithm G and to ompute
L = M1 1 M2 2 : : : Mn 11 :
But (see proposition 2.13):
Ms 1 = In + ms eTs ; s = 1 : n 1
and, hen e:
nX1
L = (In + m1 eT1 )(In + m2 eT2 ) : : : (In + mn 1 eTn 1 ) = In + mseTs + S;
s=1
where S is a sum of matrix terms of the form:

T = : : : mp eTp mq eTq : : :
with p < q. Sin e eTp mq = pq = 0, for all p < q, then T = 0 and S = 0. Thus,
2
1 0 ::: 0 0 3
6
6 21 1 : : : 0 0 777
nX1 6
6 31 32 : : : 0 0 77
L = In + mseTs = 66 . .
s=1 6 ::: . : : : 777
6
4 n 1;1 n 1;2 : : : 1 0 5
n1 n2 : : : n;n 1 1
and this means that the algorithm G omputes the Doolittle de omposition, both
matri es L (without its diagonal elements) and U overwriting the matrix A.
The Crout de omposition an be omputed taking in (2.36) LU as the Doolittle
de omposition and D = diag(u11 ; u12 ; : : : ; unn). Then, L0U 0 is the Crout de ompo-
sition of A.
However, there are some other possibilities to organize the omputation to obtain
the so alled ompa t fa torizations. We shall derive su h a Crout fa torization,
omputed elements
1 ?
0
ass. .. asj . .. = lss 1 .. . usj. ..
... ...
ais i lis 0
... ... 1

omputed elements s
Figure 2.9: Intermediate stage in Crout fa torization.
based on the idea of su esive omputation of a olumn of L and of a row of U . The

pro edure may be started from the identity A = LU , resulting:
Ae1 = LUe1 = Le1
i.e.
li1 = ai1 ; i = 1 : n (2.39)
and
eT1 A = eT1 LU = l11 eT1 U;
i.e.
a1j = l11 u1j ; j = 2 : n;
where l11 6= 0 (sin e l11 = a11 = A[1℄ 6= 0). Then,
u1j = a1j =l11 ; j = 2 : n: (2.40)
Let suppose now that the rst s 1 olumns of L and s 1 rows of U are known
( omputed); see gure 2.9.
From the A = LU identity we have
sX1
ais = lik uks + lis ; i = s : n;
k=1
where the only unknown s alar is lis. Hen e,

sX1
lis = ais lik uks ; i = s : n: (2.41)
k=1
2.9. CHOLESKY FACTORIZATION 47
Also: sX1
asj = lsk ukj + lss usj ; j = s + 1 : n;
k=1
where, now, the only unknown s alar is usj . It an be shown that the nonsingularity
of A[s℄ implies lss 6= 0. Hen e,
!
sX1
usj = asj lsk ukj =lss ; j = s + 1 : n: (2.42)
k=1
So, the pro edure started by (2.39) and (2.40) an be ontinued to ompletion
by (2.41) and (2.42), if A[s℄, s = 1 : n 1, are all nonsingular, whi h guarantee that
the division in (2.42) may be exe uted. The resulting algorithm is:
Algorithm 2.44 (CROUT { Crout fa torization)(Given a matrix
A 2 Rnn whi h has all leading prin ipal matri es A[s℄, i = 1 : n 1,
nonsingular, this algorithm omputes the Crout fa torization A = LU ,
where L is lower triangular and U is unit upper triangular. L and U
are stored over the orresponding elements of A (without the diagonal
elements of U , whi h are 1).)
( 0. ai1 li1 = ai1 ; i = 1 : n)
1. for j = 2 : n
1. a1j u1j = a1j =l11
2. for s = 2 : n
1. for i = s : n
1. ais lis = ais Pks =11 lik uks
2. if s = n then stop
3. for j = s + 1 : n P
1. asj usj = asj ks =11 lsk ukj =lss
The oating point operations asimptoti ally totalize NCROUT 2n3 =3 as in G,

GP P or GCP algorithms and the memory used is MCROUT n2 .
To avoid the ne essity for A[s℄, s = 1 : n 1, to be nonsingular, only row pivoting
is possible. It may be introdu ed after instru tion 2.1 of the algorithm above.
2.9 Cholesky fa torization

An important spe ial ase of LU fa torization whi h an be performed by an eÆ ient
and reliable algorithm is that of positive denite matri es.
Denition 2.45 A square symmetri (A = AT ) matrix A is alled positive denite

if for any nonzero ve tor x 2 Rn
xT Ax > 0 (2.43)
and positive semidenite if
xT Ax 0:
Denition 2.46 If A is a real n n matrix and I = fi1 ; i2 ; : : : ; ir g is a set of indi es
su h that i1 < i2 < : : : < ir n, then the r r matrix A0 dened by a0kl = ai i , for
k; l = 1 : r, is alled prin ipal submatrix of A.
k l
Re all that if I = f1; 2; : : : ; rg, then A0 = A[r℄ is alled leading prin ipal subma-
trix of A, of order r.
For positive denite matri es, we have the following:
Proposition 2.47 A prin ipal submatrix of a positive denite matrix is positive
denite.
Proof. Let the set I = fi1 ; i2 ; : : : ; ir g dene a prin ipal submatrix A0 of the given
positive denite matrix A 2 Rnn. For any nonzero ve tor x 2 Rr , dene the ve tor
y 2 Rn by:
yi = xk ; k = 1 : r
yi = 0; i 62 I :
k
Obviously, x 6= 0 implies y 6= 0, and from the positiveness of A:

xT A0 x = yT Ay > 0;
i.e. A0 is positive denite. 2
Corollary 2.48 All leading prin ipal submatri es of a symmetri positive denite
matrix, and the matrix itself, are nonsingular.
Proof. It is suÆ ient to show that a positive denite matrix is nonsingular and
then to apply proposition 2.47. Assume that a positive denite matrix is singular.
Then, there exists a nonzero ve tor x 2 Rn so that Ax = 0 and therefore xT Ax = 0,
whi h ontradi ts the denition of positiveness. 2
The theoreti al basis for the Cholesky de omposition is given by:
Theorem 2.49 For any symmetri positive denite matrix A 2 Rnn , there exists
a unique lower triangular matrix L 2 Rnn with positive diagonal elements su h
that:
A = LLT ;
whi h is alled the Cholesky fa torization.
2.9. CHOLESKY FACTORIZATION 49
Proof. The matrix A is positive denite; hen e, a ording to orollary 2.48, A[s℄,
s=1:n 1, are nonsingular and, therefore, A has a unique LDU fa torization (see
theorem 2.43). From the matrix symmetry (A = AT ):
A = L0 DU 0 = (U 0 )T D(L0 )T :
The uniqueness of the LDU de omposition and the relation above imply:
U 0 = (L0 )T :
Hen e, there exists a unique unit lower triangular matrix L0 2 Rnn and a unique
diagonal matrix D 2 Rnn su h that:
A = L0 D(L0 )T : (2.44)
Moreover, using (2.43) for xj = (L0 ) T ej , (2.44) leads to:
xTj Axj = eTj (L0 ) 1 L0 D(L0 )T (L0 ) T ej = eTj Dej = djj > 0;
for all j = 1 : n. Hen e, all diagonal elements of D are positive. Then, denoting:
p p p
E = diag( d11 ; d22 ; : : : ; dnn );
we have D = E 2 and (2.44) an be written as:
A = L0 EE (L0 )T = LLT ;
where
pd , i L= =1 :LnE. is a lower triangular matrix with positive diagonal elements lii =
0
ii
We proved the existen e of the Cholesky fa torization A = LLT ; the uniqueness
derives from the uniqueness of the LDU de omposition and the uniqueness of the
matrix E (with positive diagonal elements) su h that D = E 2 . 2
There are some few algorithms to ompute the Cholesky fa torization of a posi-
tive denite matrix, whi h dier in the ordering of s alar omputations. A possible
order is to al ulate from top to down the j -th olumn of L, for j = 1 : n. Identifying
A = LLT (only the lower left triangle of A is represented sin e A is symmetri )
2 3 2 3 2 3
a11 l11 l11 : : : ls1 : : : ln1
6 : : : ... 6 : : : ... . . . : : : 777
6 7 6 7 6
6
7
7 6 0 7
7
6
6
6
6 as1 : : : ass
7
7 = 6
6 ls1 : : : lss
7
7
6
6 lss : : : lns 777
6
6
4 ::: . . . 775 6
6
4 ::: . . . 775 6
6
4 0 . . . : : : 75
an1 : : : ans : : : ann ln1 : : : lns : : : lnn lnn
for the rst olumn we have:

2 ) l11 = a11
a11 = l11
p
ai1 = li1 l11 ) li1 = ai1 =l11 ; i = 2 : n (2.45)
i.e. the omputational pro ess an start.
Now, assuming that the rst s 1 olumns of L were omputed, the identi ation
of the (s; s) element in A = LLT yields:
sX1
ass = 2 + l2 ;
lsk ss (2.46)
k=1
where the only unknown s alar is lss. A ording to theorem 2.49, if A is positive
denite, there exists a unique positive lss su h that (2.46) holds. Hen e,
sX1
ass 2 >0
lsk
k=1
and v
u
u sX1
lss = ta
ss
2:
lsk (2.47)
k=1
Also,
sX1
ais = lik lsk + lis lss; i = s + 1 : n;
k=1
where, in the given order of omputation, the only unknown is lis. Hen e:
!
sX1
lis = ais lik lsk =lss ; i = s + 1 : n: (2.48)
k=1
Formulas (2.45) and (2.47), (2.48) dene the algorithm given below:

Algorithm 2.50 (CHOL { Cholesky fa torization) (Given a sym-
metri matrix A 2 Rnn , this algorithm establishes if the matrix is
positive denite. If so, it overwrites the lower triangle with the matrix
L of the Cholesky fa torization A = LLT .)
1. if a11 0 then
1. Print('A is not positive denite')
2. stop p
2. a11 l11 = a11
3. for i = 2 : n
1. ai1 li1 = ai1=l11
2.10. APPLICATIONS OF THE LU FACTORIZATIONS 51
4. for s = 2 : n P
1. ass ks =11 lsk2
2. if 0 then
1. Print('A is not positive denite')
2. stop
3. ass lss = p
4. if s = n then stop
5. for i = s + 1 : n P
1. ais lis = ais ks =11 lik lsk =lss
The above algorithm is the best test for the positive deniteness of a matrix.
The fa t is based on the fa t that the Cholesky fa torization of a symmetri matrix
an be arried out to ompletion if and only if the matrix is positive denite (see
theorem 2.49).
The CHOL algorithm requires about NCHOL = n3=3 ops and additional n
square root al ulations. The memory needed is about MCHOL = n2=2 (a symmetri
matrix is usually stored by one of its triangular parts).
2.10 Appli ations of the LU fa torizations

The LU fa torizations are useful in many omputations involving a square matrix.
For the general ase, for reliability reasons, the LU fa torization must be omputed
with a suitable pivoting strategy. However, to emphasize the LU de omposition
utility, we shall assume that a generi given matrix A 2 Rnn really has su h a
fa torization.
2.10.1 Linear system solving
The solution of the linear system Ax = b an be found using the following s heme:
1. A = LU ( ompute the LU fa torization)
2. Solve the lower triangular system Ly = b
3. Solve the upper triangular system Ux = y
The omputational eort is about the same as in the Gaussian elimination and
the main part of it is done in the fa torization pro ess.
When the linear system has multiple right hand side, i.e. has the form
AX = B;
where B 2 Rnp, and X will result of same dimension, this system an be redu ed
to p systems with simple right hand side, Axj = bj , j = 1 : p, where bj and xj are
the j -th olumns of B and X , respe tively. However, the omputational eort is not
p times bigger; sin e A is the same for all p systems, the LU fa torization an be
done only on e, and then only triangular systems have to be solved. The algorithm
is the following:
2. for j = 1 : p
1. Solve the lower triangular system Ly = bj
2. Solve the upper triangular system Uxj = y
The number of operations will be only about 2n3 =3 + 2pn2.
2.10.2 Matrix inversion
If a nonsingular matrix A has a LU de omposition, then

A 1 = U 1L 1;
so an inversion pro edure an be:
2. U X = U 1
3. L Y = L 1
3. A 1 XY
The number of operations is 2n3.
2.10.3 Determinant al ulus
Obviously, det(A) = det(LU ) = det(L) det(U ) = (Qni=1 lii) (Qni=1 uii). So, a LU
fa torization gives an immediate possibility of determinant evaluation.
2.11 Numeri al ondition of linear systems

2.11.1 Matrix norms
In order to evaluate the numeri al ondition of a linear system, we need some ele-
mentary notions of matrix norms.
Denition 2.51 A fun tion : Rmn ! R+ is a matrix norm on Rmn if:
2.11. NUMERICAL CONDITION OF LINEAR SYSTEMS 53
1. (A) > 0, 8A 2 Rmn , A 6= 0.
2. (A) = jj (A), 8A 2 Rmn, 8 2 R.
3. (A + B ) (A) + (B ), 8A; B 2 Rmn.
This denition has some immediate onsequen es, given by
Corollary 2.52 If : Rmn ! R+ is a matrix norm, then:
1. (0) = 0.
2. ( A) = (A), 8A 2 Rmn.
3. The matrix sequen e (Ak )k2N is onvergent and has the limit A if and only if
the s alar sequen e ( (Ak A))k2N is onvergent to zero, i.e.
lim Ak = A , klim
k!1 !1
(Ak A) = 0:
To make use of matrix norms independently of matrix dimensions, we introdu e

the following:
Denition 2.53 A fun tion : 1 mn ! R is a family of matrix norms
S
m=1;n=1 R +
if for ea h m; n > 0, the restri tion of to Rmn is a matrix norm. A family of
matrix norms is onsistent if
(AB ) (A) (B )
whenever the produ t AB is dened.
If n = 1, then is a family of ve tor norms.
The following theorem provides a pra ti al way to dene on rete matrix norms:
Theorem 2.54 Let be a family of ve tor norms. For A 2 Rmn , dene:
kAk = max
(x)=1
(Ax):
Then, k k : S1m=1;n=1 Rmn ! R+ is a onsistent family of matrix norms.

The family of ve tor norms kk is said to be subordinate to the family of ve tor
norms .
The most natural family of ve tor norms used in theorem 2.54 is one of the
p-norms (or Holder norms), for p = 1; 2; 1, dened by:
!1=p
m
X
kxkp = jxijp
i=1
i.e., respe tively: m

X
kxk1 = jxij
i=1
v
um
uX
kxk2 = t
i (eu lidian norm)
x2
i=1
kxk1 = imax jx j:
=1:m i
The matrix 1-norm and 1-norm, dened like in theorem 2.54, may be easily
omputed: m !
X
kAk1 = kmax
xk =1
kAxk1 = jmax
=1:n
jaij j
1 i=1
0 1
n
X
kAk1 = kxmax
k =1
kAxk1 = imax
=1:m
jaij jA:
1 j =1
The matrix p-norms, applied to ve tors seen as one olumn matri es, give the
same values as the orresponding ve tor p-norms. Hen e, there is no possibility of
onfusion in using the same symbol k kp for the matrix norm and the ve tor norm.
Another matrix norm widely used is the Frobenius norm, dened by:
v
uXm X
n
kAkF = u
t a2 ij :
i=1 j =1
The family of Frobenius norms is onsistent.

2.11.2 Numeri al ondition of linear systems
Let return now to the problem of numeri al ondition of linear determined nonsin-
gular systems of form Ax = b, where A 2 Rnn and b 2 Rn are given.
For a perturbation
(A; b) ! (A + A; b + b) (A; b)
in the input data, the perturbation of the solution x
x ! x + x x
is su h that
(A + A)(x + x) = b + b:
Hen e, sin e Ax = b:
A x + A x + A x = b:
Negle ting the produ t A x, we obtain:
x A 1 A x + A 1 b
and, for a onsistent family of matrix norms k k,
kxk kA 1 k kAk kxk + kA 1 k kbk:
If x 6= 0, a relative error bound is given by:
kxk kA 1 k kAk kAk + kA 1 k kbk
kxk kAk kxk
= kA 1 k kAk kAk + kA 1 k kAk kbk
kAk kAk kxk
k

A k
kA 1 k kAk kAk + kbk :k b k
Hen e, the relative errors kAk=kAk and kbk=kbk in the input data are amplied
(A) = kA 1 k kAk (2.49)
times, independently of the algorithm used to solve the system.
The real number (A) dened in (2.49) is alled the ondition number of the
matrix A and provides a measure for the numeri al ondition of the given linear
system. A smaller orresponds to a better numeri al ondition.
Remark 2.55 A usual way to verify a omputed solution x of the linear system
Ax = b is to al ulate a norm for the residual r = b Ax:
krk = kb Axk
and to on lude that if krk is "small", then x is a "good" (a urate) solution. In
fa t this on lusion is not true if the matrix A is ill onditioned. Indeed, the true
solution is x = A 1b, so:
x x = A 1 b A 1 (b r) = A 1 r
and, hen e, the relative error has the following bound:
kx xk = kA 1 rk kA 1 k krk (A) krk (A)
kr k
kxk kA bk1 kA rk1 kAk kA bk1 kbk
for any onsistent norm k k. So the solution test "r is small" is valid only if the
matrix A has a good numeri al ondition (i.e. small ondition number).
2.11.3 Numeri al stability of dire t algorithms for linear systems
The ee ts of rounding errors on the algorithms des ribed in this hapter is usually
expressed in terms of numeri al stability whi h involve a ba kward error analysis.
An algorithm for solving the determined linear system Ax = b is alled numeri-
ally stable if the omputed solution x is the exa t solution of a linear system:
(A + E )x = b
where the perturbation matrix E has a small norm ( ompared to kAk). Thus, the
solution error has an equivalen e in initial data error.
To evaluate bound for kE k is a very diÆ ult task. Sometimes there are no
su h analiti al bounds and the numeri al stability of an algorithm may be stated by
pra ti al experien e.
We present here only a brief numeri al stability hara terization of several im-
portant algorithms:
The solutions of triangular systems are usually omputed with high a ura y
and the orresponding algorithms are stable.
The algorithm GCP of Gaussian elimination with omplete pivoting is numer-
i ally stable.
The Gaussian elimination with partial pivoting is not un onditionally stable,
but in omputing pra ti e it is onsidered stable.
The Crout redu tion algorithm with row pivoting is also pra ti ally a stable
algorithm.
The Cholesky fa torization algorithm for positive denite matri es is un on-
ditionally stable. So that, it must be used whenever the system matrix is
symmetri and positive denite.
Problems
P 2.1 A; B 2 R
n n
are nonsingular lower bidiagonal matri es (a = 0 for i < j or i > j +1).
Give eÆ ient algorithms for:
ij
the multipli ations Ax and ABx, where x 2 .

a. R
n
b.the matrix multipli ation AB.

solving the linear system Ax = , where 2 .
.
n
R
d.solving the linear system Ax = e , where e is the j -th unit ve tor.

j j
solving the linear system ABx = , where 2 .

e. R
n
omputing the inverse of A.

f.
P 2.2 Let A 2 be a nonsingular lower bidiagonal matrix, and T = AA . Give
R
n n T
eÆ ient algorithms to:

ompute det(T ).
a.
ompute the Frobenius norm of T .

b.
solve the linear system T x = b, where b 2 .

.
n
R
ompute the inverse of T .

d.
ompute the Cholesky fa torization of T (prove rst that T is symmetri and positive
e.
denite).
P 2.3 Let H 2 be a nonsingular upper Hessenberg matrix (h = 0, for i > j + 1).
R
n n
ij
If all the prin ipal leading submatri es of H are nonsingular, adapt the Gaussian
a.
elimination algorithm for solving the linear system Hx = b, where b 2 ; al ulate the
R
n
number of operations.
Adapt the GP P algorithm for the same problem.
b.
Adapt the Crout algorithm for omputing the LU fa torization.

.
P 2.4 Let onsider the matri es: H 2 , nonsingular upper Hessenberg, and R 2
R
n n
R
, unit upper triangular. Present eÆ ient algorithms for:
n n
omputing the matrix multipli ation HR.

a.
omputing the ve tor HRb, with b 2 .

b. R
n
solve the linear system HRx = b, with b 2 .

. R
n
When all prin ipal leading submatri es of H are nonsingular, the Crout fa torization
d.
of A = HR an be obtained using one of the following two s hemes:
S heme 1. 1. Compute A = HR.
2. Compute the Crout fa torization of A: A = LU .
S heme 2. 1. Compute the Crout fa torization of H : H = LU .
2. Compute U = UR .
Whi h one is the most eÆ ient ?
P 2.5 a. Propose an algorithm to solve the linear system Ax = b, where A 2 is R
n n
nonsingular and b 2 , using only real arithmeti .

C
n
Give an algorithm for solving the matrix equation AX = B, where A 2 is

b. R
n n
nonsingular and B 2 . R
n m
P 2.6 Give an eÆ ient algorithm to solve the linear system A x = b, where A 2 is

k
R
n n
nonsingular, b 2 and k 2 , k > 1.

R
n
N

P 2.7 Let B = RA A0 , with A; R 2 , nonsingular, R upper triangular. The LU
n
R
n
fa torization of A exists and it is known (A = LU ).

Write an algorithm for omputing the LU fa torization of B, B = L~ U~ .
a.
Propose an algorithm for solving the linear system Bx = d, where d 2 .

b. R
2n
Cal ulate the number of operations for both these algorithms.

P 2.8 Let A 2 be a nonsingular matrix with all the leading prin ipal matri es
R
2n 2n
nonsingular, A = AA AA , with A ; A ; A ; A 2 and A is upper triangular.

1
3
2
4
1 2 3 4
n
R
n
3
a.Write an algorithm for solving the linear system Ax = b, with b 2 . R

2n
b. The same, keeping only the assumption that A is nonsingular.

P 2.9 Let A 2 be a nonsingular tridiagonal matrix (a = 0, for i > j +1 or i < j 1).
R
n n
ij
a.Adapt the Gaussian elimination algorithm to this type of matrix.

b. Give the algorithm that solves Ax = b, with b 2 . n
R
. If A is symmetri and positive denite, adapt the Cholesky fa torization algorithm

for A.
P 2.10 Let u; v 2 be two nonzero ve tors, and A = I + uv .
R
n
n
T
a.Give an algorithm to ompute the Frobenius norm of A.

b. Give an eÆ ient algorithm to ompute the determinant of A. When is A nonsingular ?
. If A is nonsingular and b 2 , write an eÆ ient algorithm for solving the linear
n
R
system Ax = b.
P 2.11 Des ribe a variant of Gaussian elimination that introdu es zeros into the olumns
of A, above the diagonal, in the order n : 1 : 2, and whi h produ es the fa torization
A = UL, where U is unit upper triangular and L is lower triangular.
P 2.12 Suppose that A 2 has an LU fa torization and that L and U are known. Give
R
n n
an algorithm whi h omputes the (i; j )-entry of A in approximately (n j ) + (n i)

1 2 2
ops.
P 2.13 If A 2 is symmetri and positive denite, propose an algorithm for the
R
n n
fa torization A = UU , where U is upper triangular and has positive diagonal elements.

T
P 2.14 If A; B 2 are nonsingular matri es, give an algorithm to solve the linear
R
n n
system (AB) x = , where 2 .

k
R
n
P 2.15 Give a variant of the algorithm LSS GP P for solving the linear system Ax = b
using Gaussian elimination with partial pivoting, without storing the multipliers.
Chapter 3
The linear least squares problem

3.1 The linear least squares problem
Let us onsider the linear system
Ax = b;
where A 2 Rmn , b 2 Rm .
Generally, an overdetermined system (m > n) has no solution and we must
hange the meaning of its solving. A natural statement of the problem is to nd a
ve tor x 2 Rn su h that the ve tor y = Ax is as near as possible to the ve tor b.
More pre isely, we want to determine a ve tor x 2 Rn that minimizes the fun tion:
(x) = (b Ax);
where is a ve tor norm on Rm .
For an underdetermined system (m < n), generally, there are more than one
solution. A natural riterion to sele t a single one is to require it to have minimum
"length", i.e. to minimize the fun tion:
(x) = (x)jAx=b ;
where is a ve tor norm on Rn.
Most of the standard te hniques for nding minima of fun tions are unapplyable
to the problem above, sin e, generally, a norm is not a dierentiable fun tion of
its argument. Thus, there are not general pro edures, independent of the norms
involved, to solve these problems.
When is the eu lidian norm, the problem of minimizing the norm of the residual
r = b Ax is alled the linear least square (LLS) problem.
The LLS problem is en ountered in many area of appli ations, su h as approxi-
mation theory of fun tions and data, statisti s, et .
59
60 CHAPTER 3. THE LINEAR LEAST SQUARES PROBLEM
Example 3.1 Let onsider a pro ess des ribed by a fun tional dependen y y =
f (u), known only from experimental data given by the set of pairs f(ui ; yi ) j i = 1 :
mg.
Our problem is to nd an approximate analyti al expression f for the fun tion
f as a linear ombination of the given n independent fun tions g1 ; g2 ; : : : ; gn that
an be evaluated in the points ui, i = 1 : m, i.e. to nd the oeÆ ients xi , i = 1 : n,
su h that the fun tion n
y = f(u) =
X
xj gj (u)
j =1
makes the residuals
ri = yi f(ui ); i = 1 : m
as small as possible.
If m > n, there are more residuals than unknowns and it will not be possible,
generally, to make all of them equal to zero.
The LLS approa h is to hoose the oeÆ ients xi so that:
!1=2
0 0 12 11=2
m
X m n
BX X
krk = ri2 = yi xj gj (ui )A C
A = kb Axk
i=1 i=1 j =1
where
b = [y1 y2 : : : ym ℄T 2 Rm
A 2 Rmn ; aij = gj (ui ); i = 1 : m; j = 1 : n;
is minimized.
Parti ularly, if gj (u) = uj 1, j = 1 : n, then f is a polinomial fun tion of degree
n 1.
The theory of LLS is intimately related to the geometry of the spa e Rn, espe-
ially to the notion of orthogonality whi h makes the LLS problem meaningful. In
the next se tion we will re all some geometri al properties of the ve tor spa e Rn.
3.2 Orthogonality. Householder and Givens transfor-
mations
Let us introdu e some denitions, propositions and theorems. The proofs not given
below an be found in any book of linear algebra.
Denition 3.2 Two ve tors x; y 2 Rm are orthogonal if
m
X
xT y = x i yi = 0:
i=1
3.2. ORTHOGONALITY. HOUSEHOLDER AND GIVENS TRANSFORMATIONS 61
The ve tors u1; u2 ; : : : ; up are orthogonal if they are pairwise orthogonal, i.e.
uTi uj = 0; 8i 6= j; i; j = 1 : p: (3.1)
If, in addition to (3.1), the ve tors ui have unit eu lidian norm:
kuik = 1; 8i = 1 : p;
they are alled orthonormal.
Denition 3.3 A set S of ve tors from Rm is alled linear subspa e of the spa e
Rm if both the following onditions hold:
1. u + v 2 S , 8u; v 2 S .
2. u 2 S , 8 2 R, 8u 2 S .
Obviously, every linear subspa e ontains the origin.
In pra ti e, a subspa e is often given as the olumn spa e of a matrix, i.e. as:
S = ImA = fy 2 Rm j 9x 2 Rn su h that y = Axg;
where A 2 Rmn .
The notion of orthogonality may be extended to linear subspa es.
Denition 3.4 The ve tor x 2 Rm is orthogonal to the linear subspa e S 2 Rm if
it is orthogonal to every ve tor in S . Two subspa es, S ; T 2 Rm , are orthogonal if
ea h ve tor s 2 S is orthogonal to ea h ve tor t 2 T .
Denition 3.5 A linear subspa e S 2 Rm is alled orthogonal omplement to the
linear subspa e T 2 Rm if the two subspa es are orthogonal and omplementary,
i.e. for any ve tor x 2 Rm , there exist unique ve tors s 2 S and t 2 T su h that
x = t + s.
Theorem 3.6 (Pythagoras) If x; y 2 Rm are orthogonal, then
kx + yk2 = kxk2 + kyk2 ;
where k k is the eu lidian norm.
Theorem 3.7 Let A 2 Rmn . The ve tor x 2 Rn is orthogonal to the linear
subspa e S = ImA if and only if
AT x = 0
(, x 2 KerAT )
In other words, the subspa e T = KerAT Rm is the orthogonal omplement of
S = ImA.
6y3
(1,0,1)

(0,1,1) k
QQQ ImA
QQ y1
-

y2 KerAT
(1,1,-1)
Figure 3.1: KerAT is the orthogonal omplement of ImA. Data from example 3.8.
2 3
1 07
Example 3.8 Let A = 4 0 1 5.
6
1 1 2 3
x1
Then, S = ImA = fy 2 R3 j y = 64 5 ; x1 ; x2 2 Rg, i.e. S is a plane of
7
x2
x1 + x2
equation y3 = y1 + y2.
T = KerAT2 = fy32 R3 j A0T y2 = 0g3=1 fy 2 R3 j y2 + y3 = 0; y1 + y3 = 0g =
1 1 7C
fy 2 R3 j y = 64 1 75 g = Im B
4 1 5A.
6
1 1
See gure 3.1.
Denition 3.9 A square matrix Q 2 Rmm is alled orthogonal if its olumns are
orthonormal, i.e.
qiT qj = 01;; ifif ii 6=

j (3.2)
= j,
where qi and qj are the i-th and, respe tively, the j -th olumn of Q.
Theorem 3.10 If the matrix Q 2 Rmm is orthogonal, then
QT Q = QQT = Im ; (3.3)
kQxk = kxk; 8x 2 Rm : (3.4)
These relations are a dire t onsequen e of (3.2) and of the denition of the
eu lidian norm.
The property (3.3) expresses the important fa ts that an orthogonal matrix is
nonsingular, its inverse is its transpose and it has orthonormal rows and olumns.
The property (3.4) shows that an orthogonal transformation onserves the eu-
lidian ve tor norm (i.e. it denes an isometry).
Theorem 3.11 If Q1 ; Q2 ; : : : ; Qp 2 Rmm are orthogonal, then the matrix:

Q = Q1 Q2 : : : Qp (3.5)
is also orthogonal.
Proof. Obviously, relations (3.2) and (3.3) are equivalent. Then (Q1 Q2 )T Q1 Q2 =
QT2 QT1 Q1 Q2 = QT2 Q2 = Im . We pro eed by indu tion; let suppose that Q from (3.5)
is orthogonal; then, Q0 = QQp+1, where Qp+1 is orthogonal, is orthogonal, by a
similar argument. 2
Two kinds of orthogonal matri es are important in LLS problem: a) Householder
matri es or elementary re e tors, and b) Givens matri es or plane rotations. Let
us introdu e them and stress their main properties.
3.2.1 Householder transformations
Denition 3.12 An elementary re e tor of order m (or Householder transforma-

tion) is a matrix U 2 Rmm , of the form:
U = Im
1 uuT (3.6)

where u 2 Rm is a nonzero ve tor and
= kuk2 =2:
When ui = 0 for i = 1 : s 1, then the elementary re e tor is alled of index s.
Obviously, an elementary re e tor Us of order m and index s has the following
stru ture: " #
I
Us = s0 1 U ; 0
where U 2 R(m s+1)(m s+1) is an elementary re e tor of order m s + 1.

Some simple onsequen es of denition 3.12 are ontained in the following
Theorem 3.13 If U 2 Rmm is an elementary re e tor, then U is symmetri and

orthogonal.
From (3.6), U T = U , i.e. U is symmetri . Moreover:

Proof.
T 2

U U = U = Im
1 uu T

Im
1
uu = Im
T 2 uuT + 1 u(uT u)uT = Im ;
2
sin e uT u = kuk2 = 2 . So that, an elementary re e tor is its own inverse. 2
The ru ial property of elementary re e tors is that they an be used to introdu e
zeros into a ve tor. Spe i ally, given a ve tor x 2 Rm , one an nd an elementary
re e tor Us of index s su h that (Us x)i = 0, for i = s + 1 : m. The re ipe for
determining the ve tor us whi h dene su h a re e tor is given in the following
theorem:
Theorem 3.14 Let s 2 1 : m 1 and a ve tor x 2 Rm su h that
m
X
2 = x2i 6= 0: (3.7)
i=s
Then the ve tor us 2 Rm given by:

0;8
< for i = 1 : s 1
uis (us )i = : xs + ; for i = s (3.8)
xi ; for i = s + 1 : n
denes an elementary re e tor of order m and index s
Us = Im
1 u uT ; s = kus k2 =2; (3.9)
s s
s
su h that
for i = 1 : s 1
8
< xi ;
(Usx)i = : ; for i = s
0; for i = s + 1 : n
Proof. Let us ompute
yi (Us x)i =

Im
1 usuT x = xi 1 usuT x = xi 1 uis(uT x):
s s i s s i s s
From (3.7) and (3.8):

m
X m
X
uTs x = uis xi = xs (xs + ) + x2i = 2 + xs (3.10)
i=1 i=s+1
and 0 1
s =
1 2 1
m
X m
X
2 i=s uis = 2 (xs + ) + i=s+1 xi =
2 2A
0 1
= 21 x2s + 2xs + 2 +
Xm
x2i A = 2 + xs :
i=s+1
Thus, from (3.8) and (3.10) we obtain

8
< xi ; for i = 1 : s 1
yi = xi uis = : ; for i = s
0; for i = s + 1 : n
and the proof is ompleted. 2
The theorem 3.14 almost gives an algorithm for omputing us and s whi h
dene the re e tor (3.9). The onstant is determined up to its sign by the fa t
that Us is orthogonal (see (3.7)). In order to hoose the sign of , observe that the
formulas for omputing uss (see (3.8)) and s:
s = ( + xs)
give a better pre ision if and xs have the same sign, avoiding an ellation of
signi ant digits in oating point arithmeti .
As we shall see, in appli ations it is often the ase that having formed the ve tor
us , we need no longer the zeroed omponents of x. But the last n s omponents of
us are identi al with the last n s omponents of x; hen e, only the s-th omponent
of x needs to be altered in overwriting x(s : m) by the nonzero elements of us.
All these onsiderations may be summed up in the following algorithm.
Algorithm 3.15 (Given s 2 1 : m 1 and a m-ve tor x, this al-

gorithm returns , s and us su h that [(I (2=s )us uTs )x℄i = 0, for
i = s + 1 : m. The nonzero omponents of us overwrite those of x.)
qP
1. sign(xs ) m x2
i=s i
2. xs uss = xs +
(2'. xi uis = xi , for i = s + 1 : m)
3. s uss
3.2.2 Givens transformations
Denition 3.16 A matrix Pij 2 Rmm of the form:

2 3
Ii 1
6
6 d 7
7
Pij = 6
6
6
Ij i 1
7
7
7
4 d 5
In j
where ; d 2 R and
2 + d 2 = 1
is alled a Givens transformation or plane rotation of order m in the (i; j ) plane.
The main properties of the Givens transformations are given by the following
Theorem 3.17 1. The Givens transformations are orthogonal, i.e.
PijT Pij = Pij PijT = Im:
2. If x 2 Rm is a ve tor with x2i + x2j 6= 0, then there exists a Givens transfor-
mation dened by:
x x
= q 2 i 2; d= q 2j 2
xi + xj xi + xj
su h that 8
<qxk ; for k 6= i; j
yk (Pij x)k = : x2i + x2j ; for k = i (3.11)
0; for k = j .
Proof. 1. Obvious, by dire t al ulation.
2. To prove (3.11), it is suÆ ient to solve the system
(Pij x)i = xi + dxj =
(Pij x)j = dxi + xj = 0
2 + d 2 = 1
for the unknowns , d and . 2
To see that Givens transformations oer an alternative to elementary re e tors,
let us observe that a sequen e of plane rotations of the form
Ps = Psm : : : Ps;s+2 Ps;s+1
may be used for zeroing the s +1 : m elements of a ve tor x 2 Rm , as an elementary
re e tor does in theorem 3.14.
3.3. ORTHOGONAL TRIANGULARIZATION. QR FACTORIZATION 67
3.3 Orthogonal triangularization. QR fa torization

Solving the LLS problem in a stable numeri al way requires a pro edure for matrix
redu tion to triangular form by orthogonal transformations. As the next theorem
states, this is always possible. We shall use elementary re e tors for su h a redu tion.
Theorem 3.18 Let A 2 Rmn ; then, there exists an orthogonal matrix U 2 Rmm
su h that
UA = R (3.12)
is upper triangular.
Proof. Note rst that when m < n we shall use the term "upper triangular" for
an "upper trapezoidal" form. To pre ise the index information, suppose, rst, that
m > n. The proof will be onstru tive, leading to an orthogonal triangularization
pro edure. Let onsider a olumn partition of the matrix A:
A = [a1 a2 : : : an ℄
and let A1 = A. We pro eed by indu tion.
Step 1. If ai1 , i = 2 : m, are not all zero, there exists an elementary re e tor of
order m and index 1, U1 2 Rmm (see theorem 3.14), su h that
(U1 a1 )i = 0; i = 2 : m:
Then the matrix 2 3
6
a(2) (2)
11 a12 : : : a(2)
1n 7
6
6 0 a(2)
22 : : : a(2)
2n
7
7
A2 = U1 A1 = [U1 a1 U1 a2 : : : U1 an ℄ = 6
6
6
(2)
0 a32 : : : a(2)
3n
7
7:
7
6
4 ::: 7
5
(2)
0 am2 : : : a(2)
mn
has all the rst olumn underdiagonal elements equal to zero.
If ai1 = 0, for i = 2 : m, then we take simply U1 = Im and pro eed to the next
step. (The unity matrix Im is orthogonal !)
Step s. Let suppose that in the rst s 1 steps of the pro edure we have obtained
the matrix 2 3
a(11s) a(12s) : : : a(1ss) : : : a(1sn)
a(22s) : : : a(2ss) : : : a(2sn)
6 7
6 7
6
6
6 ... :::
7
7
7
6 7
As = Us 1 : : : U2 U1 A = 66 ( s)
ass : : : (s) 7 :
asn 77
6
6
6 0 a(ss+1
) :::
;s a(ss+1
) 7
;n 7
6 7
4 ::: 5
(s)
ams : : : a(mn
s)
Now, if a(iss) , i = s + 1 : m, are not all zero, there exists an elementary re e tor of
order m and index s, Us 2 Rmm , su h that (see theorem 3.14):
(Usa(ss) )i = 0; i = s + 1 : m:
(Otherwise we take Us = Im and go to the next step.) Taking into a ount that
any re e tor of index s does not hange a ve tor whose last m s elements are zero
(Usx = x (1=s )us(uTs x) = x, be ause uTs x = 0), it results that the matrix:
As+1 = Us As = [Us a(1s) Usa(2s) : : : Us a(ns) ℄
has the rst s 1 olumns un hanged and in the s-th olumn the underdiagonal
elments are zero.
Thus, the pro edure of zeroing subdiagonal elements an be started, as in step
1, and, on e started, it an be ontinued, as in step s. So, when m > n, after n
steps, we obtain an upper triangular matrix:
" #
R = An+1 = Un Un 1 : : : U2 U1 A = R0 ;
0
where R0 2 Rnn is a square upper triangular matrix. The matrix:
U = UnUn 1 : : : U2 U1 (3.13)
is orthogonal, sin e it is a produ t of orthogonal matri es, and, hen e, we have
obtained the triangularization (3.12).
When m n, the same result is obtained after m 1 steps. The proof is
omplete. 2
In order to implement the above pro edure su h that the matrix R overwrites
the matrix A we an use the following s heme:
1. for s = 1 : min(m 1; n)
1. Determine Us su h that (Usas)i = 0, for i = s + 1 : m
2. A UsA
In order to ompute Us (i.e. the ve tor us and the s alar s), we shall use the
algorithm 3.15. Observe that an e onomi al way to store us is to use the zeroed
elements memory lo ations for the storage of uis, i = s + 1 : m. For uss and s one
an use two additional rows (m + 1 and m + 2) of the array A, like in the following
3.3. ORTHOGONAL TRIANGULARIZATION. QR FACTORIZATION 69
diagram for the situation after the s-th step:
2 3
r11 r12 : : : r1s r1;s+1 : : : r1n
6
6 u21 r22 : : : r2s r2;s+1 : : : r2n 7
7
6
6
6
... ... ... ... ... 7
7
7
6 7
6
6
us1 us2 : : : rss rs;s+1 : : : rsn 77
6
6 us+1;1 us+1;2 : : : us+1;s a(ss+1
+1) : : :
;s+1 a(ss+1
+1) 7
;n 7
6
6
6
... ... ... ... ... 7
7
7
6
s+1) 7
6
6 um1 um2 : : : ums a(m;s
s+1)
+1 : : :
(
amn 77
6 7
4 u11 u22 : : : uss 5
1 2 : : : s
To ompute A Us A, we must ompute only aj Us aj , for j = s +1 : m. But
Us aj = (Im (1=s )usuTs )aj = aj us
with m
X
!
= uTs aj =s = uks akj =s
k=s
and the rst s 1 elements of the olumn aj remain un hanged be ause uis = 0, for
i = 1 : s 1.
En apsulating the above ideas we obtain the following algorithm. In the algo-
rithm, the statement "pass to next step" is used to indi ate that for the urrent
value of the for loop ounter, the instru tions of the(s) y le body that follow this
statement are not exe uted. This is the ase when ais = 0, for i = s : m (in su h
ases, Us = Im).
Algorithm 3.19 (ORTHT { orthogonal triangularization) (Given
a matrix A 2 Rmn , this algorithm overwrites the matrix A by the
upper triangular matrix R and the ve tors us and s alars s whi h dene
the elementary re e tors Us, s 2 1 : p, p = min(m 1; n), su h that
R = Un Un 1 : : : U2 U1 A.)
1. p = min(m 1; n)
2. for s = 1 : p q
1. sign(ass) Pmi=s a2is
2. if = 0 then am+1;s 0, am+2;s 0, pass to next step
3. am+1;s uss = ass +
4. am+2;s uss
5. ass rss =
6. for j = s + 1 : n
1. (Pmk=s uksakj ) =s
2. for i = s : m
1. aij aij uis
The omplexity of the above algorithm is given by NORT HT = 4(mnp (m +

n) 2 + p3 ) ops and MORT HT mn oating point lo ations. If m n, the most fre-
p2 3
quently o urring ase, the algorithm requires about 2(mn2 n3=3) ops. Moreover,
it is shown that the algorithm ORT HT is numeri ally stable.
Note that the algorithm does not ompute expli itly the orthogonal matrix U of
(3.13), but only stores all the ne essary elements to do it.
An immediate appli ation of algorithm ORT HT is the omputation of the so
alled QR fa torization of the matrix A. The denition and the existen e of the QR
fa torization are given by the following
Theorem 3.20 If A 2 Rmn , has linearly independent olumns, then it an be
written uniquely in the form
A = QR; (3.14)
where Q 2 Rmn has orthonormal olumns and R 2 Rnn is upper triangular with
positive diagonal elements. The matrix fa torization (3.14) is alled QR fa toriza-
tion.
Proof. For all x 2 Rn , x 6= 0, we have Ax 6= 0. Hen e, xT AT Ax = kAxk2 > 0

for all x 6= 0, i.e. the matrix B = AT A is positive denite. Then, by Cholesky
fa torization, there exists a unique upper triangular matrix with positive diagonal
elements su h that B = RT R. Let Q = AR 1. Then QT Q = R T AT AR 1 =
R T RT RR 1 = In , i.e. Q has orthogonal olumns and (3.14) holds. The uniqueness
of the QR fa torization results from the uniqueness of the Cholesky fa torization.
2
The use of the Cholesky fa torization for omputing the matri es Q and R from
(3.14) is not re ommended be ause of the ill numeri al ondition of matrix B = AT A.
A mu h better method is based on the orthogonal triangularization (3.12). The
matrix A having linearly independent olumns and m n, (3.12) may be written:
" #
0
A= = Q0 Q00 R 0 =QR;
0 0 (3.15)

UT R
where Q0 2 Rmn and R0 2 Rnn. This relation is the QR fa torization (3.14) up

to the signs of the diagonal elements of R.
3.4. SOLVING THE LINEAR LEAST SQUARES PROBLEM 71
The matrix Q0 an be omputed from:
" # " # " # " #
Q0 = Q In
0 =
In
0 = (Up : : : U2 U1)
UT T In
0 = U1U2 : : : Up 0
In
using the following omputational s heme:

" #
1. Q0 = In
0
2. for s = p : 1 : 1
1. Q0 UsQ0
However, it should be stressed that in most appli ations, the matrix Q U T or
the matrix Q0 from (3.15) is not needed; it is rather suÆ ient to know the fa tored
form (3.13) of U . So that, writing an expli it algorithm to implement the above
s heme remain an exer ise to the reader.
3.4 Solving the linear least squares problem

Let return to the LLS problem of minimizing the eu lidian norm of the overdeter-
mined linear system residual:
min krk = xmin
x2R 2R
n
kb Axk; n
(3.16)
where A 2 Rmn , m > n, and b 2 Rm are given.
First, we are interested to establish the onditions under whi h the LLS problem
has a solution and when this solution is unique. We shall develop furtherly an
algorithm to ompute this (pseudo)solution of an overdetermined linear system.
Theorem 3.21 Let A 2 Rmn and b 2 Rm be given. The linear least squares
problem (3.16) always has a solution. The solution is unique if and only if the
matrix has linearly independent olumns.
Proof. Let S = ImA and T = S ? its orthogonal omplement in Rm . Then, the
ve tor b an be uniquely written as:
b = b1 + b2 ; b1 2 S ; b2 2 T : (3.17)
Obviously, the norm being a positive fun tion of its argument, (x) kb Axk is
minimum i (x) = 2(x) is minimum. Also, b1 2 S , Ax 2 S for all x imply that
b1 Ax 2 S ; hen e, b2 and b1 Ax are orthogonal (see gure 3.2). Then, applying
theorem 3.6,
2 (x) = kb Axk2 = kb1 Ax + b2 k2 = kb1 Axk2 + kb2 k2 kb2 k2 :
CO C
6 b C
b2 CC b Ax
C
C

b1 1 SoS bC1C Ax
XXX S C
X zSC
Ax XXXX
S = ImA
T = S?
Figure 3.2: Illustration for theorem 3.21.
Thus, 2(x) will be minimum when kb1 Axk2 is minimum. But b1 2 S = ImA, so
that we an always nd a x 2 Rn satisfying
Ax = b1 : (3.18)
For this x , kb1 Ax k2 = 0, whi h is ertainly a minimum, and
min kb Axk = kb Axk = kb2 k:
x2Rn
(3.19)
As we an see in the gure, (3.19) expresses the well known fa t that the min-
imum distan e from a point to a subspa e is given by the eu lidian length of the
perpendi ular from the point to the subspa e.
Now, from (3.18), the solution x is unique i A(x x0) = 0 imply x = x0 , i.e.
KerA = f0g, or, equivalently, A has linearly independent olumns. 2
Corollary 3.22 If A 2 Rmn has linearly independent olumns, then the LS solu-
tion of the overdetermined system Ax = b an be written in the form:
x = A+ b; (3.20)
where the matrix
A+ = (AT A) 1 AT (3.21)
is alled the pseudo-inverse or the Moore-Penrose generalized inverse of A.
As the matrix A has linearly independent olumns, the matrix B = AT A
Proof.
is nonsingular. Hen e, AT r = AT (b Ax ) = AT b2 , where b2 2 T ?S = ImA is
3.4. SOLVING THE LINEAR LEAST SQUARES PROBLEM 73
dened in (3.17). But b2 ?ImA implies b2Av = 0 for all v 2 Rn, parti ularly for
v = e1 ; e2 ; : : : ; en . Hen e, bT2 A = 0, or AT b2 = 0 (or b2 2 ImA), and AT (b Ax ) = 0,
i.e. the LS solution of the overdetermined linear system Ax = b is the lassi al
solution of the determined nonsingular system
AT Ax = AT b (3.22)
whi h gives (3.20) and (3.21). 2
The linear system (3.22) is alled the system of normal equations for the LLS
problem.
The omputation of the LLS solution by solving the system of normal equations
(3.22) is not re ommended, be ause of the ill numeri al ondition of the matrix
B = AT A.
A numeri ally stable way to ompute the LLS solution is by the mean of the
orthogonal triangularization algorithm ORT HT . Indeed, the orthogonal transfor-
mations onserving the eu lidian norm, for all orthogonal matri es U 2 Rmm we
have:
kb Axk = kU (b Ax)k = kd Rxk; (3.23)
where
d = Ub; R = UA: (3.24)
If the matrix U is that from theorem 3.21, then R is an upper triangular matrix, i.e.
R an be written in the form "
R 0 #
R= 0 ;
with R0 a square n n upper triangular matrix. R has linearly independent olumns

be ause A has the same property. Hen e, R0 is a nonsingular matrix. Now, using
the partition " #
d0
d = d00 ; (3.25)
where d0 2 Rn and d00 2 Rm n , relation (3.23) an be written:
" #
kb Axk =

d0 R0 x

q
= kd0 R0 xk2 + kd00 k2 ;

d00

whi h is minimum when d0 R0x = 0. Hen e, the LLS solution of the overdetermined
system Ax = b an be omputed as the solution of the upper triangular nonsingular
system:
R 0 x = d0 :
(Note that the same result is immediately obtained by substituting the QR

fa torization of A, A = Q0R0, in the normal system (3.22): (R0)T (Q0 )T Q0R0x =
(R0)T (Q0)T b, whi h implies R0x = (Q0)T b d0.)
From (3.24), (3.25) and (3.13) it results that:
d0 = [In 0℄d = [In 0℄Ub = [In 0℄Un : : : U2 U1 b:
Thus, the ve tor d0 an be omputed using the s heme:
1. d b
2. for s = 1 : n
1. d Usd
3. d d(1 : n)
0
The minimal residual ve tor r = b Ax = b2 has the signi an e of the

orthogonal proje tion of the ve tor b onto the linear subspa e T = S ? = (ImA)T =
KerAT , while the ve tor b1 = Ax has the signi an e of the proje tion of b onto
the linear subspa e S = ImA.
The two orthogonal proje tions of the ve tor b onto ImA and KerAT are best
omputed with the expressions:
" #
b1 = U1 U2 : : : Un d0
0
" #
b2 = U1 U2 : : : Un d00 0
whi h result from
" # " #!
b= = d0 0
UT d UT 0 + d00 = b1 + b2 :
Summarizing the above ideas, we obtain the following:
Algorithm 3.23 (LLS { linear least squares) (Given a matrix A 2
Rmn , m > n, with linearly independent olumns, and a ve tor b 2 Rn ,
this algorithm omputes the LLS solution of the overdetermined linear
system Ax = b. Also, the algorithm omputes the orthogonal proje tions
b1 and b2 onto the linear subspa es ImA and KerAT , respe tively.)
f The orthogonal triangularization of A g
1. [Us; s = 1 : n; R℄ = ORT HT (A)
f Cal ulation of d, overwriting the ve tor b g
2. for s = 1 : n
3.5. SOLVING UNDERDETERMINED LINEAR SYSTEMS 75
1. (Pmk=s uksbk ) =s
2. for i = s : m
1. bi bi uis
f Computation of the LLS solution g
3. x = UT RIS (R; b)
f Computation of the proje tions g
4. b1(1 : n) b(1 : n); b1(n + 1 : m) 0
5. b2(1 : n) 0; b2(n + 1 : m) b(n + 1 : m)
6. for s = n P : m1 : 1
1. ( k=s uksb1k ) =s
2. for i = s : m
1. b1Pi m b1i uis
3. ( k=s uksb2k ) =s
4. for i = s : m
1. b2i b2i uis
The main omputational eort is onsumed for orthogonal triangularization.
Hen e, the asymptoti number of oating point operations is the same as in ORT HT .
Algorithm 3.23 is a very stable way of solving the LLS problem and, if ne essary,
of omputing the orthogonal proje tions of a given ve tor onto two omplementary
linear subspa es dened by a given matrix.
3.5 Solving underdetermined linear systems
The linear system Ax = b, where A 2 Rmn and b 2 Rn, with more unknowns
than equations (n > m), have, generally, an innite number of solutions. A natural
riterion, in the least square sense, for the sele tion of a solution, is its eu lidian
norm.
The solution of minimum eu lidian norm is alled normal solution of Ax = b.
The onditions for the existen e and uniqueness of the normal solution are given by:
Theorem 3.24 If the matrix A 2 Rmn , m < n, has linearly independent rows,
then the system Ax = b has a unique normal solution, i.e. a unique ve tor x 2 Rn
su h that:
kx k = min kxk:
Ax=b
The normal solution an be written in the form:
x = A# b; (3.26)
where
A# = AT (AAT ) 1 (3.27)
is alled the normal pseudo-inverse of A.
Proof. We shall give a onstru tive proof, whi h will lead us to an algorithm
for the omputing of the normal solution. If A has linearly independent rows, then
its transpose AT has linearly independent olumns. Theorem 3.18 says that there
exists an orthogonal matrix U su h that:
" #
UAT = R ;
0
where R is a square m m nonsingular upper triangular matrix; then:
A = [RT 0℄U = [L 0℄U; (3.28)
where L = RT is a nonsingular lower triangular m m matrix. Thus, the system
Ax = b be omes:
[L 0℄Ux = b: (3.29)
Denoting Ux = y and partitioning:
" #
y= y0 ;
y00
where y0 2 Rm and y00 2 Rn m , (3.29) an be written in the form Ly0 = b. Hen e,
the set of all solutions of the system Ax = b is:
" #
X = fx 2 Rn jx= UT L 1 b ; y00 2 Rn m g:
y00
Sin e orthogonal transformations onserve the eu lidian norm, for all x 2 X :
" # " #
kxk =

T L 1b

=

L 1b

q
= kL 1 bk2 + ky00 k2 :
U
y00

y00

Hen e:
min kxk = min
Ax=b x2X
kxk = kL 1 bk
and the unique x 2 X having this property is:
" #
x = U T L 1b ; (3.30)
0
the normal solution of the system Ax = b.
In order to show that the solution (3.30) an be written in the form (3.26), (3.27),
let observe that, using (3.28), we have:
" #
AT = UT LT
0
3.5. SOLVING UNDERDETERMINED LINEAR SYSTEMS 77
and " #! 1
(AAT ) 1= [L 0℄UU T LT = (LLT ) 1 = L TL 1:
0
Hen e:
" # " #
A# b = AT (AAT ) 1 b = U T LT L TL 1b = U T L 1b = x ;
0 0
whi h ompletes the proof. 2
The omputation of the normal solution follows the proof above:
Algorithm 3.25 (USS { underdetermined system solving) (Given a
matrix A 2 Rmn , m < n, with linearly independent rows, and a ve tor
b 2 Rn , this algorithm omputes the normal solution x 2 Rn of the
underdetermined linear system Ax = b, i.e. the solution of minimum
eu lidian norm.)
f The orthogonal triangularization of AT g
1. [Us; s = 1 : m; R℄ = ORT HT (AT )
f Solve the lower triangular system RT y0 = b, with y0 overwriting b g
2. b = UT RIS (R; b)
f Computation of the normal solution from (3.30) g
3. x(1 : m) b; x(m + 1 : n) 0
4. for s = mP: n 1 : 1
1. ( k=s uksxk ) =s
2. for i = s : n
1. xi xi uis
The statement 4 implements (3.30) written in the form:
" #
x = U1 U2 : : : Um y0
0
by the following s heme:
" #
1. x y0
0
2. for s = m : 1 : 1
1. x Usx
Most of the oating point operations are onsumed in the rst statement, for
the orthogonal triangularization. The above algorithm is the best one, from the
numeri al point of view, for omputing the normal solution of an underdetermined
system with full rank matrix.
Problems
P 3.1 Let H 2 (m > n) be an upper Hessenberg matrix (h = 0 for i > j + 1).

R
m n
Give algorithms for:

ij
a. omputing the QR fa torization of H .

b.solving the overdetermined linear system Hx = b, where b 2 is given. R
m
Use Householder and Givens transformations and ompare the two versions.
P 3.2 The same problem, when H 2 is lower Hessenberg (h = 0, for j > i + 1).
R
m n
ij
P 3.3 a.
Adapt the orthogonal triangularization algorithm ORT HT to the matrix A =
A , where A 2 is upper triangular and A 2
1 n n . (m n) n
A 2
1 R 2 R
b.The same when A is also upper triangular.

2
P 3.4 Show that in (3.15), the olumns of Q0 2 form an orthogonal basis for ImA
R
m n
and the olumns of Q00 2 form an orthogonal basis for KerA .

R
m (m n) T
P 3.5 Show that a triangular orthogonal matrix is diagonal.

P 3.6 Let onsider the ve tor x 2 , of unit norm. Find an orthogonal Y 2
R
n
R
n (n 1)
su h that x Y = 0 (or [x Y ℄ orthogonal).

T
Let A 2 , m > n, a rank n matrix whose QR fa torization is known (A = QR,

m n
Q 2 , R 2 , see (3.15)). Let A~ = [A z ℄, where z 2

P 3.7 R
m m m n
. Give an algorithm to m
ompute the QR fa torization of A~.

R R R
P 3.8 Let A 2 , m n, a rank n matrix whose QR fa torization is known (A = QR,

m
R
n
Q 2 , R 2 , see (3.15)). Let A~ = wA , where w 2 is a ve tor.

T
m m m n n
R R R
a.Show that the olumns of A~ are linearly independent.

b.Give an algorithm to ompute the QR fa torization of A~.
P 3.9 Let A 2 be a " olumn" matrix, A 6= 0. Give an algorithm to ompute the
R
n 1
pseudo-inverse of A.
P 3.10 Let u; v 2 be two ve tors of equal norm, kuk = kvk. Give algorithms for:
R
n
a. omputing an orthogonal matrix Q 2 su h that Qu = v. n

R
n
b. omputing an orthogonal matrix W 2 su h that W u?v. R

n n
P 3.11 Let a; b 2 be two non olinear ve tors.

R
n
a.Give an algorithm for omputing the s alar x 2 su h that kax bk is minimum. R 2

Give a geometri interpretation of x.
b.Give an algorithm for omputing the proje tion of a onto the dire tion of b.
Chapter 4
Iterative te hniques in matrix

algebra
The iterative methods are based on a step by step onstru tion of a sequen e on-
vergent to the solution of a given numeri al problem. A ertain riterion is used
for ending the iterations and the result is a trun ated sequen e with the last term
onsidered as a satisfa tory approximate of the solution.
In this hapter we shall be ex lusively on erned of iterative methods for deter-
mined linear system solving. Sin e the time required for suÆ ient a ura y in the
iterative te hniques ex eeds that required for dire t te hniques su h as the Gaussian
elimination method, the rst ones are seldom used for solving linear systems of small
dimension. However, for large systems with a high per entage of zero entries these
te hniques prove to be eÆ ient in terms of omputer storage and time requirements.
Systems of this type arise frequently in the numeri al solution of the boundary-value
problems and partial-dierential equations.
In order to appre iate the properties of su h iterative te hniques we need rst
some results on ve tor and matrix sequen e onvergen e.
4.1 Ve tor and matrix sequen es. Convergen e

A ve tor or matrix norm (see se tion 2.11.1) an be interpreted as a measure for
ve tor or matrix "size", or a measure for the distan e between the ve tor (the matrix)
and the origin of the orresponding spa e. Consequently, a ve tor norm on Rn allows
us to introdu e the on ept of distan e on Rn.
Denition 4.1 Let k k : Rn ! R+ be a ve tor norm on Rn . Then the fun tion
d : Rn Rn ! R+ (4.1)
79
80 CHAPTER 4. ITERATIVE TECHNIQUES IN MATRIX ALGEBRA
dened by
d(x; y) = kx yk (4.2)
is alled a distan e on Rn indu ed by the norm k k.
Example 4.2 The eu lidean distan e between two ve tors x; y 2 Rn is
v
u n
uX
d2 (x; y) = kx yk2 = t (xi yi )2 : (4.3)
i=1
The distan e indu ed by the 1-norm is

n
X
d1 (x; y) = kx yk1 = jxi yij (4.4)
i=1
and the distan e indu ed by the 1-norm is dened by

d1 (x; y) = kx yk1 = max jxi yi j: (4.5)
i=1:n
Denition 4.3 The distan e between two matri es A; B 2 Rmn is

d(A; B ) = kA B k; (4.6)
where k k is an appropriate matrix norm.
The on ept of distan e on Rn an be used to dene a limit of a sequen e of
ve tors in Rn . Similarly, the distan e on Rmn allows us to introdu e the limit of
a matrix sequen e.
Denition 4.4 A sequen e (x(k) )k2N of ve tors in Rn is alled onvergent to x 2
Rn , with respe t to the norm k k, if and only if the sequen e of real numbers
(k )k2N dened by
k = kx(k) xk
is onvergent to zero. Similarly a matrix sequen e (A(k) )k2N with A(k) 2 Rmn is
alled onvergent to A 2 Rmn , with respe t to the matrix norm k k, if and only if
the sequen e of real numbers (k )k2N dened by
k = kA(k) Ak
onverges to zero.
In studying iterative matrix te hniques, it is of parti ular interest to know when
powers of a matrix be ome small, i.e., when all the entries of Ak approa h zero.
Matri es of this type are alled onvergent.
4.2. ITERATIVE TECHNIQUES FOR LINEAR SYSTEM SOLVING 81
Denition 4.5 A square matrix A 2 Rnn is alled onvergent if
lim (Ak )ij = 0; i = 1 : n; j = 1 : n
k!1
(4.7)
or, equivalently
lim Ak = 0:
k!1
(4.8)
In order to derive the onditions under whi h a square matrix is onvergent, we
must introdu e the on epts of eigenvalue and of spe tral radius of a (square) matrix.
These on epts are very important in a more general framework and we shall pay a
spe ial attention to them in a future hapter. For the moment we need only a few
basi fa ts about them.
Denition 4.6 If A 2 Rnn (or A 2 Cnn) the polynomial of degree n
p() = det(I A) (4.9)
is alled the hara teristi polynomial of A. The n zeros (over C) of the hara teristi
polynomial are alled the eigenvalues of the matrix A.
Denition 4.7 Let A 2 Rnn (or A 2 Cnn). If = f1 ; 2 ; ; n g is the set
of its eigenvalues then the spe tral radius (A) of the matrix A is dened by the
nonnegative number
(A) = imax (j j):
=1:n i
(4.10)
An important onne tion between the spe tral radius of a matrix and the on-
vergen e of the matrix is stated subsequently.
Theorem 4.8 A square real matrix A 2 Rnn is onvergent (i.e. limk!1Ak = 0)
if and only if the spe tral radius of the matrix A satises
(A) < 1: (4.11)
The proof an be found in [?℄. 2
4.2 Iterative te hniques for linear system solving

An iterative te hnique to solve the determined linear system
Ax = b; A 2 Rnn ; b 2 Rn (4.12)
with A nonsingular onsists in nding an initial approximation x(0) of the solution
x and onstru ting a sequen e of ve tors (x(k) )k2N that onverges to x. Most of
these iterative te hniques involve a onversion pro ess of the system (4.12) into an
equivalent system of the form
x = Tx + (4.13)
for a ertain matrix T 2 Rnn and ve tor 2 Rn. After the initial ve tor x(0) is
sele ted, the sequen e of approximate solutions is onstru ted re urrently using the
formulae
x(k+1) = T x(k) + ; k = 1; 2; : : : (4.14)
The matrix T in (4.13) is hosen su h that the omputational eort to generate
x(k+1) in (4.14) is reasonable small and the onvergen e speed is suÆ iently great.
The main result on the onvergen e of the sequen e dened in (4.14) with initial
ve tor x(0) is given below.
Theorem 4.9 For any x(0) 2 Rn , the sequen e (x(k) )k2N dened by (4.14) for
6= 0, onverges to the unique solution of the system (4.13) if and only if
(T ) < 1: (4.15)
Proof. The system (4.13) is equivalent to the system (4.12). Hen e the matrix I T
is nonsingular and
x = A 1 b = (I T ) 1 (4.16)
(The invertibility of the matrix I T an be derived from (4.15). Indeed, if the
spe tral radius (T ) satises (4.15) then the eigenvalues of T satisfy jij < 1; i = 1 :
n. Hen e j1 i j > 1 ji j > 0; i = 1 : n , i.e. 1 i 6= 0; i = 1 : n. But i = 1 i
are the eigenvalues of the matrix I T and onsequently I T is nonsingular). Now
it an be easily he ked that if m
X
Sm = T i (4.17)
i=0
then
(I T )Sm = I T m+1: (4.18)
From (4.15) we have that T is onvergent and onsequently limm!1 T m+1 = 0.
From (4.18) we obtain
1
X
(I T ) 1 = T i (4.19)
i=0
and from (4.14), we get by indu tion
!
kX1
x(k) = T k x(0) + Ti : (4.20)
i=0
Assuming (T ) < 1, and using Theorem 4.8, from (4.20) we get
!
kX1
lim
k!1
x(k) = lim
k!1
T k x(0) + klim
!1
Ti = (I T ) 1 = x: (4.21)
i=0
Conversely, suppose that the sequen e (x(k) )k2N onverges to x for any x(0) . From
(4.13) and (4.14)
e(k) := x x(k) = T x + T x(k 1) = T (x x(k 1) ) = T e(k 1) : (4.22)
By indu tion, from (4.22) we have
e(k) = T k e(0) ; e(0) = x x(0) : (4.23)
The onvergen e, for any x(0) , of the sequen e (x(k) )k2N is equivalent to the onver-
gen e, for any e(0) , of the sequen e (e(k) )k2N , dened by (4.23), to the ve tor e = 0.
Let e(0)
i be n initial ve tors su h that the matrix
E (0) = [e(0)
1 ; ; en ℄
(0) (4.24)
is nonsingular. Then
lim E (k) = klim
k!1 !1
T k E (0) = 0 (4.25)
from where
lim T k = 0
k!1
(4.26)
i.e. the matrix T is onvergent. By Theorem 4.8, (4.15) is true and the proof ends.
2
The ne essary and suÆ ient onvergen e ondition (4.15) if often diÆ ult to
he k. A useful riterion to prove the onvergen e to the system solution of a
parti ular sequen e is given below.
Proposition 4.10 If the matrix T 2 Rnn satises
kT k < 1; (4.27)
for a onsistent family of matrix norms k k, then the sequen e (x(k) )k2N dened by
(4.14) is onvergent to the solution x of (4.12), for any initial approximation x(0) .
Proof. Let i 2 (T ), where (T ) is the set of all eigenvalues of the matrix T . Then
det(i In T ) = 0 and there exists a nonzero ve tor y 2 Rn (or y 2 Cn) alled
eigenve tor, su h that
T y = i y: (4.28)
Hen e
ji jkyk = kT yk kT kkyk
i.e.
jij kT k < 1; 8i = 1 : n: (4.29)
The inequality (4.29) implies (4.15) and, by Theorem 4.9, the orresponding sequen e
is onvergent. 2
In order to obtain some parti ular iterative methods for linear system solving,
onsider the following de omposition of the matrix A given in (4.12)
A = L + D + U; (4.30)
where L is the stri tly lower triangular part of A, D is the diagonal part of A and
U is the stri tly upper-triangular part of A i.e.
2
0 0 0 0 3 2
a11 0 0 0 3
6 a
6 21 0 0 0 777 6 0 a
6 22 0 0 777
L = 66
6
77 ; D = 66 77 ;
6
4 an 1;1 an 1;2 0 05 4 0 0 an 1;n 1 0 5

an1 an2 2an;n 1 0 0 0 0 0 ann
0 a12 a1;n 1 a1n 3
6 0 0 a2;n 1 a2n 777
6
U = 66
6
7:
4 0 0 0 an 1;n 75
0 0 0 0 0
(4.31)
In the following development we assume that aii 6= 0 8i = 1 : n , i.e., the
matri es D; L + D; D + U are all nonsingular. Sin e the matrix A is nonsingular,
this assumption is always satised after appropriate an appropriate row ( olumn)
permutation (see Remark 4.15).
4.2.1 Ja obi iterative method
Let
TJ = D 1 (L + U ) (4.32)
and
= D 1 b: (4.33)
In this ase (4.13) be omes
x = D 1 (L + U )x + D 1 b (4.34)
or
(D + L + U )x = b
and sin e (4.34) is equivalent to (4.12), (4.14) be omes
x(k+1) = D 1 (L + U )x(k) + D 1 b; k = 1; 2; (4.35)
whi h denes the so alled Ja obi iterative method. In s alar form (4.35) an be
written 0
n
1
(k+1) X ( )
xi = bi aij xi A =aii ; i = 1 : n
k
(4.36)
j =1;j 6=i
whi h implements (4.35) in the form of the following linear system solving
Dx(k+1) = (L + U )x(k) + b: (4.37)
The above method should be ompleted with an adequate riterion for ending the
sequen e of iterations. A possible way is to iterate (4.36) until
kx(k+1) x(k) k (4.38)
kx(k+1) k
is smaller than some pres ribed toleran e " > 0. To this end, any onvenient norm
an be used but the most usual one is the k k1 norm. Due to the small speed of
onvergen e or even due to the possible sequen e divergen e, su h a riterion might
be not fullled in a reasonable great number of iterations.
Noti e that there is no need to store all the ve tors in the sequen e. An additional
ve tor updated at ea h step is suÆ ient.
Algorithm 4.11 (J ITER { Ja obi iterative method)
(Given:
- a nonsingular matrix A 2 Rnn with aii 6= 0 for i = 1 : n
- a nonzero ve tor b 2 Rn
- an initial approximation x 2 Rn
- a toleran e " > 0
- a maximum number of iterations N
the algorithm omputes an approximate solution x 2 Rn of the linear
system Ax = b or prints a message that the number of iterations was
ex eeded without attaining the given toleran e)
1. k 0
2. while k N
1. for i = 1 : n Pn
1. yi (bi j=1;j6=i aij xj )=aii
2. if ky xk=kyk < "
1. x y
2. STOP
3. x y
4. k k + 1
3. Print('The maximum number of iterations was ex eeded without
attaining the given toleran e')
Using now for onvergen e the suÆ ient ondition (4.27) we obtain the following
proposition.
Proposition 4.12 If the matrix A is stri tly diagonally dominant, i.e.,
n
X
jaii j > jaij j; i = 1 : n (4.39)
j =1;j 6=i
then the ve tor sequen e given by Ja obi's iterative method is onvergent to the
solution x.
Proof. From (4.39) we have
n
X jaij j < 1; 8i = 1 : n:
ja j
j =1;j 6=i ii
Hen e 0 1
kTJ k1 = k D 1 (L + U )k1 = imax

j aij j A < 1
n
X
=1:n j =1;j 6=i jaii j
and, sin e kk1 is a onsistent family of matrix norms, we on lude with Proposition
4.10 that the Ja obi sequen e is onvergent.
4.2.2 Gauss-Seidel iterative method
A possible improvement of the Ja obi iterative method is suggested by the following

remarks:
- for omputing x(k+1) the omponents of x(k) are used;
- the omputations are made in su essive order x(1k+1) ; : : : ; x(nk+1) .
Sin e at step i > 1, x(1k+1) ; x(2k+1) : : : ; x(ik+1)
1 have been already omputed and
are supposedly better approximations to the exa t solutions x1 ; x2 ; : : : ; xi 1 than
x(1k) ; x(2k) : : : ; x(ik)1 , it seems reasonable to ompute x(ik+1) using the last omputed
values. To be more spe i , instead of (4.36) it seems better to use
0 1
i 1
X n
X
x(k+1) = b
i i aij x(jk+1) aij x(jk) A =aii ; i = 1 : n: (4.40)
j =1 j =i+1
This orresponds to the following hoi e of the matrix T in (4.13) and (4.14)
TGS = (L + D) 1 U: (4.41)
This method depi ted in (4.40) and based on the lower-triangular linear system
solving
(L + D)x(k+1) = Ux(k) + b (4.42)
is alled the Gauss-Seidel iterative method.
Pro eeding as in Algorithm 4.11, the Gauss Seidel method an be implemented
as follows.
Algorithm 4.13 (GS ITER { Gauss-Seidel iterative method)
(Given:
- a nonsingular matrix A 2 Rnn with aii 6= 0 for i = 1 : n
- a nonzero ve tor b 2 Rn
- an initial approximation x 2 Rn
- a toleran e " > 0
- a maximum number of iterations N
the algorithm omputes an approximate solution x 2 Rn of the linear
system Ax = b or prints a message that the number of iterations was
ex eeded without attaining the given toleran e)
1. k 0
2. while k N
1. for i = 1 : n P
1. yi (bi ij=11 aij yj Pnj=i+1 aij xj )=aii
2. if ky xk=kyk < "
1. x y
2. STOP
3. x y
4. k k + 1
3. Print('The maximum number of iterations was ex eeded without
attaining the given toleran e')
Using for (4.41) the k k1 family of matrix norms, the following result, similar
to the one stated in Proposition 4.12, an be proved (exer ise to the reader).
Proposition 4.14 If the matrix A is stri tly diagonal-dominant then the ve tor
sequen e dened by the Gauss-Seidel method is onvergent to the solution x for any
initialization.
Remark 4.15 Noti e that Algorithms 4.11 and 4.13 work if aii 6= 0; i = 1 : n. If
this is not the ase, a reordering of the equations (row permutations of the ma-
trix A) an be performed su h that aii 6= 0; 8i = 1 : n. Sin e the system !is
nonsingular this is always possible. Indeed, let = (1) 1 2 n
(2) (n)
be a permutation
h of the set f 1 ; 2 ; :::;
i ng and the orresponding permutation ma-
trix P = e(1) e(2) e(n) 2 R . Let also Sn denote the set of all
n n
permutations (n!) of the set f1; 2; ; ng. It is well known that

det A = 2S ( 1)sign a(1);1 a(2);2 a(n);n :
n
Consequently, as det A 6= 0 there exists 2 Sn su h that

a(1);1 a(2);2 a(n);n 6= 0
(sin e if all the terms equal zero then det A = 0) and hen e
a(i);i 6= 0; i = 1 : n:
Clearly
(P A)(i; i) = a(i);i 6= 0
and we on lude that P A is a solution to our problem. 2
The Propositions 4.12 and 4.14 suggest that in order to speed up the onvergen e
the equations should be arranged su h that aii is as large as possible.
When we have introdu ed the Gauss-Seidel method we laimed that it seems to
be superior to the Ja obi method. In fa t there are linear systems for whi h the
Ja obi method onverges and the Gauss-Seidel method does not and onversely. In
general, it is not known whi h of the two te hniques, Ja obi or Gauss-Seidel will
work better on a parti ular ase.
4.2.3 Relaxation methods
Let us dene, for an approximation x of the system solution x, the residual ve tor
r = b Ax: (4.43)
In an iterative method, a residual ve tor is asso iated with ea h omputation of
an approximation of the solution ve tor. The obje tive of an iterative method is
to generate a sequen e of approximations for whi h the asso iated residual ve tor
onverges to zero.
In Gauss-Seidel method, the approximate solution is, at the step at whi h x(ik+1)
is omputed
x = [x(1k+1) ; x(2k+1) ; :::; x(ik+1) (k)
1 ; xi ; ; xn ℄
(k) T (4.44)
and, if we denote by
ri(k+1) = [r1(ki +1) ; r2(ki +1) ; :::; rni
(k+1) ℄T (4.45)
the residual ve tor at this step, then from (4.43) and (4.44)
i 1
X n
X
rii(k+1) = bi aij x(jk+1) aij x(jk) aii x(ik) : (4.46)
j =1 j =i+1
Consequently
i 1
X n
X
aii x(ik) + rii(k+1) = bi aij xj(k+1) aij x(jk) : (4.47)
j =1 j =i+1
Re all that in Gauss-Seidel method x(ik+1) is omputed by (4.40) and (4.47) an be

rewritten as
aii x(ik) + rii(k+1) = aii x(ik+1) (4.48)
or
r(k+1)
x(ik+1) = x(ik) + ii : (4.49)
aii
Moreover, with (4.40), after the i-th omponent of x(k+1) was omputed
i
X n
X
(k+1) = b
ri;i aij x(jk+1) aij x(jk) = 0: (4.50)
+1 i
j =1 j =i+1
The redu tion to zero of one of the entries of the residual ve tor
( k+1)
is not ne essarily
the most eÆ ient way to redu e the norm of the ve tor ri+1 . It an be shown
that by introdu ing a s alar parameter ! the modied Gauss-Seidel method given
in (4.49)
r(k+1)
x(ik+1) = x(ik) + ! ii (4.51)
aii
leads, for ertain hoi es of ! > 0, to a signi antly faster onvergen e.
The equation (4.51) an be obtained in matrix form as follows.
Consider the de omposition of the matrix A depi ted in (4.31) and modify the
Gauss-Seidel method by redening the matrix TGS in (4.41) as
1
T! = (L + D) 1 ((1
1 )D + U ): (4.52)
! !
In the modied method, the equation whi h denes an iteration is

(L + 1 D)x(k+1) = ((1 1 )D + U )x(k) + b;
! !
(4.53)
where ! is a s alar parameter used to speed up the sequen e onvergen e. It is easy
to show that (4.52) and (4.53) orrespond to the updating (4.51) of the elements of
x.
Methods based on (4.51) or, equivalently, on (4.52) and (4.53) are alled relax-
ation methods.
For 0 < ! < 1 the pro edures are alled under-relaxation methods and an
be used for obtaining the onvergen e of some systems whi h do not onverge in
Gauss-Seidel method.
For ! = 1 one re overs the Gauss-Seidel method.
For ! > 1 the pro edures are alled over-relaxation methods and are urrently
abbreviated SOR (Su essive Over-Relaxation). These last methods an be used for
speeding up the onvergen e of systems whi h onverge by using the Gauss-Seidel
algorithm. SOR are parti ularly useful for solving linear systems o urring in the
numeri al solution of ertain partial dierential equations. Generally, the optimal !
ranges in the interval (1; 2). For more details see [?℄.
4.3 Iterative renement of the omputed solution

As we have already seen, when solving linear nonsingular systems Ax = b, the
perturbations in the initial data (A ! A + A, b ! b + b) are amplied, in terms
of a spe ied kAkkA 1 k =: (A) times. Here k k stands for a ertain spe ied
family of matrix norms. The number (A) is alled the numeri al ondition of the
matrix A. It is worth mentioning that the initial error ampli ation is independent
of the algorithm used. Moreover, the omputations in an approximate arithmeti
leads to additional errors. If the algorithm employed in the linear system solving is
a numeri ally stable one, this additional errors are of the same size as those due to
the rounding errors in initial data when exa t al ulations are performed.
For ill onditioned systems it is parti ularly important to have a possibility for
improving the a ura y of a omputed system solution. Su h a possibility is shown
below.
Let x 2 Rn be an approximate solution of the system Ax = b and let
r = b Ax (4.54)
be the residual ve tor and x = A 1 b. Hen e, if
e=x x (4.55)
4.3. ITERATIVE REFINEMENT OF THE COMPUTED SOLUTION 91
then
Ae = Ax Ax = b (b r) = r: (4.56)
When solving the system Ae = r one expe ts xnew given by (4.55)
xnew = x + e (4.57)
to be more a urate than x. Here e stands for the omputed solution of the system
(4.56). Unfortunately, the naive oating point exe ution of the above omputations
renders an xnew that is no more a urate than x. This is due to the fa t that the
omputed residual ve tor (4.54) has few, if any, orre t signi ant digits (re all the
disastrous ee ts of oating point subtra tion of nearby numbers).
In [?℄ it is shown that a more a urate solution an be obtained if the omputation
of the residual ve tor is made in double pre ision (i.e. with a mantissa having a
double number of (binary) digits).
In order to improve the a ura y of a omputed solution x, for example by
Gaussian elimination with partial pivoting, the following sequen e of statements
an be iterated.
1. Compute in double pre ision r = b Ax.
2. Solve Ae = r (using the Gaussian multipliers and permutation from the initial
omputation of x).
3. x = x + e.
This iterative improvement of the solution a ura y is known as iterative rene-
ment. An error analysis of the iterative renement an be found in [?℄.
Problems
P 4.1 Show that the Ja obi iterative method onverges for 2 2 symmetri positive denite
systems.
P 4.2 Show that if A is stri tly diagonally dominant (i.e. ja j > P 6 ja j), then the
T n
Ja obi's iterative method is onvergent.

jj i=1;i=j ij

P 4.3
2 1
Consider the linear system Ax = b, where A = 1 2 . Whi h iterative method
will onverge faster: Ja obi or Gauss-Seidel ?
Chapter 5
Eigenvalues and eigenve tors

Many obje ts and pro esses of the real world are des ribed by mathemati al models
alled dynami al systems. The linear dynami al systems form an important lass
whi h is widely used in various s ienti and te hni al areas. A nite dimensional
linear system is often represented by a set of real matri es and its properties are
expressed in terms of these matri es. Some of the ru ial properties of linear dy-
nami al systems, su h as stability, depend on a set of numbers and ve tors related to
a system matrix alled eigenvalues and eigenve tors. This is one possible motivation
for studying the so alled algebrai eigenvalue problem.
Sin e the eigenvalues of a real matrix may be omplex numbers and the eigen-
ve tors may be omplex ve tors, a natural setting for the eigenvalue problem is the
spa e Cn. We begin with the presentation of the main features of this spa e.
5.1 The spa e Cn

For our purposes a omplex n-ve tor x is a olle tion of omplex numbers x1, x2, ...,
xn ordered in a olumn 2 3
x1
6 7
6 x2 7
x= 6
6 ... 7;
7 xi 2 C: (5.1)
4 5
xn
Clearly, any omplex n-dimensional ve tor an be written in the form
x = u + iv; (5.2)
where u; v 2 Rn and i is the imaginary unit. The set of all omplex n-dimensional
ve tors and the set of all omplex m n matri es is denoted by Cn and Cmn,
respe tively.
93
94 CHAPTER 5. EIGENVALUES AND EIGENVECTORS
The spa es Rn and Cn feature a striking similarity. Sin e the axioms of the eld
of real numbers are also satised by the eld of omplex numbers all the properties
of the ve tor spa e Rn are also fullled in Cn.
However, the denition of the inner produ t highlights a dieren e between the
two spa es. If x 2 Cn denote
xH = xT (5.3)
its onjugate transpose.
Denition 5.1 If x; y 2 Cn then the inner produ t of x and y is the omplex
number
< x; y >= yH x: (5.4)
Clearly, < y; x >= < x; y >. The above denition of the inner produ t allows us to
introdu e the on ept of orthogonality in Cn as follows.
Denition 5.2 Two ve tors x; y 2 Cn are orthogonal if
xH y = 0: (5.5)
Sin e Cn with the inner produ t (5.4) form a Hilbert spa e, the inner produ t in
Cn an be used, in the usual way, to dene the orresponding eu lidian ve tor norm
on Cn (indu ed by the inner produ t). Noti e that xH x is always a nonnegative
real number.
Theorem 5.3 The fun tion k:k : Cn ! R+ dened by
p
kxk := kxk2 = xH x (5.6)
is a ve tor norm on Cn.
The proof is an easy exer ise for the reader. More generally, all the denitions and
theorems on norms in Rn an be extended to Cn; however, when dealing with the
2-norm, the transpose must be repla ed by the onjugate transpose.
Denition 5.4 The matrix A 2 Cnn is Hermitian if
AH = A: (5.7)
Theorem 5.5 If A 2 Cnn is Hermitian then the number xH Ax is real for any
x 2 Cn .
5.2. EIGENVALUES AND EIGENVECTORS 95
Proof. We have (xH Ax)H = (xH Ax)T = xT AT (xH )T = xH AH x. Hen e xH Ax is a

s alar whi h equals its onjugate, i.e. is a real number. 2
Using Theorem 5.5 we an introdu e the on ept of positive denite omplex
matrix.
Denition 5.6 A Hermitian matrix A 2 Cnn is positive denite if
xH Ax > 0 8x 2 Cn ; x 6= 0: (5.8)
The analogous on ept to that of an orthogonal real matrix is in Cn the unitary
matrix introdu ed below.
Denition 5.7 A matrix A 2 Cnn is unitary if
AH A = I (5.9)
i.e., if it has orthonormal ( omplex) olumns.
In a similar way to the real ase, one an dene elementary unitary matri es
su h as omplex re e tors and omplex rotations. However, it is rather unusual in
pra ti e to work with omplex matri es sin e in most situations it is possible to
redu e the problem to the real ase.
5.2 Eigenvalues and eigenve tors

Denition 5.8 Let A 2 Cnn. A ve tor x 2 Cnn is an eigenve tor of A, orre-
sponding to the eigenvalue 2 C, if the following properties hold
a) x 6= 0 (5.10)
b) Ax = x:
Clearly, if x is an eigenve tor of A orresponding to the eigenvalue , the ve tor
y = x is also an eigenve tor of A orresponding to the same eigenvalue , for any
2 C; 6= 0. Thus, the eigenve tors are determined only by their dire tion.
Some important properties of the eigenvalues of a given square matrix are stated
subsequently.
Theorem 5.9 Let A 2 Cnn. The number 2 C is an eigenvalue of A if and only
if the matrix I A is singular. Moreover, the matrix A has exa tly n eigenvalues,
multipli ity ounted, whi h are the zeros of the hara teristi polynomial
p() = det(In A ): (5.11)
If A 2 Rnn , the omplex eigenvalues o ur in onjugate pairs.
Proof. If is a xed eigenvalue of A, then there exists a nonzero (eigen)ve tor x su h

that Ax = x, i.e., (In A)x = 0 and onsequently In A is singular. Conversely,
if for some 2 C the matrix In A is singular, then there exists a nonzero ve tor
x 2 Cn su h that (In A)x = 0, i.e., Ax = x and thus is an eigenvalue of A.
Sin e In A is singular if and only if is a root of the algebrai equation of
order n
p() = 0 (5.12)
( alled the hara teristi equation of A), we infer that (5.12) has n roots (multipli -
ity ounted) whi h oin ide with the eigenvalues of A. If A is a real matrix, the
hara teristi polynomial p() has real oeÆ ients and onsequently the omplex
roots o ur in onjugate pairs. 2
Denote by
(A) = f1 ; 2 ; :::; n g = f 2 C j det(I A) = 0g (5.13)
the set of all eigenvalues of a matrix A 2 Cnn. (A) is alled the eigenvalue spe -
trum of A. We are interested to determine matrix transformations that preserve the
eigenvalue spe trum of a given matrix. Noti e that the eigenvalues and eigenve tors
are not preserved when the matrix is premultiplied or postmultiplied by another
matrix. In parti ular, the elementary row or olumn operations, su h as permuta-
tions, may hange the eigenvalues. The eigenvalues are preserved by the so alled
similarity transformations introdu ed below.
Denition 5.10 Two matri es A; B 2 Cnn are alled similar if there exists a
nonsingular matrix T 2 Cnn su h that
B = T AT 1 : (5.14)
If T is a unitary (orthogonal in the real ase) matrix, then A and B are alled
orthogonally similar.
Theorem 5.11 If A; B 2 Cnn are similar then they share the same eigenvalue
spe trum
(A) = (B ): (5.15)
Moreover, if T is a similarity transformation as in (5.14) and if xA is an eigenve tor
of A orresponding to the eigenvalue 2 (A), then the ve tor
xB = T xA (5.16)
is an eigenve tor of the matrix B , orresponding to the same eigenvalue.
5.2. EIGENVALUES AND EIGENVECTORS 97
Proof. Let A and B be related by (5.14). Then
det(I B ) = det(I T AT 1) = det(T (I A)T 1 )
= det T det(I A)det(T 1 ) = det(I A)
i.e., (5.15) is true. If xA is an eigenve tor of A orresponding to the eigenvalue ,
then AxA = xA or T AT 1T xA = T xA form where BxB = xB and onsequently
assertion (5.16) is also true. 2
In most of our developments we shall be on erned with the generi ase of the so
alled nondefe tive matri es, i.e., the matri es whi h have a omplete set of linearly
independent eigenve tors.
Theorem 5.12 Let A 2 Cnn be a matrix with n linear independent eigenve tors
and let V 2 Cnn be a matrix built up of these eigenve tors as olumns. Then:
V 1 AV = Cnn (5.17)
is a diagonal matrix.
Proof. We have V = [v1 ; v2 ; :::; vj ; :::; vn ℄ and Avj = j vj , j = 1 : n, where j are
the eigenvalues of A. Hen e
AV = [Av1 ; Av2 ; :::; Avj ; :::; Avn ℄ = [1 v1 ; 2 v2 ; :::; j vj ; :::; n Avn ℄
= [v1; v2 ; :::; vj ; :::; vn ℄diag(1 ; 2 ; :::; j ; :::; n ) = V :
Sin e the ve tors vj , j = 1 : n, are linearly independent, the matrix V is nonsingular
and (5.17) follows.
It an be shown that if a n n matrix has n distin t eigenvalues then it is
nondefe tive and hen e, from Theorem 7.12., we infer that after a ertain similarity
transformation it has a anoni al diagonal form.
When a matrix has not a diagonal form, it an be redu ed, by a similarity
transformation, to the so alled Jordan anoni al form:
J = diag[J1 ; J2 ; :::; Jq ℄; (5.18)
where
k 1 0 0 0
2 3
6 0 k 1 0 0 77
6
Jk = 66 777 2 Cn n :
6 k k
(5.19)
4 0 0 0 k 1 5
0 0 0 0 k
Although the Jordan anoni al form is essential in matrix analysis, however it is not
suitable for numeri al purposes due to its stru ture sensibility to numeri al perturba-
tions in the matrix entries. This is the reason for using in all numeri al developments
on erning matrix eigenvalues and eigenve tor of a more robust stru ture alled real
(or omplex) S hur form.
5.3 The real S hur form. De ation te hniques

We are interested in a matrix form whose stru ture is insensitive to numeri al per-
turbations and, at the same time, allows an easy omputation of its eigenvalues.
Denition 5.13 A matrix S 2 Rnn is alled to be in the real S hur form (RSF)
if 2 3
S11 S12 S1q
S = 66 0 S22 S2q 77 ;
6 7
4 5

(5.20)
0 0 : : : Sqq
where Sii 2 R11 or Sii 2 R22 and all 2 2 diagonal blo ks have omplex eigen-
values.
An immediate onsequen e of the S hur stru ture is given in the following Propo-
sition.
Proposition 5.14 Let S 2 Rnn be in RSF. Then:
(S ) = [qi=1 (Sii ): (5.21)
Hen e, the real eigenvalues of a RSF an be determined by a simple inspe -
tion of the main diagonal and the omplex ones an be omputed by solving the
orresponding se ond order algebrai equations
det(I Sii ) = 0: (5.22)
The omplex S hur form (CSF) is an upper triangular stru ture. The eigenvalues
of a matrix in CSF are, of ourse, its diagonal elements.
Let A be a real n n matrix and v 2 Rn one of its real eigenve tors (if any).
Then the asso iate eigenvalue an be omputed as follows:
vT Av
= vvTvv = :
T
vT v
(5.23)
If v has a unit eu lidian norm then
= vT Av: (5.24)
This remark allows us to determine, by a similarity transformation, a real eigenvalue
whenever a orresponding unit eigenve tor is available. Indeed, let v be su h an
eigenve tor and V 2 Rnn 1 a matrix hosen su h that
V = vi V
h i
(5.25)
5.3. THE REAL SCHUR FORM. DEFLATION TECHNIQUES 99
is orthogonal. Su h a matrix exists sin e, if U 2 Rnn is an elementary re e tor
whi h zeroes the last n 1 omponents of v, then Uv = kvke1 = e1 and v =
U T e1 is the rst olumn of the orthogonal matrix V = U T = U .
Let us apply now an orthogonal similarity transformation, dened by (5.25), to
the matrix A. We get
" #
vT A v V
h i
A1 = V 1 AV = V T AV = V T =
vT Av vT AV vT AV
" # " #
= V T Av V T AV = 0 V T AV : (5.26)
In fa t, in (5.26) we have performed a step of the de ation pro edure. The pro edure
an be further ontinued in a similar way on the redu ed dimensional matrix
B = V T AV : (5.27)
The resulting orthogonal transformation on B an be expressed in terms of a di-
mensional extended orthogonal transformation on the matrix A1 dened in (5.26).
For omplex eigenvalues the asso iated eigenve tors are omplex and a omplex
arithmeti must be used. It an be shown that for a pair of omplex onjugate
eigenvalues the two omplex de ation steps an be ondensed and omputed as a
double real step whi h emphasizes a 2 2 diagonal blo k of the orresponding S hur
form.
As a on lusion, by using the de ation pro edure, the following important result
an be stated.
Theorem 5.15 For any square matrix A 2 Rnn there exists an orthogonal matrix
Q 2 Rnn su h that
QT AQ = S (5.28)
is in the RSF.
Thus, any matrix is orthogonally similar with a RSF. Sin e

(A) = (S ) (5.29)
and the eigenvalues of S an be easily omputed, the de ation te hnique suggests a
way for omputing the eigenvalues of a matrix. Unfortunately, the de ation pro e-
dure requires at ea h step an eigenve tor, whi h, in turn, an not be omputed by
dire t methods without the a priori knowledge of the orresponding eigenvalue.
Consequently, the de ation pro edure must be ompleted with a method for
omputing an eigenve tor without knowing the asso iate eigenvalue. Su h a method
is ne essarily an iterative one, be ause the possible existen e of a dire t method

ontradi ts the well-known algebra theorem that states the impossibility of solving
by a nite sequen e of elementary operations an algebrai equation of degree greater
than four.
The most used methods for iterative eigenve tor omputing are the power method
and the inverse power method.
5.4 Power and inverse power methods

These methods are based on the following result.
Proposition 5.16 Let A 2 Rnn be a square matrix,
(A) = f1 ; 2 ; :::; n g (5.30)
its eigenvalue spe trum and xi, i = 1 : n, the orresponding eigenve tors asso iated
with the eigenvalues i. Then:
a)
(Ak ) = fk1 ; k2 ; :::; kn g (5.31)
for any k 2 N, and xi ; i = 1 : n, is an eigenve tor asso iated with the eigenvalue
ki .
b)
(A I ) = f1 ; 2 ; :::; n g (5.32)
and xi is also an eigenve tor asso iated with i .
) If 62 (A) then A I is nonsingular and
((A I ) 1 ) =

1 ; 1 ; :::; 1 (5.33)
1 2 n
xi ; i = 1 : n, is also an eigenve tor asso iated with 1 .
i
Proof. Using the hypothesis, for any i 2 1 : n; xi 6= 0, we have

Axi = i xi (5.34)
a) We have A2 xi = iAxi = 2i xi and by indu tion, it follows immediately that
Ak xi = ki xi (5.35)
for all k 2 N, and (5.31) is true. Clearly xi is an eigenve tor orresponding to ki .
b) From (5.34)
(A I )xi = ixi xi = (i )xi: (5.36)
5.4. POWER AND INVERSE POWER METHODS 101
) If A I is nonsingular then 0 62 (A I ), i.e., i 6= 0, 8i 2 1 : n. From
(5.36)
1 x = (A I ) 1 x ; i = 1 : n; (5.37)
i i
i
whi h ompletes the proof. 2
In Proposition we have given some types of matri es that maintain a set of
eigenve tors, although the set of eigenvalues is hanged. This provides a possibility
to dene a ve tor sequen e onvergent to an eigenve tor of a given matrix.
5.4.1 The power method
Let A 2 Rnn be a matrix having a dominant eigenvalue, i.e., an eigenvalue hav-

ing the magnitude stri tly greater than the magnitude of the others. Suppose the
eigenvalues are ordered by their magnitude
j1 j > j2 j j3 j ::: jnj: (5.38)
Let y(0) 2 Rn be an initial unit ve tor having a nonzero omponent on the dire tion
of the eigenve tor asso iated with the dominant eigenvalue 1 2 (A). If A has a
omplete set of linearly independent eigenve tors
x1 ; x2 ; :::; xn (5.39)
then y(0) an be uniquely written as
n
X
y(0) = i xi (5.40)
i=1
with
1 6= 0: (5.41)
If we dene a re urrent unit ve tor sequen e (y(k) )k2N
y(k) = k Ay(k 1) ; k = 1; 2; (5.42)
where 1
k = (5.43)
kAy k 1) k
(
is a s aling fa tor, then it is easy to show, by indu tion, that
y(k) = ~k Ak y(0) ; (5.44)
where ~k is a umulative s aling fa tor (~k = 1=kAk y(0) k). From (5.40), (5.44) and
(5.31) we have
!
n n n
= ~k 1 1 x1 + i ( i )k xi :
X X X
y(k) = ~k Ak xi = ~k i ki xi k (5.45)
i=1 i=1 i=2 1
Using (5.38) we get

i

< 1; i = 2 : n (5.46)
1
from where
i k
lim
k!1 1
=0
and
k!1
lim y(k) = x1; (5.47)
where is a nonzero s alar su h that k x1 k = 1. Hen e, the ve tor sequen e dened
by the re urren e (5.42) is onvergent to an eigenve tor (5.47) asso iated with the
dominant eigenvalue of the matrix A. The onvergen e speed is determined by the
ratio j2 =1 j. The smaller the value of j2=1 j, the greater the onvergen e speed.
Consequently, the power method works eÆ iently for the matri es with a strong
dominant eigenvalue and a sparse stru ture (to easily multiply Ay(k 1) ).
From a pra ti al point of view, the power method is attra tive due to the sim-
pli ity of its re urren e (5.42). A drawba k to the method is that it onverges slowly
when 1 is not strongly dominant. A ure for this problem is to transform the matrix
A so that the transformed matrix has a strongly dominant eigenvalue and the same
eigenve tors. The fa t that the speed of onvergen e of the power method depends
on the ratio of the magnitude of the largest two eigenvalues suggests that one should
rst transform the matrix so that it has a very large eigenvalue of multipli ity one.
This idea is synthetised in the so alled inverse power method.
5.4.2 The inverse power method
Suppose again that A 2 Rnn has the eigenvalues i; i = 1 : n (not ne essarily in any
spe ial order), orresponding to the linearly independent eigenve tors xi, i = 1 : n,
and suppose that is a lose approximation to 1 . Then, by Proposition 7.16, the
matrix (I A) 1 has the eigenvalues ( i) 1 ; i = 1 : n, orresponding to the
same eigenve tors.
Consequently, if we hoose a starting ve tor y(0) non-defe tive with respe t to
x1 , i.e., satisfying (5.40) and (5.41) we an dene for the matrix (I A) 1 , using
the power method, the unit ve tor sequen e
y(k) = k (I A) 1 y(k 1) ; k = 1; 2; ::: (5.48)
Now, if is mu h loser to 1 than to i; i = 2 : n, then ( 1 ) 1 will be mu h
larger than ( i) 1 ; i = 2 : n, i.e.,
maxi=2:n j( i ) 1j (5.49)
j( 1 ) 1j
5.4. POWER AND INVERSE POWER METHODS 103
is mu h smaller than 1, and onsequently y(k) is very rapidly onvergent to x1 .

The re urren e (5.48), with k a s aling ve tor, denes the power method for the
matrix (I A) 1 and it is known as the inverse power method for the matrix A.
Of ourse, for performing the iteration (5.48), one has not to ompute the matrix
(I A) 1 but to solve a linear system as in the following s heme.
1. Solve (I A)y(k) = y(k 1) with unknown y(k) .
2. y(k) y(k) =ky(k) k:
Solving the above system requires about 2n3=3 oating point operations, whi h
is a high pri e for the omputation of a single eigenve tor. Fortunately, the inverse
power method is usually performed on Hessenberg matri es (as we will see in the
QR algorithm) whi h redu es the omputational eort to approximately n2 ops
per iteration.
The inverse power method is one of the best methods for iterative omputing an
eigenve tor of a given matrix.
An important variant of the inverse power method is the Rayleigh quotient it-
eration whi h, at ea h iteration, uses in (5.48), as an approximation to a matrix
eigenvalue , the urrent Rayleigh quotient
=
(y(k 1) )H Ay(k 1) = (y(k 1) )H Ay(k 1) : (5.50)
ky(k 1) k2
This hoi e of improves the onvergen e speed of the ve tor sequen e to the orre-
sponding eigenve tor. Namely, it an be shown that the Rayleigh quotient iteration
assures a so- alled quadrati onvergen e, i.e.,
ky(k+1) x1 k ky(k) x1k2 (5.51)
where is a onstant s alar. This is a very fast onvergen e (for example, if ky(0)
x1 k " and = 1, then ky(k) x1 k "2 ). k
The formal algorithms for eigenve tor omputation that implement the power
and inverse power methods are left as exer ises for the reader (see Problem 5.4). As
an ending riterion for the iterative pro ess one an use
ky(k) y(k 1) k < tol; (5.52)
ky(k) k
where tol is a pres ribed toleran e.
5.5 The QR algorithm

The eigenvalue and eigenve tor problem an be solved as follows:
1. Find an eigenve tor by using the power or inverse power method.
2. Compute the orresponding eigenvalue by Rayleigh quotient.
3. Apply the de ation pro edure in order to redu e the problem dimension.
After a nite number of the above steps one an ompute all the eigenvalues of
a given matrix A. This pro edure is elegantly en apsulated in a famous algorithm
known as the QR algorithm.
The QR algorithm is an iterative de ation pro edure whi h re urrently onstru ts
a matrix sequen e, with all the matri es orthogonally similar to the given matrix
and onvergent to a S hur form. When only real arithmeti is used the sequen e
limit is the real S hur form of the initial matrix.
When the QR algorithm is applied to a full matrix it is prohibit expensive re-
quiring approximately n3 oating point operations for ea h iteration. Consequently,
the matrix must be rst redu ed by a nite number of similarity transformations to
a form as lose as possible to the (real) S hur form . This form is the upper Hes-
senberg stru ture. Fortunately, the QR iterations preserve the upper Hessenberg
stru ture and onsequently, the iteration on su h a parti ular matrix requires only
approximately n2 oating point operations. The redu tion to Hessenberg form is
onsidered as the rst stage of the QR algorithm.
5.5.1 Redu tion to Hessenberg form
It is a famous lassi result in mathemati s that there is no algorithm, involving

only a nite number of additions, multipli ations, divisions and root extra tion
whi h solves a general polynomial equation of degree greater or equal to 5. Be ause
the eigenvalue omputation is equivalent to solving the hara teristi equation there
is no dire t (nite) algorithm whi h redu es a matrix (of arbitrary dimension n) to
a form loser to the S hur one than the upper Hessenberg form. Re all that the
matrix H 2 Rnn is in upper Hessenberg form if hij = 0 8i > j + 1, i.e.:
2 3
h11 h12 h13 h1;n 1 h1n 7
6
6 h21 h22 h23 h2;n 1 h2n 77
6
6 0 h32 h33 h3;n 1 h3n 77
H=6 0 0 h43 (5.53)
6
6 h4;n 1 h4n 77 :
6 7
6 7
4 0 0 0 hn 1;n 1 hn 1;n 75
6
0 0 0 hn;n 1 hnn
For redu ing a matrix A 2 Rnn to the upper Hessenberg form by orthogonal simi-
larity transformations, we shall use elementary re e tors. Re all that an elementary
5.5. THE QR ALGORITHM 105
re e tor Us of order n and index s is an orthogonal matrix
" #
us uTs I s 1 0
Us = In 2
kusk2 = 0 Us 2 R ; (5.54)
nn
where
us = [0; ; 0; us;s ; us+1;s ; ; uns ℄T : (5.55)
Given a ve tor x 2 Rn there exists always a re e tor su h that
(Usx)i = 0 for i = s + 1 : n (5.56)
(see Chapter 3.)
Theorem 5.17 For any matrix A 2 Rnn , there exists an orthogonal matrix U 2
Rnn su h that the matrix
H = UAU T (5.57)
is upper Hessenberg.
Proof. We shall give a onstru tive proof based on the following omputational
s heme:
1. for s = 1 : n 2
1. Compute an elementary re e tor Us+1 su h that
(Us+1A)i;s = 0 for i = s + 2 : n
2. A Us+1A
3. A AUs+1
whi h overwrites the matrix A by the matrix
A H = Un 1 U3 U2 AU2 U3 Un 1 : (5.58)
Denoting
U = Un 1 :::U3 U2 (5.59)
we have
U T = U2T U3T :::UnT 1 = U2 U3 :::Un 1 ; (5.60)
be ause the elementary re e tors are symmetri . Consequently, (5.58) an be writ-
ten in the form (5.57). It only remains to show that the above omputational s heme
really reates an upper Hessenberg stru ture. We show this by nite indu tion.
Step s=1: It is known (see Chapter 3) that there exists an elementary re e tor
U2 su h that (U2 A)i;1 = 0 for i = 3 : n, i.e., whi h reates the upper Hessenberg
stru ture on the rst olumn. The matrix U2 has the stru ture
" #
U2 = 0 U :1 0 (5.61)
2
Hen e, the postmultipli ation of U2 A by U2 does not hange the rst olumn of U2 A.
i.e., the zeroes in the rst olumn of U2 A are preserved in U2 AU2.
Step s=k: Suppose that in the rst k 1 steps we have obtained the upper
Hessenberg stru ture in the rst k 1 olumns.
A Ak := Uk U2 AU2 Uk : (5.62)
Then, there exists an elementary re e tor Uk+1 su h that (Uk+1 A)i;k = 0 for i =
k + 2 : n, i.e., the premultipli ation by Uk+1 reates the upper Hessenberg stru ture
in the k-th olumn whi h is not altered by postmultiplying with Uk+1. The stru ture
" #
I
Uk+1 = 0 Uk 0 (5.63)
k+1
assures that the postmultipli ation by Uk+1 preserves the rst k olumns of the
matrix (Uk+1Ak ).
Hen e, the omputational s heme presented at the beginning of the proof pro-
vides the redu tion of a given matrix to the upper Hessenberg form by orthogonal
similarity transformation. This ompletes the proof. 2
We leave to the reader the task of writing a detailed algorithm, based on the
s heme given above (Problem 5.6. The algorithm is quite stable, this means that
the omputed Hessenberg form in a oating point arithmeti is an exa t Hessenberg
form of a slightly perturbed matrix A + E . Here E stands for a perturbation matrix
whose entries are of magnitude omparable to the rounding errors.
5.5.2 The iterative stage of the QR algorithm
One of the most eÆ ient methods for nding all the eigenvalues of a matrix is an
iterative te hnique, alled the QR algorithm, whi h uses, in an impli it manner,
the power and inverse power methods to assure the iterative redu tion of a matrix
to the S hur form, by ex lusively using orthogonal similarity transformations. To
improve the eÆ ien y of the method, the given matrix is rst redu ed to the upper
Hessenberg form.
Suppose A 2 Rnn is a matrix in upper Hessenberg form. The QR algorithm
produ es a sequen e of orthogonally similar matri es
A = A1 ; A2 ; ; Ak ; Ak+1 ; (5.64)
onvergent to the S hur ((quasi)-upper triangular) form of the matrix A.
The expli itly shifted QR algorithm is dened by the following re urren e:
1:Ak k In = Qk Rk ; k = 1; 2; (5.65)
2:Ak+1 = Rk Qk + k In
where the s alar k , alled the origin shift, is used to speed up the onvergen e. In
the rst step of (5.65) the shifted matrix Ak k In is fa torized in a produ t of an
orthogonal matrix Qk and an upper triangular matrix Rk . (See the QR fa torization
in Chapter 3). In the se ond step, the next matrix Ak+1 is obtained multiplying the
Qk and Rk matri es in reverse order and vanishing the shift by adding k In .
The main properties of the matrix sequen e (5.64), alled the QR sequen e and
generated by (5.65), are given by the following theorem.
Theorem 5.18 a) If the initial matrix A1 = A of the QR matrix sequen e is upper
Hessenberg then all the matri es in sequen e have the upper Hessenberg stru ture.
b) All the matri es of the QR sequen e are orthogonally similar and share the
same eigenvalue spe trum.
) By an appropriate hoi e of the origin shift k , the QR sequen e is onvergent
to a real S hur form of the initial matrix A.
Proof. The rst two properties an be easily derived.
a) If Ak in (5.65) is an upper Hessenberg matrix, so is the shifted matrix Ak
k In. If k 62 (Ak ), whi h is the usual ase, then Ak k In is nonsingular and so
is the upper triangular matrix Rk . Hen e
Qk = (Ak k In )Rk 1 : (5.66)
The inverse of an upper triangular matrix is upper triangular and it is easy to he k
that the produ t between an upper Hessenberg matrix and an upper triangular one is
upper Hessenberg. Hen e Qk is an orthogonal upper Hessenberg matrix. It follows,
by a straightforward omputation, that Rk Qk is upper Hessenberg and, learly, so
is Ak+1. By indu tion, if A1 = A is upper Hessenberg then all Ak ; k = 2; 3; ::: are
upper Hessenberg matri es.
b) From (5.65)
Rk = QTk (Ak k In ); (5.67)
whi h introdu ed in (5.65) gives:
Ak+1 = QTk (Ak k In )Qk + k In = QTk Ak Qk ; (5.68)
i.e., Ak+1 and Ak are orthogonally similar and share the same eigenvalue spe trum.
Applying (5.68) repeatedly we get
Ak = QTk 1 Ak 1 Qk = QTk 1 QTk 2Ak 2 Qk 2 Qk 1 =
= QTk 1QTk 2 Q1 A1Q1 Qk 2Qk 1 = Q~ Tk AQ~ k ; (5.69)
where
Q~ k = Q1 Q2 Qk 1 (5.70)
is an orthogonal matrix as a produ t of orthogonal matri es. Hen e, all the matri es
in the QR sequen e are orthogonally similar and onsequently share the same spe -
trum. The resulting orthogonal transformation (5.70) an be onstru ted re urrently
by the relation
Q~ k+1 = Q~ k Qk ; Q~ 0 = In ; k = 0; 1; 2; (5.71)
) We prove now that by an adequate hoi e of the shifts k the QR sequen e is
onvergent to the S hur form of the matrix. To this end, we show that the o-
diagonal element in the last row of Ak an be onstrained to approa h zero very
swiftly. Moreover, all the subdiagonal elements of Ak may approa h zero somewhat
more slowly. The reason is twofold:
(i) The QR sequen e, with an appropriate hoi e of the origin shifts, implements
a version of the Rayleigh quotient iteration and onsequently onverges quadrati ally.
(ii) The QR sequen e implements simultaneously an extension of the power
method. Thus, while the method onverges to a ertain eigenvalue, it also attempts
to nding the others, albeit more slowly.
In order to justify the above assertions let us rst onsider (5.65) from where, if
k 62 (A), it results
Rk = QTk (Ak k In ); (5.72)
whi h with (5.69) and (5.71) be omes
Rk = QTk (Q~ Tk AQ~ k k In) = QTk Q~ Tk (A k In )Q~ k = Q~ Tk+1 (A k In )Q~ k (5.73)
or
Q~ k+1Rk = (A k In )Q~ k : (5.74)
We begin with the se ond assertion (ii). The rst olumn of (5.74) is
Q~ k+1r1(k) = (A k In )~q1(k) ; (5.75)
where q~1(k) = Q~ k e1 is the rst olumn of Q~ k and r1(k) = Rk e1 is the rst olumn of
Rk . Sin e Rk is upper triangular we have
r1(k) = r11
(k) e
1 (5.76)
and (5.75) an be written in the equivalent form
q~(k+1) =
1 (A In)~q(k) (5.77)
1 (k) k 1
r11
whi h expresses the power method's re urren e relations for omputing an eigen-
ve tor of the matrix A using a s alar shift parameter k . As it has been shown in
Se tion 5.4, the rst olumn of the orthogonal transformation matrix Q~ k onverges
to an eigenve tor asso iated with the dominant eigenvalue of the matrix A k I .
A ording to the de ation pro edure (see Se tion 5.3) the rst olumn of the matrix
Q~ Tk AQk = Ak onverges to the S hur form of A, i.e., the subdiagonal element in the
rst olumn of A is asymptoti ally vanishing. The speed of onvergen e depends
on the ratio of the absolute values of the rst and se ond dominant eigenvalues of
A k In .
In order to stress out the Rayleigh quotient iteration, let us derive from (5.74)
the dependen e between the last olumns of the matri es Q~ k+1 and Qk , respe tively.
Rewriting (5.74) in the form
Rk Q~ Tk = Q~ Tk+1 (A k In ) (5.78)
and separating the last row of (5.78) we get
eTn Rk Q~ Tk = eTn Q~ Tk+1(A k In): (5.79)
But Rk is upper triangular and its last row is
eTn Rk = [0 0 rnn (k) ℄ = r(k) eT :
nn n (5.80)
Hen e (5.79) an be rewritten as
(~qn(k) )T rnn
(k) = (~ qn(k+1) )T (A k In ) (5.81)
or
q~n(k+1) = rnn (k) (AT In) 1 q~(k) ;
k n (5.82)
whi h denes the inverse power iteration for nding an eigenve tor of the matrix
AT . It is straightforward to prove that AT has the same eigenvalue spe trum as
A but, in general, has other eigenve tors. Fortunately, if the last olumn of the
orthogonal similarity transformation is an eigenve tor of AT then the o-diagonal
elements of the last row of A are vanishing in a similar way as in the standard
de ation pro edure.
Indeed, onsider the orthogonal transformation
h i
Q = Q vn (5.83)
with
AT vn = vn : (5.84)
Then
" # " # " #
T T T
Q AQ =
T Q h
A Q vn = T
i
Q AQ Q Av n A h
= 0 n
T
(5.85)
vT
n n v AQ vT Av
n n
where the expression for the se ond blo k row in the last equality follows from (5.83)
and (5.84)
vnT AQ = n vnT Q = 0 (5.86)
vnT Avn = n vnT vn = n :
The quadrati onvergen e of qn(k) in (5.82) to an eigenve tor of AT an be a hieved

by hoosing k as in the Rayleigh quotient iteration (5.50)
k =
(~qn(k))T AT q~(k) = (~q(k))T AT q~(k) = (~q(k))T Aq~(k) = eT Q~ T AQe
~ n = eTn Ak en = a(nnk) :
(k)
kq~n k n n n k
(5.87)
The hoi e (5.87) of the origin shifts guarantees an ex ellent onvergen e of the QR
sequen e to the S hur form. Thus the proof of the ) part of the Theorem ends. 2
In the above proof of onvergen e the expli itly QR algorithm is dened by the
re urren e (5.65), with the hoi e (5.87) of the origin shifts. When the element
a(n;n
k)
1 satises a ondition like
ja(n;n
k) (k)
1 j < tol(jan 1;n 1 j + jann j);
(k)
where tol is a pres ribed toleran e level, we an onsider a(n;n

k)
1 to be numeri ally
(k)
negligible and set it to zero. Thus ann is the omputed eigenvalue of A. After this
step is ompleted, the dimension of our problem is redu ed by one. Usually, 1.5-2
QR iterations are enough to ompute one eigenvalue of a given Hessenberg matrix.
Another problem to be solved is to ompute in real arithmeti a 2 2 diagonal
blo k of the real S hur form. To remain in the real arithmeti eld it is ne essary to
ompress two QR iterations (5.65) into a single one, and to establish how to hoose
an adequate double origin shift. The details an be found in [?℄, [?℄.
Standard implementations of the QR algorithm use an iteration version alled
impli itly shifted QR algorithm. The ore of this version is an intensive use of the
algorithm of redu ing a matrix to upper Hessenberg form. In the impli itly shifted
variant ea h iteration onsists of two orthogonal similarity transformations whi h
have the same ee t as an expli itly shifted iteration. The rst transformation,
essentially dened by the rst olumn of the matrix Qk in (5.65), assures the desired
origin shift but ruins the upper Hessenberg stru ture. The se ond transformation
rebuilds the upper Hessenberg form by using the algorithm depi ted in the proof of
Theorem 5.17. For details the reader is referred to [?℄ and [?℄.
Eigenve tor omputation. On e the S hur form and the eigenvalues of the
given matrix obtained, the asso iated eigenve tors an be omputed by solving
the orresponding singular linear system or by few iterations of the inverse power
method. It is important to stress out that in fa t in many appli ations instead of
the eigenve tors one an use in an adequate manner the olumns of the transfor-
mation Q, dened in Theorem 5.15. These olumns are the so alled S hur ve tors
asso iated with the given matrix. The orthogonal transformation matrix Q an be
omputed by aggregating all transformations during the QR algorithm exe ution.
Problems

P 5.1 Prove that if A = A0 1 A2
A3 , then (A) = (A ) [ (A ).
1 3

P 5.2 Compute a real S hur form of the matrix A = 2 3 .1 2

P 5.3 Compute a real S hur form of the matrix A = 11 11 .
P 5.4 Write detailed algorithms for the power and inverse power methods.
P 5.5 Whi h will be the result of the power method applied to:
2
5 8 13
A = 4 0 1 2 5; B = 11 31 ;
0 0 2
with appropriate initial approximations ? What does "appropriate" mean in these ases ?
P 5.6 Write a detailed algorithm to redu e a given matrix A 2 to upper Hessenberg R
n n
form by orthogonal similarity transformations. Use the omputational s heme from the
proof of Theorem 5.17.
P 5.7 There are p linear systems of the form (A + I )x = b to be solved, where A 2
, 2 , b 2 , for k = 1 : p.
k k k
n n n
R k R k R
Show that x an be omputed from the solution of the system (H + I )y = d ,

a.
where H 2 is upper Hessenberg and orthogonally similar to A.
k k k k
n n
R
Outline an algorithm for solving the p systems (A + I )x = b , using the idea of

b.
.
k k k
Compare the number of operations to that obtained using Gaussian elimination.

.

P 5.8 a. Consider the symmetri matrix A = d0 d0 , with d 6= 0. Compute the eigen-
values and eigenve tors of A.
Consider
b.
the diagonal matrix D = diag (d ; : : : ; d ), where d 6= d , 8i 6= j , and
1
0
n i j
B= D 0 2 D . Show that there exists a permutation matrix P (as produ t of

R
2n 2n
elementary permutation matri es) su h that

P BP = diag(D ; : : : ; D ); where D = d0 d0 :
T
1 n i
i
Give an algorithm to ompute the eigenvalues and eigenve tors of B.

.

P 5.9 a. Let A = 2 be a symmetri matrix, with 6= 0. Give an
R
2 2
algorithm to ompute the eigenvalues and the eigenve tors of A (verify that the eigenve tors
are orthogonal).
b. Consider the symmetri matrix B 2 R

n n
:
2 3

6
6 1 7
7
B= 6
6 ... 7
7; 6= 0:
6 7
4 1 5

Give an algorithm to ompute the eigenvalues and eigenve tors of B.
P 5.10 Show that a real symmetri matrix has real eigenvalue spe trum and pairwise or-
thogonal eigenve tors. What omputational fa ilities appear in the symmetri QR algo-
rithm ?
P 5.11 Suppose A 2 has distin t eigenvalues and B 2 ommutes with A:
C
n n n
C
n
AB = BA. Show that if Q AQ = T is the ( omplex) S hur de omposition of A, then

H
Q BQ is upper triangular.
H
P 5.12 Give an algorithm for redu ing a matrix A 2 to Hessenberg form, H = R

n n
T AT , where T is a produ t of elementary triangular (Gaussian) transformations.

1
P 5.13 (The Hessenberg form similar to a given matrix is not unique) Suppose A 2 n
R
n
and z 2 . Give an algorithm for omputing an orthogonal Q su h that QAQ is upper

R
n T
Hessenberg and Q z is a multiple of e .

T
1
P 5.14 Let H 2 be upper Hessenberg.

R
n n
Des ribe the details of an expli itly shift QR step, using the shift = h .
a. nn
Suppose that an orthogonal Q 2 was determined su h that:

b. R
n n

~H := Q HQ = ; 1
0 H
T
where 2 (H ) and H 2

1
is upper Hessenberg. Give an algorithm for
R
(n 1) (n 1)
omputing an eigenve tor asso iated with . 1
P 5.15 Let H 2 be upper Hessenberg. Write an algorithm whi h veries if H is a

R
n n
real S hur form (quasi upper triangular).

P 5.16 Give an algorithm to ompute the eigenvalues of A = I + uv , where u; v 2 . n
T
R
n
Chapter 6
The singular value

de omposition
As we have already seen, the orthogonality plays a striking role in matrix ompu-
tations. The least-square (LS) problem and eigenvalue-eigenve tor problem solving
intensively use orthogonal transformations.
Another important appli ation of orthogonal transformations is the solving of
an extremely useful stru tural matrix de omposition alled singular value de ompo-
sition (SVD). Among other features, the SVD enables to eÆ iently solve the matrix
rank omputation problem. The important mathemati al on ept of matrix rank is
perfe tly lear in the exa t arithmeti setting. However, in the presen e of roundo
errors, all matri es are, generi ally, of full rank. The SVD allows us to introdu e the
pra ti al on ept of numeri al rank. The SVD is also the best tool for solving the
general least square problem, i.e., to ompute the minimum norm ve tor x whi h
minimizes the eu lidian norm of the residual krk = kb Axk, in the ase of rank
de ien y of the matrix A.
6.1 Denitions. Properties

Before stating the SVD, let us re all rst some results from the orthogonal trans-
formations theory, established in Chapter 3.
1. The orthogonal transformations preserve the eu lidian norm, i.e., if Q 2
Rmm is an orthogonal matrix, x 2 Rm and k k is the eu lidian ve tor norm then
kQxk = kxk: (6.1)
2. The matrix 2-norm k k2 and the Frobenius matrix norm k kF are also
invariant with respe t to orthogonal transformations, i.e., for any orthogonal matrix
113
114 CHAPTER 6. THE SINGULAR VALUE DECOMPOSITION
Q 2 Rmm and any matrix A 2 Rmn we have

kQAk2 = kmax
xk =1
kQAxk2 = kAk2 ; (6.2)
2
0 11=2
m X
X n
kQAkF = j(QA)ij j2 A = kAkF : (6.3)
i=1 j =1
3. In parti ular, it is easy to prove that for any orthogonal matri es Q and Z ,
of appropriate dimensions, we have
kQAZ k2 = kAk2 ; (6.4)
kQAZ kF = kAkF : (6.5)
4. The matrix rank is dened as follows.
Denition 6.1 Let A 2 Rmn . The rank of the matrix A is the maximal number
of linearly independent olumns of A, or, equivalently,
rank(A) = dim(ImA); (6.6)
where dim stands for the dimension of the argument spa e.
5. It an be shown (see any linear algebra book) that
rank(A) = rank(AT ) (6.7)
and, thus, any matrix has the same maximal number of independent rows and
independent olumns. Hen e the rank of a given matrix A is the maximal dimension
of a nonsingular square submatrix of A.
6. It is an easy exer ise for the reader to show that
rank(A) = rank(T AS ) (6.8)
for any nonsingular matri es T and S of appropriate dimensions.
Now we are able to dene SVD by using the following theorem.
Theorem 6.2 (Singular Value De omposition-SVD) If A 2 Rmn then there
exist two orthogonal matri es U 2 Rmm and V 2 Rnn su h that
" #
U T AV = = 0 00 ; (6.9)
6.1. DEFINITIONS. PROPERTIES 115
where
= diag(1 ; 2 ; :::; r ) 2 Rrr ; (6.10)
with
1 2 ::: r > 0: (6.11)
The expressions (6.9)-(6.11) dene the singular value de omposition of the matrix
A.
Proof. If A = 0 then (6.9)-(6.11) are satised by r = 0, U = Im, V = In, = 0. If

A 6= 0 then kAk2 6= 0 and, by using a nite indu tion, we shall obtain the matrix
stru ture in (6.9).
Step 1. Let x 2 Rm be a unit eu lidian norm ve tor (kxk2 = 1) for whi h
kAk2 = kmax
xk =1
kAxk2 = kAxk2 (6.12)
2
(the existen e of su h a ve tor is assured by the theorem on erned with the matrix
2-norm). Dene the ve tor y 2 Rn as
Ax
y= (6.13)
kAk2 ;
where x is dened in (6.12). Then
kyk2 = yT y = kA1k kAxk22 = 1:
q q
(6.14)
2
As we have seen in the previous hapter (see the de ation te hnique), on e given a
unit 2-norm
h ve tor
i q 2 Rp there exists a matrix Q 2 Rp(p 1) su h that the matrix
Q = q Q is orthogonal (for example Q an hoosen to be a re e tor). Hen e,
let U 1 2 Rm(m 1) and V 1 2 Rn(n 1) be su h that the matri es
h i
U1 = y U 1 2 Rmm (6.15)
h i
V1 = x V 1 2 Rnn (6.16)
be orthogonal. Then
" # " #
yT h i yT Ax yT AV 1
1 := U1T AV1 = U T1
A x V1 = U T1 Ax U T1 AV 1
: (6.17)
From (6.13)
yT Ax = yT ykAk2 = kAk2 (6.18)
U T1 Ax = U T1 ykAk2 = 0: (6.19)
Denoting
1 := kAk2
wT := yT AV 1 (6.20)
B1 := U T1 AV 1
(6.17) be omes " #
1 = 1 wT
U T AV
1 1= 0 B1 (6.21)
and applying (6.4)
k1 k2 = kAk2 = 1 : (6.22)
On the other hand,
" # 2 " # 2

1 w1

=

12 + wT w

= (12 + wT w)2 + kB1 wk22

2
B1 w
2
(12 + wT w)2 = (12 + kwk22 )2 (6.23)
and from the onsisten y property of the matrix 2-norm we have
" # 2 " # 2

1 w1

k1 k22 w1

= 12 (12 + kwk22 ): (6.24)
2 2
Now, (6.23) and (6.24) imply
12 + kwk22 12 ; (6.25)
i.e.
kwk2 = 0 (6.26)
or
w = 0: (6.27)
Hen e " #
1 = 1 1 = 0 B1 U T AV 1 0(6.28)
and the obje tive of the rst step is attained.
Step s. Suppose that in the rst s 1 steps of the diagonalization pro edure
we have obtained
" #
T T T
s 1 = Us 1 U2 U1 AV1 V2 Vs 1 = 0 Bs 1
s 1 0 (6.29)
where
s 1 = diag(1; 2 ; :::; s 1 ); with 1 2 s 1 > 0: (6.30)
If Bs 1 = 0 the pro edure is ompleted. If Bs 1 6= 0 then there exists U~s 2

R(m s+1)(m s+1) and V~s 2 R(n s+1)(n s+1) both orthogonal (by the same ar-
guments as in step 1) su h that
" #
T s 0
U~s Bs 1 V~s = 0 B ; s = kBs 1 k2 > 0:
(6.31)
s
Dening " # " #

I s 0 I s 0
Us =
0 U~s ; Vs = 0 V~s ; (6.32)
whi h are both orthogonal, it results
" #
s = Us s 1Vs = 0 Bs ;
T s 0
(6.33)
with s a diagonal matrix with positive diagonal elements. To omplete the proof
it is suÆ iently to show that
s s 1: (6.34)
But this inequality holds sin e the expression (6.31), written for the previous step,
we have "
" # #

~
s 1 = kBs 2 k2 = Us 1 s 1 0 V T

~s 1 =

s 1 0

0 Bs 1 2 0 Bs 1
2
" # " #" #
= max

s 1 0
z max

s 1 0 0

kzk2 =1 0 B s 1 2 kz k 2 = 1
0 Bs 1 y 2

z = [0 y ℄
T T
= kmax
yk =1
2
kBs 1yk2 = kBs 1 k2 = s:
Hen e, the diagonalization pro edure an be started and ontinued. Thus, for some
r p = min(m; n) (6.35)
we shall obtain for r < p
Br = 0 (6.36)
or Br does not exist for r = p. Finally,
" #
T T T T r 0
r = Ur U2 U1 AV1 V2 Vr = U AV = 0 0 ; (6.37)
where
U = U1 U2 :::Ur (6.38)
V = V1 V2 :::Vr
are orthogonal matri es, r := r ; := r and the proof ends. 2
Remark 6.3 Let uj = Uej ; j = 1 : m, and vj = V ej , j = 1 : n, be the j -th olumns

of the matri es U and V dened in (6.9), respe tively. Rewriting (6.9) in the form
AV = U (6.39)
or
AT U = V T (6.40)
we get
Avj = j uj ; j = 1 : p; p = min(m; n): (6.41)
AT uj = j vj
Theorem 6.2 and the above Remark allow us to introdu e the following denition.
Denition 6.4 The positive numbers i , i = 1 : p; p = min(m; n)
1 2 r > 0; r+1 = r+2 = = p = 0 (6.42)
are alled the singular values of the matrix A.
The olumns uj 2 Rm of the orthogonal matrix U are alled the left singular
ve tors of the matrix A.
The olumns vj 2 Rn of the orthogonal matrix V are alled the right singular
ve tors of the matrix A.
The SVD reveals a great deal about the stru ture of a given matrix. We shall
give some properties in the Propositions below.
Let A 2 Rmn and its SVD be given by Theorem 6.2 and Denition 6.4.
Proposition 6.5
rank(A) = r (6.43)
i.e. the rank of a matrix is given by the number of its nonzero singular values.
Proof. The orthogonal matri es U T and V are learly nonsingular. Then form (6.8)
and (6.9)-(6.11)
rank(A) = rank(U V T ) = rank = rank = r: 2
Let now introdu e the following partitions of U and V :
U = [u1 u2 ::: ur ur+1 ::: um ℄; V = [v1 v2 ::: vr vr+1 ::: vn ℄
U (:; 1 : r) =: U1 ; U (:; r + 1 : m) =: U2 ; V (:; 1 : r) =: V1 ; V (:; r + 1 : n) =: V2 :
Proposition 6.6 a)
ImA = ImU1 (6.44)
b)
KerA = ImV2 (6.45)
i.e., the rst r olumns of the matrix U form an orthonormal basis for the linear
subspa e ImA and the last n r olumns of the matrix V form an orthonormal basis
for the linear subspa e KerA.
Proof. a)
ImA = fy 2 Rm j9x 2 Rn su h that y = Axg:
Let y 2 ImA. Then, from (6.9)
" # " #
0 h
y = Ax = U V T x = U 0 0 V T x = U1 U2
i
0
0 0 V x = U1 z
T
h i
where z = 0 V T x.
Hen e y 2 ImU1 and
ImA ImU1: (6.46)
1
Conversely, if y 2 ImU1, i.e., y = U1x for some x 2 Rr then for z = x we have
" #
h i z
y = U1 z = U1 0r;n r
w
for an arbitrarily hosen w 2 Rn r , or equivalently
" #
y = U1 [ 0℄V T V z = U V T z~ = Az~
w
" #
where z~ = V z 2 Rn . Hen e y 2 ImA and
w
ImU1 ImA: (6.47)
From (6.46) and (6.47) the desired equality (6.44) follows.
b)
KerA = fx 2 RnjAx = 0 2 Rm g:
Let x 2 KerA. Then Ax = 0, i.e.
" #" # " #
Ax = U V x = U 0 0 0 V T V Tx
V2T x = U 0 = 0: (6.48)
T 1 1
But U and are nonsingular and onsequently

V1T x = 0; (6.49)
i.e.,
x 2 KerV1T : (6.50)
Now, it is easily he kable that
KerV1T = ImV2 : (6.51)
The olumns of V form an orthogonal basis of Rn and hen e the olumns of V1
and the olumns of V2 form orthogonal basis for the two orthogonal omplementary
subspa es S = ImV1 and T = ImV2 = S ?. Hen e any ve tor x 2 T is orthogonal to
S , parti ularly to every olumn of V1 , i.e., xT V1 = 0 or, V1T x = 0 and onsequently
x 2 KerV1T or equivalently T KerV1T . Conversely, if x 2 KerV1T , i.e., V1T x = 0
or, equivalently xT V1 = 0 or x ? ImV1 , i.e., x 2 T ) KerV1T T . Hen e (6.51) is
true. From (6.48), (6.50) and (6.51) we have
KerA ImV2 : (6.52)
Conversely, if x 2 ImV2 = KerVT1 then V1T x = 0 and from (6.48) we get Ax = 0,
i.e., x 2 KerA. Hen e
ImV2 KerA: (6.53)
The in lusions (6.52) and (6.53) prove the desired equality (6.45) and the proof ends.
2
The following proposition stresses out a lose relation between the SVD and two
parti ular matrix norms.
Proposition 6.7 a)
kAk2 = 1 (6.54)
b) q
kAkF = 12 + 22 + + r2
i.e., the 2 norm of a matrix oin ides with the maximum singular value and the
Frobenius norm equals the square root of the sum of the squared singular values.
Proof. a) From (6.4) we have
kAk2 = kU T AV k2 = kk2 = kmax
xk =1
kxk2
2
q
= x +x max
2 ++x =1
2 2
12 x21 + 22 x22 + + r2 x2r =
1 2 n
r
= max (12 x21 + 22 x22 + + r2 x2r ):
x21 +x22 ++x2n =1
(6.55)
But 1 2 r > 0. Hen e

f (x1 ; x2 ; ; xn ) = 12 x21 + 22 x22 + + r2 x2r 12 x21 + ( 2 )2 x22 + + ( r )2 x2r
1 1
12 (x21 + x22 + + x2r ) 12 (x21 + x22 + + x2n)
and onsequently
max f 12 kxk2 12:
kxk =1
2
(6.56)
h i
The equality in (6.56) holds for x = 1 0 0 T , and we infer that
max f = 12
kxk =1 2
and from (6.55) we on lude that (6.54) is true.

b) From (6.5)
q
kAkF = kU T AV kF = kkF = k1kF = 12 + 22 + + r2
and the proof ends. 2
The omputation of the SVD is based on the following main result.
Theorem 6.8 The nonzero singular values i ; i = 1 : r, of a matrix A 2 Rmn are
the positive square roots of the nonzero eigenvalues of the symmetri semipositive
denite matrix
B = AT A (6.57)
i.e., if 1 2 r > 0, are the r nonzero eigenvalues of B then
p
i = i ; i = 1 : r: (6.58)
Proof. Using Theorem 6.2, there exists two orthogonal matri es U 2 Rmm and
V 2 Rnn su h that (6.9) holds and we get
" #
2
B = A A = V U U V = V V = V
T T T 2 T 0 T (6.59)
0 0 V :
Hen e, B = AT A and 2 are orthogonally similar and both share the same spe trum.
Then
(B ) = (2 ) = f12 ; 22 ; ; r2 ; i = 0; i = r + 1 : ng (6.60)
whi h proves (6.58). 2
Theorem 6.2 shows that a symmetri positive semidenite matrix fa torized as
in (6.57) has only real nonnegative eigenvalues. Moreover, any symmetri positive
semidenite matrix has su h property.
Proposition 6.9 Let B 2 Rnn be a symmetri matrix and let

(B ) = f1 ; 2 ; ; n g and x1 ; x2 ; ; xn , be its spe trum and the orresponding
set of eigenve tors, respe tively. Then the following assertions are true.
1.
(B ) R: (6.61)
2.
xTi xj = 0; 8i 6= j (6.62)
i.e. all the eigenve tors are orthogonal.
3. The real S hur form of B is diagonal.
4. If, in addition, B 0 then
j 0; j = 1 : n: (6.63)
Proof. From the previous hapters we know that any matrix is orthogonally similar
with a real S hur form
S = QT BQ; (6.64)
where Q is orthogonal and S is quasi upper triangular. Sin e
S T = QT B T Q = QT BQ = S; (6.65)
i.e., we infer that S is a blo k diagonal matrix, ea h blo k having at most dimension
2. The 2 2 dimensional diagonal blo ks are symmetri and, as it an be easily
he ked, their spe trum belongs to R. From here we on lude that S is a real
diagonal matrix whi h is exa tly assertion 3. From (B ) = (S ) we have that
(6.61) also holds. Using (6.64) rewritten as
QS = BQ (6.66)
we get
sjj qj = Bqj ; j = 1 : n; (6.67)
where qj = Qej ; j = 1 : n, is the j -th olumn of the matrix Q. From (6.67) we
have that the eigenve tors of B are all olinear with the orresponding olumns of
an orthogonal matrix, and (6.62) follows.
If the symmetri matrix B is positive semidenite, then xT Bx 0; 8x 2 Rn .
Taking su essively x = Qej = qj ; j = 1 : n, the above inequality be omes
qjT Bqj = eTj QT BQej = eTj Sej = sjj = j 0; j = 1 : n; (6.68)
and onsequently the assertion 4 is true and the proof ends. 2
Theorem (6.8) suggests a pro edure for omputing the singular values of a given
matrix A by using the QR algorithm for omputing the eigenvalues of B = AT A.
6.2. THE SVD ALGORITHM 123
This pro edure is not re ommended from a numeri al point of view due to the pos-
sible ill numeri al ondition of B whi h equals the squared numeri al ondition of
A. In order to avoid the expli it omputation of B , a numeri ally sound algorithm
(Golub and Kahan) a ting ex lusively on the matrix A was developed. This algo-
rithm, known as the SVD algorithm, is presented in the next se tion.
6.2 The SVD Algorithm

As we have already seen, the singular values of a matrix A 2 Rmn are the non-
negative square roots of the symmetri semipositive denite matrix B1 = AT A (see
(6.60) of Theorem 6.8). Moreover, there are other important relationships between
the SVD of" a matrix#A and the RSF (Real S hur form) of the matri es B2 = AAT
and B3 = A0 A0 .
T
Theorem 6.10 Let A 2 Rmn , with m n, and onsider its SVD

" #
= U T AV = 01 ; 1 = diag(1 ; 2 ; ; n ): (6.69)
Then the matri es B2 and B3 dened above have the following RSF
S2 = U T (AAT )U = diag(12 ; 22 ; ; n2 ; 0| ; {z ; 0}) (6.70)
m n
" #
S3 = QT 0 AT Q = diag(1 ; 2 ; ; n ; 1 ; 2 ; ; n ; 0| ; {z
; 0}) (6.71)
A 0 m n
where the orthogonal matrix Q is
" #
1
Q= p
V V p0 (6.72)
2 U1 U1 2U2
with
U1 = U (:; 1 : n); U2 = U (:; n + 1 : m):
Proof. a) First, from the SVD of A we have

" # " #
AA = U V V U = U 0
T T T T 1 h i
T 2 0
1 0 U = U 0 0 U T = US2U T
1
and onsequently S2 = U T AAT U .

b) Se ond, from the SVD of A and the partition

U = [|{z} U2 ℄
U1 |{z}
n m n
we have " #
A = U VT =
h
U1 U2
i 1 VT = U1 1V T
0
and
AT = V 1U1T :
Hen e
" # " # " #" #" #
B3 = A0 A0 = 0 V 1 U1T = U01 V0 0 1 0 U1T
T
U1 1 V T 0 1 0 VT 0 :
(6.73)
Let " #
S := 0 01 2 Rnn ; 1 = diag(1 ; ; n )
1
and dene the similarity transformation
" # " #
T := I n 0 ;T =
1 I n 0
:
In In In In
Then " #
S~ = T ST 1 = 1 1 (6.74)
0 1
and
(S ) = (S~) = (1 ) [ (
1) = f1 ; ; ng [ f 1 ; ; ng:
For obtaining the RSF of the symmetri matrix S we need to determine a set of
(orthonormal) eigenve tors for it. To this end, we nd rst a set of eigenve tors
yj 2 R2n , j = 1 : 2n, for S~. Further, the eigenve tors xj 2 R2n of S will be
omputed as
xj = T 1 yj : (6.75)
" #
yj0
Hen e, for j = j ; j = 1 : n, and with the partition yj = yj00 the system
S~yj = j yj (6.76)
an be split as
(i j )yij0 + iyij00 = 0 ; i = 1 : n: (6.77)
( i j )yij00 = 0
The system (6.77) is satised by the nonzero ve tor yj given by
yij00 = 0; i = 1 : n
yij = 0; i = 1 : n n fj g
0 (6.78)
yjj = j 6= 0
0
For j = j ; j = 1 : n, the system (6.76) be omes

(i + j )yij0 + iyij00 = 0 ; i = 1 : n; (6.79)
( i + j )yij00 = 0
whi h is satised by a nonzero ve tor yj00 if
00 = j 6= 0
yjj (6.80)
the other omponents of yj00 being zero
00 = 0; i = 1 : n n fj g:
yjj (6.81)
From the rst set of equations (6.79) we have that yj00 an be
yij0 = 0; i = 1 : n n fj g
2j yjj0 + j j = 0 (6.82)
or
0 = 1 j :
yjj (6.83)
2
Let now
R := diag(1 ; 2 ; ; n ) (6.84)
K := diag(1 ; 2 ; ; n ):
Then from (6.78) and (6.80)-(6.83), the set of eigenve tors of S~ an be rewritten in
the form "
1K
#
Y= 0 K R 2 2 R2n2n (6.85)
and from (6.75) the orresponding set of eigenve tors of S is
" #" # " #
n 0
I R 1K R 1K
X =T Y = I I
1
0 K = R 21 K
2 2 (6.86)
n n
i.e. " #
j ej
xj = e 2 R2n ; j = 1 : n; (6.87)
j j
" #
xj =
1 j nej n 2 R2n; j = n + 1 : 2n: (6.88)
2 j nej n
For unity eu lidian norm eigenve tors, it is ne essary to have

2j + 2j = 1 ) j = p12 ; j = 1 : n;
1 2 + 1 2 = 1 ) j = p2; j = 1 : n: (6.89)
4 j 4 j
Consequently the eigenve tor matrix X (6.86) be omes
" #
X=p I I
1 I n I n
(6.90)
2 n n
and a dire t he k shows that it is orthogonal. Hen e the RSF of S is
" #" #" # " #
T 1 n n 0
~ = X SX = 2 In In 1 0 In In = 0 1 : (6.91)
I I 1 I n I n 1 0
Now from (6.73) we have

" # " # " #
0 A T
B3 = A 0 = U 0 S V T 0 = 0 V 0 U1
T
1
" # " #
= U01 V0 X ~ X T VT
0 U1T
0 = Q~ ~ Q~ T ; (6.92)
where
" # " # " # " #
Q~ = 0 V 0 V 1 In In = p12 V V
U1 0 X = U1 0 p2 In In U1 U1 2 R(m+n)n :
(6.93)
Finally, it is straightforward to see that the matrix
# " #
h
~
Q= Q U = p 0 1 V V p 0 (m+n)(m+n) (6.94)
2 2 U1 U1 2U2 2 R
" #
is orthogonal ( = 2V0V 2UU 0 = Im+n ) and
T
QQT 1
2 T
2 3
"
~ # 0 0
B3 = Q 0 gg2mn n QT ) QT B3 Q = 64 0
1
0 0 1 0 75 2
0 0 0
Theorem 6.8 and the above one suggest the possibility of omputing the singular
values of a matrix A as the eigenvalues of one of the matri es B1, B2 or B3, whi h
in turn, an be omputed by using the QR Algorithm. However, for omputing the
SVD of A the orthogonal matri es U and V are needed in addition.
For example, let B = B1 = AT A and let Q be the orthogonal matrix produ ed
by the QR algorithm
QT BQ = diag(1 ; 2 ; : : : ; n ) =: (6.95)
with i 0; i = 1 : n (be ause of AT A 0).
The rst step in obtaining the SVD of A is the orthogonal ordering of the diagonal
elements of . If Pij = [e1 ; : : : ; ei 1 ; ej ; ei+1 ; : : : ; ej 1; ei ; ej+1; : : : ; en ℄ is the (i; j )-
permutation matrix (whi h is orthogonal) then
PijT Pij = Pij Pij = diag(1 ; : : : ; i 1 ; j ; i+1 ; : : : ; j 1 ; i ; j +1 ; : : : ; n ) (6.96)
and any ordering algorithm an be used for obtaining
P T P = diag(12 ; 22 ; ; n2 ) (6.97)
with 1 2 : : : n. Dening
V = QP (6.98)
we have
V T BV = V T AT AV = diag(12 ; 22 ; : : : ; n2 ) (6.99)
where, as we have already seen, i, i = 1 : n, are the singular values of A and V is
one of the two orthogonal matri es dening the SVD of A.
For omputing the matrix U let i , i = 1 : r, be the nonzero singular values of A
1 = diag(1 ; 2; ; r ) (6.100)
and onsider the partition
h i
V = V1 V2 ; V1 = V (:; 1 : r): (6.101)
Then (6.66) be omes
" # " #
V1T AT A h V V i = 21 0 gr
0 0 gn r =: (6.102)
2
V2T 1 2
i.e.
V1T AT AV1 = 21 ; V2T AT AV2 = 0
or
1 1V1T AT AV1 = 1; AV2 = 0: (6.103)
Let
U1 = (1 1 V1T AT )T = AV1 1 2 Rmr : (6.104)
The matrix U1 has orthonormal olumns

U1T U1 = 1 1 V1T AT AV1 1 1 = Ir (6.105)
and if U2 2 Rm(m r) is a ompletion of U1 to an orthogonal matrix
U = [|{z}
U1 |{z}U2 ℄ (6.106)
r m r
then from (6.103) and (6.104) we have

" # " # " #
U T AV = U1T h
A V1 V2
i
= U1T AV1 U1T AV2 = U1T U1 1 0
U2T U2T AV1 U2T AV2 U2T U1 1 0
" #
= 01 00 gr
gm r (6.107)
|{z} |{z}
r n r
whi h is exa tly the SVD of A and onsequently the matrix U an be " omputed
#
from (6.104) and (6.106). More dire tly, from (6.107), the matrix 0 0 is1 0
upper triangular. So U T (AV ) an be seen as the orthogonal triangularization of
the matrix AV 2 Rmn . Sin e su h a triangularization does not ne essarily give a
diagonal ordered matrix it should be ompleted with a olumn pivoting strategy
(see [?℄).
The QR fa torization with olumn pivoting omputes a QR fa torization of a
matrix C 2 Rmn (m n) with permuted olumns
CP = QR; (6.108)
where P 2 Rnn is a permutation matrix, Q 2 Rmm is orthogonal and, in the
general ase, the upper triangular matrix R has the following stru ture
" #
R= R11 R12 gr ; r = rank(C ); R11 nonsingular: (6.109)
0 0 gn r
In order to obtain the fa torization (6.108)-(6.109) let us ompute an orthogonal
triangularization with olumn pivoting by using Householder re e tors. If, at step
s, we have omputed the re e tors U1 ; U2 ; ; Us 1 and the permutation matri es
P1 ; P2 ; ; Ps 1 su h that
"
C (s 1) C (s 1) # gs 1
Us 1 U1 CP1 P2 Ps 1 = Cs 1 =
0 C22(s 1) gm s + 1 (6.110)
11 12
where C11(s 1) is upper triangular and nonsingular. Now, suppose that

(s 1) = h (s 1) (s 1) i
(6.111)
C22 zs zs+1 zn(s 1)
is the olumn partition of C22(s 1) and let l 2 s : n be the smallest index su h that
kzl(s 1) k2 = max
i2s:n
(kzi(s 1) k2 ): (6.112)
Noti e that if r = rankC = s 1 then kzl(s 1) k2 = 0 and the fa torization is a hieved.
For improving the eÆ ien y of the fa torization method, noti e that there is no need
to ompute at ea h step the norms in (6.112) sin e the orthogonal transformations
leave them un hanged and
" #
QT z = w gg1k 1 ) kQT z k = 2 + kwk22 ) kwk2 = kz k 2
(6.113)
holds for any orthogonal matrix Q 2 Rkk and any z 2 Rk .
Permuting, at the urrent step s, the olumns l and s, we get nally that the
fa torization (6.108) is given by
Q = U1 U2 Ur (6.114)
P = P1 P2 Pr :
This algorithm was developed by Businger and Golub and is based on the following
omputational s heme.
1. Q Im
2. for j = 1 : n
1. j kC (1 : m; j )k2
3. r 0
4. Find the smallest l 2 1 : n su h that = l = maxj=1:n(j )
5. while > 0
1. r r + 1
2. pr l
3. C (1 : m; r) $ C (1 : m; l)
4. l $ r
5. Compute the re e tor Ur su h that Ur C (r + 1 : m; r) = 0
6. C Ur C
7. Q QUr
8. for j = r + 1 : n
1. j j 2rj
9. if r < n
1. Find the smallest l 2 r + 1 : n su h that l = maxj2r+1:n(j )
else
1. 0.
The ve tor p 2 Nr ontains all the olumn permutations. Clearly, the upper
triangular matrix R overwrites the matrix C and nally
jr11 j jr22 j jrrr j > 0: (6.115)
In our parti ular ase C = AV and the result of the orthogonal triangularization
with olumn pivoting
" #
QT CP = QT AV P = R11 R12 gr (6.116)
0 0
is an orthogonal upper triangular matrix
(QT AV P )T QT AV P = P T V T AT QQT AV P = P T P
= Pr P2 P1 P1P2 Pr = diag(p2 ; p2 ; ; p2 ; 0; ; 0)
1 2 r
(6.117)
whi h implies with (6.115) that pi = i; i = 1 : r.
In on lusion, a possible s heme for the SVD omputing is the following.
1. Compute B = AT A
2. Apply to the matrix B the symmetri QR Algorithm whi h omputes
the matrix V and = diag(1 ; 0)
3. Compute C = AV
4. Compute the matrix U := Q form the QR fa torization with olumn
pivoting of the matrix C , i.e., su h that CP = UR.
" #
The omputed matri es U; V and RP 1 = 01 (for m n) dene the SVD
of A.
However, the expli it omputation of the matrix B = AT A an lead to a loss of
information, as shown in the following example
2 3
1 1 "
1 : 000001 1
#
A = 4 0:001 0 5 ) A A =
6 7 T
1 1 : 000001 :
0 0:001
" #
In a oating point format with less than 7 de imal digits, we have AT A = 11 11 .
A better method for omputing the SVD has been proposed by Golub and Kahan
in 1965. Their algorithm determines U and V simultaneously by impli itly applying
the symmetri QR algorithm to B = AT A (i.e. working only on the matrix A without
expli itly forming the matrix B ).
This SVD algorithm has two stages.
1. The rst stage onsists of redu ing A to an upper bidiagonal form J su h that
the tridiagonal matrix T = J T J oin ides with the one produ ed by the rst step
of the QR symmetri algorithm applied to B .
2. The se ond stage onsists of iteratively redu ing the matrix J to diagonal
form by vanishing asymptoti ally the overdiagonal elements. This an be a hieved
using the bilateral orthogonal transformations that orrespond to an impli it QR
symmetri step applied to B .
Let us work now on some details.
Stage 1: Bidiagonalization
Suppose A 2 Rmn and m n. We show next how to ompute the orthogonal
matri es U 2 Rmm and V 2 Rnn su h that
2 3
d1 f1
6
6 d2 f2 7
7
6
6
6
... ... 7
7
7
J = U T AV = 6
6
6
... fn 1
7
7
7
(6.118)
6 7
6
6
dn 7
7
4 5
is bidiagonal. The matri es U and V will be omputed as produ ts of Householder

transformations
U = U1 U2 Un (6.119)
V = V2 V3 Vn 1
using the following omputational s heme
1. U Im
2. V In
3. for s = 1 : n
1. Compute the re e tor Us su h that (Us A)(s + 1 : m; s) = 0
2. A UsA
3. U UUs
4. if s < n 1
1. Compute the re e tor Vs+1 su h that (AVs+1)(s; s + 2 : n) = 0
2. A AVs
3. V V Vs
Noti e that at step s we have Us = diag(Is 1; U s) and the zeros reated in the
rst s 1 rows and s olumns are not destroyed. Similarly, Vs+1 = diag(Is; V s+1)
does not ae t the zeros reated in the rst s olumns and s 1 rows. Finally, the
matrix A is overwritten by the bidiagonal matrix
A Un U2 U1 AV2 V3 Vn 1 = U T AV: (6.120)
The detailed algorithm orresponding to the above s heme is given below.
Algorithm 6.11 JQ (Bidiagonalization) Given the matrix A 2 Rnn
the algorithm omputes the orthogonal matri es U 2 Rmm and V 2
Rnn and the bidiagonal matrix J su h that J = U T AV . The matrix J
overwrites the matrix A.
1. U eye(m)
2. V eye(n)
3. for s = 1 : n
% Compute Us, A qUPsA and U = UUs
1. sign(ass) mi=s a2is
2. uss ass +
3. for i = s + 1 : m
1. uis ais
4. s uss
5. ass
% A Us A
6. for i = s + 1 : m
1. ais 0
7. if s < n
1. for j = s P+1 :n
1. = ( mi=s uisaij )=s
2. for i = s + 1 : m
1. aij aij uis
% U UUs
8. for i = 1 P: mm
1. ( k=s uik uks)=s
2. for j = s : m
1. uij uij ujs
% Compute Vs+1, A AVs+1 and V V Vs+1
9. if s < n 1
% Vs+1 q
1. sign(as;s+1) Pnj=s+1 a2sj
2. vs+1;s+1 as;s+1 +
3. for j = s + 2 : n
1. vj;s+1 as;j
4. s+1 vs+1;s+1
% A AVs+1
5. as;s+1
6. for j = s + 2 : n
1. asj 0
7. for i = s P +1 :m
1. ( ni=s+1 vj;s+1aij )= s+1
2. for j = s + 1 : n
1. aij aij vj;s+1
% V V Vs+1
8. for i = 1 P: nn
1. ( j=s+1 vij vj;s+1)= s+1
2. for j = s + 1 : n
1. vij vij vj;s+1
The omputational eort is
n
X
Nop = 2p + 4(n s)(m s) + 4(m s)(n s) + 4m(m s) + 4n(n s))
s=1
n
X n
X n
X
= 2np + 8 (nm s(n + m) + s2 ) + 4m (m s) + 4n (n s)
s=1 s=1 s=1
= 2np + 8(n2m (n + m)n2=2 + n3=3 + 4m(mn n2=2) + 4n(n2 n2=2)
= 2np + 4n2 m 4n3=3 + 4mn(m n=2) + 2n3 :
Remark 6.12 A more eÆ ient algorithm for a umulating the transformations an
be obtained if (6.119) are implemented out of the main loop by a ba kward a u-
mulation. For the hairy details see [?℄.
Stage 2: Iterative diagonalization
The iterative diagonalization produ es a matrix sequen e
J1 = J; J2 ; ; Jk ; (6.121)
" #
onvergent to the diagonal matrix = 01 00 su h that the matrix sequen e
T1 = J1T J1 ; ; Tk = JkT Jk ; (6.122)
" #
is the QR symmetri sequen e with impli it shift onvergent to 01 00 2 Rnn,
2
a RSF of the tridiagonal matrix T = T1.

Suppose 2 3
d1 f1
6
6 d2 f2 7
7
6
6
6
... ... 7
7
7
Jk := J = 6
6
6
... fn 1
7
7
7
(6.123)
6 7
6
6
dn 7
7
4 5
has a bidiagonal form and apply an impli it shift QR algorithm step to the tridiag-
onal matrix
T := Tk = JkT Jk =: J T J: (6.124)
An impli it shift QR step onsists of
1. Computation of the shift
:= T (n; n) = d2n (6.125)
or, better, using the Wilkinson shift, whi h is the losest to d2n eigenvalue of the
matrix "
2 + d2
#
f d
T (n 1 : n; n 1 : n) = nd 2 f n 1 f 2 + d2 :
n 1 f n 1 (6.126)
n 1 n 1 n 1 n
2. Compute an orthogonal matrix U1 su h that U1 e1 is the rst olumn of the

expli it shift QR step transformation
2 3 2 3
t11 d21
6
6 t21 7
7
6
6 f1 d1 7
7
U1 e1 = 06
66 = 7
70 6
66
7
7: (6.127)
...
6
4
7
7
5
... 6
4
7
7
5
0 0
The matrix U1 an be a Givens rotation P12 su h that
2 3 2 3
d21
6
6 f1d1 7
7 0 777 1
6
6
6
P12T 66 0 7
7 = 0. 77 = e1:
6
6 (6.128)
6
4
... 7
7
5
6
.. 75
6
4
0 0
3. Compute
C = P12T T P12 (6.129)
that outlines an altered tridiagonal matrix in the position (3; 2).
4. Apply the HQ algorithm (adequately updated for the symmetri ase, i.e.,
the tridiagonalization algorithm) to the matrix C and obtain the matrix
T = QT CQ (6.130)
in tridiagonal form. The matrix Q an be a sequen e of rotations:
Q = P23 Pn 1;n ; (6.131)
su h that the new matrix is
T QT CQ = QT P12T T P12 Q = PnT+1;n P23T P12T T P12 P23 Pn 1;n : (6.132)
The idea of Golub-Kahan SVD step is detailed as follows.
1. Apply the above P12 Givens rotation to the matrix J instead of T
K = JP12 (6.133)
whi h alters the bidiagonal stru ture of the matrix J in the (1; 2) position.
2. Redu e the altered stru ture to the bidiagonal form by bilateral orthogonal
transformations.
J J 0 = Un 1 Un 2 U1 KV2 Vn 1 (6.134)
where Us, Vs an be Givens rotations (or Householder re e tors).
d1 f1 0
2 3
6
6 0 d2 f2 7
7
Example 6.13 Let m = 5, n = 3, J = 6
6 0 0 d3 7
7.
6
4 0 0 0 7
5
0 0 0
1. 2
0 32 3
2
03
6 0 7 s 0 6 ? 7
6 7 6 7
K = JP12 = 66 0 0 77 4 s 0 5 = 666 0 0 777 :
6 76 7
4 0 0 0 5 0 0 1 4 0 0 0 5
0 0 0 0 0 0
2.
2
?3 2
3 2
03
6 0 7 6 0 7 6 0 7
6 7 6 7 6 7
K U1 K = 666 0 0 777 ) K KV2 666 0 ? 777 ) K U2 K 666 0 0 777 :
4 0 0 05 4 0 0 05 4 0 0 05
0 0 0 0 0 0 0 0 0
From the above example it is lear that the omputational s heme for (6.134) is
for s = 1 : n 1
1. Compute the Givens rotation Us := Ps;s+1 su h that (UsK )(s + 1; s) = 0
2. K UsK
3. if s < n 1
1. Compute the Givens rotation Vs+1 := Ps+1;s+2
su h that (KVs+1)(s; s + 2) = 0
2. K KVs+1
The new matrix K = J 0 is bidiagonal and
T 0 = J 0T J 0 = (Un 1 U1 JP12 V2 Vn 1 )T = Un 1 U1 JP12 V2 Vn 1
= VnT 1 V2T P12T J T U1T UnT 1Un 1 U1JP12 V2 Vn 1 = QT J T JQ (6.135)
and
Qe1 = P12 V2 Vn 1 e1 = P12 e1
is the same as in the QR-step for the tridiagonal matrix T .
Consequently, Jk = J , dening the SVD sequen e is su h that Tk = JkT Jk is the
QR sequen e for the matrix B = AT A and, hen e, is onvergent to a diagonal form.
It is possible that the diagonal limit matrix = J1 has no ordered diagonal
elements. The ordering an be immediately a hieved by a suitable permutation
sequen e.
Algorithm 6.14 (Golub-Kahan SVD step) Given the ve tors d 2
Rn and f 2 Rn 1 that determine the bidiagonal matrix
2 3
d1 f1
6
6 d2 f2 7
7
6
6
6
... ... 7
7
7
J= 6
6
6
... fn 1
7
7
7
6 7
6
6
dn 7
7
4 5
the algorithm omputes the su eeding matrix J 0 = QT JQ in the SVD

sequen e. The matrix J 0 overwrites the given matrix J . The new ve tors
d0 and f 0 overwrite the ve tors d and f .
1. % Compute the Wilkinson shift
Æ (fn2 2 + d2n 1 fn2 1 d2n )=2
fn2 1 + d2n Æ+sign(Æ(f)pÆ12d+(f1 )

2
n n
)
1 dn 1 2
2. y = d1 ; z = f1d1
n
" # " #
3. % Compute ; s su h that PT y = 0 ?
12 z
p
1. r y2 + z2
2. y=r
3. s z=r
4. % Compute J JP12 . Let be the outbidiagonal nonzero element
1. d1 sf1
2. f1 sd1 + f1
3. d1
4. d2 d2
5. sd2
% Redu tion to bidiagonal form (6.134)
5. for k = 1q: n 1
1. r d2k + 2
2. dk =r
3. s =r
4. dk r
5. fk + sdk+1
6. dk+1 sfk + dk+1
7. fk
8. fk+1
9. if k < nq 1
1. r = fk2 + 2
2. = fk =r
3. s = =r
4. fk r
5. dk+1 sfk+1
6. fk+1 dk+1s + fk+1
7. dk+1
8. dk+2
The above SVD-step must be ompleted with the a umulation of the transfor-
mations. The number of ops required is
Nop ' 2np + 24n
0 ' 6mn
Nop
00 ' 6n2
Nop
where Nop0 and Nop00 are the ops required for a umulating U and V , respe tively.
Typi ally, after few SVD-steps of the above algorithm fn 1 be omes negligible and
the problem dimension de reases. The following riterion is used to ontrol the
pro ess of zeroing the supradiagonal entries
if jfij "(jdi j + jdi+1j) then fi = 0 (6.136)
and for zeroing the diagonal elements the usual riterion is used
if jdi j "kJ k then di = 0 (6.137)
where " is a small multiple of the unit roundo.
The SVD algorithm onsists of Golub-Kahan iterative steps applied on the ir-
redu ible part of the bidiagonal matrix J and simultaneously, based on the above
riterion, monitors the supradiagonal and diagonal zeros. The omputed SVD is an
exa t SVD for a slightly perturbed matrix A + E .
Algorithm 6.15 (The SVD Algorithm) Given a matrix A 2 Rmn
(m n) and the toleran e level " the above algorithm overwrites A
with U T AV = D + E , where U 2 Rmm , V 2 Rnn are orthogonal,
D 2 Rmn is diagonal, E satises kE k ukAk2 and u is the unit
roundo.
1. Bidiagonalization. Apply Algorithm 6.14 (JQ)
to ompute the bidiagonal matrix
A J = U T AV
2. while q < n
1. for i = 1 : n1
1. if jai;i+1 j "(jaii j + jai+1;i+1j)
1. ai;i+1 0
2. Find the largest q and2the smallest p su h3 that in the partition
A11 0 0 gp
A(1 : n; 1 : n) = 64 0 A22 0 75 gn p q
0 0 A33 gq
the bidiagonal matrix A22 is irredu ible
(i.e. with all the supradiagonal entries zero)
3. if q < n
1. for i = p + 1 : n q + 1
if aii 0
1. ai;i+1 0
else
1. Apply the SVD step (Alg. 9.1) to the matrix A22
2. A diag(Ip; U; Iq+m n )T A diag(Ip; V; Iq )
6.3. APPLICATIONS OF THE SVD 139
6.3 Appli ations of the SVD

6.3.1 On omputing the rank of a matrix
The omputed diagonal matrix is after ordering and without zeroing the diagonal
entries 2 3
1
6
6 2 7
7
6 ... 7
= J1 = UAV T = 6
6
6
7
7; (6.138)
6
6
n 777
4 5
where
1 2 n 0: (6.139)
If we onsider now negligible the singular values that satisfy a riterion like (6.137),
where the matrix norm is, for example, the Frobenius one
v
u n
uX
i "kkF = "t 2 i (6.140)
i=1
and if the index i = r + 1 is the rst whi h satises (6.140), the matrix be omes
2 3
1
6
6 2 7
7
6
6
6
... 7
7
7
6 7
6
6
r 7
7
= 6
6 0 7
7 (6.141)
6
6 ... 7
7
6 7
6
6
6
0 7
7
7
4 5
and the numeri al rank of A is

rankA = rank = r: (6.142)
Clearly, the rank is an intrinsi property of the matrix itself but, due to roundo in
a omputer one should always expe t to nd full rank matri es (i.e. r = n) unless a
threshold Æ is hosen below whi h elements of will be onsidered zero. Usually, the
threshold is hosen of order "kAkF (where " is the relative pre ision of the omputer
used).
6.3.2 The general least square problem
The general least square problem (GLS) onsists in solving overdetermined or un-
derdetermined linear systems
Ax = b (6.143)
whith rank de ient matrix A, i.e.,
rankA < min(m; n): (6.144)
Generally, in su h a ase the linear system (6.143) has no solution but has an innite
number of (pseudo)solutions in the least square (LS) sense. In fa t, the problem is
to ompute the minimum eu lidean norm (pseudo)solution
kx k = min minkb Axk kxk: (6.145)
x 2Rn
The following theorem shows how one an use the SVD to ompute this normal
(pseudo)solution.
Theorem 6.16 Suppose A 2 Rn , m n and rankA = r < n. If

U T AV =
is the SVD of A then the minimum norm pseudosolution to (6.143) is given by
r
X uTi b
x = vi (6.146)
i=1 i
where ui = Uei 2 Rm and vi = V ei 2 Rn are the i-th olumn of U and V ,

respe tively. Moreover, the minimum residue is
m
X
( )2 = kAx bk22 = (uTi b)2 : (6.147)
i=r+1
Proof. For any x 2 Rn we have

krk22 = kb Axk22 = kb U V T xk22 = kU (U T b (V T x))k22 :
Let us dene now
d = U T b; y = V T x:
Then " # 2
d(1 : r ) (1 : r; 1 : r )y (1 : r )

krk2 = kd yk2 =
2 2
d(r + 1 : m)

2
= kd(1 : r) (1 : r; 1 : r)y(1 : r)k22 + kd(r + 1 : m)k22 ;
whi h attains its minimum for arbitrary y(r + 1 : n) and
2 3
uT1 b=1
6 7
1 (1 : r; 1 : r)d(1 : r) = 6 uT2 b=2 7
y(1 : r) = 6
6 ... 7:
7
4 5
uTr b=r
Then the set of all minimizers is of the form
" #
x=Vy =V y(1 : r)
y(r + 1 : n)
among whi h, the minimum norm ones satisfy
y(r + 1 : n) = 0:
Consequently r
X uTi b
x = V (:; 1 : r)y(1 : r) = vi
i=1 i
and Xm
( )2 = kb Ax k22 = kd(r + 1 : m)k22 = (uTi b)2
i=r+1
Noti e that the omputational s heme is straightforward.
If m < n (underdetermined systems) the normal (pseudo)solution an be om-
puted in a similar way. The rank de ient underdetermined systems an have no
solution and onsequently we must dene and ompute a LS solution that is given
by (6.146).
6.3.3 Orthonormal basis for linear subspa es
The SVD of matrix A 2 Rmn produ es orthonormal basis for all linear subspa es
dened by A.
By theorem (6.2), if U T AV = is the SVD of A then
(a) S1 = ImA = ImU (:; 1 : r) Rm
(b) T1 = KerA = ImV (:; r + 1 : n) Rn (6.148)
( ) S2 = KerAT = ImU (:; r + 1 : m) Rm
(d) T2 = ImAT = ImV (:; 1 : r) Rn :
The proof of the rst two equalities is given in Proposition 6.6; ( ) and (d) follow
immediately from (a) and (b), sin e U and V are orthogonal; however, we present
here dire t proofs for the last two equalities.
( ) Suppose that x 2 KerAT . Then

AT x = 0 ) (U V T )T x = 0 ) V T U T x = 0 ) T U T x = 0
" #" #
0 U (: ; 1 : r ) Tx
) 0 0 U (:; r + 1 : m)T x = 0 ) 1U (:; 1 : r)x = 0 ) (U (:; 1 : r))T x = 0:
1
But x 2 Rm and the olumns of U form a basis of Rm . Consequently, there exists

y su h that
" # " #
( U (: ; 1 : r )) Tx
x = Uy ) y = U x = y(r + 1 : m) = y(r + 1 : m)
T 0
" #
)x=U 0
y(r + 1 : m) = U (:; r + 1 : m)y(r + 1 : m)
i.e.,
x 2 ImU (:; r + 1 : m) ) KerAT ImU (:; r + 1 : m):
Conversely, if
x 2 ImU (:; r + 1 : m) ) x = U (:; r + 1 : m)z ) AT x = V T U T U (:; r + 1 : m)z
" #
=V T U1T U2 z =
U2T
" #
=V T 0 = 0 ) x 2 KerAT ) ImU (:; r + 1 : m) KerAT :
Im r z
(d) Suppose x 2 ImAT . Then
" # " #
x= ) x = V 01 00 U T z = V y0 = V (:; 1 : r)y
AT z
) x 2 ImV (:; 1 : r) ) ImAT ImV (:; 1 : r)

Conversely, if
" # "
y g r 1y #
x 2 ImV (:; 1 : r) ) x 2 V (:; 1 : r)y = V 0 gn r = V 1
0
1 =
" # " #" #
= V 01 1 1y = V 01 00 1 y
1
"
1y #
= V U U 1 = AT z ) ImV (:; 1 : r) ImAT :
T
6.3.4 The pseudoinverse omputation
As we have already seen, the pseudoinverse of a full rank matrix A 2 Rmn is dened
by
m > n A+ = (AT A) 1 AT
m = n A+ = A 1 (6.149)
m < n A+ = AT (AAT ) 1 :
The general denition is as follows.
Denition 6.17 Let A 2 Rmn . The pseudo-inverse of A is the unique matrix
X 2 Rnm that satises the four Moore-Penrose onditions
(i) AXA = A
(ii) XAX = X (6.150)
(iii) (AX )T = AX
(iv) (XA)T = XA:
It is straightforward to he k these onditions for the full rank ase.
In the de ient rank ase, let U T AV = be the SVD of A.
Then
" #" #
A = U V T = [|{z}
U2 ℄ 0 0 0 V T
V2T ; where V = [|{z} V2 ℄ ) A = U1 1 V1T :
U1 |{z} 1 1 V1 |{z}
r m r r n r
It is easy to show that the pseudo-inverse of is

" #
X = + = 1 0 ggrn
1
0 0 r 2 Rnm
(6.151)
|{z} |{z}
r m r
Theorem 6.18 Let A 2 Rmn and U T AV = its SVD. Then the pseudo-inverse
of A is
A+ = V + U T : (6.152)
Moreover, X = A+ is the unique minimal F -norm solution to the problem
min kAX
X 2Rnm
I m kF (6.153)
and for any ve tor b the solution to the GLS system Ax = b an be written as
x = A+ b: (6.154)
Proof. The matrix dened in (6.151) veries all four Moore-Penrose onditions.
Indeed,
(i) AA+ A = U V T V + U T U V T = U +V T = U V T = A:
(ii) A+AA+ = V +U T U V T V +U T = V +U T = A+:
(iii) (AA+ )T = (U V T V +U T )T = (U +U T )T = U (+)T U T =
= U +U T = U V T V +U T = AA+:
(iv) (A+ A)T = (V +U T U V T )T = V (+)T V T = V +V T =
= V +U T U V T = A+A:
The Frobenius matrix norm is preserved under orthogonal transformations. Conse-
quently
rF (X ) = kAX Im kF = kU T AV V T X U T kF = kV T X U T kF
" #" #
= k V T XU Im kF =

1 0 Y11 Y12
Im =

0 0 Y21 Y22 F
" #
=1Y11 Ir

;

1 Y12

0 Im r F
where Y := V T XU 2 Rnm and the partition of Y is onformaly to the partition
of . Clearly, rF (X ) is minimum when1 both k1Y12 kF and k1 Y11 Ir kF are
minimum, i.e., for Y12 = 0 and Y11 = 1 . Hen e, all the matri es X 2 Rnm that
minimizes rF (X ) are of the form
" #
X = V Y UT =V 1 1 0 T
Y21 Y22 U :
" #

But kX kF = Y12 Y22

1 1 0
is minimum for Y21 = 0; Y22 = 0. Consequently the

F
minimal F -norm solution for the problem (6.153) is X = V +U T = A+ .
Finally, rewrite the solution (6.146) of the GLS problem in the matrix form
2 3 2 3
uT1 b=1 uT1 b=1
x = v1
h
vr
i
6
4
... 7
5 = V (:; 1 : r) 64 ... 7
5 =
uTr b=r uTr b=r
2 3
uT1 b=1 2 3
6 ... 77 6 1 1 .
6
6
6
T b= 7
7 6
6
.. 7
7
7
u
0 777 = V 666 r 1 = V +U T b = A+b
6 r 7 6 7 T
V r6 7U b
6
6
6 ... 75 4 ... 7
7
5
4
0 0
Problems

P 6.1 Whi h are the singular values of A = 11 11 ?
P 6.2 Show that kQAk = kAV k = kAk , where A 2 , and Q 2 R
m n m
R
m
,V 2 n
R
n
are orthogonal.
F F F
p
P 6.3 Show that if A 2 , then kAk kAk rank(A)kAk .
R
m n
2 F 2
P 6.4 Prove that if is the largest sigular value of A, then

1
y Ax
1 = 2Rmax
T
m 2Rn kyk2kxk2 :
y ;x
P 6.5 Show that if A 2 , m n, and rank(A) < m, then the SVD of A is not unique
R
m n
(however, noti e that the singular values of A are unique).

Give expli it formulae for omputing the SVD of A = .
P 6.6
Æ
P 6.7 Whi h is the SVD of A = uv , where u 2 , v 2 ? Whi h is the rank of A ?
T
R
m
R
n
P 6.8 Give an algorithm to ompute the SVD of A = I + uv , where u; v 2 are two T
R
n
non olinear ve tors.

n
P 6.9 Show that if A 2 has rank n, then kA(A A) A k = 1.

R
m n T 1 T
2
P 6.10 Prove that the ondition number of a nonsingular matrix A 2 , (A) := R

n n
2
kAk kA k = = , where and are the largest and the smallest singular values of
2
1
2 1 1
A.
n n

Let A = A1 A2
, where A 2 , A 2 . Sket h an algorithm (equiv-
0 A3
m n p p
P 6.11 R 1 R
alent to algorithm JQ) to eÆ iently redu e A to bidiagonal form by bilateral orthogonal

transformations.
Solutions to the problems
P1.1 The result depends on the order of omputation; y1 = (x1 + x2 ) + x3 = 0, y2 =
x1 +(x2 + x3 ) = 0:001. The exa t result is y2 (relative error equal to 0). For y1 , the relative
error is j0 0:001j=0:001 = 1 (i.e. 100%).
P1.2 The omputed result is y = fl(fl(x1 + x2 ) + x3 ) = fl((x1 + x2 )(1 + 1 ) + x3 ) =
[(x1 + x2 )(1 + 1) + x3 ℄(1 + 2), with j1 j; j2j , and of order unity. It follows that: t
jy yj 1 + jx + x j : 1 2
jx + x + x j
t
jy j 1 2 3
P1.3 Suppose that input data are ae ted by errors, so that (a + a; b + b) is used
instead of (a; b). The result will be x + x. From (x + x)(a + a) = (b + b), negle ting
ax, it results that x=x = a=a b=b. Thus, the problem is always well onditioned
(small relative input errors imply small relative output error).
Sin e x = fl( b=a) = ( b=a)(1 + ) = b(1 + )=a = b=a, with jj , the t
algorithm is numeri ally stable (b is lose to b).

P1.4 the problem inherits the ill onditioning of the sum (for example, when ja + a j 1 2
is small and ja j, ja j big). The "algorithm" x = (b + b )=(a + a ) is stable.
1 2 1 2 1 2
P1.5 The next oating point number is x = 0:100 : : : 01 ; then, x 1 = (the 1 t+1
relative error bound for hopped rounding).

P1.6 " 0:5
r . t+1
P2.1 b. Hint: C = AB is lower tridiagonal. . The algorithm is:

1. x 1 b1=a11
2. for i=2:n
1. x (b i i a i;i 1 x
i 1 )=a ii
d. Hint: x = 0, for i = 1 : j 1. e. Solve rst Ay = , then Bx = y. f. The inverse

is lower triangular:
i
1. for j=1:n
1. x 1=a
2. i = j + 1 : n
jj jj
for
1. x a ij i;i 1 x
i 1;j =a ii
147
P2.2 b. Hint: only here T must be omputed; T is tridiagonal and symmetri ; that
is, only the diagonal and the rst under diagonal must be omputed. see the previous .
problem. ompute X = A like in the previous problem, and then T = X X ; X is
d.
1 1 T
lower triangular, X is upper triangular and T is symmetri . T = AA is not already

T 1
e.
T
the Cholesky fa torization, sin e the diagonal elements of A are not ne essarily positive. Let
T = LL be the Cholesky fa torization. It is very natural to try to prove that L is lower
T
bidiagonal (then, by uni ity, L is the Cholesky fa tor). For any s 2 1 : n 1:

t =a +a =l +l 2 2 2 2
=a a =l l :
ss s;s 1 ss s;s 1 ss
t s+1;s s+1;s ss s+1;s ss
It is again natural to try to prove that jl j = ja j; for the rst relation above, the signs are
not important. For the se ond, if a is negative, we take l = a and l = a .
ij ij
s+1;s s+1;s
The algorithm will be:

ss ss ss
1. for s=1:n
1. l ja j
2.
ss ss
if s < n then
1. if a < 0 then l +1
ss s ;s a +1
s ;s
else l +1 a +1
s ;s s ;s
P2.3 a.It is obvious that the multipliers will be zero for i > j + 1. Then, the
Gaussian elimination will have the form:
ij
1. for s=1:n 1
1. h h =h
s;s+1 s+1;s
2. j = s + 1 : n
ss
for
1. h h s+1;j s+1;j h +1 h
s ;s sj
The ve tor b will be modied a ording to these parti ular values of the multipliers:
1. s = 1 : n 1
for
1. b b
s+1 h b s+1 s+1;s s
Then, an upper triangular system must be solved.

b.Hint: the partial pivoting does not hange the upper Hessenberg stru ture. .
Noti e that L is lower bidiagonal.
P2.4 b. Hint: ompute y = Rb, then Hy. Solve Hy = b, like in the previous .
problem, then Rx = y.
P2.5 a. If b = + id, where ; d 2 , the system an be written as A[y z℄ = [ d℄,
R
n
where y; z 2 and x = y + iz. It is a parti ular ase of , for m = 2.

R
n
b
the matrix equation AX = B onsists of m linear systems: Ax = b , for j = 1 : m

b.
(x and b are the j -th olumn of X and B, respe tively). Using the algorithm:
j j
j j
1. j = 1 : m
for
1. use LSS GPP to solve Ax = b j j
148
is not a good idea, sin e the number of operations would be 2mn =3. Instead, GP P an be 3
used only on e to triangularize A, like in the following:

1. [M; U; p℄ = GP P (A)
2. j = 1 : m
for
1. s = 1 : n 1
for
1. b $ b p(s);j
1. i = s + 1 : n
sj
for
1. b b b
2. x = UT RIS (U; b )
ij ij is sj
j j
The number of operations is 2n =3 + O(mn ). 3 2
P2.6 Again, it is not a good idea, nor to ompute A (2kn ops), neither to use the k 3
algorithm based on the idea that A(A x) = b, applied re ursively: k 1
1. j = 1 : k
for
1. use LSS GPP to solve Ax = b
2. b x
whi h implies 2kn =3 ops. Again, GP P an be used only on e to nd the solution of all
3
the systems in the statement 1.1 of the s heme above. Thus, this s heme is modied to:
1. [M; U; p℄ = GP P (A)
2. j = 1 : k
for
1. s = 1 : n 1
for
1. b $ b p(s)
1. i = s + 1 : n
s
for
1. b b b
2. b = UT RIS (U; b)
i i is s
3. x b
The number of operations is only 2n =3 + O(kn ). 3 2
P2.7 a. Suppose that:

L~ = X L 0 ; U~ = U Y :
L 0 U
Then
B = L~ U~ = XU A LY
XY + A :
Thus LY = 0 and, be ause A nonsingular implies L, U nonsingular, Y = 0; XU = R, thus
X = RU , and X is upper triangular.
1

A 0 x = d ) Ax = d
b.
1 1 1 1
R A x d Rx + Ax = d
2 2 1 2 2
First solve Ly = d , Ux = y and x is obtained (2n ops). Then ompute f = d Rx

1 1 1
2
2 1
(n ops); then solve Ly = f , Ux = y and x is obtained (2n ops). The total is of
2
2 2
2
149
only 5n ops. The above omputational s heme an be applied to solve any blo k lower
2
triangular systems.
P2.8 a. Use Gaussian elimination; a = 0, for i > j + n; the multipliers will respe t
the same relation. Use Gaussian elimination with partial pivoting, whi h will not ae t
ij ij
b.
the stru ture of A.
P2.9 a.
1. for s=1:n 1
1. a s+1;s a +1 =a
2. a
s ;s ss
s+1;s+1 a +1 +1s ;s ; a +1 a
s ;s s;s+1
P3.1 a. We present only an algorithm for orthogonal triangularization (adaptation of

ORT HT ). With Householder re e tors (U = I (u u )= , with u = [0 : : : u u 0 : : : 0℄):
s s
T
s s
T
s ss s+1;s
1. s = 1 : n
for
q
1. sign(a ) a + a ss
2
ss
2
s+1;s
2. u a + , u a s+1;s s+1;s
3. u
ss ss
4. a r =
s ss
5. j = s + 1 : n
ss ss
for
1. (u a + u a )= s+1;s s+1;j
2. a a u
ss sj s
2. a
sj sj ss
as+1;j u s+1;j s+1;s
With Givens rotations:

1. s = 1q: n
for
1. t a +a 2
ss
2
s+1;s
2. a =t, d a =t s+1;s
3. a r = t
s ss s
4. j = s + 1 : n
ss ss
for
1. t a + d a s+1;j
2. a d a + a
s sj s
s+1;j s+1;j
3. a t
s sj s
sj
In this ase, the se ond variant is better (the reverse situation is met for general matri-
es).
P3.2 Both the use of Householder re e tors or Givens rotations will destroy the lower
Hessenberg stru ture (the algorithm will be identi al to ORT HT ).
P3.3 a. Hint: elementary re e tors U = I (u u )= , with T
u = [0 : : : 0 u 0 : : : 0 u : : : u ℄ will be used.
s s s s
T
s ss n+1;s ms
b.u = [0 : : : 0 u 0 : : : 0 u : : : u ℄, where p = min(s + n; m).

T
s ss ps ms
P3.4 Sin e A = Q0R0 and R0 is nonsingular, ImA = ImQ0 (and Q0 has orthogonal
olumns). KerA = (ImA)? ; but (Q00 ) Q0 = 0, be ause U is orthogonal; thus Q00?Q0 and
T T T
ImQ00?ImA.
150
P3.5From q q = 0, for j > 1, prove rst that q = 0 (Q is upper triangular). Continue
T
1j
by nite indu tion.
1 j
P3.6 Let U 2 be a Householder re e tor su h that Ux = e , or x = U e . Then,

R
n n
1
T
1
U = U = [x Y ℄, and Y = U (:; 2 : n) is the desired matrix.
T
P3.7 Let B = Q A~ = [Q A Q z℄ = [R y℄ 2 . B is already upper triangular,

T T T
R
m (n+1)
unless its last olumn, y = Q z. Let U 2 be the Householder re e tor for whi h
T m m
(U y) = 0, for i > n. Then, U B = [R U y℄ = R~ is upper triangular; U Q A~ = R~ implies

n R
T
that A~ = QU R~ and then Q~ = QU .

n i n n n
n n

P3.8
1 0 ~ w
Take Z = 0 Q . Then, Z A = R is upper Hessenberg. The QR
T
fa torization of this matrix an be omputed as in problem 3.1.

P3.10 a. Let Q , Z be elementary re e tors of order n and index 1 su h that Q u =
1 1 1
Z v = kuke . Then, Z Q u = v and Q = Z Q is the desired orthogonal matrix. Attention:
1 1
T
1
T
1
Q is not omputed using matrix multipli ation. If Z = I 2zz and Q = I 2qq (we
1 1
T T
1 1
take kzk = kqk = 1), then Z Q = I 2zz 2qq + 4z(z q)q . 1
T
1
T T T T
b.The rst olumn of Z has the same dire tion as v (Z e = v=kuk). Consider an other
1 1 1
olumn of Z , denoted by y; obviously, y?v. Then, use the algorithm from to ompute
1 a
an orthogonal W su h that W u = yP; hen e, W u?Pv.
P3.11 a. x = (a a) a b = ( T
a b )= 1
a . There is no numeri al problem
T n n 2
in using the pseudo-inverse, sin e a a is a s alar.

i=1 i i i=1 i
T
b.The orthogonal proje tion of a onto Imb is by, where y 2 is su h that kby ak is R
minimum. Of ourse, y = (b b) b a. T 1 T

T = D 1(L + U ) = 0 a12=a11 , thus the eigenvalues of T are
P4.1
p a12 =a22 0
1 2 = a12 = a11 a22 . Sin e A is positive denite, then y Ay > 0 for y = [x 1℄; then, T T
a11 x2 + 2a12 x + a22 > 0 for any x 6= 0 and the dis riminant 4(a12 a11 a22 ) is negative. In
;
on lusion, j1 2j < 1. ;

P
P4.2 k D 1 (L + U )k1 = max =1: =1 ja =a j < 1. j n
n
i ij jj

P4.3 For Ja obi's method: T = D 1 (L + U ) = 1=02 10=2 , so that (T ) = 1=2.
J J

For Gauss-Seidel method, T = (L + D) 1U = 00 11==42 ; thus, (t ) = 1=4. Sin e
GS GS
(T ) < (T ), the Gauss-Seidel method onverges faster.

GS J
P5.1 Suppose 2 (A) and

A1 A2 x1 = x1 :
0 A3 x2 x2
If x 6= 0, then A x = x and 2 (A ); if x = 0, then A x = x , and 2 (A ).
2 3 2 2 3 2 1 1 1 1
Hen e, (A) (A ) [ (A ). But (A) and (A ) [ (A ) have the same ardinality.
1 3 1 3
Another way to solve this problem is dire tly from det(I A) = det(I A ) det(I 1
A ).
3
151
P5.2 The eigenvalues of A are both equal to 1. The unique asso iated eigenve -
p 1
tor is x = (1= 2) 1 (with kxk = 1). Taking Q = [x y℄, where x y = 0, thus T
p
y = (1= 2) 11 (with kyk = 1, too), it results that Q AQ = 01 41 . Noti e T
that the S hur form an be obtained, although A is defe tive.

P5.3 The eigenvalues of A are not real, thus a RSF of A is quite A.
P5.4 Only two ve tor variables are required. The entry data are x { the initial approx-
imation (of unit norm), " { the toleran e, N { the maximum number of steps; here is the
power method:
1. k=1:N
for
1. y Ax
2. y y=kyk
3. ky xk < "
if then
1. x y, stop
2. print('The method is not onvergent')
P5.5 For A, the result is e , sin e Ae = 5e and (A) = f5; 2; 1g. For B, the method
1 1 1
will not onverge, be ause there is no dominant eigenvalue: (B) = f1; 1g.
P5.7 a. If H = UAU , with U 2 , orthogonal, then (A + I )x = b is equivalent
T n
R
n
to (UAU + I )Ux = Ub .
k k k
T
k k k
b.The algorithm is:

1. ompute orthogonal U su h that UAU = H , upper Hessenberg T
2. k = 1 : p
for
1. d Ub
2. solve (H + I )y = d
k
3. x U y
k
T
k
The main omputational eort is in step 1, i.e. O(n ) ops. In 2.2, the solution of 3
an upper Hessenberg system may be obtained in O(n ) ops, see . The total is of
2
P2.3
O(n + pn ) ops.
3 2
. If Gaussian elimination is used p times to solve (A + I )x = b , then O(pn ) 3
operations will be ne essary.

k k k
P5.8 b. For larity, we onsider only the ase when n is even. First, transform B by sim-
ilarity, applying the elementary permutation matri es P ,P , ..., P
n+1;2n , n+2;2n 1 3n=2;3n=2+1
obtaining the matrix C = QAQ whi h has nonzero elements only on the se ondary diag-
T
onal; more pre isely, these elements are, from the upper right to the lower left orner: d , 1
d , ..., d , d , ..., d , d .
2 n n 2 1
Then, applying the EPMs P , P , . . . , P , C is brought to a matrix with

2;2n 4;2n 2 n;n+2
2 2 diagonal blo ks, i.e. diag(D ; D ; : : : ; D ; D ). The permutation of this diagonal blo ks
1 3 4 2
to the desired form an be now easily performed (with a sorting algorithm).
152
P5.9 b. the eigenvalues are those of A and 1 with multipli ity n 2. If 2 (B) and
x is the asso iated eigenve tor, then:
x1 + x = x1 ;
x1 + x = x ;
n
x = x ; for i = 2 : n 1:
n n
i i

If 2 (A), then the rst two relations are fullled for the eigenve tor of A, y = 2 x1
xn
R , and x = 0, for i = 2 : n 1. Otherwise, = 1 and x = x = 0 (and the eigenve tors

2
1
are, for example, e ; : : : ; e ).
i n
2 n 1
If 2 (A) and Ax = x, then BAx = Bx, A(Bx) = (Bx) and, sin e the
P5.11
eigenvalues of A are distin t, A has a omplete set of linearly independent eigenve tors and
Bx = x (Bx is an eigenve tor both for A and B ). Sin e the redu tion to S hur form based
on the de ation pro edure is done using eigenve tors, the same similarity transformation
brings A and B to the S hur form (similar arguments hold for the real ase).
The idea is the same as for the orthogonal redu tion in Theorem 5.17, but using
P5.12
ELTs.
Idea: nd U , elementary re e tor, su h that U z = kzke . Compute A0 =
P5.13 1 1 1
U AU . Then, transform A0 to Hessenberg form H = Q0 U AU (Q0 ) ; Q = Q0 U is the de-
1
T
1
T T
1
sired orthogonal transformation, sin e Q0 = U : : : U , and thus Qz = U : : : U kzke =
1
1 2 1 2 1
kz ke .
n n
P5.14 a. H I is upper Hessenberg; like in , its QR fa torization an be P3.1

omputed using a sequen e of Givens rotations P P : : : P (H I ) = R, where 12 23 1;n
Q=P : : : P . The matrix multipli ation RQ an be performed without expli itly
n
T T
1;n 12
omputing Q, by RQ = RP : : : P . The s heme of the QR step is:
n
T T
n 1;n 12
1. h
2. H H I
nn
3. ompute Givens rotations P , . . . , P su h that P : : : P H = R

12 1;n 12 1;n
4. H RP : : : P
n n
T T
5. H H + I
n 1;n 12
All omputations ould be performed in pla e, in A.

Obviously, He
b. ~ = e , and H (Qe ) = (Qe ). Hen e Qe is the desired eigenve -
1 1 1 1 1 1 1
tor.
If Q = Q is the Householder re e tor for whi h Qu = kuke , then A0 =
P5.16
T
1
Q AQ = I + kuke v Q is upper triangular and a0 = 1 for i 2; thus, 1 is an eigenvalue
T
1
T T
of multipli ity n 1. a0 = 1 + kukv q , where q = Qe is the rst olumn of Q. But

ii
T
1 1 1
q = u=kuk, hen e a0 = 1 + v u; this is the n-th eigenvalue of A.
11
T
1 11
The matrix is symmetri , then there exists an orthogonal Q su h that Q AQ = D

P6.1
T
is diagonal; this is also the SVD of A (up to a reordering of the eigenvalues). Hen e the
singular values of a symmetri matrix are its eigenvalues; in our ase, = 2, = 0. 1 2
P Let denote B := A A, C := AA . It is easy to prove that kAk = tra e(B) =

P6.2
T T 2
b and also kAk = tra e(C ). Then, kQAk = tra e(A Q QA) = tra e(B ) = kAk
F
n 2 2 T T 2
and kAV k = tra e(C ).

i=1 ii F F F
2
F
153
P6.3 kAk = P r , where r = rank(A) and kAk = .
2 r
i=1
2
i
2
1 2 1
Hint: if A = U V is the SVD of A, then (U , V are nonsingular):

F
T
P6.4
z x
max y Ax = 2Rmax z U AV x
m 2Rn kUz k kV wk = 2Rmax
T T T T
y2Rm 2Rn kyk kxk ;x 2 2

m 2Rn kz k kwk :
z ;w 2 2 z ;w 2 2
P6.5 Sin e the singular values of A are the eigenvalues of A A, they are uniquely T
dened, and so is (whose diagonal is ordered). But U and V may not be unique. For
example, if rank(A) < m, then in (6.106) U is uniquely dened, but U is hosen as an 1 2
orthogonal ompletion to U and may take several values (even when rank(A) = m 1,
1
there are two possibilities: U = w or U = w, where w 2 is the unique unit norm
2 2 R
m
ve tor su h that U w = 0). T

1
Another situation whi h allows multiple denitions of U and V is when several singular
values are equal.
P6.7 If U 2 , V 2 are Householder re e tors su h that U u = kuke 2
1 R
m m
1 R
n n
1 1
R
m
, and V v = kvke 2 , then:
1 1 R
n

U AV = kukk vk 0 2 :
0 0
T m n
1 1 R
Obviously, rank(A) = 1 if u 6= 0, v 6= 0, and zero otherwise.

P6.8 Let w := Qv?u, where Q is omputed like in problem 3.10, and w := w= kwk,
u := u=kuk. Compute an orthogonal C 2 su h that U = [u w C ℄ and V = [w u C ℄
R
n (n 2)
are orthogonal (use the QR fa torization of [u w℄; see also problem 3.6). Then
2 3 2 3
u (I + uv )w u (I + uv )u 0
T T
u uv w 1 + u uv u 0
T T T T T T
U (I +uv )V = 4 w (I + uv )w w (I + uv )u 0 5 = 4 1

T T T T T
0 0 5T
0 0 I 0 0 I n 2 n 2
and the problem has been redu ed to the 2 2 ase (anyway, n 2 singular values of A are
equal to 1).
P6.10 A = (U V ) = V U , and thus the singular values of A are
1 T 1 1 T 1
f ; : : : ; g. Then, kA k = 1= .
1
1
n
1 1
2 n
P6.11 Using Householder or Givens transformations, one an introdu e zeros as follows

(m = 7, n = 6, p = 3; we use Givens rotations, for larity):
2 3 2 3

6 7 6 0 7
6 7 6 7
6 7 6 0 7
6 7 6 7
A=6
6 77 ; A P P A = 66 77 ; 12 13
6
6 77 6
6 77
4 5 4 5

2 3 2 3
0 0 0 0
6 0 7 6 0 7
6 7 6 7
6 0 7 6 0 7
6 7 6 7
A AP P = 6
6 45 77 ; A P P P A = 66
46 77 ; 45 46 47
6
6 77 6
6 0 77
4 5 4 0 5
0
154
2 3
0 0 0 0
6 0 77
6
6 0 77
6
A AP P = 6
6
23 ? 0
24 77 :
6
6 0 77
4 0 5
0
After these transformations A is bidiagonal on the rst row. A similar problem is
obtained, but of dimension (m 1) (n 1); the square p p blo k shifted one diagonal
position.

333

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

333

Uploaded by

Copyright:

Available Formats

1

\Politehni a" University of Bu harest

3 The linear least squares problem 59

Main features of numeri al

1.1 The oating point representation

Example 1.4 In ten based numeration system the two sequen es

-1.5 -1 -0.5 0 0.5 1 1.5 -

01001100 01101001 10111000 00001011

As the interval [ M; M ℄ is an in nite set of real numbers, ea h x 2 F repre-

1.2 Computer arithmeti

1.3 Numeri al ondition and numeri al stability

where a; b 2 R are two given numbers

where j1 j; j2 j  2 t (1 =  + 1 + 1 ).

Dire t methods for solving

where aij 2 R, bi 2 R, for i = 1 : m, j = 1 : n, are given and xj , j = 1 : n, are the

2.1 Types of linear systems

Finding a solution to the linear system (2.2) means to ompute a ve tor x 2 Rn

 Iterative methods, based on the re ursive building of a ve tor sequen e onver-

2.2 Triangular systems

In general, if we know x1, x2, .. ., xi 1, we an solve the i-th equation:

Proposition 2.13 a) An ELT matrix is nonsingular and

(M1a1 )i = a0;11 ; for i=1

Step s. Let suppose that the matrix

After step s Final ontents

The required number of ops is:

2.4 Pivoting strategies

(APij )ek = : Aej ; for k = i

2.4.1 Partial pivoting

must hoose is  s. See gure 2.5.

1. Determine is so that jai sj = maxi=s:n jaisj.

2. Inter hange the is-th and s-th rows, i.e. A PsA.

Figure 2.5: Step s of Gaussian

3. Determine the orresponding ELT matrix Ms.

3. for j = s : n finter hange rows s and isg

nonsingularity of all matri es in the right hand side of (2.17). 2

1. Find is and js so that jai j j = maxi=s:n;j=s:n jaij j.

2. Inter hange the is-th and s-th rows, i.e. A PsA.

In GCP, if ai j = 0, then all aij , i = s : n, j = s : n, are zero and the algorithm

2.5 Solving determined linear systems

2.6 Matrix inversion

Let L 2 Rnn be a lower triangular matrix. Assume that L is nonsingular, i.e.

where L(11j) is the (j 1) (j 1) upper left submatrix of L and

The number of ops required is (exer ise: prove this result):

The ne essary memory is MLINV  n2=2 oat lo ations.

Figure 2.7: Data storage in matrix inversion.

In this way, a matrix inversion an be done with only two additional (n 1)

2.7 Determinant al ulus

2.8 LDU and LU matrix fa torizations

The following important fa t shows under whi h onditions a LDU fa torizations

where S is a sum of matrix terms of the form:

based on the idea of su esive omputation of a olumn of L and of a row of U . The

where the only unknown s alar is lis. Hen e,

The oating point operations asimptoti ally totalize NCROUT  2n3 =3 as in G,

2.9 Cholesky fa torization

De nition 2.45 A square symmetri (A = AT ) matrix A is alled positive de nite

Obviously, x 6= 0 implies y 6= 0, and from the positiveness of A:

for the rst olumn we have:

Formulas (2.45) and (2.47), (2.48) de ne the algorithm given below:

2.10 Appli ations of the LU fa torizations

If a nonsingular matrix A has a LU de omposition, then

2.11 Numeri al ondition of linear systems

To make use of matrix norms independently of matrix dimensions, we introdu e

Then, k k : S1m=1;n=1 Rmn ! R+ is a onsistent family of matrix norms.

i.e., respe tively: m

As the interval [ M; M ℄ is an innite set of real numbers, ea h x 2 F repre-

where j1 j; j2 j 2 t (1 = + 1 + 1 ).

Iterative methods, based on the re ursive building of a ve tor sequen e onver-

must hoose is s. See gure 2.5.

The ne essary memory is MLINV n2=2 oat lo ations.

The oating point operations asimptoti ally totalize NCROUT 2n3 =3 as in G,

Denition 2.45 A square symmetri (A = AT ) matrix A is alled positive denite

Formulas (2.45) and (2.47), (2.48) dene the algorithm given below:

Then, k k : S1m=1;n=1 Rmn ! R+ is a onsistent family of matrix norms.

. If A is symmetri and positive denite, adapt the Cholesky fa torization algorithm

Denition 3.12 An elementary re e tor of order m (or Householder transforma-

where U 2 R(m s+1)(m s+1) is an elementary re e tor of order m s + 1.

Denition 3.16 A matrix Pij 2 Rmm of the form:

P 3.8 Let A 2 , m n, a rank n matrix whose QR fa torization is known (A = QR,