Ill-conditioned matrices

 The condition number of matrix

 Norm of a vector
 Norm of a matrix
 Condition number via matrix norm
 Estimation of errors in solving systems of
linear algebraic equations

A.V. Dyskin, CRE, UWA GENG2140 Slide 49

Points to note
 There exist systems of linear equations that
• Theoreticallyy have unique
q solution but
• Practically cannot be solved using a computer with the
given accuracy
 Method of identification of such systems-
condition number
 Determination of condition number
• Norm of a vector
• Norm of a matrix
• Condition number
 Meaning of condition number

A.V. Dyskin, CRE, UWA GENG2140 Slide 50

1
 Detection of internal structure
Transmitter/receiver
vp(1)=2.564 km/s, vs(1) =2.190 km/s
flection.

vp(2)=3.552 km/s, vs(2) =3.034 km/s

x1 vp((1)), vs((1))
Transition zone. No refl

Arrival times: tp=1.343 s, ts=1.572 s

x2 vp(2), vs(2) Arrival times of the signals

2 x1 2 x2 2x 2x
tp  (1)
 ( 2) , t s  (11)  ( 22)
vp vp vs vs
Reflection from the boundary
Equations

vp - compressive wave velocity 0.780 x1  0.563x2  1.343

vs - shear wave velocity 0.913x1  0.659 x2  1.572

A.V. Dyskin, CRE, UWA GENG2140 Slide 51

The system
(Forsythe, Malcolm and Moler, 1977)

 0.780 0.563  x1  1.343 

     
 0.913 0.659  x2  1.572 
A x b Ax= b

The exact answer is x=(1,

( , 1))

A.V. Dyskin, CRE, UWA GENG2140 Slide 52

2
 Solution
Gaussian elimination with partial pivoting on a computer with =10, t=3:

Partial pivoting
 0.780 0.563  x1  1.343   0.913 0.659  x1  1.572   0.913 0.659  x1   1.572 
 0.913 0.659  x   1.572    0.780 0.563  x   1.343    0     
0.001 x2    0.000001
  2      2    

 0.913 0.659  x1   1.572   0.913 0  x1   1.573   1 0  x1   1.72 

                
 0 1  x2    0.001  0 1  x2    0.001  0 1  x2    0.001

The solution: x*=(1.72, -0.001). The error e=x*-x=(-0.71, 1.001)

is of the order of the true solution.

A.V. Dyskin, CRE, UWA GENG2140 Slide 53

Analysis of the Example

 Higher accuracy of computation can improve
the situation.
situation For instance,
instance the solution
obtained on a computer with t=6 is: (1.00000,
1.00000) which is very accurate.
 The residual is very small:

  0.0019 
r  Ax   b   
  0.0023 
 Determinant is very small: det(A)=0.000913

A.V. Dyskin, CRE, UWA GENG2140 Slide 54

3
 Counter-example
 0.03 0.01 x1   0.04 
      The exact solution is x=(1, 1).
 0.005 0.03  x2   0.035 

Gaussian elimination on a computer with =10, t=3:

 0.03 0.01 x1   0.04   0.03 0.01  x1   0.04   0.03 0.01 x1   0.04 

                 
 0.005 0.03  x2   0.035   0 0.0283 x2   0.0283  0 1  x2   1 
 0.03 0  x1   0.03   1 0  x1  1
           
 0 1  x2   1   0 1  x2  1

The solution: x*=(1, 1) is very accurate.

However, the determinant of the matrix is even less than in the Example.

A.V. Dyskin, CRE, UWA GENG2140 Slide 55

Images of unit vectors for the

matrix from the Counter-example
 0.03 0.01
A    0.04
Unit vectors:  0.005 0.03 
o o o
u=(cos  , sin ),  108
v=Au o
72 o
36
o o 0.02
108 72 o
144
o
144 o
36
min |Au|= 0.023 max |Au|= 0.037 o
o o 0
180 0
1 -0.04 -0.02 0 0.02 0.04
o
o
180 o
216 324 o
o o 324
252 288 o
216 -0.02

o
max |Au| 288
 252
o
min |Au|
-0.04

A.V. Dyskin, CRE, UWA GENG2140 Slide 56

4
 Images of unit vectors for the
matrix from the Example
 0.780 0.563 
Unit vectors: A   
o o
 0.913 0.659 
u=(cos  , sin  ),  o
1 o 36
-6 72
v=Au min |Au|= 1.6*10 o
o 0
o
108 72
o 0.5 o
144 o 324
36
o
108
o o max |Au|= 1.48
180 0
0
1 -0.8 -0.4 0 0.4 0.8
o
o 288
216 o o
324 144 -0.5
o o
252 288
o
252
o -1
180
max |Au| 6
 216
o
min |Au|

A.V. Dyskin, CRE, UWA GENG2140 Slide 57

The condition number of a matrix

max Ax Ax
max
x 1 x x0

cond( A)  or equivalently
or, equivalently, cond( A) 
Ax
min Ax min
x0 x
x 1
(Here ||x|| denote a length of vector x.)
Matrices with high condition numbers are called ill-
conditioned, otherwise they are called well-conditioned.
Properties

1. If A is singular, min Ax  0 , hence, cond(A)=.

x 1
2. cond(A)1 3. cond(cA)=cond(A) for any scalar c
A.V. Dyskin, CRE, UWA GENG2140 Slide 58

5
 Questions
 What does it mean "high condition number"?
 Is there a better (easier) way to compute the
condition number?
To answer these questions the notions of norm of vector and
norm of matrix are necessary.

 Norm of a vector, properties

 Norm of a matrix based on the vector norm
 Properties of matrix norm
 The condition number via matrix norm
 Meaning of the condition number

A.V. Dyskin, CRE, UWA GENG2140 Slide 59

Norm of a vector
Norm of a vector x is a real number ||x|| such that:
(i ) x  0 if x0
(ii ) 0 0
(iii ) cx  c  x for all scalars c
(iv) xy  x  y
n
x 1   xi
i 1

The most
common n

x
2
x2 i - Euclidian length
vector i 1

norms:
x   max xi
i 1,, n

A.V. Dyskin, CRE, UWA GENG2140 Slide 60

6
 Examples

 1 
  x 1  1  0 .1  3  1  5 .1
 0.1
x  x 2  1  0.01  9  1  3.32
3
 
 1
  x 
 max1, 0.1, 3, 1  3

Although these three norms are different, the results are of the
same order of magnitude. The chose of a particular norm is
dictated by the convenience to perform the analysis.

A.V. Dyskin, CRE, UWA GENG2140 Slide 61

Norm of a matrix

A  max Ax or equivalently
or, equivalently, A  max
Ax
x 1 x0 x

The particular value of the matrix norm is determined by a

particular chose of the vector norm. Thus, if
 a11 a12  a1n 
 
a a22  a2 n  n n
A   21

a
   

th
then A 1  max
j 1,, n
 aij , A   max  aij
i 1,, n
 n1 an 2  ann  i 1 j 1

There is no simple formula for Euclidean norm.

A.V. Dyskin, CRE, UWA GENG2140 Slide 62

7
 Properties of matrix norm
For any norm

(i ) A  0 if A0
(ii ) 0 0
(iii ) cA  c  A for all scalars c
(iv) A B  A  B

((v)) For anyy matrix A and vector x ||||Ax||

|| ||||A|||| ||||x||||
Moreover, there always exists a vector xA, such that ||AxA||= ||A|| ||xA||

(vi) For any two matrices A and B ||AB|| ||A|| ||B||

A.V. Dyskin, CRE, UWA GENG2140 Slide 63

Example 1
 1 2  1  0.5 
 
 0 .5  3 0 1 
A
0  1.5 0 1 
 
 2 1 1 
 3

A 1  max1  0.5  0  2 2  3  1.5  3 1  0  0  1 0.5  1  1  1

 max3.5 9.5 2 3.5  9.5

 1  2  1  0.5   4.5 
   
 0. 5  3  0  1  4.5 
A  max   max   7
0  1.5  0  1 2.5
   
 2  3 11   7 
   

A.V. Dyskin, CRE, UWA GENG2140 Slide 64

8
 Example 2
 1 2  1  0.5   1
   
 0. 5  3 0 1    1
A , A 1  9.5, x   , x 1  1111  4
0  1 .5 0 1  1
   
 2 3 1 1  
 1

 1 2  1  0 .5   1   1 .5 
    
 0 . 5  3 0 1    1   2 .5 
Ax   
0  1.5 0 1   1    0 .5 
    
 2 3 1 1   1   3 

Ax 1  1.5  2.5  0.5  3  7.5, A 1  x 1  9.5  4  38, Ax 1  A 1  x 1

A.V. Dyskin, CRE, UWA GENG2140 Slide 65

Example 3
 1 2  1  0.5   0
   
 0.5  3 0 1   1
A , A 1  9.5, x   , x 1 1
0  1.5 0 1  0
   
 2 1 1   0
 3  

 1 2  1  0.5  0   2 
    
 0.5  3 0 1  1    3 
Ax   
0  1.5 0 1  0    1.5 
    
 2 1 1  0   3 
 3

Ax 1  2  3  1.5  3  9.5, A 1  x 1  9.5 1  9.5, Ax 1  A 1  x 1

A.V. Dyskin, CRE, UWA GENG2140 Slide 66

9
 Condition number via matrix
norm
A1x y 1
Let y=A-1x, x=Ay. Then A1  max  max 
x 0 x y 0 Ay Ay
min
y 0 y
Ay
max
y 0 y
cond( A)   A A1
Ay
min
y 0 y

Therefore cond( A)  A A1

A.V. Dyskin, CRE, UWA GENG2140 Slide 67

Condition number for the matrix

from Example and Counter-example
The Example
 0.780 0.563   659000  563000 
A   , A1   , A 1  1.69, A1  1.57  106
 0.913 0.659   913000 780000  1

cond( A)  2.66  106

The Counter-example
 0.03 0.01  35.3  11.8 
A   , A1   , A 1  0.04, A1  47.1
 0.005 0.03    5.88 35.3  1

cond( A)  1.884

A.V. Dyskin, CRE, UWA GENG2140 Slide 68

10
 Estimation of errors in solving
linear algebraic equations 1
Ax=b
1. Error in the right part A(x+x)=b+b
x is the error in the solution, x. The error satisfies the system Ax=b

x  A1b  x  A1 b , Ax  b  A x  b
x b b
 A1 A  Cond ( A)
x b b

x b The condition number of a matrix is

 Cond ( A)
x b the magnification factor of the
relative error.

A.V. Dyskin, CRE, UWA GENG2140 Slide 69

Estimation of errors in solving

linear algebraic equations 2
Ax=b
2. Error in the matrix (A+A)x=b
x is the error in the solution, x. The error satisfies the system Ax=b

Ax  b  Ax  b  AA1b

x b AA1b A
 Cond
C d( A)  Cond
C d( A)  Cond
C d( A)
x b b A

The condition number of a matrix is the magnification

factor of the relative error.

A.V. Dyskin, CRE, UWA GENG2140 Slide 70

11
 Structure of this chapter
Ill-conditioned systems
and matrices

Condition number

Condition number
Matrix norm Vector norm
via matrix norm

Condition number as
error multiplicator

A.V. Dyskin, CRE, UWA GENG2140 Slide 71

Summary
 Amongst systems of linear algebraic equations there exist
such that amplify errors in the matrix or right hand parts -
ill-conditioned systems
 In 2D
2D, ill-conditioned systems are recognised by
slenderness of the image of a circle. Generally they are
characterised by large condition number.
 The condition number is defined as
max Ax
x 1
cond( A) 
min Ax
x 1
 P
Properties
i
• If detA=0, then cond(A)=; cond(A)1; cond(cA)=cond(A)
 Method of computation – through matrix norm
 Meaning – multiplication factor of errors in the right-hand
part or the system.

A.V. Dyskin, CRE, UWA GENG2140 Slide 72

12