MA3023 Semester 5, 2019

1 System of Linear Equations

You already know how to solve system linear equation of two variables. When the variables
and the number of equations increase, solving the system of linear equations is a challenging
problem. In general,

a11 x1 + a12 x2 + a13 x3 + · · · · · · · · · + a1n xn = b1 (1)

is an nth order linear equation, where x1 , x2 , x3 , · · · xn are variables, a11 , a12 , a13 x3 , · · · a1n are
coefficients and b1 is a constant.
Following shows the system of linear equations.

a11 x1 + a12 x2 + a13 x3 + · · · · · · · · · + a1n xn = b1

a21 x1 + a22 x2 + a23 x3 + · · · · · · · · · + a2n xn = b2
a31 x1 + a32 x2 + a33 x3 + · · · · · · · · · + a3n xn = b3
..
.
an1 x1 + an2 x2 + an3 x3 + · · · · · · · · · + ann xn = bn (2)

There are three possible solutions of system of linear equations:

(i) A unique solution (a consistent set of equations),

(ii) No solution (an inconsistent set of equations),

(iii) An infinite number of solutions

Any system of linear equations can be represent in matrix form,

~ =B
AX ~ (3)

where A is called as a coefficient matrix, X ~ is a variable or unknown matrix and B

~ is a constant
matrix. Thus, the system of linear equation (2) can be written in matrix form AX ~ =B ~ where
0 1 0 1 0 1
a11 a12 a13 · · · · · · · · · a1n x1 b1
B a21 a22 a23 · · · · · · · · · a2n C B x2 C B b2 C
B C B C B C
B a31 a32 a33 · · · · · · · · · a3n C ~ B x3 C ~ B C
A=B C , X = B C and B = B b3 C.
B .. .. .. .. C B . C B .. C
@ . . . . A @ .. A @.A
an1 an2 an3 · · · · · · · · · ann xn bn
In the most general case, the numberof variables is not required to be the same as the
number of equations. However,in most practical problems,they are the same. The number of
equations of a system can range from two to hundreds, thousands, and even millions.
There are two fundamentally di↵erent approaches for solving systems of linear equations:
Direct elimination methods and Iterative methods. We mainly discuss some direct methods in
this class. You may find itrative methods by referring any numerical book.

1.1 Direct Methods

Direct methods give the exact solution to the system in a finite number of steps. In practice, of
course, the solution obtained will be contaminated by the round-o↵ error that is involved with
the arithmetic being used. Analyzing the e↵ect of this round-o↵ error and determining ways
to keep it under control will be a major component.

1 All right reserved, Dep of Math, UoM

 MA3023 Semester 5, 2019

1.1.1 Gaussian Elimination

The system of linear equations can be reduced to triangular form (or upper-triangular ma-
trix) by employing the row operations and can be solved for the unknowns by a backward
substitution. This process is called as Gaussian Elimination. The useful row operations are:

(i) Any row (equation) may be multiplied by a non-zero constant (scaling).

(ii) The order of the rows (equations) may be interchanged (pivoting).

(iii) Any row (equation) can be replaced by a weighted linear combination of that row (equa-
tion) with any other row (equation) (elimination).

In the Gaussian Elimination process, first form the augmented matrix [A|B],
0 . 1
a11 a12 a13 · · · · · · · · · a1n .. b1
B C
B a21 a22 a23 · · · · · · · · · a2n ... b2 C
B C
B .
. C
A = B a31 a32 a33 · · · · · · · · · a3n . b3 C .
B C
B .. .. .. .. .. .. C
@ . . . . . .A
.
an1 an2 an3 · · · · · · · · · ann .. bn

Then, if a11 6= 0, perform the operations (elimination) corresponding to

⇣a ⌘
j1
Rj R1 ! Rj
a11
for each j = 2, 3, ....., n to eliminate the coefficient of x1 in each of the rows. The resulting
matrix has the form, 0 1
(1) .
· · · · · · · · · a1n .. b1
(1) (1) (1) (1)
a a12 a13
B 11 (2) .
C
B 0 a(2) (2)
a23 · · · · · · · · · a2n .. b2 C
(2)
B 22 C
B (2) .. (2) C .
B 0 a(2) (2)
a33 · · · · · · · · · a3n . b3 C
B 32 C
B .. .. .. .. .. .. C
@ . . . . . . A
(2) (2) (2) .. (2)
0 an2 an3 · · · · · · · · · ann . bn
⇣ a(2) ⌘
(2) j2
Next, if a22 6= 0, perform operation Rj (2)
R2 ! Rj to eliminate x2 in each row j =
a22
3, 4, ......, n. Likewise, eliminate xi in each row below the ith for all values of i = 1, 2, ..., n 1 by
⇣ a(i) ⌘
ji (i)
performing the operation Rj (i)
Ri ! Rj for each j = i + 1, i + 2, ..., n, provided aii 6= 0.
aii
If the system has a unique solution, the final matrix has the form
0 1
(1) (1) (1) (1) .. (1)
a a12 a13 · · · · · · · · · a1n . b1
B 11 C
B 0 a(2) a(2) · · · · · · · · · a(2) ... b(2) C
B 22 23 2n 2 C
B (3) (3) .. (3) C .
B 0 0 a · · · · · · · · · a . b C
B 33 3n 3 C
B .. .. .. .. .. .. C
@ . . . . . . A
(n) .
0 · · · · · · 0 ann .. bn
(n)
0 0

This matrix represents a linear system with the same solution set as the original system.

2 All right reserved, Dep of Math, UoM

 MA3023 Semester 5, 2019

The new linear system is triangular,

(1) (1) (1) (1) (1)
a11 x1 + a12 x2 + a13 x3 + · · · · · · · · · + a1n xn = b1
(2) (2) (2) (2)
a22 x2 + a23 x3 + · · · · · · · · · + a2n xn = b2
(3) (3) (3)
a33 x3 + · · · · · · · · · + a3n xn = b3
..
.
a(n) (n)
nn xn = bn (4)

so backward substitution can be performed. Solving the nth equation for xn gives
(n)
bn
xn = (n)
(5)
ann

Solving the (n 1)th equation for xn 1 and using the known value for xn yields
(n 1) (n 1)
bn 1 an
1,n xn
xn 1 = (n 1)
(6)
an 1,n 1

Continuing this process to obtain all the variables.

It can be shown that an arbitrary linear system AX~ = B
~ requires O(n3 /3) arithmetic
~
operations to determine X.

1.1.2 Pivoting
The pivot element for a specific column is the entry or element that is used to place zeros in
the other entries in that column. The elimination procedure described so far fails immediately
if the first pivot element a11 is zero. The procedure also fails if any subsequent pivot element
(k)
aii is zero. To avoid this situation, row interchange was needed. This row interchange has the
(k)
form Rk $ Rp , where p is the smallest integer greater than k with apk 6= 0.
Even when the pivot elements are not zero (but very small compared to the other entries
in that column), it is often necessary to perform row interchanges to reduce round-o↵ error.
As an example, the exact solution of the system

0.003000x1 + 59.14x2 = 59.17

5.291x1 6.130x2 = 46.78 (7)

is x1 = 10 and x2 = 1. Using Gaussian elimination with four digit aithmetic rounding (finite-
digit arithmetic), the approximation solution to the system (7) is x1 = 10.00 and x2 = 1.001.
There is a huge di↵erence between the exact and approximated solution of x1 . When performing
the backward substitution for x1 , any error in the numerator can be dramatically increased
(1)
because of the division by a small value of a11 = 0.003. To avoid this, row interchanges has to
be performed.
(k)
If the pivot element akk is small relative to the entries in the same column that is below the
diagonal, select an element in the same column that has the largest absolute value; specifically,
we determine the smallest p k such that
(k) (k)
|apk | = max |aik |
kin

and perform Rk $ Rp .

3 All right reserved, Dep of Math, UoM

 MA3023 Semester 5, 2019

Scaled pivoting
Another situation that the round-o↵ error can dominate the calculations is when the mag-
nitudes of the pivot elements are smaller than the magnitudes of the other elements in the
equations (or row) containing the pivot elements. In such cases, scaling is employed to select
the pivot elements.
Let si = max1jn |aij |. If si 6= 0, select the row. Rp , p i with

|aki |
max
ikn sk

and perform the row interchange Ri $ Rp if i 6= p.

1.1.3 LU factorization
Another approach to solve system of linear equations is factorization. Suppose that A has
been factored into the triangular form A = LU , where L is lower triangular and U is upper
triangular. Then we can solve for X~ more easily by using a two-step process.

AX~ =B
~ (8)
~ =B
LU X ~ (9)

• First we let Y~ = U X
~ and solve the lower triangular system LY~ = B ~ for Y~ using forward
substitution. Since L is triangular, determining Y~ from this equation requires only O(n2 )
operations.

• Once Y~ is known, the upper triangular system U X

~ = Y~ requires only an additional O(n2 )
~ using backward substitution.
operations to determine the solution X

Solving a linear system AX ~ =B~ in factored form means that the number of operations needed
to solve the system AX ~ = B~ is reduced from O(n3 /3) to O(2n2 ). The reductions from the
factorization come at a cost; determining the specific matrices L and U requires O(n3 /3)
operations. But once the factorization is determined, systems involving the matrix A can be
~
solved in this simplified manner for any number of vectors B.
To see which matrices have an LU factorization and to find how it is determined, see the
following theorem.

Theorem 1 If Gaussian elimination can be performed on the linear system AX ~ =B ~ without

row interchanges, then the matrix A can be factored into the product of a lower-triangular
(i)
aji
matrix L and and an upper-triangular matrix U , that is, A = LU , where mji = (i) ,
0 (1) (1) (1) 1 aii
(1) 0 1
a11 a12 a13 · · · · · · · · · a1n 1 0 0 ········· 0
B (2) C B m21
B 0 a(2) 0C
(2)
22 a23 · · · · · · · · · a2n C B 1 0 ········· C
B (3) C
(3) C B m31 m32 C
U =B B 0 0 a 33 · · · · · · · · · a 3n C, and L = B
1 · · · · · · · · · 0 C.
B .. .. .. .. C B .. .. .. .. C
@ . . . . A @ . . . .A
0 0 0 · · · · · · 0 ann
(2) m n1 m n2 m n3 · · · · · · m n,n 1 1

Specifying the diagonal elements of either L or U makes the factoring unique. The procedure
based on unity elements on the main diagonal of L is called the Doolittle’s method.

4 All right reserved, Dep of Math, UoM

 MA3023 Semester 5, 2019

ALGORITHM

~ =B
This factorization can be used if AX ~ can be solved using Gaussian elimination without
row interchanges. Practically, for some matrices, row interchanges needed to perform in order
to reduce round-o↵ error. Now consider the modifications of factorization that must be made
when row interchanges are required.
Permutation Matrix
An n ⇥ n permutation matrix P = [pij ] is a matrix obtained by rearranging the rows of In ,
the identity matrix. This gives a matrix with precisely one nonzero entry in each row and in
each column, and each nonzero entry is a 1.
Two important properties of permutation matrix, P , are,
(i) P A permutes or rearrange the rows of A
1 1
(ii) P exists and P = PT.
By using the permutation matrix, the equations in the system can be rearranged to proceed
Gaussian elimination without row interchanges. That is, for any nonsingular matrix A, a
permutation matrix P exists for which the system
~ = PB
P AX ~ (10)

5 All right reserved, Dep of Math, UoM

 MA3023 Semester 5, 2019

can be solved without row interchanges. As a consequence, this matrix P A can be factored
into
P A = LU,
1
where L is lower triangular and U is upper triangular. Because P = P T , this produces the
factorization
A = P 1 LU = (P T L)U. (11)
The matrix U is still upper triangular, but P T L is not lower triangular unless P = I.

Theorem 2 A strictly diagonally dominant matrix A is nonsingular. Moreover, in this case,

~ =B
Gaussian elimination can be performed on any linear system of the form AX ~ to obtain its
unique solution without row or column interchanges, and the computations will be stable with
respect to the growth of round-o↵ errors.

1.2 Positive Definite Matrices

A matrix A is positive definite if it is symmetric and if ~xT A~x > 0 for every n-dimensional
vector ~x 6= 0.

Theorem 3 The symmetric matrix A is positive definite if and only if Gaussian elimination
without row interchanges can be performed on the linear system AX ~ =B ~ with all pivot elements
positive. Moreover, in this case, the computations are stable with respect to the growth of round-
o↵ errors.

The following two corollaries occur when constructing the proof of the above theorem.

Corollary 1 The matrix A is positive definite if and only if A can be factored in the form
LDLT , where L is lower triangular with 1s on its diagonal and D is a diagonal matrix with
positive diagonal entries.

Corollary 2 The matrix A is positive definite if and only if A can be factored in the form
LLT , where L is lower triangular with nonzero diagonal entries.

The matrix L in these Corollaries are not the same. LLT factorization is called Cholesky
or positive definite decomposition.

6 All right reserved, Dep of Math, UoM

 MA3023 Semester 5, 2019

ALGORITHM

The above Algorithm provides a stable method for factoring a positive definite matrix into
the form A = LDLT , but it must be modified to solve the linear system AX ~ = B.~ To do this,
we delete the STOP statement from Step 5 in the algorithm and add the following steps to
solve the lower triangular system LY~ = B:
~

7 All right reserved, Dep of Math, UoM

 MA3023 Semester 5, 2019

CHOLESKY ALGORITHM

To factor the positive definite n ⇥ n matrix A into LLT , where L is lower triangular:
INPUT the dimension n; entries aij , for 1  i, j  n of A.
OUTPUT the entries lij , for 1  j  i and 1  i  n of L.
(The entries of U = LT are uij = lji , for i  j  n and 1  i  n.)

~ =B
To solve the system AX ~ delete the STOP statement from Step 7 and add the following
steps.

1.3 Tridiagonal Matrices

The special case of a system of linear equations that is tridiagonal, that is, has nonzero elements
only on the diagonal plus or minus one column, is one that occurs frequently. Large tridiagonal
systems arise naturally in a numberof problems, especially in the numerical solution of di↵eren-
tial equations by implicit methods. For tridiagonal sets, the procedures of LU decomposition,
forward and backsubstitution each take only O(n) operations.
Consider the matrix equation
AX~ =B ~ (12)

8 All right reserved, Dep of Math, UoM

 MA3023 Semester 5, 2019

where 0 1
a11 a12 0 ··· ··· 0 0 0
Ba21 a22 a23 ··· ··· 0 0 0 C
B C
B 0 a32 a33 a34 ··· 0 0 0 C
B . .. .. .. .. .. .. C
B . C
B . . . . . . . C
A=B . .. .. .. .. .. .. C.
B .. . . . . . . C
B C
B0 0 0 · · · an 2,n an an 2,n 1 0 C
B 3 2,n 2 C
@0 0 0 ··· ··· an 1,n 2 an 1,n 1 an 1,n A
0 0 0 ··· ··· 0 an,n 1 an,n
Suppose the tridiagonal matrix A can be factored into the triangular matrices L and U ,
that also have the form tridiagonal form.

0 1 0 1
l11 0 0 · · · ··· 0 1 u12 0 · · · · · · 0
B C
Bl21 l22 0 · · ·
B ··· 0 CC B0 1 u23 . . . 0 C
B . C B C
B 0 l32 l33 . . ··· 0 C B ... ... C
B0 0 1 0 C
L=B
B ... . . . . . . . . . .. C
.. C and U = B .. C
B . . C B ... . . . . . . . . . . . . . C
B. C B C
@ .. .. . . . . A B .. .. . . . . C
. . . ln 1,n 1 0 @. . . . 1 un 1,n A
0 0 0 · · · ln,n 1 ln,n 0 0 0 ··· 0 1
There are (2n 1) undetermined entries of L and (n 1) undetermined entries of U , which
totals (3n 2), the number of possible nonzero entries of A. The 0 entries of A are obtained
automatically.
The multiplication involved with A = LU gives, in addition to the 0 entries,

a11 = l11 (13)

ai,i 1 = li,i 1 , for each i = 2, 3, ..., n (14)
aii = li,i 1 ui 1,i + lii , for each i = 2, 3, ..., n (15)
ai,i+1 = lii ui,i+1 , for each i = 1, 2, ..., n 1 (16)

This factorization is called as Crout factorization. Once an entry L or U is computed,

the corresponding entry in A is not needed. So the entries in A can be overwritten by the
entries in L and U with the result that no new storage is required.
The Crout Factorization Algorithm can be applied whenever lii 6= 0 for each i = 1, 2, ..., n.
This ensures when the coefficient matrix of the system is positive definite or that it is strictly
diagonally dominant. An additional condition that ensures this algorithm can be applied is
given in the next theorem.

Theorem 4 Suppose that A = [aij ] is tri diagonal with ai,i 1 ai,i+1 6= 0, for each i = 2, 3, ..., n
1. If |a11 | > |a12 |, |aii | |ai,i 1 | + |ai,i+1 |, for each i = 2, 3, ..., n 1, and |ann | > |an,n 1 |,
then A is nonsingular and the values of lii described in the Crout Factorization Algorithm are
nonzero for each i = 1, 2, ..., n.

9 All right reserved, Dep of Math, UoM

 MA3023 Semester 5, 2019

CROUT ALGORITHM for Tridiagonal Linear Systems

1.4 Condition of a Linear System

Any nonsingular systems of linear equations have a solution and it can always be obtained
by Gauss elimination. However, there are two major difficulties in the application of Gauss
elimination(or its variations): (a) the presence of round- o↵ errors, and (b) ill-conditioned
systems.
Round-o↵ errors occur when exact infinite precision numbers are approximated by finite
precision numbers. In most computers, single precision representation of numbers typically
contains about 7 significant digits, double precision representation typically contains about
14 significant digits, and quad precision representation typically contains about 28 significant
digits. Round-o↵ errors can never be completely eliminated. However, they can be minimized
by using high precision arithmetic and pivoting.

1.4.1 System Condition

Consider the following three systems of linear equations.

(a) (b) (c)

x1 + x2 = 2 x1 + x2 = 2 x1 + x2 = 2
x1 + 1.0001x2 = 2.0001 x1 + 0.9999x2 = 2.0001 x1 + 1.0001x2 = 2

We may solve each system by using Gaussian elimination. The solution of system (a) is
x2 = 1, x1 = 1. The system (b) is quite similar to system (a), the only di↵erence is a22 . It
changed slightly from 1.0001 to 0.9999. So, we will expect a solution close to the solution of
system (a) as the solution of system (b) but it is x2 = 1, x1 = 3 which is greatly di↵erent

10 All right reserved, Dep of Math, UoM

 MA3023 Semester 5, 2019

from the solution of system (a). Sytem (c) is also a modification of system (a) in which b2
is changed slightly from 2.0001 to 2. The solution to the system (c) is x2 = 0, x1 = 2 which
is also far from the solution of (a). This illustrates that very small changes in any of the
elements of A or B ~ can cause extremely large changes in the solution, X. ~ Such a system is ill-
conditioned. An ill-conditioned problem is one in which a small change in any of the elements
of the problem causes a large change in the solution of the problem. If a small change in any
of the elements of the problem causes only a small change in the solution of the problem then
it is a well-conditioned problem.
With infinite precision arithmetic, ill-conditioning is not a problem. However with finite
precision arithmetic, round-o↵ errors e↵ectively change the elements of A and B ~ slightly, and if
the systemis ill-conditioned, large changes(i.e., errors) can occur in the solution. Ill-conditioning
is quantified by the condition number of a matrix, which is defined in terms of the norms of
the matrix and its inverse.

1.4.2 Norms

VECTOR NORMS
Let <n denote the set of all n-dimensional column vectors with real-number components.
To define a distance in <n we use the notion of a norm, which is the generalization of the
absolute value on <, the set of real numbers.

Definition 1 A vector norm on <n is a function, k· k, from <n into < with the following
properties:
(i) k~x|
geq0 for all ~x 2 <n ,
(ii) k~xk = 0 if and only if ~x = 0 ,
(iii) k↵~xk = |↵|k~xk for all ↵ 2 < and ~x 2 <n ,
(iv) k~x + ~y k  k~xk + k~y k for all ~x, ~y 2 <n ,

The norm of a scalar is its absolute value. Thus, k↵k = |↵|. There are several definitions of
the norm of a vector.

Definition 2 The l1 , l2 and l1 norms for the vector ~x = (x1 , x2 , ..., xn )T are defined as follows
respectively.
n
X
(i) k~xk1 = |xi |
i=1

nX
n o1/2
(ii) k~xk2 = x2i (Euclidean Norm)
i=1

(iii) k~xk1 = max |xi |

1in

Definition 3 If ~x = (x1 , x2 , ..., xn )T and ~y = (y1 , y2 , ..., yn )T are vectors in <n , the l2 and l1
distances between ~x and ~y are defined by
nX
n o1/2
2
k~x ~y k2 = (xi yi ) , k~x ~y k1 = max |xi yi |
1in
i=1

11 All right reserved, Dep of Math, UoM

 MA3023 Semester 5, 2019

MATRIX NORMS
In a similar manner, the norm of a matrix is defined.

Definition 4 A matrix norm on the set of all n ⇥ n matrices is a real-valued function, k· k,

defined on this set, satisfying for all n ⇥ n matrices A and B and all real numbers ↵:
(i) kAk 0,

(ii) kAk = 0 if and only if A is a zero matrix ,

(iii) k↵Ak = |↵|kAk,

(iv) kA + Bk  kAk + kBk,

(v) kABk  kAkkBk

Theorem 5 If k· k is a vector norm on <n , then

kAk = max kA~xk

kxk=1

is a matrix norm.

Matrix norms defined by vector norms are called the induced matrix norm associated
with the vector norm. Every vector norm produces an associated induced matrix norm. The
matrix norms induced by vector norms l1 , l2 and l1 respectively, are:
n
X
(i) kAk1 = max |aij | (Maximum column sum)
1jn
i=1

(ii) kAk2 = ⇢(AT A) where ⇢(A) = max( ) is called spectral radius and is an eigen value of
A
n
X
(iii) kAk1 = max |aij | (Maximum row sum)
1in
j=1

1.4.3 Condition Number

Consider a system of linear equation
~ =B
AX ~ (17)
Then,
~ = kAXk
kBk ~  kAkkXk
~ (18)
~ is altered by B, which causes
Consider the following modification of equation (17) where B
a change in the solution X.
A(X~ + X) = B ~+ B (19)
Thus,
A X= B (20)
and
1
X=A B (21)
be the solution of system (20). Therefore,

k Xk  kA 1 kk Bk (22)

12 All right reserved, Dep of Math, UoM

 MA3023 Semester 5, 2019

Next, by the equations (18) and (22),

~
kBkk ~
Xk  kAkkXkkA 1
kk Bk (23)

Thus,

k Xk k Bk
 kAkkA 1 k (24)
~
kXk ~
kBk
~
It can be shown by a similar analysis that perturbing the matrix A instead of the vector B
gives
k Xk k Ak
 kAkkA 1 k (25)
~ + Xk
kX kAk

Definition 5 The condition number of the nonsingular matrix A relative to a norm k· k is

(A) = kAkkA 1 k

A matrix A is well-conditioned if (A) is close to 1, and is ill-conditioned when (A) is

significantly greater than 1.

References

(i) Numerical Methods for Scientific and Engineering Computation, M.K. Kain, S.R.K. Iyenger,
R.K. Jain

(ii) Numerical Analysis, Richard L. Burden, J.Douglas Faires.

(iii) Numerical Linear Algebra, Lloyd N. Trefethen, David Bau III.

13 All right reserved, Dep of Math, UoM