MATH321 Lec2 Solutions To Systems of Linear Equations, Eigenvalues and Eigenvectors PDF

Solutions to Systems of Linear Equations
Angelo R. dela Cruz, ECE
ECE-MATH321 engradc (c)2011
Topics
z z z z z z z z z
Rank of a Matrix Homogeneous and Non-Homogeneous Cramers Rule Inverse Method QR-Decomposition Gaussian Elimination Lower-Upper (LU) Decomposition Cholesky Factorization Eigenvalues and Eigenvectors
Rank of a Matrix, r
z
It is the greatest value of r for which there exists an r x r submatrix of A with non vanishing determinants. The rank of an m x n matrix can at most be equal to the smaller of the numbers m and n but may be less.
r m < n OR r n < m
Rank of a Matrix, r
z
An n x n matrix has a rank less than n if and only if A is singular matrix.
r < n if A = 0
z
Full rank can be achieved if and only if A is a square and non-singular matrix.
r = n if A 0
Example
z
Determine the rank of the matrices given below
4 3 2 1 a.) A = 6 4 2 8
2 4 3 6 b.) A = 3 6 5 9 1 2 1 3
System of Linear Equations: General Form

z
n unknowns with m equations
a11 x1 + a12 x2 + ... + a1n xn = b1 a21 x1 + a22 x2 + ... + a2 n xn = b2 a31 x1 + a32 x2 + ... + a3n xn = b3 ...... am1 x1 + am 2 x2 + ... + amn xn = bm
z
Where aij are coefficients of the unknowns xk.

General Form
z
Using matrix form (m x n)
a11 a12 ... a1n x1 b1 a a ... a x b 21 22 2 n 2 = 2 : : : : : : am1 am 2 ... amn xn bm
Ax = b
Homogeneous and Non-Homogeneous

z
If vector b is a null vector i.e.
Ax=0
thus, The given linear equations are homogeneous
z
If vector b is not a null vector thus, the given linear equations are non-homogeneous
Solution to Homogeneous Linear Equations

z
If the system of linear equations is homogeneous i.e. b = 0, and A is non-singular square matrix (full-rank), therefore the system has only trivial solution i.e.
x1 = x2 = x3 = = xn = 0
Solution to Homogeneous Linear Equations

z
If the system of linear equations is homogeneous i.e. b = 0, and A is not a full rank matrix whose r < m, consider only r equations whose coefficient matrix has a rank r and omitting the other mr equations. There will be an infinite solution to the system
Examples
z
Determine the solution of the following: a. 2x
4y = 0, x + y = 0 b. x + y + z = 0 2x y 5z = 0 x y 5z = 0
Examples
z
Determine the solution of the following:
a.
x + y + z = 0, 2x y 7z = 0, x y 5z = 0 b. 2x + y + z = 0 4x + 3y z = 0
Solution to Non-Homogeneous Linear Equations

z
If the system of linear equations is non-homogeneous i.e. b 0, and A is not full-rank matrix: Let
a12 a 22 : am2 ... a1n a11 a 21 ... a 2 n , B= : : : ... a mn a m1 a12 a 22 : am2 ... a1n ... a 2 n : : ... a mn b1 b2 : bn
a11 a 21 A= : a m1

z 1.
a.
Rules: If rank(A) = rank(B), exact solution exist

If rank(A) < m, consider only r equations whose coefficient matrix has rank r. Omit the other m-r equations Transpose n-r unknowns to the right member of each equation, retaining the r unknowns in the left member whose coefficient matrix has a rank r. Solve the system using Cramers rule
b.
c.
Cramers Rule
b1 b2 D( x1 ) = : : bn a12 a 22 : : an2 .. .. a1n .. .. a 2 n : : : : : :
a11 a 21 D( x 2 ) = : : a n1
b1 b2 : : bn
.. .. a1n .. .. a 2 n : : : : : : .. .. a nn
.. .. a nn
a11 a 21 D( x n ) = : : a n1
a12 a 22 : : an2
.. .. b1 .. .. b2 : : : : : :
D ( xn ) D( x1 ) D ( x2 ) x1 = , x2 = , ..., xn = A A A
.. .. bn
Examples
z
Solve the following systems of linear equations

a.
2x 5y + 13z = 24 3x + y 6z = 2 x y + z = 4, x + y z = 0 2x 3y + z = 4, 3x + 2y 3z = 1
b.
Solution to Non-Homogeneous Linear Equations: QR-Decomposition

Amxn = QmxnRnxn
r11 r12 0 r 22 ... q n ] : 0 0 .. .. r1n r2 n .. : 0 rnn
[a1
a2
... an ] = [q1 q 2
Where:
ak column vectors of A qk orthonormal column vectors R upper triangular matrix (also known as projection matrix)
QR-Decomposition Steps
Consider a 4x3 matrix A
[a1
Where:
a 2 a3 ] = [q1 q 2
r11 r12 q3 ] 0 r 22 0 0
r13 r23 r33
ak 4x1 column vectors of A qk 4x1 orthonormal vectors R 3x3 upper triangular matrix
QR-Decomposition Steps
1 , r11 = q r12 = q 1 a 2 , 2 , r22 = q r13 = q 1 a 3 , r23 = q 2 a 3 , 3 , r33 = q 3 q q3 = 3 q
1 = a1 q 1 q q1 = 1 q
2 = I q 1q 1T a 2 q 2 q q2 = 2 q
T T q 3 = I q 1q 1 I q 2 q 2 a 3
)(
QR-Decomposition General Steps

k = I q 1q 1T I q 2 q 2 T .. . I q k 1q k 1T a k q
T T 2 1 1 2
( )( = (I q q q q
)(
... q k 1q
T k 1
)a
= a k r1, k q 1 r2 , k q 2 r3, k q 3 ... rk 1, k q k 1 k q qk = k q k rkk = q rij = q i a j

T i j
QR-Decomposition
Case 1: Overdetermined System of Linear Equations The Least-Square Method
Ax = b QRx = b QT QRx = QT b IRx = Q b

T
Note that if Q is orthonormal matrix,
QTQ = I
Rx = Q b
T
Note that since
R is upper triangular matrix,

x can be solved using backward substitution
QR-Decomposition
Case 2: Underdetermined System of Linear Equations The Least-Norm Method
AT = QR R y =b x = Qy
T
y can be solved using forward substitution

Examples
z
Solve the following systems of linear equations

a.
2x 5y + 13z = 24 3x + y 6z = 2 x y + z = 4, x + y z = 0 2x 3y + z = 4, 3x + 2y 3z = 1
b.

z 2.
Rules: If rank(A) rank(B), the system has no exact solution and the system can be solved by estimation if m > n (more equations than unknowns) Typical solution to best estimate the unknowns is Method of Least Squares using QRDecomposition method
Method of Least Squares

z
Since the solution is not exact, The residual error (vector), e is the distance vector of the actual vs. estimate and the loss function, (scalar) is the square norm of the error:
0 e = b Ax min = e e
T
QR-Decomposition Example
Obtain the best solution given an overdetermined system of equations
1 1 4 1 1 4 2 x1 0 x2 = 1 4 2 0 x3 1 1 0 0

z
If the system of linear equations is non-homogeneous i.e. b 0, and A is a full-rank matrix r = n Can be solved by:
z z z z z
a a a ... 11 12 1 n Inverse method Gaussian elimination a a a ... 21 22 2n A= LU-decomposition : : : : QR-Decomposition Cholesky Factorization an1 an 2 ... ann
Inverse Method
z
Since Ax = b
Premultiply both sides of the equation by A-1
A Ax = A b
-1 -1
Ix = A b
-1
x=A b
-1
Example
1.
Solve for the unknowns, x and y using Inverse method and compare it to the solution using Cramers rule
2x + 3y = 5 3x y = 4
Gaussian Elimination
z
This method involves replacing equations by obtaining its triangular matrix form
u11 x1 + u12 x2 + ... + u1n xn = c1 u22 x2 + ... + u2 n xn = c2 ............ : unn xn = cn

z
The unknowns are computed using backward substitution
Example
z
Determine the solution using Gaussian elimination method
3x + 4y + 2z w = 2 4x + 2y 3z + 2w = 4 5x + 3y 4z + 3w = 4 3x + y + z + 2w = 2
Lower-Upper (LU) Decomposition Method

z z
From the form Ax
=b Decompose A = LU where L is a lower triangular matrix and U is an upper triangular

matrix
1 0 ... 0 l 1 ... 0 21 L= : : : : ln1 ln 2 ... 1
u11 u12 ... u1n 0 u u ... 22 2n U= : : : : 0 ... unn 0

Lower-Upper (LU) Decomposition Method

z
After decomposing A = LU, therefore
LUx = b. Let y = Ux, thus Ly = b where y can be

solved by forward substitution After solving y, since y = Ux, x can be solved by backward substitution
Example
z
Determine the solution using LU-decomposition and QR-decomposition method
3x + 4y + 2z w = 2 4x + 2y 3z + 2w = 4 5x + 3y 4z + 3w = 4 3x + y + z + 2w = 2
Cholesky-Decomposition
If A is real symmetric positive definite square matrix
A = LLT
for 4x4 matrix
l11 0 0 0 l11 l21 l31 l l22 0 0 0 l22 l32 21 A= l31 l32 l33 0 0 0 l33 l41 l42 l43 l44 0 0 0
l41 l42 l43 l44
A is positive definite square matrix iff its eigenvalues are real > 0
The elements of L can be computed easily using the Cholesky algorithm shown below
j 1
l j , j = a j , j l j ,k
k =1
j 1 1 li , j = ai , j li ,k l j ,k , l j, j k =1
for i > j
Example: Obtain the Cholesky Decomposition of A
4 A= 2 2 l11 = l21 l31
2 10 2 2 5 0 0 l11 l21 l31 0 l l22 0 l 22 32 l32 l33 0 0 l33 2

Eigenvalues and Eigenvectors

z
Given a square matrix A, a non-trivial vector v and a scalar that satisfy the relationship
Av = v
z
Where
eigenvalue of A v corresponding eigenvector


z
Matrix A has n eigenvalues Generally, eigenvalues are not always real and distinct If A is real symmetric therefore, its eigenvalues are real For each eigenvalue, there is a corresponding eigenvector.
1 > 2 > 3 > .. > n
z z

Note
Av v = 0 Iv = v Av Iv = 0 (A I) v = 0
Note that v = 0 is a possible solution but it is a trivial solution!


z
To get a non-trivial solution
det (A I) = 0
z
This leads to a polynomial equation f() = 0 of degree equal to rank of A The roots of f() are the eigenvalues of A
Eigenvectors
z
The eigenvector v(i), that corresponds to eigenvalue i is computed as
(A iI) v(i) = 0
z
The non-trivial solution of v(i) is the eigenvector of the corresponding eigenvalue
Eigenvectors
z
The eigenvector v(i), of the corresponding eigenvalue i is not unique, thus a normalization factor is needed to have a common eigenvector
(i ) 2
+ v2 + ... + vn
(i ) 2
(i ) 2
(i )
(i )
Eigen-Decomposition
A = QQ1
A = [v1 v2 11 0 0 22 ... v n ] : 0 .. 0 0 0 0 [v 1 .. : 0 nn .. v2 ... v n ]
1
Where:
Q eigenvector matrix arrange in column vectors diagonal matrix whose elements are the eigenvalues of A
Example
z
Solve for the eigenvalues and the eigenvector of the greatest positive eigenvalue. Decompose A = QQ1
5 3 2 A= 2 6 3 3 2 4

MATH321 Lec2 Solutions To Systems of Linear Equations, Eigenvalues and Eigenvectors PDF

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

MATH321 Lec2 Solutions To Systems of Linear Equations, Eigenvalues and Eigenvectors PDF

Uploaded by

Copyright:

Available Formats

Solutions to Systems of Linear Equations

Angelo R. dela Cruz, ECE

ECE-MATH321 engradc (c)2011

An n x n matrix has a rank less than n if and only if A is singular matrix.

Determine the rank of the matrices given below

System of Linear Equations: General Form

n unknowns with m equations

Where aij are coefficients of the unknowns xk.

Using matrix form (m x n)

a11 a12 ... a1n x1 b1 a a ... a x b 21 22 2 n 2 = 2 : : : : : : am1 am 2 ... amn xn bm

Homogeneous and Non-Homogeneous

If vector b is a null vector i.e.

Solution to Homogeneous Linear Equations

Solution to Homogeneous Linear Equations

Determine the solution of the following: a. 2x

Determine the solution of the following:

Solution to Non-Homogeneous Linear Equations

ECE-MATH321 engradc (c)2011

Solution to Non-Homogeneous Linear Equations

Rules: If rank(A) = rank(B), exact solution exist

Solve the following systems of linear equations

Solution to Non-Homogeneous Linear Equations: QR-Decomposition

r13 r23 r33

QR-Decomposition General Steps

= a k r1, k q 1 r2 , k q 2 r3, k q 3 ... rk 1, k q k 1 k q qk = k q k rkk = q rij = q i a j

Ax = b QRx = b QT QRx = QT b IRx = Q b

Note that if Q is orthonormal matrix,

Note that since

R is upper triangular matrix,

x can be solved using backward substitution

y can be solved using forward substitution

Solve the following systems of linear equations

Solution to Non-Homogeneous Linear Equations

Method of Least Squares

Solution to Non-Homogeneous Linear Equations

ECE-MATH321 engradc (c)2011

u11 x1 + u12 x2 + ... + u1n xn = c1 u22 x2 + ... + u2 n xn = c2 ............ : unn xn = cn

The unknowns are computed using backward substitution

ECE-MATH321 engradc (c)2011

Determine the solution using Gaussian elimination method

Lower-Upper (LU) Decomposition Method

From the form Ax

=b Decompose A = LU where L is a lower triangular matrix and U is an upper triangular

u11 u12 ... u1n 0 u u ... 22 2n U= : : : : 0 ... unn 0

Lower-Upper (LU) Decomposition Method

After decomposing A = LU, therefore

LUx = b. Let y = Ux, thus Ly = b where y can be

ECE-MATH321 engradc (c)2011

Determine the solution using LU-decomposition and QR-decomposition method

for 4x4 matrix

l41 l42 l43 l44

ECE-MATH321 engradc (c)2011

4 A= 2 2 l11 = l21 l31

2 10 2 2 5 0 0 l11 l21 l31 0 l l22 0 l 22 32 l32 l33 0 0 l33 2

Eigenvalues and Eigenvectors

eigenvalue of A v corresponding eigenvector

Eigenvalues and Eigenvectors

1 > 2 > 3 > .. > n

Eigenvalues and Eigenvectors

Note that v = 0 is a possible solution but it is a trivial solution!

Eigenvalues and Eigenvectors

To get a non-trivial solution