CME 434 notes - Matrix Equations

These notes are intended to complement the text for the course, covering
more of the derivation and the theory. This is a draft. There may be typos.
Please inform me if anything looks wrong.

Contents
1 Matrices 1
1.1 Introduction to matrices . . . . . . . . . . . . . . . . . . . . . 1
1.2 Transpose operation . . . . . . . . . . . . . . . . . . . . . . . 2
1.3 Matrix addition . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.4 Matrix multiplication by a scalar . . . . . . . . . . . . . . . . 3
1.5 Matrix multiplication by vectors . . . . . . . . . . . . . . . . . 4
1.6 From linear equations to matrices . . . . . . . . . . . . . . . . 5
1.7 Gauss elimination . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.8 Multiplying two matrices . . . . . . . . . . . . . . . . . . . . . 8
1.9 Matrix inversion . . . . . . . . . . . . . . . . . . . . . . . . . . 10
1.10 Non-invertible Matrices . . . . . . . . . . . . . . . . . . . . . . 13
1.11 Determinants . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
1.12 Matrices in Matlab . . . . . . . . . . . . . . . . . . . . . . . . 16

1 Matrices
1.1 Introduction to matrices
A matrix is a two-dimensional array of numbers with certain arithmetic prop-
erties. It is usually represented inside brackets. For example:

" #
1 2 1
A= (1)
0 4 1
This matrix has 2 rows and 3 columns, and hence is called a 2 3 matrix.
The entries of a matrix are represented by two subscripts, the first for the
row and the second for the column. Hence, Aij is the entry in the ith row,
jth column. In the above example, then A12 = 2, while A21 = 0. A general
2 3 matrix may be written

" #
A11 A12 A13
A= (2)
A21 A22 A23

1
 A matrix with either one row or one column is called a vector. A vector
with only one row is called a row vector, and in this class will be written in
triangular braces. For example

v =< 1 6 1 > (3)

In general, we can write the vector

v =< v1 v2 ... vn > (4)

where n is the number of items in the vector.

Similarly a vector with only one column is called a column vector, and in
this class will be with curly braces. For example

1


v= 6 (5)
1

In general, a column vector may be written

v1


v



v= 2 (6)

...


vn

where n is the number of items in the vector.

1.2 Transpose operation

The main or diagonal (or just diagonal) of a matrix is the line of diagonal
entries from top left toward the bottom right, i.e. the entries A11 , A22 , down
to Ann . If the matrix is square (has the same number of rows as columns),
then Ann is the bottom right corner. Otherwise, n is the smaller of the
number of rows and the number of columns.
The transpose of a matrix is that matrix flipped about the main diag-
onal. The general definition is ATij = Aji . So if

" #
1 2 1
A= (7)
0 4 1

2
 then

1 0
T
A = 2

4 (8)
1 1

The transpose of an n m matrix is an m n matrix. The transpose of

a row vector is a column vector, and vice versa.

1.3 Matrix addition

If two matrices have the same size then they can be added simply by adding
the corresponding components of the two matrices. That is, if C = A + B,
then Cij = Aij + Bij for each entry ij. For example

" # " # " # " #

1 2 1 3 1 7 1 + 3 2 + 1 1 7 4 3 8
+ = = (9)
0 4 1 2 0 4 0 2 4 + 0 1 + 4 2 4 3

Matrices of different sizes cannot be added.

1.4 Matrix multiplication by a scalar

A matrix multiplied by a scalar results in all the terms being multiplied by
that scalar. In other words, if B = cA, then Bij = cAij . For example,

" # " # " #

10 12 2 10 2 12 20 24
2 = = (10)
20 0 2 20 2 0 40 0

This process can also be done in reverse, where a common factor is pulled
out of terms to simplify the expression. One matrix we will see is

" # " #
EA
e L
EA
L
EA 1 1
k = = (11)
EA
L
EA
L L 1 1

Using the formulas in the last two sections, we can now define matrix
subtraction

A B = A + (1 B) (12)

3
1.5 Matrix multiplication by vectors
Matrix-vector multiplication comes up often. The formula for matrix-vector
multiplication is as follows: If Ax = b, then

X
bi = Aij xj (13)
j

This is most easily seen by example. Heres one:

" # 5 ( ) ( )
1 2 3 15+26+37 38


6 = = (14)
0 4 1 05+4617 17

7

If the number of columns in the matrix is not equal to the number of rows
in the vector, then that matrix and vector cannot be multiplied.
Square matrices are of particular interest. A special square matrix is the
identity matrix, which is usually called I. It is the matrix extension of the
number 1. We know that 1 x = x for any number x. For the identity matrix

Iv = v (15)

for any vector v that has the same number of rows as the number of
columns in the identity matrix. The identity matrix has the form of having
1s on the main diagonal from top left to bottom right, and zeros elsewhere.
The 3 3 identity matrix, for example, is

1 0 0
0 1 0 (16)

0 0 1

The reader should verify that for any vector v with length 3, Iv = v. In
other words

1 0 0 v1
v1


0 1 0 v = v2 (17)

2
0 0 1 v3
v3

This should not be too difficult.

4
1.6 From linear equations to matrices
One of the most important applications of linear algebra (arguably the most
important) is the solution of systems of linear equations. A set of linear
equation may look like

2d1 + 4d2 2d3 = 2 (18)

d1 + 2d2 = 1 (19)
2d1 + d3 = 2 (20)
We are familiar with the traditional methods for solving such equation
from algebra. In order to solve with linear algebra, we need to transform this
into a matrix equation. We do this by performing matrix-vector multiplica-
tion in reverse. We first create a vector of unknowns

d1

d2 (21)


d3

and then figure out the coefficient for each one in each equation. The resulting
matrix equation looks like

2 4 2 d1

2

1 2 0 d2 = 1 (22)

2 0 1


d3
2

If we multiply the matrix and vector back, we can check that we get our
original 3 equations. To fill out our matrix, we ask ourselves, what is the
coefficient of d1 in the first equation? That number (2 here) goes in the
11 position of the matrix. Then, what is the coefficient of d2 in the first
equation? That goes in the 12 position, and so on until the first row is
completed. Then we go to the second row, and ask, what is the coefficient
of d1 in the second equation, and put that in the 21 position. We continue
until all the rows and columns are filled up.
Once we create a matrix equation, we are in a position to use linear
algebra to solve the system of equations.

1.7 Gauss elimination

The most common technique to solve a set of linear equations is called Gauss
elimination. Gauss elimination makes use several rows operations that can

5
be performed on a system of equations without changing the answer. The
operations of interest are:
1) We can multiply a given row by a scalar
2) We can add a scalar multiple of one row to another
3) We can interchange 2 rows.
These operations correspond to multiplying the corresponding equation
by a scalar, adding a multiple of one equation to another, and changing
the order of two equations, respectively. We know from algebra that these
operations will not change the answers.
Of course, whatever we apply to one side of the equation we must apply
to the other. Here is an example of how to use these operations to solve a
problem.
Consider the problem

2 4 2 d1
2


1 2 0 d2 = 1 (23)

2 0 1


d3
2

The goal is to manipulate the matrix until the lefthand side is the identity
matrix. Typically, we first eliminate the entries below the diagonal, starting
with the first column, changing the diagonal entries to 1 along the way. Then
we start with the last column, eliminating upwards.
So our first step is to get the first diagonal term equal 1. We accomplish
this by multiplying that row by 1/2.

R1 1/2 1 2 1 d1
1

1 2 0 d2 = 1 (24)

2 0 1


d3
2
Notice that, like all algebra, anything we do to the lefthand side of the
equation we have to do to the right. Next we eliminate K21 by adding 1
times the first row.

1 2 1 d1
1


R1 + R2 0 0 1 d2

= 0 (25)
2 0 1


d3
2

Now we eliminate K31

1 2 1 d1
1


0 0 1 d2 = 0 (26)

2 R1 + R3 0 4 1


d3
4

6
 We now wish to set K22 = 1. Typically, we would accomplish this by
multiplying the second row by the reciprocal of K22 . However, we cannot do
this when K22 = 0. In this case then, we interchange the second row with
the next row below that has a nonzero entry in second column (here, row 3).

1 2 1 d1
1


R3 0 4 1 d2

= 4 (27)

R2 0 0 1
d3
0

Now, multiply the second row by 1/4.

1 2 1 d1
1

1
R2 4 0 1 1/4

d2 = 1 (28)

0 0 1
d3
0
The next step would be to finish zeroing the second column below the
main diagonal, but this is done in this problem. We next multiple K33 by its
reciprocal, but we are lucky that this is also done. So we begin eliminating
the numbers above the diagonal, starting with the right column.

1 2 1 d1
1


1
R2 + 4 R3 0 1 0 d2

= 1 (29)

0 0 1
d3
0

And continue

R1 + R3 1 2 0 d1
1

0 1 0 d2 = 1 (30)


0 0 1 d3
0
And now continue to the next column

R1 2 R2 1 0 0 d1
1


0 1 0 d = 1 (31)

2
0 0 1 d3
0

And we are done. Since Id = d

d1
1


d2 = 1 (32)


d3
0

7
 It is a good idea to check our answers by plugging into our original prob-
lem, and indeed

2 4 2 1
2

1 2 0 1 = 1 (33)

2 0 1


0
2

Those students interested in more advanced techniques can look at meth-

ods like LU decomposition and Conjugate Gradient Methods, but it will not
be required for this class.

1.8 Multiplying two matrices

Consider two systems of equations Ax = b and By = x, where A, B, and
b are known, while x and y are variables. Suppose we would like to solve for
y. We would like to be able to write

ABy = b (34)

but what is the proper way to multiply AB? Lets look at an example
where x and y have length 2. In this case we can rewrite the equations in
non-matrix form as

A11 x1 + A12 x2 = b1 (35)

A21 x1 + A22 x2 = b2 (36)

and

B11 y1 + B12 y2 = x1 (37)

B21 y1 + B22 y2 = x2 (38)

Substituting the second pair of equations in to the first, we obtain

A11 (B11 y1 + B12 y2 ) + A12 (B21 y1 + B22 y2 ) = b1 (39)

A21 (B11 y1 + B12 y2 ) + A22 (B21 y1 + B22 y2 ) = b2 (40)

Gathering the terms for y1 and y2 ,

8
 (A11 B11 + A12 B21 )y1 + (A11 B12 + A12 B22 )y2 = b1 (41)
(A21 B11 + A22 B21 )y1 + (A21 B12 + A22 B22 )y2 = b2 (42)

Returning to matrix form

" #( ) ( )
A11 B11 + A12 B21 A11 B12 + A12 B22 y1 b1
= (43)
A21 B11 + A22 B21 A21 B12 + A22 B22 y2 b2
or

2
X 2
X

A1k Bk1 A1k Bk2 ( ) ( )
y1 b1

k=1 k=1


2 2
= (44)
X X y2 b2
A2k Bk1 A2k Bk2

k=1 k=1

In general if C = AB, then

n
X
Cij = Aik Bkj (45)
k=1

where n is the number of columns in A and the number of rows in B. If the

number of columns in A is not the same as the number of rows in B, then
the matrices cannot be multiplied. In general, if A is an m n matrix, and
B is an n p matrix, then C is m p.
For example

" # 4 0 1 2
1 2 3
1 3 1 4

0 4 1
0 5 2 1
" #
1 4 + 2 1 + 3 0 1 0 + 2 3 + 3 5 1 1 + 2 1 + 3 2 1 2 + 2 4 + 3 1
=
0 4 + 4 1 1 0 0 0 + 4 3 1 5 0 1 + 4 1 1 2 0 2 + 4 4 1 1
" #
2 21 7 13
= (46)
4 7 2 15

Notice that matrix-vector multiplication is just a special case of matrix-

matrix multiplication.

9
 An important note is that AB 6= BA. The size of the matrices may not
even be the same. Even if they are the same size, generally the answer will
not be the same. For example

" #" # " #

1 1 1 0 2 0
= (47)
0 0 1 0 0 0

while

" #" # " #

1 0 1 1 1 1
= (48)
1 0 0 0 1 1

Another important theorem is that

(AB)T = B T AT (49)

Proof:
Let C = AB. Then

   
(AB)T = CT = C ji (50)
ij ij

while

  n n n
B T AT B Tik ATkj =
X X X
= B ki Ajk = Ajk B ki = C ji (51)
ij
k=1 k=1 k=1

Hence the two sides are equal.

1.9 Matrix inversion

For a square matrix A, there often (but not always) exists an inverse matrix,
A1 , such that

AA1 = A1 A = I (52)

Notice that here, the inverse is the same on the left and right, even though
in general matrix multiplication is not commutative.

10
 The same process of Gauss elimination can be used to invert a matrix.
For example Let

4 2 2
4 6 8
A= (53)

2 2 4

We wish to solve the equation

4 2 2 1 0 0
1
4 6 8 A = 0 1 0 (54)

2 2 4 0 0 1

We simply use the row operations on both sides of the equation as before.
The goal is to change A into the identity matrix, which will make the right
hand side A1 , as we will see. We start, as before, by multiplying the first
row by 1/4.

1

R
4 1
1 .5 .5 .25 0 0
1
4 6 8 A = 0 1 0 (55)


2 2 4 0 0 1

Next, subtract 4 times the first row from the second

1 .5 .5 .25 0 0
1
R2 4R1
0 4 6 A = 1 1 0

(56)
2 2 4 0 0 1

Then add twice the first row to the third

1 .5 .5 .25 0 0
1
0 4 6 A = 1 1 0 (57)

R3 + 2R1 0 3 5 .5 0 1

Now divide row 2 by 4 to get the 1 on the diagonal

1 .5 .5 .25 0 0
1
.25R2 0 1 1.5 A = .25 .25 0

(58)
0 3 5 .5 0 1

11
 Now get rid of the 32 term by subtracting the 3 times the second row
from the third

1 .5 .5 .25 0 0
1
0 1 1.5 A = .25 .25 0 (59)

R3 3R2 0 0 .5 1.25 .75 1

Next, multiply row 3 by 2 to get 1 along the diagonal

1 .5 .5 .25 0 0
1
0 1 1.5 A = .25 .25 0 (60)

2R3 0 0 1 2.5 1.5 2

Now we can work our way back up the upper triangle to to zero out those
entries.

1 .5 .5 .25 0 0
1
R2 1.5R3 0 1 0 A = 4 2.5 3

(61)
0 0 1 2.5 1.5 2

Then

R1 .5R3 1 .5 0 1 .75 1
1

0 1 0 A =

4 2.5 3
(62)
0 0 1 2.5 1.5 2

Finally,

R1 .5R2 1 0 0 1 .5 .5
1
0 1 0 A = 4 2.5 3 (63)

0 0 1 2.5 1.5 2

Since

1 0 0
1 1 1
0 1 0 A = IA = A (64)

0 0 1

we can conclude

12

1 .5 .5
A1 = 4 2.5 3

(65)
2.5 1.5 2

Having the inverse is convenient for solving systems of equations, since if

we are trying to solve a system of equations Ax = b

A1 Ax = A1 b (66)
x = A1 b (67)

However, we see that inverting a matrix is more work than solving a

system of equations. For large systems of equations, it take about 3 times
as long to invert a matrix as it does to solve a system of equations. In finite
elements, this is typically where most of the computer time is spent. A good
finite element program never inverts a matrix when it can solve a system
of equations. However, it may make sense to invert a matrix that is known
beforehand, especially if it will be used many times (there are techniques,
e.g. LU decomposition, that allow for solving a system of equations multiple
times without computing the inverse of the matrix, but these are beyond the
scope of this class).

1.10 Non-invertible Matrices

We may find during the process of Gauss elimination that we cannot proceed
further, because one of the columns has become 0 everywhere on and below
the main diagonal. For example, look at the matrix equation

2 4 2 d1
2


1 2 1 d2 = 1 (68)

2 0 1


d3
2

Proceeding as before

R1 1/2 1 2 1 d1
1


1 2 1 d2 = 1 (69)

2 0 1


d3
2

Then

13

1 2 1 d1
1

R1 + R2 0 0 0 d2

= 0 (70)
2 0 1


d3
2

Then

1 2 1 d1
1

0 0 0 d2 = 0 (71)

2 R1 + R2 0 4 1 d3


4

Then

1 2 1 d1
1

R3 0 4 1 d2

= 4 (72)

R2 0 0 0
d3
0

Looking at the third row, there is no way to get the last entry to be 1
without spoiling the other entries in that row. This matrix is non-invertible,
or singular. Since the last equation is true (multiplying out, we get 0 = 0)
the system of equations has infinitely many solutions. If it were false (e.g. 0
= 4), the system would have no solutions.
One way to check whether a matrix is singular is to see if the determinant
is zero. Of course, to do this we need to know what the determinant is...

1.11 Determinants
The determinant, as we will learn later, is a scale factor of a linear map
between two spaces of the same dimension. This will make more sense later.
For now, we just discuss how to determine it and what it means for solving
equations.
For a 2-by-2 matrix, the determinant is

" #  
a b a b 

det =  = ad bc (73)

c d c d 



This may be thought of as multiplying the elements on each diagonal to-

gether, with those going up to the right being negative.
For larger matrices, we define the determinant recursively.

14
det A = A11 det (cof(A11 )) A12 det (cof(A12 )) + ... + (1)n+1 A1n det (cof(A1n ))(74)

where cof(Aij ) is the matrix formed by eliminating row i and column j from
the original matrix A. For example, if

a b c
A= d e f (75)

g h i

then

" #
d f
cofA12 = (76)
g i

which is formed by eliminating row 1 and column 2 from the original matrix
A. In this way, we can determine the determinant of an n n by taking the
determinants of smaller (n 1) (n 1) matrices.
Instead of using the first row, we can use any row or column. To keep
signs positive, we add the term if i + j is even, and subtract if it is odd.
As an example, lets examine a 3 3 matrix

 

 a b c  
 e f 
 
 d f 
 
 d e 

d e f  = a  b  + c (77)
       
 h i   g i   g h 
 
 
 g h i 
= aei ahf bdi + bgf + cdh cge (78)

If we examine closely, we notice that we can calculate this by multiplying

the components of the main diagonal, then adding the diagonal next it, wrap-
ping around when we reach the end (bf g), then adding the next diagonal,
wrapping around again (cdh). We then subtract the three diagonals going
up to the right. This is convenient for 3 3 matrices, but it is important to
note that this idea we use in 2 2 and 3 3 matrices does not extend to
larger matrices. Fortunately, we will rarely need to compute this for larger
matrices in this class.
If the determinant of a matrix is 0, the matrix is not invertible (singular),
and vice versa. This means that any system of equations solved with this
matrix will either have 0 or infinitely many solutions, and computer methods
that attempt to solve the system may fail.

15
1.12 Matrices in Matlab
(More examples of how to do linear algebra in Matlab will be posted later)
The strength of this approach is that calculators and computers can solve
these systems of equations very rapidly. This section shows how to solve a
system in one computer program: Matlab. I can usually figure out with a
little time how most of the graphing calculators work, so feel free to stop by
and I will look. See also
http://www.math.umbc.edu/ e campbell/Calculators/TI8x/linalg.html
for TI calculators.
Matlab treats everything as a matrix (scalars are 1 1 matrices, vectors
are n 1 matrices). To create a matrix, start with a left bracket, [, in
the command window followed by the numbers. Numbers in the same row
should be separated by spaces or commas, while changing rows should be
accomplished by using a semicolon or return. When done, enter a right
bracket, ] and hit return. The program will then display the matrix.

16
Next enter the vector and hit enter.

17
To solve, type inv(K) * b or K \ b.
And you are done.

18
19