AEM ADV03

Introductory Mathematics
2017/2018
Table of Contents:

Introductory Mathematics ........................................................................................................................... 4

1 Function Expansion & Transforms .......................................................................................................... 7
1.1 Infinite series ..................................................................................................................................... 7
1.2 Convergence ...................................................................................................................................... 9
1.3 Power Series .................................................................................................................................... 15
1.3.1 Taylor Series ............................................................................................................................. 15
1.3.2 Fourier Series ............................................................................................................................ 20
1.3.3 Complex Fourier series ............................................................................................................. 22
1.3.4 Termwise Integration and Differentiation ................................................................................ 23
1.3.5 Fourier series of Odd and Even functions........................................................................... 25
1.4 Integral Transform........................................................................................................................... 27
1.4.1 Fourier Transform..................................................................................................................... 27
1.4.2 Laplace Transform .................................................................................................................... 29
2. Vector Spaces, vector Fields & Operators ............................................................................................ 32
2.1 Scalar (inner) product of vector fields............................................................................................. 33
2.1.1 Lp norms ................................................................................................................................... 34
2.2 Vector product of vector fields........................................................................................................ 36
2.3 Vector operators .............................................................................................................................. 37
2.3.1 Gradient of a scalar field .......................................................................................................... 37
2.3.2 Divergence of a vector field ..................................................................................................... 40
2.3.3 Curl of a vector field ................................................................................................................ 42
2.4 Repeated Vector Operations – The Laplacian................................................................................. 44
3. Linear Algebra, Matrices & Eigenvectors ............................................................................................ 49
3.1 Basic definitions and notation ......................................................................................................... 49
3.2 Multiplication of matrices and multiplication of vectors and matrices ........................................... 51
3.2.1 Matrix multiplication ................................................................................................................ 51
3.2.2 Traces and determinants of square Cayley products ................................................................ 52
3.2.3 The Kronecker product ............................................................................................................. 52
3.3 Matrix Rank and the Inverse of a full rank matrix .......................................................................... 54
3.3.1 Full Rank matrices .................................................................................................................... 54

2
 3.3.2 Solutions of linear equations .................................................................................................... 55
3.3.3 Preservation of positive definiteness ........................................................................................ 55
3.3.4 A lower bound on the rank of a matrix product ....................................................................... 56
3.3.5 Inverse of products and sums of matrices ................................................................................ 56
3.4 Eigensystems ................................................................................................................................... 57
3.5 Diagonalisation of symmetric matrices ........................................................................................... 60
3.6 Matrix Factorization ........................................................................................................................ 62
3.6.1 Similarity Transform ................................................................................................................ 62
3.6.2 LU decomposition .................................................................................................................... 63
3.6.3 QR decomposition .................................................................................................................... 64
3.6.4 Singular Value decomposition.................................................................................................. 67
3.7 Solution of linear systems ............................................................................................................... 70
3.7.1 Direct methods .......................................................................................................................... 70
3.7.2 Iterative methods ...................................................................................................................... 70

3
Introductory Mathematics

What is Mathematics?
Different schools of thought, particularly in philosophy, have put forth radically different definitions of
mathematics. All are controversial and there is no consensus.

Leading definitions
1. Aristotle defined mathematics as: The science of quantity. In Aristotle's classification of the
sciences, discrete quantities were studied by arithmetic, continuous quantities by geometry.
2. Auguste Comte's definition tried to explain the role of mathematics in coordinating phenomena in
all other fields: The science of indirect measurement, 1851. The ``indirectness'' in Comte's
definition refers to determining quantities that cannot be measured directly, such as the distance to
planets or the size of atoms, by means of their relations to quantities that can be measured directly.
3. Benjamin Peirce: Mathematics is the science that draws necessary conclusions, 1870.
4. Bertrand Russell: All Mathematics is Symbolic Logic, 1903.
5. Walter Warwick Sawyer: Mathematics is the classification and study of all possible patterns, 1955.

Most contemporary reference works define mathematics mainly by summarizing its main topics and
methods:

6. Oxford English Dictionary: The abstract science which investigates deductively the conclusions
implicit in the elementary conceptions of spatial and numerical relations, and which includes as
its main divisions geometry, arithmetic, and algebra, 1933.
7. American Heritage Dictionary: The study of the measurement, properties, and relationships of
quantities and sets, using numbers and symbols, 2000.

Other playful, metaphorical, and poetic definitions:

8. Bertrand Russell: The subject in which we never know what we are talking about, nor whether
what we are saying is true, 1901.
9. Charles Darwin: A mathematician is a blind man in a dark room looking for a black cat which isn't
there.
10. G. H. Hardy: A mathematician, like a painter or poet, is a maker of patterns. If his patterns are
more permanent than theirs, it is because they are made with ideas, 1940.

Field of Mathematics

Mathematics can, broadly speaking, be subdivided into the study of quantity, structure, space, and change
(i.e. arithmetic, algebra, geometry, and analysis). In addition to these main concerns, there are also

4
subdivisions dedicated to exploring links from the heart of mathematics to other fields: to logic, to set
theory (foundations), to the empirical mathematics of the various sciences (applied mathematics), and
more recently to the rigorous study of uncertainty.

Mathematical awards

Arguably the most prestigious award in mathematics is the Fields Medal, established in 1936 and now
awarded every four years. The Fields Medal is often considered a mathematical equivalent to the Nobel
Prize.

The Wolf Prize in Mathematics, instituted in 1978, recognizes lifetime achievement, and another major
international award, the Abel Prize, was introduced in 2003. The Chern Medal was introduced in 2010 to
recognize lifetime achievement. These accolades are awarded in recognition of a particular body of work,
which may be innovational, or provide a solution to an outstanding problem in an established field.

A famous list of 23 open problems, called Hilbert's problem, was compiled in 1900 by German
mathematician David Hilbert. This list achieved great celebrity among mathematicians, and at least nine
of the problems have now been solved. A new list of seven important problems, titled the Millennium
Prize Problems, was published in 2000. A solution to each of these problems carries a $1 million reward,
and only one (the Riemann hypothesis) is duplicated in Hilbert's problems.

Mathematics in Aeronautics

Mathematics in aeronautics includes calculus, differential equations, and linear algebra, etc.

Calculus 1

Calculus has been an integral part of man's intellectual training and heritage for the last twenty-five
hundred years. Calculus is the mathematical study of change, in the same way that geometry is the study
of shape and algebra is the study of operations and their application to solving equations. It has two major
branches, differential calculus (concerning rates of change and slopes of curves), and integral calculus
(concerning accumulation of quantities and the areas under and between curves); these two branches are
related to each other by the fundamental theorem of calculus. Both branches make use of the fundamental
notions of convergence of infinite sequences and infinite series to a well-defined limit. Generally, modern
calculus is considered to have been developed in the 17th century by Isaac Newton and Gottfried Leibniz,
today calculus has widespread uses in science, engineering and economics and can solve many problems
that algebra alone cannot.

1
Extracted from: Boyer, Carl Benjamin. The history of the calculus and its conceptual development. Courier Dover Publications, 1949.

5
Differential and integral calculus is one of the great achievements of the human mind. The fundamental
definitions of the calculus, those of the derivative and the integral, are now so clearly stated in textbooks
on the subject, and the operations involving them are so readily mastered, that it is easy to forget the
difficulty with which these basic concepts have been developed. Frequently a clear and adequate
understanding of the fundamental notions underlying a branch of knowledge has been achieved
comparatively late in its development. This has never been more aptly demonstrated than in the rise of the
calculus. The precision of statement and the facility of application which the rules of the calculus early
afforded were in a measure responsible for the fact that mathematicians were insensible to the delicate
subtleties required in the logical development of the discipline. They sought to establish the calculus in
terms of the conceptions found in the traditional geometry and algebra which had been developed from
spatial intuition. During the eighteenth century, however, the inherent difficulty of formulating the
underlying concepts became increasingly evident, and it then became customary to speak of the
“metaphysics of the calculus”, thus implying the inadequacy of mathematics to give a satisfactory
exposition of the bases. With the clarification of the basic notions --which, in the nineteenth century, was
given in terms of precise mathematical terminology-- a safe course was steered between the intuition of
the concrete in nature (which may lurk in geometry and algebra) and the mysticism of imaginative
speculation (which may thrive on transcendental metaphysics). The derivative has throughout its
development been thus precariously situated between the scientific phenomenon of velocity and the
philosophical noumenon of motion.

The history of integration is similar. On the one hand, it had offered ample opportunity for interpretations
by positivistic thought in terms either of approximations or of the compensation of errors, views based on
the admitted approximate nature of scientific measurements and on the accepted doctrine of superimposed
effects. On the other hand, it has at the same time been regarded by idealistic metaphysics as a
manifestation that beyond the finitism of sensory percipiency there is a transcendent infinite which can be
but asymptotically approached by human experience and reason. Only the precision of their mathematical
definition --the work of the nineteenth century-- enables the derivative and the integral to maintain their
autonomous position as abstract concepts, perhaps derived from, but nevertheless independent of, both
physical description and metaphysical explanation.

6
1 Function Expansion & Transforms

A series expansion is a representation of a particular function of a sum of powers in one of its variables,
or by a sum of powers of another function 𝑓𝑓(𝑥𝑥). There are many areas in engineering, such as the motion
of fluids, the transfer of hear or processing of signals where the application of certain quantities involves
functions as independent variables. Therefore, it is important for us to understand how to solve each
function in the equations. In this chapter, we will cover infinite series, convergence and power series.

Furthermore, in engineering, transforms in one form to another plays a major role in analysis and design.
An area of continuing importance is the use of Laplace, Fourier, and other transforms in fields such as
communication, control and signal processing. These will be covered later in this chapter.

1.1 Infinite series

Many important differential equations appearing in practice cannot be solved in terms of these functions.
Solutions can often only be expressed as infinite series, that is, infinite sums of simpler functions such as
polynomials, or trig functions.

A polynomial series may be written as:

𝑃𝑃(𝑥𝑥) = � 𝑎𝑎𝑘𝑘 𝑥𝑥 𝑘𝑘 (1)

𝑘𝑘=0

where {𝑎𝑎𝑘𝑘 , 𝑘𝑘 = 0, 1, … ∞} are the expansion coefficients of the power series.

If {𝑎𝑎𝑘𝑘 }, 𝑘𝑘 = 0, 1, … is a sequence of numbers, the ordered sum of all its terms, namely:

� 𝑎𝑎𝑘𝑘 = 𝑎𝑎𝑜𝑜 + 𝑎𝑎1 + ∙∙∙ (2)

𝑘𝑘=0

is called an infinite series. There are a few infinite series that you’ll need to know as well as you know the
multiplication table. These are:

1. The geometric series

∞
1
1 + 𝑎𝑎 + 𝑎𝑎2 + 𝑎𝑎3 + 𝑎𝑎4 + 𝑎𝑎5 + ⋯ = � 𝑎𝑎𝑘𝑘 = (3)
1 − 𝑎𝑎
𝑘𝑘=0

We can also write geometric series as:

7
 ∞

� 𝑎𝑎𝑟𝑟 𝑘𝑘 (4)
𝑘𝑘=0

where 𝑎𝑎 is the first term and 𝑟𝑟 is the common ratio. This series will converge if | 𝑟𝑟 | < 1 and
diverges if otherwise.

2. The 𝑝𝑝 −series or hyperharmonic series

∞
1
� (5)
𝑘𝑘 𝑝𝑝
𝑘𝑘=1

This series will converge if 𝑝𝑝 > 1.

When 𝑝𝑝 = 1, the 𝑝𝑝 −series becomes a harmonic series which diverges.

Other series you need to know include the following which are known as the power series:

∞
𝑎𝑎
𝑎𝑎2 𝑎𝑎𝑘𝑘
𝑒𝑒 = 1 + 𝑎𝑎 + +⋯ =� (6)
2! 𝑘𝑘!
0

∞
𝑎𝑎3 𝑎𝑎2𝑘𝑘+1 (7)
sin 𝑎𝑎 = 𝑎𝑎 − + ⋯ = �(−1)𝑘𝑘
3! (2𝑘𝑘 + 1)!
0

∞
𝑎𝑎2 𝑎𝑎2𝑘𝑘 (8)
cos 𝑎𝑎 = 1 − + ⋯ = �(−1)𝑘𝑘
2! (2𝑘𝑘)!
0

∞
𝑎𝑎2 𝑎𝑎3 𝑘𝑘+1
𝑎𝑎𝑘𝑘 (9)
ln(1 + 𝑎𝑎) = 1 − + − ⋯ = �(−1) (|𝑎𝑎| < 1)
2 3 𝑘𝑘
0

It is better to understand their derivation instead of just memorizing the series. For example, the cosine
series is the derivative of the sine. Therefore, by knowing the sine series, you can differentiate it term by
term to obtain the cosine series. The logarithm of (1 + 𝑎𝑎) is an integral f 1/(1 + 𝑎𝑎) so you can obtain its
series from that of the geometric series.

8
1.2 Convergence
Does an infinite series converges? Does the limit as 𝑁𝑁 → ∞ of the sum below exist?

𝑁𝑁

� 𝑢𝑢𝑘𝑘 (10)
1

There are a few methods used to prove convergence of a series. These methods are:

1. The 𝑛𝑛 −th term rule

If you have a series with a sum of

� 𝑎𝑎𝑛𝑛 < ∞ (11)

𝑛𝑛=0

Where the lim 𝑎𝑎𝑛𝑛 = 0, then the series converges.

𝑛𝑛→∞

Example 1: Determine whether this infinite series below converges or diverges using the 𝑛𝑛 −th term rule:

∞
4𝑛𝑛2 − 𝑛𝑛3
�
7 − 3𝑛𝑛3
𝑛𝑛=1

Solution: The first step is to rewrite the equation above as:

4𝑛𝑛2 − 𝑛𝑛3
lim
𝑛𝑛→∞ 7 − 3𝑛𝑛3

Since we have both 𝑛𝑛3 in the numerator and denominator, so we can divide the series by 𝑛𝑛3 , and this gives
us

4
− 1
lim 𝑛𝑛
𝑛𝑛→∞ 7
−3
𝑛𝑛3

As 𝑛𝑛 → ∞, we can then rewrite the equation above as:

9
 4
− 1 −1 1
lim 𝑛𝑛 = =
𝑛𝑛→∞ 7 −3 3
−3
𝑛𝑛3

Hence,
4𝑛𝑛2 − 𝑛𝑛3 1
lim 3
= ≠ 0
𝑛𝑛→∞ 7 − 3𝑛𝑛 3

Therefore, the series diverges.

Example 2: Determine whether this infinite series below converges or diverges using the 𝑛𝑛 −th term rule:
∞
1
�
𝑛𝑛2
𝑛𝑛=1

Solution: By applying the n-th term rule, the equation above can be re-written as:

1
lim =0
𝑛𝑛→∞ 𝑛𝑛2

Therefore, this series converges.

2. The ratio test

To apply this comparison test, you will need a stable of known convergent series. Assuming you have a
series of

∞
(12)
� 𝑎𝑎𝑘𝑘
𝑘𝑘=0

Then, let

𝑎𝑎𝑘𝑘+1 (13)
𝑃𝑃 = lim � �
𝑘𝑘→∞ 𝑎𝑎𝑘𝑘

If 𝑃𝑃 < 1→ the series converges

𝑃𝑃 > 1→ the series diverges
𝑃𝑃 = 1→ then it is inconclusive

10
Example 3: Determine whether this infinite series below converges or diverges using the ratio test.

∞
(−10)𝑘𝑘
�
42𝑘𝑘+1 (𝑘𝑘 + 1)
𝑘𝑘=0

Solution: The first thing to do is to identify the series above for 𝑎𝑎𝑘𝑘 ,

(−10)𝑘𝑘
𝑎𝑎𝑘𝑘 =
42𝑘𝑘+1 (𝑘𝑘 + 1)

To compute 𝑎𝑎𝑘𝑘+1, we need to substitute 𝑘𝑘 + 1 for all 𝑘𝑘 in the equation of 𝑎𝑎𝑘𝑘 above, therefore:

(−10)𝑘𝑘+1 (−10)𝑘𝑘+1
𝑎𝑎𝑘𝑘+1 = =
42(𝑘𝑘+1)+1 ((𝑘𝑘 + 1) + 1) 42𝑘𝑘+3 (𝑘𝑘 + 2)

Now, we can define P as:

1
𝑃𝑃 = lim � 𝑎𝑎𝑘𝑘+1 ∙ �
𝑘𝑘→∞ 𝑎𝑎𝑘𝑘

Substituting the values of 𝑎𝑎𝑘𝑘+1 and 𝑎𝑎𝑘𝑘 into the equation above, we get:

(−10)𝑘𝑘+1 42𝑘𝑘+1 (𝑘𝑘 + 1)

𝑃𝑃 = lim � 2𝑘𝑘+3 ∙ �
𝑘𝑘→∞ 4 (𝑘𝑘 + 2) (−10)𝑘𝑘

−10 (𝑘𝑘 + 1)
𝑃𝑃 = lim � �
𝑘𝑘→∞ 42 (𝑘𝑘 + 2)

10 𝑘𝑘 + 1
𝑃𝑃 = lim
16 𝑘𝑘→∞ 𝑘𝑘 + 2

When 𝑘𝑘 → ∞, the ratio of (𝑘𝑘 + 1)/(𝑘𝑘 + 2) becomes 1, and therefore:

10
𝑃𝑃 = <1
16

Since 𝑃𝑃 < 1, therefore the series converges.

11
3. The Root test

The root test is similar to that of the ratio test. Assuming you have a series of

∞
(14)
� 𝑎𝑎𝑘𝑘
𝑘𝑘=0
Then, let

𝑛𝑛
1 (15)
𝑟𝑟 = lim �|𝑎𝑎𝑘𝑘 | = lim |𝑎𝑎𝑘𝑘 |𝑘𝑘
𝑘𝑘→∞ 𝑘𝑘→∞

If 𝑟𝑟 < 1→ the series converges

𝑟𝑟 > 1→ the series diverges
𝑟𝑟 = 1→ then it is inconclusive

Example 4: Determine whether this infinite series below converges or diverges using the root test.

∞
𝑘𝑘 𝑘𝑘
�
31+2𝑘𝑘
𝑘𝑘=1

Solution: Using the root test rule, we can re-write the equation above as:

1
𝑘𝑘
𝑘𝑘 𝑘𝑘
𝑟𝑟 = lim � �
𝑘𝑘→∞ 31+2𝑘𝑘

𝑘𝑘 ∞
𝑟𝑟 = lim = = ∞
𝑘𝑘→∞ 1+2 32
3𝑘𝑘

Therefore, since 𝑟𝑟 = ∞ > 1 , this series diverges.

Example 5: Determine whether this infinite series below converges or diverges using the root test.

∞ 𝑘𝑘
5𝑘𝑘 − 3𝑘𝑘 3
�� 3 �
7𝑘𝑘 + 2
𝑘𝑘=0

Solution: Using the root test rule, we can re-write the equation above as:

12
 1
3 𝑘𝑘 𝑘𝑘
5𝑘𝑘 − 3𝑘𝑘
𝑟𝑟 = lim �� � �
𝑘𝑘→∞ 7𝑘𝑘 3 + 2

5𝑘𝑘 − 3𝑘𝑘 3
𝑟𝑟 = lim � �
𝑘𝑘→∞ 7𝑘𝑘 3 + 2

Therefore, solving for 𝑟𝑟, we get

2
𝑟𝑟 = � �
9

Since 𝑟𝑟 < 1, this series converges.

4. Absolute Convergence

An absolutely convergent series is convergent. Any absolute series that converges, then the original series
converges too.

If: �|𝑎𝑎𝑘𝑘 | Converges

𝑘𝑘=1

� 𝑎𝑎𝑘𝑘 Converges too!

Then:
𝑘𝑘=1

5. Leibniz Theorem

Leibniz Theorem is used for alternating series. For example, we have a series as below:

∞
(16)
�(−1)𝑘𝑘 𝑎𝑎𝑘𝑘
𝑘𝑘=0

When 𝑎𝑎𝑘𝑘 > 0, 𝑎𝑎𝑘𝑘 > 𝑎𝑎𝑘𝑘+1 and 𝑎𝑎𝑘𝑘 → 0, then the series converges. Also, if 𝑎𝑎𝑘𝑘 is a decreasing sequence and
that the

13
 lim 𝑎𝑎𝑘𝑘 = 0
𝑘𝑘→∞

Then the series is convergent.

Example 6: Determine whether this infinite series below converges using Leibniz Theorem.

∞
(−1)𝑘𝑘+1
�
𝑘𝑘
𝑘𝑘=1

Solution: First, let’s look at 𝑎𝑎𝑘𝑘 , the equation above can be re-written as:

∞
1
�(−1)𝑘𝑘+1 ∙
𝑘𝑘
𝑘𝑘=1

Therefore,

1
𝑎𝑎𝑘𝑘 =
𝑘𝑘

Now, if we apply a limit to 𝑎𝑎𝑘𝑘 ,

1
lim 𝑎𝑎𝑘𝑘 = lim =0
𝑘𝑘→∞ 𝑘𝑘→∞ 𝑘𝑘

And we can see that 𝑎𝑎𝑘𝑘 > 𝑎𝑎𝑘𝑘+1 , therefore this series converges.

Of all the methods discussed above, it can be concluded that root test is more powerful because it can be
shown that

𝑎𝑎𝐾𝐾+1
If: 𝑃𝑃 = lim exist
𝑘𝑘→∞ 𝑎𝑎𝑘𝑘

1
𝑟𝑟 = lim (𝑎𝑎𝑘𝑘 )𝑘𝑘 exist too!
Then: 𝑘𝑘→∞

Therefore, 𝑃𝑃 = 𝑟𝑟. However, 𝑟𝑟 may exist even when 𝑃𝑃 does not!

14
1.3 Power Series
We must therefore give meaning to an infinite sum of constants, using this to give meaning to an infinite sum of
functions. When the functions being added are the simple powers (𝑥𝑥 − 𝑥𝑥𝑜𝑜 )𝑘𝑘 , the sum is called a Taylor (power)
series and if 𝑥𝑥𝑜𝑜 = 0, a Maclaurin series.

When the functions are trig terms such as 𝑠𝑠𝑠𝑠𝑠𝑠(𝑘𝑘𝑘𝑘) or 𝑐𝑐𝑐𝑐𝑐𝑐(𝑘𝑘𝑘𝑘), the series might be a Fourier series, certain infinite
sums of trig functions that can be made to represent arbitrary functions, even functions with discontinuities. This
type of infinite series is also generalized to sums of other functions such as Legendre polynomials. Eventually,
solutions of differential equations will be given in terms of infinite sums of Bessel functions, themselves infinite
series.

1.3.1 Taylor Series

Having understood sequences, series and power series, now we will focus to one of the main topic: Taylor
polynomials. The Taylor polynomial approximation is given by:

1 𝑥𝑥
𝑓𝑓(𝑥𝑥) = 𝑝𝑝𝑛𝑛 (𝑥𝑥) + � (𝑥𝑥 − 𝑡𝑡)𝑛𝑛 𝑓𝑓 (𝑛𝑛+1) (𝑡𝑡)𝑑𝑑𝑑𝑑 (17)
𝑛𝑛! 𝑎𝑎

Where the 𝑛𝑛-th degree Taylor polynomial 𝑝𝑝𝑛𝑛 (𝑥𝑥) is given by:

𝑓𝑓 ′ (𝑎𝑎) 𝑓𝑓 (𝑛𝑛) (𝑎𝑎)

𝑝𝑝𝑛𝑛 (𝑥𝑥) = 𝑓𝑓(𝑎𝑎) + (𝑥𝑥 − 𝑎𝑎) + ⋯ + (𝑥𝑥 − 𝑎𝑎)𝑛𝑛 (18)
1! 𝑛𝑛!

When 𝑎𝑎 = 0, the series is also called Maclaurin series.

There are 2 conditions apply:

1. 𝑓𝑓 (𝑥𝑥), 𝑓𝑓 (1) (𝑥𝑥), ⋯ , 𝑓𝑓 (𝑛𝑛+1) (𝑥𝑥) are continuous in a closed interval containing 𝑥𝑥 = 𝑎𝑎.
2. 𝑥𝑥 is any point in the interval.

A Taylor series represents a function for a given value as an infinite sum of terms that are calculated from
the value of the function’s derivatives.

Therefore, the Taylor series of a function 𝑓𝑓 (𝑥𝑥) for a value 𝑎𝑎 is the power series, and can be written as:

∞
𝑓𝑓 𝑛𝑛 (𝑎𝑎)
𝑓𝑓(𝑥𝑥) = � (𝑥𝑥 − 𝑎𝑎)𝑛𝑛 (19)
𝑛𝑛!
𝑛𝑛=0

15
Example 7: Find the Maclaurin series of a function 𝑓𝑓 (𝑥𝑥) = 𝑒𝑒 𝑥𝑥 and its radius of convergence.

Solution: So, if 𝑓𝑓 (𝑥𝑥) = 𝑒𝑒 𝑥𝑥 , then 𝑓𝑓 (𝑛𝑛) (𝑥𝑥) = 𝑒𝑒 𝑥𝑥 , so 𝑓𝑓 (𝑛𝑛) (0) = 𝑒𝑒 0 = 1 for all 𝑛𝑛.

Therefore, the Taylor series for 𝑓𝑓 at 0 (which is the Maclaurin series), so:

∞ ∞
𝑓𝑓 𝑛𝑛 (0) 𝑘𝑘
𝑥𝑥 𝑛𝑛 𝑥𝑥 𝑥𝑥 2 𝑥𝑥 3
𝑓𝑓(𝑥𝑥) = � (𝑥𝑥) = � = 1+ + + +⋯
𝑛𝑛! 𝑛𝑛! 1! 2! 3!
𝑛𝑛=0 𝑛𝑛=0

To find the radius of convergence, let 𝑎𝑎𝑛𝑛 = 𝑥𝑥 𝑛𝑛 /𝑛𝑛! . Then,

𝑎𝑎𝑛𝑛+1 𝑥𝑥 𝑛𝑛+1 𝑛𝑛! |𝑥𝑥|

� �= � ∙ 𝑛𝑛 � = →0<1
𝑎𝑎𝑛𝑛 (𝑛𝑛 + 1)! 𝑥𝑥 𝑛𝑛 + 1

So, by Ratio Test, the series converges for all 𝑥𝑥 and the radius of convergence is 𝑅𝑅 = ∞

The conclusion we can draw from example 7 is that if 𝑒𝑒 𝑥𝑥 has a power series expansion at 0, then:

∞
𝑥𝑥
𝑥𝑥 𝑛𝑛
𝑒𝑒 = �
𝑛𝑛!
𝑛𝑛=0

So now, under what circumstances is a function equal to the sum of its Taylor series? Or if 𝑓𝑓 has
derivatives of all orders, when is it that equation (19) is true?

With any convergent series, this means that 𝑓𝑓(𝑥𝑥) is the limit of the sequence of partial sums. In the case
of Taylor series, the partial sums can be written as in as equation (18), where:

𝑓𝑓 ′ (𝑎𝑎) 𝑓𝑓 ′′ (𝑎𝑎) 𝑓𝑓 (𝑛𝑛) (𝑎𝑎)

𝑝𝑝𝑛𝑛 (𝑥𝑥) = 𝑓𝑓(𝑎𝑎) + (𝑥𝑥 − 𝑎𝑎) + (𝑥𝑥 − 𝑎𝑎)2 ⋯ + (𝑥𝑥 − 𝑎𝑎)𝑛𝑛
1! 2! 𝑛𝑛!

For the example of the exponential function 𝑓𝑓 (𝑥𝑥) = 𝑒𝑒 𝑥𝑥 , the results from example 7 shows that the Taylor
polynomials at 0 (or Maclaurin polynomials) with 𝑛𝑛 = 1, 2 and 3 are:

𝑝𝑝1 (𝑥𝑥) = 1 + 𝑥𝑥

𝑥𝑥 2
𝑝𝑝2 (𝑥𝑥) = 1 + 𝑥𝑥 +
2!

16
 𝑥𝑥 2 𝑥𝑥 3
𝑝𝑝3 (𝑥𝑥) = 1 + 𝑥𝑥 + +
2! 3!

In general, 𝑓𝑓 (𝑥𝑥) is the sum of its Taylor series if

𝑓𝑓(𝑥𝑥) = lim 𝑝𝑝𝑛𝑛 (𝑥𝑥) (20)

𝑛𝑛→∞

If we let

𝑅𝑅𝑛𝑛 (𝑥𝑥) = 𝑓𝑓(𝑥𝑥) − 𝑝𝑝𝑛𝑛 (𝑥𝑥) so that 𝑓𝑓(𝑥𝑥) = 𝑝𝑝𝑛𝑛 (𝑥𝑥) + 𝑅𝑅𝑛𝑛 (𝑥𝑥) (21)

Then, 𝑅𝑅𝑛𝑛 (𝑥𝑥) is called the remainder of the Taylor series.

If we show that lim 𝑅𝑅𝑛𝑛 (𝑥𝑥) = 0, then it follows that from equation (21):
𝑛𝑛→∞

lim 𝑝𝑝𝑛𝑛 (𝑥𝑥) = lim [𝑓𝑓(𝑥𝑥) − 𝑅𝑅𝑛𝑛 (𝑥𝑥)] = 𝑓𝑓 (𝑥𝑥) − lim 𝑅𝑅(𝑥𝑥) = 𝑓𝑓(𝑥𝑥)
𝑛𝑛→∞ 𝑛𝑛→∞ 𝑛𝑛→∞

We have therefore proved the following:

If 𝑓𝑓(𝑥𝑥) = 𝑝𝑝𝑛𝑛 (𝑥𝑥) + 𝑅𝑅𝑛𝑛 (𝑥𝑥), where 𝑝𝑝𝑛𝑛 is the 𝑛𝑛th degree Taylor Polynomial of 𝑓𝑓 at 𝑎𝑎 and

lim 𝑅𝑅𝑛𝑛 (𝑥𝑥) = 0 (22)

𝑛𝑛→∞

for |𝑥𝑥 − 𝑎𝑎| < 𝑅𝑅, then 𝑓𝑓 is equals to the sum of its Taylor series on the interval |𝑥𝑥 − 𝑎𝑎| < 𝑅𝑅.

Therefore, if 𝑓𝑓 has 𝑛𝑛 + 1 derivatives in an interval 𝐼𝐼 that contains the number 𝑎𝑎, then for 𝑥𝑥 in 𝐼𝐼 there is a
number 𝑧𝑧 strictly between 𝑥𝑥 and 𝑎𝑎 such that the remainder term in the Taylor series can be expressed as

𝑓𝑓 (𝑛𝑛+1) (𝑧𝑧)
𝑅𝑅𝑛𝑛 (𝑥𝑥) = (𝑥𝑥 − 𝑎𝑎)𝑛𝑛+1 (23)
(𝑛𝑛 + 1)!

Example 8: Find the Maclaurin series for sin 𝑥𝑥 and prove that it represents sin 𝑥𝑥 for all 𝑥𝑥.

Solution: First, we arrange our computation in two columns as follows:

𝑓𝑓(𝑥𝑥) = sin 𝑥𝑥 𝑓𝑓(0) = 0

𝑓𝑓 (1) (𝑥𝑥) = cos 𝑥𝑥 𝑓𝑓 (1) (0) = 1
𝑓𝑓 (2) (𝑥𝑥) = −sin 𝑥𝑥 𝑓𝑓 (2) (0) = 0

17
 𝑓𝑓 (3) (𝑥𝑥) = −cos 𝑥𝑥 𝑓𝑓 (3) (0) = 1
𝑓𝑓 (4) (𝑥𝑥) = −cos 𝑥𝑥 𝑓𝑓 (4) (0) = 0

Since the derivatives repeat 𝑛𝑛 a cycle of four, we can write the Maclaurin series as follow:

𝑓𝑓 (1) (0) 𝑓𝑓 (2) (0) 2 𝑓𝑓 (3) (0) 3 𝑓𝑓 (4) (0) 4 1 0 −1 3 0 4

𝑓𝑓(0) + 𝑥𝑥 + 𝑥𝑥 + 𝑥𝑥 + 𝑥𝑥 + ⋯ =0+ 𝑥𝑥 + 𝑥𝑥 2 + 𝑥𝑥 + 𝑥𝑥 + ⋯
1! 2! 3! 4! 1! 2! 3! 4!

𝑥𝑥 3 𝑥𝑥 5 𝑥𝑥 7
= 𝑥𝑥 − + − +⋯
3! 5! 7!

∞
𝑥𝑥 2𝑘𝑘+1
𝑘𝑘
= �(−1)
(2𝑘𝑘 + 1)!
𝑘𝑘=0

You can try with different types of functions, and you will get a Maclaurin series table that looks like this:

∞
1
= � 𝑥𝑥 𝑛𝑛 = 1 + 𝑥𝑥 + 𝑥𝑥 2 + 𝑥𝑥 3 + ⋯ 𝑅𝑅 = 1
1 − 𝑥𝑥
𝑛𝑛=0
∞ 𝑛𝑛
𝑥𝑥 𝑥𝑥 𝑥𝑥 2 𝑥𝑥 3
𝑒𝑒 𝑥𝑥 = � =1+ + + +⋯ 𝑅𝑅 = ∞
𝑛𝑛! 1! 2! 3!
𝑛𝑛=0
∞
𝑥𝑥 (2𝑛𝑛+1)𝑛𝑛
𝑥𝑥 3 𝑥𝑥 5 𝑥𝑥 7
sin 𝑥𝑥 = �(−1) = 𝑥𝑥 − + − + ⋯ 𝑅𝑅 = ∞
(2𝑛𝑛 + 1)! 3! 5! 7!
𝑛𝑛=0
∞
𝑥𝑥 2𝑛𝑛 𝑥𝑥 2 𝑥𝑥 4 𝑥𝑥 6
𝑛𝑛
cos 𝑥𝑥 = �(−1) =1− + − +⋯ 𝑅𝑅 = ∞
(2𝑛𝑛)! 2! 4! 6!
𝑛𝑛=0
∞
𝑥𝑥 2𝑛𝑛+1 𝑥𝑥 3 𝑥𝑥 5 𝑥𝑥 7
tan−1 𝑥𝑥 = �(−1)𝑛𝑛 = 𝑥𝑥 − + − + ⋯ 𝑅𝑅 = 1
2𝑛𝑛 + 1 3 5 7
𝑛𝑛=0
∞
𝑛𝑛−1
𝑥𝑥 𝑛𝑛 𝑥𝑥 2 𝑥𝑥 3 𝑥𝑥 4
ln(1 + 𝑥𝑥) = �(−1) = 𝑥𝑥 − + − + ⋯ 𝑅𝑅 = 1
𝑛𝑛 2 3 4
𝑛𝑛=0

18
Example 9: Find the first 3 terms of the Taylor series for the function sin 𝜋𝜋𝜋𝜋 centered at 𝑎𝑎 = 0.5. Use
𝜋𝜋 𝜋𝜋
your answer to find an approximate value to sin �2 + 10�

Solution: Let us first do the derivatives for the function given:

𝑓𝑓(𝑥𝑥) = sin 𝜋𝜋𝑥𝑥 .

Therefore, 𝑓𝑓 (1) 𝑥𝑥 = 𝜋𝜋 cos 𝜋𝜋𝜋𝜋 ,
𝑓𝑓 (2) 𝑥𝑥 = −𝜋𝜋 2 sin 𝜋𝜋𝜋𝜋 ,
𝑓𝑓 (3) 𝑥𝑥 = −𝜋𝜋 3 cos 𝜋𝜋𝜋𝜋 ,
𝑓𝑓 (4) 𝑥𝑥 = 𝜋𝜋 4 sin 𝜋𝜋𝜋𝜋
And so,

Substituting this back into equation (17), we get:

1 2 1 4
𝜋𝜋 �𝑥𝑥 − 2� �𝑥𝑥 − 2�
sin 𝜋𝜋𝜋𝜋 = sin + × (−𝜋𝜋)2 + × 𝜋𝜋 4 + ⋯
2 2! 4!

1 2 1 4
�𝑥𝑥 − 2� �𝑥𝑥 − 2�
= 1 − 𝜋𝜋 2 + 𝜋𝜋 4 +⋯
2! 4!

Therefore,

1 2 1 4
1 1 �10� �10�
sin 𝜋𝜋 � + � = 1 − 𝜋𝜋 2 + 𝜋𝜋 4 +⋯
2 10 2! 4!

= 1 − 0.0493 + 0.0004
= 0.9511

19
1.3.2 Fourier Series

As mentioned previously, a Fourier series decomposes periodic functions into a sum of sines and cosines
(trigonometric terms or complex exponentials). For a periodic function 𝑓𝑓(𝑥𝑥), periodic on [−𝐿𝐿, 𝐿𝐿], its
Fourier series representation is:

∞
𝑛𝑛𝑛𝑛𝑛𝑛 𝑛𝑛𝑛𝑛𝑛𝑛
𝑓𝑓(𝑥𝑥) = 0.5𝑎𝑎0 + � �𝑎𝑎𝑛𝑛 cos � � + 𝑏𝑏𝑛𝑛 sin � �� (24)
𝐿𝐿 𝐿𝐿
𝑛𝑛=1

where 𝑎𝑎0 , 𝑎𝑎𝑛𝑛 and 𝑏𝑏𝑛𝑛 are the Fourier coefficients and they can be written as:

1 𝐿𝐿
𝑎𝑎0 = � 𝑓𝑓(𝑥𝑥)𝑑𝑑𝑑𝑑
2𝐿𝐿 −𝐿𝐿 (25)

1 𝐿𝐿 𝑛𝑛𝑛𝑛𝑛𝑛
𝑎𝑎𝑛𝑛 = � 𝑓𝑓(𝑥𝑥) cos � � 𝑑𝑑𝑑𝑑
𝐿𝐿 −𝐿𝐿 𝐿𝐿 (26)

1 𝐿𝐿 𝑛𝑛𝑛𝑛𝑛𝑛
𝑏𝑏𝑛𝑛 = � 𝑓𝑓(𝑥𝑥) sin � � 𝑑𝑑𝑑𝑑 (27)
𝐿𝐿 −𝐿𝐿 𝐿𝐿

where period, 𝑝𝑝 = 2𝐿𝐿. Equation (24) is also called Real Fourier series.

There are 2 conditions apply:

1. 𝑓𝑓(𝑥𝑥) is a piecewise continuous is piecewise continuous on the closed interval [−𝐿𝐿, 𝐿𝐿]. A function
is said to be piecewise continuous on the closed interval [𝑎𝑎, 𝑏𝑏] provided that it is continuous there,
with at most a finite number of exceptions where, at worst, we would find a removable or jump
discontinuity. At both a removable and a jump discontinuity, the one-sided limits 𝑓𝑓(𝑡𝑡 + ) =
lim+ 𝑓𝑓(𝑥𝑥) and 𝑓𝑓(𝑡𝑡 − ) = lim− 𝑓𝑓(𝑥𝑥) exist and are finite.
𝑥𝑥→𝑡𝑡 𝑥𝑥→𝑡𝑡

2. A sum of continuous and periodic functions converges pointwise to a possibly discontinuous and
non-periodic function. This was a startling realisation for mathematicians of the early nineteenth
century.

20
Example 10: Find the Fourier series of (𝑥𝑥) = 𝑥𝑥 2 , −1 < 𝑥𝑥 < 1

Solution: In this example, period, 𝑝𝑝 = 2, but we know that 𝑝𝑝 = 2𝐿𝐿, therefore, 𝐿𝐿 = 1.

First, let us find 𝑎𝑎0 . From equation (25),

1 𝐿𝐿
𝑎𝑎0 = � 𝑓𝑓(𝑥𝑥)𝑑𝑑𝑑𝑑
2𝐿𝐿 −𝐿𝐿

1 1 2 1
𝑎𝑎0 = � 𝑥𝑥 𝑑𝑑𝑑𝑑 =
2 −1 3

Next, let us find 𝑏𝑏𝑛𝑛 . From equation (27),

1 𝐿𝐿 𝑛𝑛𝑛𝑛𝑛𝑛
𝑏𝑏𝑛𝑛 = � 𝑓𝑓(𝑥𝑥) sin � � 𝑑𝑑𝑑𝑑
𝐿𝐿 −𝐿𝐿 𝐿𝐿

1 1 2
𝑏𝑏𝑛𝑛 = � 𝑥𝑥 sin 𝑛𝑛𝑛𝑛𝑛𝑛 𝑑𝑑𝑑𝑑 = 0
1 −1

Finally, we will find 𝑎𝑎𝑛𝑛 . From equation (26),

1 𝐿𝐿 𝑛𝑛𝑛𝑛𝑛𝑛
𝑎𝑎𝑛𝑛 = � 𝑓𝑓(𝑥𝑥) cos � � 𝑑𝑑𝑑𝑑
𝐿𝐿 −𝐿𝐿 𝐿𝐿

1 1 2
𝑎𝑎𝑛𝑛 = � 𝑥𝑥 cos 𝑛𝑛𝑛𝑛𝑛𝑛 𝑑𝑑𝑑𝑑
1 −1

Solving using integration by parts, we get:

2𝑥𝑥𝑥𝑥𝑥𝑥𝑥𝑥𝑥𝑥𝑥𝑥𝑥𝑥 1
𝑎𝑎𝑛𝑛 = �
𝑛𝑛2 𝜋𝜋 2 −1
2
𝑎𝑎𝑛𝑛 = 2 2 [(−1)𝑛𝑛 + (−1)𝑛𝑛 ]
𝑛𝑛 𝜋𝜋

4(−1)𝑛𝑛
𝑎𝑎𝑛𝑛 =
𝑛𝑛2 𝜋𝜋 2

Therefore, the Fourier series can be written as:

21
 ∞
2)
1 4(−1)𝑛𝑛
𝑓𝑓(𝑥𝑥 = + � 2 2 cos(𝑛𝑛𝑛𝑛𝑛𝑛)
3 𝑛𝑛 𝜋𝜋
𝑛𝑛=1

1.3.3 Complex Fourier series

A function 𝑓𝑓(𝑥𝑥) can also be expressed as a Complex Fourier series and can be defined to be:

+∞

𝑓𝑓(𝑥𝑥) = � 𝑐𝑐𝑛𝑛 𝑒𝑒 𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖/𝐿𝐿 (28)

−∞

where

1 𝜋𝜋
𝑐𝑐𝑛𝑛 = � 𝑓𝑓(𝑥𝑥)𝑒𝑒 −𝑖𝑖𝑖𝑖𝑖𝑖 (29)
2𝜋𝜋 −𝜋𝜋

We know that:

𝑒𝑒 𝑖𝑖𝑖𝑖 = cos 𝑥𝑥 + 𝑖𝑖 sin 𝑥𝑥

𝑒𝑒 −𝑖𝑖𝑖𝑖 = cos 𝑥𝑥 − 𝑖𝑖 sin 𝑥𝑥

(30)
𝑒𝑒 𝑖𝑖𝑖𝑖 − 𝑒𝑒 −𝑖𝑖𝑖𝑖 = 2𝑖𝑖 sin 𝑥𝑥

𝑒𝑒 𝑖𝑖𝑖𝑖 + 𝑒𝑒 −𝑖𝑖𝑖𝑖 = 2 cos 𝑥𝑥

Therefore, from equation (29),

1 𝜋𝜋
𝑐𝑐𝑛𝑛 = � 𝑓𝑓(𝑥𝑥)𝑒𝑒 −𝑖𝑖𝑖𝑖𝑖𝑖
2𝜋𝜋 −𝜋𝜋

1 1 𝜋𝜋 1 𝜋𝜋
𝑐𝑐𝑛𝑛 = � � 𝑓𝑓(𝑥𝑥) cos 𝑛𝑛𝑛𝑛 𝑑𝑑𝑑𝑑 − 𝑖𝑖 � 𝑓𝑓(𝑥𝑥) sin 𝑛𝑛𝑛𝑛 𝑑𝑑𝑑𝑑�
2 𝜋𝜋 −𝜋𝜋 𝜋𝜋 −𝜋𝜋

Hence, we can write:

1
⟹ 𝑐𝑐𝑛𝑛 = (𝑎𝑎 − 𝑖𝑖𝑏𝑏𝑛𝑛 ) , 𝑛𝑛 > 0
2 𝑛𝑛

22
 1
⟹ 𝑐𝑐𝑛𝑛 = (𝑎𝑎 + 𝑖𝑖𝑏𝑏−𝑛𝑛 ) , 𝑛𝑛 < 0
2 −𝑛𝑛
⟹ 𝑐𝑐𝑛𝑛 = 𝑎𝑎0 , 𝑛𝑛 = 0

Example 11: Write the complex Fourier transform of 𝑓𝑓(𝑥𝑥) = 2 sin 𝑥𝑥 − cos 10𝑥𝑥

Solution: Here, we can expand the function by substituting the sin and cos functions from equation (30),
we get:

𝑒𝑒 𝑖𝑖𝑖𝑖 − 𝑒𝑒 −𝑖𝑖𝑖𝑖 𝑒𝑒 10𝑖𝑖𝑖𝑖 + 𝑒𝑒 −10𝑖𝑖𝑖𝑖

𝑓𝑓(𝑥𝑥) = 2 −
2𝑖𝑖 2

1 1 1 1
𝑓𝑓(𝑥𝑥) = 𝑒𝑒 𝑖𝑖𝑖𝑖 − 𝑒𝑒 −𝑖𝑖𝑖𝑖 − 𝑒𝑒 10𝑖𝑖𝑖𝑖 − 𝑒𝑒 −10𝑖𝑖𝑖𝑖
𝑖𝑖 𝑖𝑖 2 2

Therefore:

1 1 1 1
𝑐𝑐1 = , 𝑐𝑐10 = − , 𝑐𝑐−1 = − , 𝑐𝑐−10 = −
𝑖𝑖 2 𝑖𝑖 2

1.3.4 Termwise Integration and Differentiation

Parseval’s Identity

Consider a Fourier series below and expand it

𝑓𝑓(𝑥𝑥) = 𝑎𝑎0 + �{𝑎𝑎𝑛𝑛 cos 𝑛𝑛𝑛𝑛 + 𝑏𝑏𝑛𝑛 sin 𝑛𝑛𝑛𝑛} = 𝑎𝑎0 + 𝑎𝑎1 cos 𝑥𝑥 + 𝑏𝑏1 sin 𝑥𝑥 + 𝑎𝑎2 cos 2𝑥𝑥 + 𝑏𝑏2 sin 2𝑥𝑥 + ⋯
𝑛𝑛=1

Square it, we get:

𝑁𝑁 𝑁𝑁
2 (𝑥𝑥) 2 2 2 2 2
𝑓𝑓 = 𝑎𝑎0 + ��𝑎𝑎𝑛𝑛 cos 𝑛𝑛𝑛𝑛 + 𝑏𝑏𝑛𝑛 sin 𝑛𝑛𝑛𝑛� + 2𝑎𝑎0 �( 𝑎𝑎𝑛𝑛 cos 𝑛𝑛𝑛𝑛 + 𝑏𝑏𝑛𝑛 sin 𝑛𝑛𝑛𝑛)
𝑛𝑛=1 𝑛𝑛=1
𝑁𝑁

+ 2𝑎𝑎1 cos 𝑥𝑥 𝑏𝑏1 sin 𝑥𝑥 + 2𝑎𝑎1 cos 𝑥𝑥 �( 𝑎𝑎𝑛𝑛 cos 𝑛𝑛𝑛𝑛 + 𝑏𝑏𝑛𝑛 sin 𝑛𝑛𝑛𝑛) + ⋯
𝑛𝑛=1
+ 2𝑎𝑎𝑁𝑁 cos 𝑁𝑁𝑁𝑁 𝑏𝑏𝑁𝑁 sin 𝑁𝑁𝑁𝑁

23
Integrate both sides, we get:

𝜋𝜋 𝜋𝜋 𝑁𝑁

� 𝑓𝑓 2 (𝑥𝑥)
𝑑𝑑𝑑𝑑 = � �𝑎𝑎0 + ��𝑎𝑎𝑛𝑛 2 cos2 𝑛𝑛𝑛𝑛 + 𝑏𝑏𝑛𝑛 2 sin2 𝑛𝑛𝑛𝑛� + ⋯� 𝑑𝑑𝑑𝑑
2
−𝜋𝜋 −𝜋𝜋 𝑛𝑛=1

𝜋𝜋 𝑁𝑁

⟹ � 𝑓𝑓 2 (𝑥𝑥)
𝑑𝑑𝑑𝑑 = 2𝜋𝜋𝑎𝑎0 + ��𝜋𝜋𝜋𝜋𝑛𝑛 2 + 𝜋𝜋𝜋𝜋𝑛𝑛 2 � + 0
2
−𝜋𝜋 𝑛𝑛=1

Parseval’s Identity can be written as:

∞
1 𝐿𝐿
� |𝑓𝑓(𝑥𝑥)|2 𝑑𝑑𝑑𝑑 = 2|𝑎𝑎0 |2 + �(|𝑎𝑎𝑛𝑛 |2 + |𝑏𝑏𝑛𝑛 |2 ) (31)
𝐿𝐿 −𝐿𝐿
𝑛𝑛=1

If:

a) 𝑓𝑓(𝑥𝑥) is continuous, and 𝑓𝑓 ′ (𝑥𝑥) is a piecewise continuous on [−𝐿𝐿, 𝐿𝐿]

b) 𝑓𝑓(𝐿𝐿) = 𝑓𝑓(−𝐿𝐿)
c) 𝑓𝑓 ′′ (𝑥𝑥) exist at 𝑥𝑥 in (−𝐿𝐿, 𝐿𝐿),

Therefore:

∞
′ (𝑥𝑥)
𝜋𝜋 𝑛𝑛𝑛𝑛𝑛𝑛 𝑛𝑛𝑛𝑛𝑛𝑛
𝑓𝑓 = � 𝑛𝑛 �−𝑎𝑎𝑛𝑛 sin + 𝑏𝑏𝑛𝑛 cos � (32)
𝐿𝐿 𝐿𝐿 𝐿𝐿
𝑛𝑛=1

Example 12: From Example 10, we found that the Fourier series is:

∞
1 4(−1)𝑛𝑛
𝑓𝑓(𝑥𝑥 = + � 2 2 cos(𝑛𝑛𝑛𝑛𝑛𝑛) , 𝑥𝑥 2
2)
3 𝑛𝑛 𝜋𝜋
𝑛𝑛=1

Solution: Applying Parseval’s to the equation above, we get:

∞
1 2 16 1
2
2 � � + � 4 4 = � 𝑥𝑥 4 𝑑𝑑𝑑𝑑 =
3 𝑛𝑛 𝜋𝜋 −1 5
𝑛𝑛=1

24
 ∞
16 2 2 8
⟹� 4 4
= − =
𝑛𝑛 𝜋𝜋 5 9 45
𝑛𝑛=1

∞
1 𝜋𝜋 4
⟹� 4=
𝑛𝑛 90
𝑛𝑛=1

1.3.5 Fourier series of Odd and Even functions

A function 𝑓𝑓(𝑥𝑥) is called an 𝑒𝑒𝑒𝑒𝑒𝑒𝑒𝑒 or 𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠 function if it has the property

𝑓𝑓(−𝑥𝑥) = 𝑓𝑓(𝑥𝑥) (33)

A function 𝑓𝑓(𝑥𝑥) is called an 𝑜𝑜𝑜𝑜𝑜𝑜 or 𝑎𝑎𝑛𝑛𝑛𝑛𝑛𝑛𝑛𝑛𝑛𝑛𝑛𝑛𝑛𝑛𝑛𝑛𝑛𝑛𝑛𝑛𝑛𝑛𝑛𝑛 function if

𝑓𝑓(−𝑥𝑥) = −𝑓𝑓(𝑥𝑥) (34)

A function that is neither even nor odd can be represented as the sum of an even and an odd function.

Cosine waves are even, so any Fourier series representation of a periodic function must have an even
symmetry. A function 𝑓𝑓(𝑥𝑥) defined on [0, 𝐿𝐿] can be extended as an even periodic function (𝑏𝑏𝑛𝑛 = 0).
Therefore, the Fourier series representation of an even function is:

∞
𝑛𝑛𝑛𝑛𝑛𝑛 2 𝐿𝐿 𝑛𝑛𝑛𝑛𝑛𝑛
𝑓𝑓(𝑥𝑥) = 0.5𝑎𝑎0 + � 𝑎𝑎𝑛𝑛 cos � � , 𝑎𝑎𝑛𝑛 = � 𝑓𝑓(𝑥𝑥) cos � � 𝑑𝑑𝑑𝑑 (35)
𝐿𝐿 𝐿𝐿 0 𝐿𝐿
𝑛𝑛=1

Similarly sine waves are odd, so any Fourier sine series representation of a periodic function must have
odd symmetry. Therefore a function 𝑓𝑓(𝑥𝑥) defined on [0, 𝐿𝐿] can be extended as an odd periodic function
(𝑎𝑎𝑛𝑛 = 0) and the Fourier series representation of an even function is:

∞
𝑛𝑛𝑛𝑛𝑛𝑛 2 𝐿𝐿 𝑛𝑛𝑛𝑛𝑛𝑛
𝑓𝑓(𝑥𝑥) = 0.5𝑎𝑎0 + � 𝑏𝑏𝑛𝑛 sin � � , 𝑏𝑏𝑛𝑛 = � 𝑓𝑓(𝑥𝑥) sin � � 𝑑𝑑𝑑𝑑 (36)
𝐿𝐿 𝐿𝐿 0 𝐿𝐿
𝑛𝑛=1

Example 13: If 𝑓𝑓(𝑥𝑥) is even, show that

2 𝐿𝐿 𝑛𝑛𝑛𝑛𝑛𝑛
(a) 𝑎𝑎𝑛𝑛 = 𝐿𝐿 ∫0 𝑓𝑓(𝑥𝑥) cos � 𝐿𝐿
� 𝑑𝑑𝑑𝑑

25
 (b) 𝑏𝑏𝑛𝑛 = 0

Solution: For an even function, we can write the equation as:

1 𝐿𝐿 𝑛𝑛𝑛𝑛𝑛𝑛 1 0 𝑛𝑛𝑛𝑛𝑛𝑛 1 𝐿𝐿 𝑛𝑛𝑛𝑛𝑛𝑛

𝑎𝑎𝑛𝑛 = � 𝑓𝑓(𝑥𝑥) cos 𝑑𝑑𝑑𝑑 = � 𝑓𝑓(𝑥𝑥) cos 𝑑𝑑𝑑𝑑 + � 𝑓𝑓(𝑥𝑥) cos 𝑑𝑑𝑥𝑥
𝐿𝐿 −𝐿𝐿 𝐿𝐿 𝐿𝐿 −𝐿𝐿 𝐿𝐿 𝐿𝐿 0 𝐿𝐿

Let x=-u, we can re-write:

1 0 𝑛𝑛𝑛𝑛𝑛𝑛 1 𝐿𝐿 −𝑛𝑛𝑛𝑛𝑛𝑛 1 𝐿𝐿 𝑛𝑛𝑛𝑛𝑛𝑛

� 𝑓𝑓(𝑥𝑥) cos 𝑑𝑑𝑑𝑑 = � 𝑓𝑓(−𝑢𝑢) cos � � 𝑑𝑑𝑑𝑑 = � 𝑓𝑓(𝑢𝑢) cos � � 𝑑𝑑𝑑𝑑
𝐿𝐿 −𝐿𝐿 𝐿𝐿 𝐿𝐿 0 𝐿𝐿 𝐿𝐿 0 𝐿𝐿

Since by definition of an even function f(-u) = f(u). Then:

1 𝐿𝐿 𝑛𝑛𝑛𝑛𝑛𝑛 1 𝐿𝐿 𝑛𝑛𝑛𝑛𝑛𝑛 2 𝐿𝐿 𝑛𝑛𝑛𝑛𝑛𝑛

𝑎𝑎𝑛𝑛 = � 𝑓𝑓(𝑢𝑢) cos � � 𝑑𝑑𝑑𝑑 + � 𝑓𝑓(𝑥𝑥) cos 𝑑𝑑𝑑𝑑 = � 𝑓𝑓(𝑥𝑥) cos 𝑑𝑑𝑑𝑑
𝐿𝐿 0 𝐿𝐿 𝐿𝐿 0 𝐿𝐿 𝐿𝐿 0 𝐿𝐿

To show that 𝑏𝑏𝑛𝑛 = 0, we can write the expression as

1 𝐿𝐿 𝑛𝑛𝑛𝑛𝑛𝑛 1 0 𝑛𝑛𝑛𝑛𝑛𝑛 1 𝐿𝐿 𝑛𝑛𝑛𝑛𝑛𝑛

𝑏𝑏𝑛𝑛 = � 𝑓𝑓(𝑥𝑥) sin � � 𝑑𝑑𝑑𝑑 = � 𝑓𝑓(𝑥𝑥) sin � � 𝑑𝑑𝑑𝑑 + � 𝑓𝑓(𝑥𝑥) sin � � 𝑑𝑑𝑑𝑑
𝐿𝐿 −𝐿𝐿 𝐿𝐿 𝐿𝐿 −𝐿𝐿 𝐿𝐿 𝐿𝐿 0 𝐿𝐿

If we make the transformation x=-u in the first integral on the right of the equation above, we obtain:

1 0 𝑛𝑛𝑛𝑛𝑛𝑛 1 𝐿𝐿 𝑛𝑛𝑛𝑛𝑛𝑛 1 𝐿𝐿 𝑛𝑛𝑛𝑛𝑛𝑛

� 𝑓𝑓(𝑥𝑥) sin � � 𝑑𝑑𝑑𝑑 = � 𝑓𝑓(−𝑢𝑢) sin �− � 𝑑𝑑𝑑𝑑 = − � 𝑓𝑓(𝑢𝑢) sin � � 𝑑𝑑𝑑𝑑
𝐿𝐿 −𝐿𝐿 𝐿𝐿 𝐿𝐿 0 𝐿𝐿 𝐿𝐿 0 𝐿𝐿

1 𝐿𝐿 𝑛𝑛𝑛𝑛𝑛𝑛 1 𝐿𝐿 𝑛𝑛𝑛𝑛𝑛𝑛
= − � 𝑓𝑓(𝑢𝑢) sin � � 𝑑𝑑𝑑𝑑 = − � 𝑓𝑓(𝑥𝑥) sin � � 𝑑𝑑𝑑𝑑
𝐿𝐿 0 𝐿𝐿 𝐿𝐿 0 𝐿𝐿

Therefore, substituting this into the equation for 𝑏𝑏𝑛𝑛 , we get

1 𝐿𝐿 𝑛𝑛𝑛𝑛𝑛𝑛 1 𝐿𝐿 𝑛𝑛𝑛𝑛𝑛𝑛
𝑏𝑏𝑛𝑛 = − � 𝑓𝑓(𝑥𝑥) sin � � 𝑑𝑑𝑑𝑑 + � 𝑓𝑓(𝑥𝑥) sin � � 𝑑𝑑𝑑𝑑 = 0
𝐿𝐿 0 𝐿𝐿 𝐿𝐿 0 𝐿𝐿

26
1.4 Integral Transform
An integral transform is any transform of the following form

𝑥𝑥2
𝐹𝐹(𝑤𝑤) = � 𝐾𝐾(𝑤𝑤, 𝑥𝑥)𝑓𝑓(𝑥𝑥) 𝑑𝑑𝑑𝑑 (37)
𝑥𝑥1

With the following inverse transform

𝑤𝑤2
𝑓𝑓(𝑥𝑥) = � 𝐾𝐾 −1 (𝑤𝑤, 𝑥𝑥)𝐹𝐹(𝑤𝑤) 𝑑𝑑𝑑𝑑 (38)
𝑤𝑤1

1.4.1 Fourier Transform

A Fourier series expansion of a function 𝑓𝑓(𝑥𝑥) of a real variable 𝑥𝑥 with a period of 2𝐿𝐿 is defined over a
finite interval −𝐿𝐿 ≤ 𝑥𝑥 ≤ 𝐿𝐿 . If the interval becomes infinite and we sum over infinitesimals, we then obtain
the Fourier integral

1 ∞
𝑓𝑓(𝑥𝑥) = � 𝐹𝐹(𝑤𝑤)𝑒𝑒 𝑖𝑖𝑖𝑖𝑖𝑖 𝑑𝑑𝑑𝑑 (39)
2𝜋𝜋 −∞

with the coefficients

∞
𝐹𝐹(𝑤𝑤) = � 𝑓𝑓(𝑥𝑥)𝑒𝑒 −𝑖𝑖𝑖𝑖𝑖𝑖 𝑑𝑑𝑑𝑑 (40)
−∞

Equation (40) is the Fourier transform of 𝑓𝑓(𝑥𝑥). The Fourier integral is also known as the inverse Fourier
transform of 𝐹𝐹(𝑤𝑤). In this example, 𝑥𝑥1 = 𝑤𝑤1 = −∞, 𝑥𝑥2 = 𝑤𝑤2 = ∞ and 𝐾𝐾(𝑤𝑤, 𝑥𝑥) = 𝑒𝑒 −𝑖𝑖𝑖𝑖𝑖𝑖 . The Fourier
transform transforms a function of one variable (e.g. time in seconds) which lives in the time domain to a
second function which lives in the frequency domain and changes the basis of the function to cosines and
sines.

Example 14: Find the Fourier transform of

1 ∶ −2 < 𝑥𝑥 < 2
𝑓𝑓(𝑥𝑥) = �
0 ∶ 𝑜𝑜𝑜𝑜ℎ𝑒𝑒𝑒𝑒𝑒𝑒𝑒𝑒𝑒𝑒𝑒𝑒

Solution: We can write the Fourier transform as in equation (40):

27
 2
1
𝐹𝐹(𝑤𝑤) = � 𝑓𝑓(𝑥𝑥)𝑒𝑒 −𝑖𝑖𝑖𝑖𝑖𝑖 𝑑𝑑𝑑𝑑
√2𝜋𝜋 −2

2
1
= � 𝑒𝑒 −𝑖𝑖𝑖𝑖𝑖𝑖 𝑑𝑑𝑑𝑑
√2𝜋𝜋 −2

2
1 𝑒𝑒 −𝑖𝑖𝑖𝑖𝑖𝑖
= � � �
√2𝜋𝜋 −𝑖𝑖𝑖𝑖 −2

−1
= �𝑒𝑒 −2𝑖𝑖𝑖𝑖 − 𝑒𝑒 2𝑖𝑖𝑖𝑖 �
𝑖𝑖𝑖𝑖√2𝜋𝜋

−1
= [(cos 2𝑤𝑤 − 𝑖𝑖 sin 2𝑤𝑤) − (cos 2𝑤𝑤 + 𝑖𝑖 sin 2𝑤𝑤)]
𝑖𝑖𝑖𝑖√2𝜋𝜋

−1
= [−2𝑖𝑖 sin 2𝑤𝑤]
𝑖𝑖𝑖𝑖√2𝜋𝜋

2 sin 2𝑤𝑤
=� � �
𝜋𝜋 𝑤𝑤

Example 15: Find the Fourier transform of

𝑡𝑡 ∶ −1 ≤ 𝑡𝑡 ≤ 1
𝑓𝑓(𝑡𝑡) = �
0 ∶ 𝑒𝑒𝑒𝑒𝑒𝑒𝑒𝑒𝑒𝑒ℎ𝑒𝑒𝑒𝑒𝑒𝑒

Solution: Recalling the Fourier transform in equation (40), we can write

∞
𝐹𝐹(𝑤𝑤) = � 𝑓𝑓(𝑥𝑥)𝑒𝑒 −𝑖𝑖𝑖𝑖𝑖𝑖 𝑑𝑑𝑑𝑑
−∞

1
= � 𝑡𝑡 ∙ 𝑒𝑒 −𝑖𝑖𝑖𝑖𝑖𝑖 𝑑𝑑𝑑𝑑
−1

By applying integration by parts, we get:

1
𝑡𝑡 𝑖𝑖𝑖𝑖𝑖𝑖 1 1 −𝑖𝑖𝑖𝑖𝑖𝑖
=� 𝑒𝑒 � − � 𝑒𝑒 𝑑𝑑𝑑𝑑
−𝑖𝑖𝑖𝑖 −1 −1 −𝑖𝑖𝑖𝑖

28
 1
We can also rewrite − 𝑖𝑖 = 𝑖𝑖, therefore:

𝑖𝑖𝑖𝑖 1
1 1 𝑖𝑖𝑖𝑖𝑖𝑖 1
= � 𝑒𝑒 𝑖𝑖𝑖𝑖𝑖𝑖 � + � ∙ 𝑒𝑒 �
𝑤𝑤 −1 𝑖𝑖𝑖𝑖 −𝑖𝑖𝑖𝑖 −1

𝑖𝑖𝑖𝑖 𝑖𝑖𝑖𝑖𝑖𝑖 1 1 𝑖𝑖𝑖𝑖𝑖𝑖 1

= � 𝑒𝑒 � + � 2 𝑒𝑒 �
𝑤𝑤 −1 𝑤𝑤 −1

𝑖𝑖 −𝑖𝑖𝑖𝑖 1
= �𝑒𝑒 + 𝑒𝑒 𝑖𝑖𝑖𝑖 � + 2 �𝑒𝑒 −𝑖𝑖𝑖𝑖 + 𝑒𝑒 𝑖𝑖𝑖𝑖 �
𝑤𝑤 𝑤𝑤

𝑖𝑖 1 −𝑖𝑖𝑖𝑖 2𝑖𝑖 1
=2 �𝑒𝑒 + 𝑒𝑒 𝑖𝑖𝑤𝑤 � + �− 2 ∙ � �𝑒𝑒 𝑖𝑖𝑖𝑖 − 𝑒𝑒 −𝑖𝑖𝑖𝑖 �
𝑤𝑤 2 𝑤𝑤 2𝑖𝑖

2𝑖𝑖 2𝑖𝑖
= cos 𝑤𝑤 − 2 sin 𝑤𝑤
𝑤𝑤 𝑤𝑤

2𝑖𝑖 1
= �cos 𝑤𝑤 − sin 𝑤𝑤�
𝑤𝑤 𝑤𝑤

1.4.2 Laplace Transform

The Laplace transform is an example of an integral transform that will convert a differential equation into
an algebraic equation. The Laplace transform of a function 𝑓𝑓(𝑥𝑥) of a variable 𝑥𝑥 is defined as the integral

∞
𝐹𝐹(𝑠𝑠) = ℒ{𝑓𝑓(𝑡𝑡)} = � 𝑓𝑓(𝑡𝑡)𝑒𝑒 −𝑠𝑠𝑠𝑠 𝑑𝑑𝑑𝑑 (41)
0

Where s is a real, positive parameter that serves as a supplementary variable. The conditions are: if 𝑓𝑓(𝑡𝑡)
is piecewise continuous on (0, ∞), and of exponential order (|𝑓𝑓(𝑡𝑡)| ≤ 𝐾𝐾𝑒𝑒 𝛼𝛼𝛼𝛼 for some 𝐾𝐾 and 𝛼𝛼 > 0), then
𝐹𝐹(𝑠𝑠) exists for 𝑠𝑠 > 𝛼𝛼. Several Laplace transforms are given in the table below, where 𝑎𝑎 is a constant and
𝑛𝑛 is an integer.

Example 16: Find the Laplace transforms of the following functions:

3 ∶ 0 < 𝑡𝑡 < 5
𝑓𝑓(𝑡𝑡) = �
0∶ 𝑡𝑡 > 0

29
Solution:
∞ 5 ∞
−𝑠𝑠𝑠𝑠 −𝑠𝑠𝑠𝑠
ℒ{𝑓𝑓(𝑡𝑡)} = � 𝑓𝑓(𝑡𝑡) 𝑒𝑒 𝑑𝑑𝑑𝑑 = � 3 ∙ 𝑒𝑒 𝑑𝑑𝑑𝑑 + � 0 ∙ 𝑒𝑒 −𝑠𝑠𝑠𝑠 𝑑𝑑𝑑𝑑
0 0 5

5
𝑒𝑒 −𝑠𝑠𝑠𝑠
= 3� � +0
−𝑠𝑠 0

𝑒𝑒 −5𝑠𝑠 1
= 3� − �
−𝑠𝑠 −𝑠𝑠

3
= (1 − 𝑒𝑒 −5𝑠𝑠 )
𝑠𝑠

Example 17: Find the Laplace transforms of the following functions:

𝑡𝑡 ∶ 0 < 𝑡𝑡 < 𝑎𝑎
𝑓𝑓(𝑡𝑡) = �
𝑏𝑏 ∶ 𝑡𝑡 > 𝑎𝑎

Solution:
∞ 𝑎𝑎 ∞
ℒ{𝑓𝑓(𝑡𝑡)} = � 𝑓𝑓(𝑡𝑡) 𝑒𝑒 −𝑠𝑠𝑠𝑠 𝑑𝑑𝑡𝑡 = � 𝑡𝑡 ∙ 𝑒𝑒 −𝑠𝑠𝑠𝑠 𝑑𝑑𝑑𝑑 + � 𝑏𝑏 ∙ 𝑒𝑒 −𝑠𝑠𝑠𝑠 𝑑𝑑𝑑𝑑
0 0 𝑎𝑎

𝑎𝑎 ∞
𝑒𝑒 −𝑠𝑠𝑠𝑠 𝑒𝑒 −𝑠𝑠𝑠𝑠 𝑒𝑒 −𝑠𝑠𝑠𝑠
=� 𝑡𝑡 − 2 ∙ 1� + 𝑏𝑏 � �
−𝑠𝑠 𝑠𝑠 0
−𝑠𝑠 𝑎𝑎

𝑎𝑎 1 1 𝑏𝑏
= 𝑒𝑒 −𝑎𝑎𝑎𝑎 �− − 2 � − 𝑒𝑒 0 �0 − 2 � − (0 − 𝑒𝑒 −𝑎𝑎𝑎𝑎 )
𝑠𝑠 𝑠𝑠 𝑠𝑠 𝑠𝑠

1 𝑏𝑏 − 𝑎𝑎 1 −𝑎𝑎𝑎𝑎
= 2
+� − 2 � 𝑒𝑒
𝑠𝑠 𝑠𝑠 𝑠𝑠

Example 17: Determine the Laplace transform of the function below:

𝑓𝑓(𝑡𝑡) = 5 − 3𝑡𝑡 + 4 sin 2𝑡𝑡 − 6𝑒𝑒 4𝑡𝑡

Solution: First, let’s break the equation one by one, we get:

30
 5
ℒ{5} = , 𝑅𝑅𝑅𝑅 (𝑠𝑠) > 0
𝑠𝑠

1
ℒ{𝑡𝑡} = , 𝑅𝑅𝑅𝑅 (𝑠𝑠) > 0
𝑠𝑠 2

2
ℒ{sin 2𝑡𝑡} = , 𝑅𝑅𝑅𝑅 (𝑠𝑠) > 0
𝑠𝑠 2 + 4

1
ℒ{𝑒𝑒 4𝑡𝑡 } = , 𝑅𝑅𝑅𝑅 (𝑠𝑠) > 4
𝑠𝑠 − 4

Therefore, by linearity property,

ℒ{𝑓𝑓(𝑡𝑡)} = ℒ{5 − 3𝑡𝑡 + 4 sin 2𝑡𝑡 − 6𝑒𝑒 4𝑡𝑡 }

= ℒ{5} − 3ℒ{𝑡𝑡} + 4ℒ{sin 2𝑡𝑡} − 6ℒ{𝑒𝑒 4𝑡𝑡 }

5 3 8 6
= − 2+ 2 −
𝑠𝑠 𝑠𝑠 𝑠𝑠 + 4 𝑠𝑠 − 4

LAPLACE TRANSFORMS
−𝟏𝟏 {𝑭𝑭(𝒔𝒔)} 𝑭𝑭(𝒔𝒔) = 𝓛𝓛{𝒇𝒇(𝒔𝒔)}
𝒇𝒇(𝒙𝒙) = 𝓛𝓛
𝑎𝑎
𝑎𝑎
𝑠𝑠
1
𝑡𝑡
𝑠𝑠 2
(𝑛𝑛!)
𝑥𝑥 𝑛𝑛
𝑠𝑠 𝑛𝑛+1
1
𝑒𝑒 𝑎𝑎𝑎𝑎
(𝑠𝑠 − 𝑎𝑎)
𝑎𝑎
sin 𝑎𝑎𝑎𝑎
(𝑠𝑠 + 𝑎𝑎2 )
2
𝑠𝑠
cos 𝑎𝑎𝑎𝑎
(𝑠𝑠 2 + 𝑎𝑎2 )
𝑎𝑎
sinh 𝑎𝑎𝑎𝑎
(𝑠𝑠 − 𝑎𝑎2 )
2
𝑠𝑠
cosh 𝑎𝑎𝑎𝑎
(𝑠𝑠 − 𝑎𝑎2 )
2

31
2. Vector Spaces, vector Fields & Operators
In the context of physics we are often interested in a quantity or property which varies in a smooth and
continuous way over some one-, two-, or three-dimensional region of space. This constitutes either a scalar
field or a vector field, depending on the nature of property. In this chapter, we consider the relationship
between a scalar field involving a variable potential and a vector field involving ‘field’, where this means
force per unit mass or change. The properties of scalar and vector fields are described and how they lead
to important concepts, such as that of a conservative field, and the important and useful Gauss and Stokes
theorems. Finally examples will be given to demonstrate the ideas of vector analysis.

There are basically four types of functions involving scalars and vectors:

• Scalar functions of a scalar, 𝑓𝑓(𝑥𝑥)

• Vector function of a scalar, 𝒓𝒓(𝑡𝑡)
• Scalar function of a vector, 𝜑𝜑(𝒓𝒓)
• Vector function of a vector, 𝑨𝑨(𝒓𝒓)

1. The vector x is normalised if x Tx = 1

2. The vectors x and y are orthogonal if x Ty = 0
3. The vectors x1, x2 , …, x𝑛𝑛 are linearly independent if the only numbers which satisfy the equation
𝑎𝑎1 x1 + 𝑎𝑎2 x2 + … + 𝑎𝑎𝑛𝑛 x𝑛𝑛 = 0 are 𝑎𝑎1 = 𝑎𝑎2 = . . . = 𝑎𝑎𝑛𝑛 = 0
4. The vectors x1, x2 , …, x𝑛𝑛 form a basis for a 𝑛𝑛 −dimensional vector-space if any vector x in the vector-
space can be written as a linear combination of vectors in the basis thus x = 𝑎𝑎1 x1 + 𝑎𝑎2 x2 + ⋯ + 𝑎𝑎𝑛𝑛 x𝑛𝑛
where 𝑎𝑎1 , 𝑎𝑎2 , ⋯ , 𝑎𝑎𝑛𝑛 are scalars.

Figure 1: Components of a vector

32
For example, a vector A from the origin in the figure above to a point P in the 3-dimensions takes the
form
𝑨𝑨 = 𝑎𝑎𝑥𝑥 𝚤𝚤̂ + 𝑎𝑎𝑦𝑦 𝚥𝚥̂ + 𝑎𝑎𝑧𝑧 𝑘𝑘� (42)

Where �𝚤𝚤̂, 𝚥𝚥̂, 𝑘𝑘��are unit vectors along the {𝑥𝑥, 𝑦𝑦, 𝑧𝑧} axes, respectively. The vector components �𝑎𝑎𝑥𝑥 , 𝑎𝑎𝑦𝑦 , 𝑎𝑎𝑧𝑧, �
are the corresponding distances along the axes. The length or magnitude of Vector 𝑨𝑨 is

|𝑨𝑨| = �𝑎𝑎𝑥𝑥 2 + 𝑎𝑎𝑦𝑦 2 + 𝑎𝑎𝑧𝑧 2 (43)

2.1 Scalar (inner) product of vector fields

The scalar product of vector fields is also called as the dot product. For example, if we have 2 vectors as
𝑨𝑨 = (𝐴𝐴1 , 𝐴𝐴2 , 𝐴𝐴3 ) and 𝑩𝑩 = (𝐵𝐵1 , 𝐵𝐵2 , 𝐵𝐵3 ), therefore,

〈𝑨𝑨, 𝑩𝑩〉 = 𝑨𝑨 ∙ 𝑩𝑩 = 𝑨𝑨𝑇𝑇 𝑩𝑩 = 𝐴𝐴1 𝐵𝐵1 + 𝐴𝐴2 𝐵𝐵2 + 𝐴𝐴3 𝐵𝐵3 (44)

We can also write

𝑨𝑨 ∙ 𝑩𝑩 = ‖𝑨𝑨‖‖𝑩𝑩‖ cos 𝜃𝜃 (45)

where 𝜃𝜃 is the angle between 𝑨𝑨 and 𝑩𝑩 satisfying 0 ≤ 𝜃𝜃 ≤ 𝜋𝜋. The inner product of vectors is a scalar.
The scalar product obeys the product laws which are listed below:

Product laws:

1. Commutative: 𝑨𝑨 ∙ 𝑩𝑩 = 𝑩𝑩 ∙ 𝑨𝑨
2. Associative: 𝑚𝑚𝑨𝑨 ∙ 𝑛𝑛𝑩𝑩 = 𝑚𝑚𝑚𝑚𝑨𝑨 ∙ 𝑩𝑩
3. Distributive: 𝑨𝑨 ∙ (𝑩𝑩 + 𝑪𝑪) = 𝑨𝑨 ∙ 𝑩𝑩 + 𝑨𝑨 ∙ 𝑪𝑪
1 1
4. Cauchy-Schwarz inequality: 𝑨𝑨 ∙ 𝑩𝑩 ≤ (𝑨𝑨 ∙ 𝑨𝑨)2 (𝑩𝑩 ∙ 𝑩𝑩)2

Note that a relation such as 𝑨𝑨 ∙ 𝑩𝑩 = 𝑨𝑨 ∙ 𝑪𝑪 does not imply that 𝑩𝑩 = 𝑪𝑪, as

𝑨𝑨 ∙ 𝑩𝑩 − 𝑨𝑨 ∙ 𝑪𝑪 = 𝑨𝑨 ∙ (𝑩𝑩 − 𝑪𝑪) = 0 (46)

Therefore, the correct conclusion is that 𝑨𝑨 is perpendicular to the vector 𝑩𝑩 − 𝑪𝑪.

Example 1: Determine the angle between 𝑨𝑨 = 〈1,3, −2〉 and 𝑩𝑩 = 〈−2, 4, −1〉.

33
Solution: All we need to do here is to rewrite equation (45) as:

𝑨𝑨 ∙ 𝑩𝑩
cos 𝜃𝜃 =
‖𝑨𝑨‖‖𝑩𝑩‖

Therefore we know that:

𝑨𝑨 ∙ 𝑩𝑩
cos 𝜃𝜃 =
‖𝑨𝑨‖‖𝑩𝑩‖

We’ll first have to compute the individual parameters

𝑨𝑨 ∙ 𝑩𝑩 = 12 ‖𝑨𝑨‖ = √14 ‖𝑩𝑩‖ = √21

Hence, the angle between the vectors is:

12
cos 𝜃𝜃 = = 0.69985 ⟹ 𝜃𝜃 = cos −1 (0.69985) = 45.58°
√14√21

2.1.1 Lp norms

There are many norms that could be defined for vectors. One type of norms is called the 𝐿𝐿𝑝𝑝 norm, often
denoted as ‖ ∙ ‖𝑝𝑝 . For 𝑝𝑝 ≥ 1, it is defined as the 𝑝𝑝 − 𝑛𝑛𝑛𝑛𝑛𝑛𝑛𝑛 and can be written as:

1
𝑛𝑛 𝑝𝑝
‖𝑥𝑥‖𝑝𝑝 = ��‖𝑥𝑥𝑖𝑖 ‖𝑝𝑝 � , 𝑥𝑥 = [𝑥𝑥1 , ⋯ 𝑥𝑥𝑛𝑛 ]𝑇𝑇 (47)
𝑖𝑖=1

There are a few types of norms such as the following:

1. ‖𝑥𝑥‖1 = ∑𝑖𝑖|𝑥𝑥𝑖𝑖 |, also called the Manhattan norm because it corresponds to sums of distances along
coordinate axes, as one would travel along the rectangular street plan of Manhattan.
2. ‖𝑥𝑥‖2 = �∑𝑖𝑖|𝑥𝑥𝑖𝑖 |2 , also called the Euclidean norm, the Euclidean length, or just the length of the
vector.
3. ‖𝑥𝑥‖∞ = 𝑚𝑚𝑎𝑎𝑎𝑎𝑖𝑖 |𝑥𝑥𝑖𝑖 |, also called the max norm or the Chebyshev norm.

34
Some relationships of norms are as below:

‖𝑥𝑥‖∞ ≤ ‖𝑥𝑥‖2 ≤ ‖𝑥𝑥‖1

‖𝑥𝑥‖∞ ≤ ‖𝑥𝑥‖2 ≤ √𝑛𝑛‖𝑥𝑥‖∞ (48)

‖𝑥𝑥‖2 ≤ ‖𝑥𝑥‖1 ≤ √𝑛𝑛‖𝑥𝑥‖2

If we define the inner product induced norm ‖𝑥𝑥‖ = �⟨𝑥𝑥, 𝑥𝑥⟩. Then,

(‖𝑥𝑥‖ + ‖𝑦𝑦‖)2 ≥ ‖𝑥𝑥 + 𝑦𝑦‖2 , ‖𝑥𝑥 + 𝑦𝑦‖2 = ‖𝑥𝑥‖2 + ‖𝑦𝑦‖2 + 2⟨𝑥𝑥, 𝑦𝑦⟩ (49)

�⃗ , determine the Manhattan norm, Euclidean length and

Example 2: Given a vector 𝑣𝑣⃗ = 𝚤𝚤⃗ − 4𝚥𝚥⃗ + 5𝑘𝑘
Chebyshev norm.

Solution: So, if we re-write the vector 𝑣𝑣⃗ as 𝑣𝑣⃗ = (1, −4,5), then we can calculate the norms easily.

A. Manhattan norm (One norm):

‖𝑣𝑣⃗‖1 = �|𝑣𝑣𝑖𝑖 |
𝑖𝑖
= |1| + |−4| + |5|
= 10

B. Euclidean norm (Two norm)

‖𝑣𝑣⃗‖2 = ��|𝑥𝑥𝑖𝑖 |2
𝑖𝑖

= �|1|2 + |−4|2 + |5|2

= √42

C. Chebyshev norm (Infinity norm)

‖𝑣𝑣⃗‖∞ = 𝑚𝑚𝑚𝑚𝑚𝑚𝑖𝑖 |𝑥𝑥𝑖𝑖 |
= 𝑚𝑚𝑚𝑚𝑚𝑚𝑖𝑖 {|1|, |−4|, |5|}
= 5

35
Therefore, we can see that
‖𝑥𝑥‖∞ ≤ ‖𝑥𝑥‖2 ≤ ‖𝑥𝑥‖1
5 ≤ √42 ≤ 10

2.2 Vector product of vector fields

The vector product of vector fields is also called as the cross product. For example, if we have 2 vectors
as 𝑨𝑨 = (𝐴𝐴1 , 𝐴𝐴2 , 𝐴𝐴3 ) and 𝑩𝑩 = (𝐵𝐵1 , 𝐵𝐵2 , 𝐵𝐵3 ), therefore,

𝑨𝑨 × 𝑩𝑩 = (𝐴𝐴2 𝐵𝐵3 − 𝐴𝐴3 𝐵𝐵2 , 𝐴𝐴1 𝐵𝐵3 − 𝐴𝐴3 𝐵𝐵1 , 𝐴𝐴1 𝐵𝐵2 − 𝐴𝐴2 𝐵𝐵1 ) (50)

The cross product of the vectors 𝑨𝑨 and 𝑩𝑩, is orthogonal to both 𝑨𝑨 and 𝑩𝑩, forms a right-handed systems
with 𝑨𝑨 and 𝑩𝑩, and has length given by:

‖𝑨𝑨 × 𝑩𝑩‖ = ‖𝑨𝑨‖‖𝑩𝑩‖ sin 𝜃𝜃 (51)

where 𝜃𝜃 is the angle between 𝑨𝑨 and 𝑩𝑩 satisfying 0 ≤ 𝜃𝜃 ≤ 𝜋𝜋. The vector product of a vector is a vector.
A few additional properties of the cross product are listed below:

1. Scalar multiplication (𝑎𝑎𝑨𝑨) × (𝑏𝑏𝑩𝑩) = 𝑎𝑎𝑎𝑎(𝑨𝑨 × 𝑩𝑩)

2. Distribution laws 𝑨𝑨 × (𝑩𝑩 + 𝑪𝑪) = 𝑨𝑨 × 𝑩𝑩 + 𝑨𝑨 × 𝑪𝑪
3. Anticommuttaion 𝑩𝑩 × 𝑨𝑨 = −𝑨𝑨 × 𝑩𝑩
4. Nonassociativity 𝑨𝑨 × (𝑩𝑩 × 𝑪𝑪) = (𝑨𝑨 ∙ 𝑪𝑪)𝑩𝑩 − (𝑨𝑨 ∙ 𝑩𝑩)𝑪𝑪

If we breakdown equation (9), we ca rewrite the cross product of vectors 𝑨𝑨 and 𝑩𝑩 as:

𝑨𝑨 × 𝑩𝑩 𝐴𝐴2 𝐴𝐴3 𝐴𝐴 𝐴𝐴3 𝐴𝐴 𝐴𝐴2 �⃗

=� � 𝚤𝚤⃗ − � 1 � 𝚥𝚥⃗ + � 1 � 𝑘𝑘
𝐵𝐵2 𝐵𝐵3 𝐵𝐵1 𝐵𝐵3 𝐵𝐵1 𝐵𝐵2

𝚤𝚤⃗ 𝚥𝚥⃗ �⃗
𝑘𝑘
= �𝐴𝐴1 𝐴𝐴2 𝐴𝐴3 �
𝐵𝐵1 𝐵𝐵2 𝐵𝐵3

Example 3: If 𝑨𝑨 = (3, −2, −2) and 𝑩𝑩 = (−1, 0, 5), compute 𝑨𝑨 × 𝑩𝑩 and find the angle between the two
vectors.

Solution: It’s a very simple solution here, all we have to do is the compute the cross product first, so

𝑨𝑨 × 𝑩𝑩 = �−2 −2� 𝚤𝚤⃗ − � 3 −2� 𝚥𝚥⃗ + � 3 −2� 𝑘𝑘

�⃗
0 5 −1 5 −1 0

36
 �⃗
= −10𝚤𝚤⃗ − 13𝚥𝚥⃗ − 2𝑘𝑘

Angle between the two vectors are given as: ‖𝑨𝑨 × 𝑩𝑩‖ = ‖𝑨𝑨‖‖𝑩𝑩‖ sin 𝜃𝜃. Rearranging equation (51), we
get:
‖𝑨𝑨 × 𝑩𝑩‖
sin 𝜃𝜃 =
‖𝑨𝑨‖‖𝑩𝑩‖

�(−10)2 + (−13)2 + (−2)2

=
�(3)2 + (−2)2 + (−2)2 �(−1)2 + (0)2 + (5)2

√273
=
√17√26

𝜃𝜃 = 51.80°

2.3 Vector operators

Certain differential operations may be done on a scalar and vector fields. This may have a wide range of
applications in physical sciences. The most important tasks are those of finding the gradient of a scalar
field and the divergence and curl of a vector field. In the following topics, we will discuss the mathematical
and geometrical definitions of these, which will rely on concepts of integrating vector quantities along
lines and over surfaces. In the midst of these differential operations is the vector operator ∇, which is
called as del (or nabla) and in Cartesian coordinates, ∇ is defined as:

𝜕𝜕 𝜕𝜕 𝜕𝜕
∇≡ 𝚤𝚤⃗ + 𝚥𝚥⃗ + 𝑘𝑘 �⃗ (52)
𝜕𝜕𝜕𝜕 𝜕𝜕𝜕𝜕 𝜕𝜕𝜕𝜕

2.3.1 Gradient of a scalar field

The gradient of a scalar field 𝜑𝜑(𝑥𝑥, 𝑦𝑦, 𝑧𝑧) is defined as

𝜕𝜕𝜕𝜕 𝜕𝜕𝜕𝜕 𝜕𝜕𝜕𝜕

grad φ = ∇φ = 𝚤𝚤⃗ + 𝚥𝚥⃗ + �⃗
𝑘𝑘 (53)
𝜕𝜕𝜕𝜕 𝜕𝜕𝜕𝜕 𝜕𝜕𝜕𝜕

Clearly, ∇φ is a vector field whose 𝑥𝑥, 𝑦𝑦 and 𝑧𝑧 components are the first partial derivatives of 𝜑𝜑(𝑥𝑥, 𝑦𝑦, 𝑧𝑧) with
respect to 𝑥𝑥, 𝑦𝑦 and 𝑧𝑧.

37
Example 4: Find the gradient of the scalar field 𝜑𝜑 = 𝑥𝑥𝑦𝑦 2 𝑧𝑧 3 .

Solution: We can easily solve this problem by using equation (12), so the gradient of the scalar field 𝜑𝜑 =
𝑥𝑥𝑦𝑦 2 𝑧𝑧 3 is

𝜕𝜕𝜕𝜕 𝜕𝜕𝜕𝜕 𝜕𝜕𝜕𝜕

grad φ = 𝚤𝚤⃗ + 𝚥𝚥⃗ + �⃗
𝑘𝑘
𝜕𝜕𝜕𝜕 𝜕𝜕𝜕𝜕 𝜕𝜕𝜕𝜕

�⃗
= 𝑦𝑦 2 𝑧𝑧 3 𝚤𝚤⃗ + 2𝑥𝑥𝑥𝑥𝑧𝑧 3 𝚥𝚥⃗ + 3𝑥𝑥𝑦𝑦 2 𝑧𝑧 2 𝑘𝑘

If we consider a surface in 3D space with 𝜑𝜑(𝒓𝒓) = 𝑐𝑐𝑐𝑐𝑐𝑐𝑐𝑐𝑐𝑐𝑐𝑐𝑐𝑐𝑐𝑐 then the direction normal (i.e. perpendicular)
to the surface at the point 𝒓𝒓 is the direction of grad 𝜑𝜑. The magnitude of the greater rate of change of 𝜑𝜑(𝒓𝒓)
is the magnitude of grad 𝜑𝜑.

∇φ

𝜑𝜑 = constant

Figure 2. Direction of gradient

In a physical situations, we may have a potential, 𝜑𝜑, which varies over a particular region and this
constitutes a field 𝐸𝐸, satisfying:

𝜕𝜕𝜕𝜕 𝜕𝜕𝜕𝜕 𝜕𝜕𝜕𝜕

𝐸𝐸 = −∇φ = − � 𝚤𝚤⃗ + 𝚥𝚥⃗ + �⃗ �
𝑘𝑘
𝜕𝜕𝜕𝜕 𝜕𝜕𝜕𝜕 𝜕𝜕𝜕𝜕

Example 5: Calculate the electric field at point (𝑥𝑥, 𝑦𝑦, 𝑧𝑧) due to a charge 𝑞𝑞1 at (2, 0, 0) and a charge 𝑞𝑞2 at
(-2, 0, 0) where charges are in coulombs and distances are in metres.

Solution: We need to understand the equation for Electric field which is given by:

38
 𝑞𝑞
𝐸𝐸 = 𝑘𝑘𝑐𝑐
𝑟𝑟

where 𝑟𝑟 is the magnitude or position and 𝑘𝑘𝑐𝑐 is the Coulomb constant and is given by:

1
𝑘𝑘𝑐𝑐 =
4𝜋𝜋𝜖𝜖0

Therefore, the potential at the point (𝑥𝑥, 𝑦𝑦, 𝑧𝑧) is

𝑞𝑞1 𝑞𝑞2
φ(𝑥𝑥, 𝑦𝑦, 𝑧𝑧) = − +
4𝜋𝜋𝜖𝜖0 �(2 − 𝑥𝑥)2 + 𝑦𝑦 2 + 𝑧𝑧 2 4𝜋𝜋𝜖𝜖0 �(2 + 𝑥𝑥)2 + 𝑦𝑦 2 + 𝑧𝑧 2

As a result, the components of the fields are

𝑞𝑞1 (2 − 𝑥𝑥) 𝑞𝑞2 (2 + 𝑥𝑥)

𝐸𝐸𝑥𝑥 = − +
4𝜋𝜋𝜖𝜖0 {(2 − 𝑥𝑥) + 𝑦𝑦 + 𝑧𝑧 }
2 2 2 3/2 4𝜋𝜋𝜖𝜖0 {(2 + 𝑥𝑥)2 + 𝑦𝑦 2 + 𝑧𝑧 2 }3/2

𝑞𝑞1 𝑦𝑦 𝑞𝑞2 𝑦𝑦
𝐸𝐸𝑦𝑦 = − +
4𝜋𝜋𝜖𝜖0 {(2 − 𝑥𝑥) + 𝑦𝑦 + 𝑧𝑧 }
2 2 2 3/2 4𝜋𝜋𝜖𝜖0 {(2 + 𝑥𝑥)2 + 𝑦𝑦 2 + 𝑧𝑧 2 }3/2

𝑞𝑞1 𝑧𝑧 𝑞𝑞2 𝑧𝑧
𝐸𝐸𝑧𝑧 = − +
4𝜋𝜋𝜖𝜖0 {(2 − 𝑥𝑥)2 + 𝑦𝑦 2 + 𝑧𝑧 2 }3/2 4𝜋𝜋𝜖𝜖0 {(2 + 𝑥𝑥)2 + 𝑦𝑦 2 + 𝑧𝑧 2 }3/2

Example 6: The function that describes the temperature at any point in the room is given by:

𝑥𝑥 𝑦𝑦
𝑇𝑇(𝑥𝑥, 𝑦𝑦, 𝑧𝑧) = 100 cos � � sin � � cos 𝑧𝑧
10 10

Find the gradient of 𝑇𝑇, the direction of greatest change in temperature in the room at point (10𝜋𝜋, 10𝜋𝜋, 𝜋𝜋)
and the rate of change of temperature at this point.

Solution: First, let’s find the gradient of the function 𝑇𝑇, which is given by equation (53):

𝜕𝜕𝜕𝜕 𝜕𝜕𝜕𝜕 𝜕𝜕𝜕𝜕

∇ 𝑇𝑇 = 𝚤𝚤⃗ + 𝚥𝚥⃗ + �⃗
𝑘𝑘
𝜕𝜕𝜕𝜕 𝜕𝜕𝜕𝜕 𝜕𝜕𝜕𝜕

𝑥𝑥 𝑦𝑦 𝑥𝑥 𝑦𝑦
= �−10 sin � � sin � � cos 𝑧𝑧� 𝚤𝚤⃗ + �10 cos � � cos � � cos 𝑧𝑧� 𝚥𝚥⃗
10 10 10 10
𝑥𝑥 𝑦𝑦
− �100 cos � � sin � � sin 𝑧𝑧� 𝑘𝑘 �⃗
10 10

39
Therefore, at the point (10𝜋𝜋, 10𝜋𝜋, 𝜋𝜋) in the room, the direction of the greatest change in temperature is:

�⃗
∇ 𝑇𝑇 = 0𝚤𝚤⃗ − 10𝚥𝚥⃗ + 0𝑘𝑘

And the rate of change of temperature at this point is the magnitude of the gradient, which is

|∇ 𝑇𝑇| = �(−10)2 = 10

2.3.2 Divergence of a vector field

The divergence of a vector field 𝑨𝑨(𝑥𝑥, 𝑦𝑦, 𝑧𝑧) is defined as the dot product of the operator ∇ and 𝑨𝑨:

𝜕𝜕𝐴𝐴1 𝜕𝜕𝐴𝐴2 𝜕𝜕𝐴𝐴3

div 𝑨𝑨 = ∇ ∙ 𝑨𝑨 = + + (54)
𝜕𝜕𝑥𝑥 𝜕𝜕𝜕𝜕 𝜕𝜕𝜕𝜕

where 𝐴𝐴1 , 𝐴𝐴2 and 𝐴𝐴3 are the 𝑥𝑥−, 𝑦𝑦 − and 𝑧𝑧 − components of 𝑨𝑨. Clearly, ∇ ∙ 𝑨𝑨 is a scalar field. Any vector
field 𝑨𝑨 for which ∇ ∙ 𝑨𝑨 = 0 is said to be solenoidal.

�⃗
Example 7: Find the divergence of a vector field 𝑨𝑨 = 𝑥𝑥 2 𝑦𝑦 2 𝚤𝚤⃗ + 𝑦𝑦 2 𝑧𝑧 2 𝚥𝚥⃗ + 𝑥𝑥 2 𝑧𝑧 2 𝑘𝑘

Solution: This is a straight forward example, using equation (54) we can solve this easily:

𝜕𝜕𝐴𝐴1 𝜕𝜕𝐴𝐴2 𝜕𝜕𝐴𝐴3

∇ ∙ 𝑨𝑨 = + +
𝜕𝜕𝜕𝜕 𝜕𝜕𝜕𝜕 𝜕𝜕𝜕𝜕

= 2(𝑥𝑥𝑦𝑦 2 + 𝑦𝑦𝑧𝑧 2 + 𝑥𝑥 2 𝑧𝑧)

Example 8: Find the divergence of a vector field 𝑭𝑭 = (𝑦𝑦𝑦𝑦𝑒𝑒 𝑥𝑥𝑥𝑥 , 𝑥𝑥𝑥𝑥𝑒𝑒 𝑥𝑥𝑥𝑥 , 𝑒𝑒 𝑥𝑥𝑥𝑥 + 3 cos 3𝑧𝑧)

Solution: Again, using equation (54) we can solve this easily:

𝜕𝜕𝐹𝐹1 𝜕𝜕𝐹𝐹2 𝜕𝜕𝐹𝐹3

∇ ∙ 𝑭𝑭 = + +
𝜕𝜕𝜕𝜕 𝜕𝜕𝜕𝜕 𝜕𝜕𝜕𝜕

= 𝑦𝑦 2 𝑧𝑧𝑒𝑒 𝑥𝑥𝑥𝑥 + 𝑥𝑥 2 𝑧𝑧𝑒𝑒 𝑥𝑥𝑥𝑥 − 9 sin 3𝑧𝑧

40
The value of the scalar div 𝑨𝑨 at point 𝑟𝑟 gives the rate at which the material is expanding or flowing away
from the point 𝑟𝑟 (outward flux per unit volume).

2.3.2.1 Theorem involving Divergence

Divergence theorem, also known as Gauss theorem relates a volume integral and a surface integral within
a vector field. Let 𝑭𝑭 be a vector field, 𝑆𝑆 be a closed surface and ℛ be the region inside of 𝑆𝑆, then:

� 𝑭𝑭 ∙ 𝑑𝑑𝑨𝑨 = � ∇ ∙ 𝑭𝑭𝑑𝑑𝑑𝑑 (55)

𝑆𝑆 ℛ

Example 9: Evaluate the following

� (3𝑥𝑥𝚤𝚤⃗ + 2𝑦𝑦𝚥𝚥⃗) ∙ 𝑑𝑑𝑨𝑨

𝑆𝑆

where 𝑆𝑆 is the sphere 𝑥𝑥 2 + 𝑦𝑦 2 + 𝑧𝑧 2 = 9.

Solution: We could parameterize the surface and evaluate the surface integral, but it is much faster to use
the divergence theorem. Since:

𝜕𝜕 𝜕𝜕 𝜕𝜕
div (3𝑥𝑥𝚤𝚤⃗ + 2𝑦𝑦𝚥𝚥⃗) = (3𝑥𝑥) + (2𝑦𝑦) + (0) = 5
𝜕𝜕𝜕𝜕 𝜕𝜕𝑦𝑦 𝜕𝜕𝜕𝜕

The divergence theorem gives:

� (3𝑥𝑥𝚤𝚤⃗ + 2𝑦𝑦𝚥𝚥⃗) ∙ 𝑑𝑑𝑨𝑨 = � 5 𝑑𝑑𝑑𝑑

𝑆𝑆 ℛ

= 5 × (Volume of sphere)

= 180π

Example 10: Evaluate the following

�⃗ ) ∙ 𝑑𝑑𝑨𝑨
� (𝑦𝑦 2 𝑧𝑧𝚤𝚤⃗ + 𝑦𝑦 3 𝚥𝚥⃗ + 𝑥𝑥𝑥𝑥𝑘𝑘
𝑆𝑆

41
where 𝑆𝑆 is the boundary of the cube defined by −1 ≤ 𝑥𝑥 ≤ 1, −1 ≤ 𝑦𝑦 ≤ 1, 𝑎𝑎𝑎𝑎𝑎𝑎 0 ≤ 𝑧𝑧 ≤ 2.

Solution: First let’s solve the divergence of the equation given:

𝜕𝜕 2 𝜕𝜕 𝜕𝜕
�⃗ � =
div �𝑦𝑦 2 𝑧𝑧𝚤𝚤⃗ + 𝑦𝑦 3 𝚥𝚥⃗ + 𝑥𝑥𝑥𝑥𝑘𝑘 (𝑦𝑦 𝑧𝑧) + (𝑦𝑦 3 ) + (𝑥𝑥𝑥𝑥)
𝜕𝜕𝜕𝜕 𝜕𝜕𝜕𝜕 𝜕𝜕𝜕𝜕

= 3𝑦𝑦 2 + 𝑥𝑥

The divergence theorem gives:

�⃗ � ∙ 𝑑𝑑𝑨𝑨 = � (3𝑦𝑦 2 + 𝑥𝑥) 𝑑𝑑𝑑𝑑

� �𝑦𝑦 2 𝑧𝑧𝚤𝚤⃗ + 𝑦𝑦 3 𝚥𝚥⃗ + 𝑥𝑥𝑥𝑥𝑘𝑘
𝑆𝑆 ℛ

2 1 1
= � � � (3𝑦𝑦 2 + 𝑥𝑥) 𝑑𝑑𝑑𝑑 𝑑𝑑𝑑𝑑 𝑑𝑑𝑑𝑑
0 −1 −1

1
= 2 � 6𝑦𝑦 2 𝑑𝑑𝑑𝑑
−1

=8

2.3.3 Curl of a vector field

The vector product (cross product) of operator and the vector A is known as the curl or rotation of A.
Thus in Cartesian coordinates, we can write:

𝚤𝚤⃗ 𝚥𝚥⃗ �⃗
𝑘𝑘
𝜕𝜕 𝜕𝜕 𝜕𝜕 �
curl 𝑨𝑨 = ∇ × 𝑨𝑨 = �� � (56)
𝜕𝜕𝜕𝜕 𝜕𝜕𝜕𝜕 𝜕𝜕𝜕𝜕
𝐴𝐴1 𝐴𝐴2 𝐴𝐴3

Therefore:

𝜕𝜕𝐴𝐴3 𝜕𝜕𝐴𝐴2 𝜕𝜕𝐴𝐴1 𝜕𝜕𝐴𝐴3 𝜕𝜕𝐴𝐴2 𝜕𝜕𝐴𝐴1

curl 𝑨𝑨 = ∇ × 𝑨𝑨 = � − 𝚤𝚤⃗, − 𝚥𝚥⃗, − �⃗ , �
𝑘𝑘 (57)
𝜕𝜕𝜕𝜕 𝜕𝜕𝜕𝜕 𝜕𝜕𝜕𝜕 𝜕𝜕𝜕𝜕 𝜕𝜕𝜕𝜕 𝜕𝜕𝜕𝜕

42
where 𝑨𝑨 = (𝐴𝐴1 , 𝐴𝐴2 , 𝐴𝐴3 ). The vector curl 𝑨𝑨 at point r gives the local rotation (or vorticity) of the material
at point r. The direction of curl 𝑨𝑨 is the axis of rotation and half the magnitude of curl 𝑨𝑨 is the rate of
rotation or angular frequency of the rotation.

�⃗
Example 11: Find the curl of a vector field 𝒂𝒂 = 𝑥𝑥 2 𝑦𝑦 2 𝑧𝑧 2 𝚤𝚤⃗ + 𝑦𝑦 2 𝑧𝑧 2 𝚥𝚥⃗ + 𝑥𝑥 2 𝑧𝑧 2 𝑘𝑘

Solution: This is a straight forward question. All we have to do is to put the equation in the form of
equation (56), we get:

𝚤𝚤⃗ 𝚥𝚥⃗ �⃗
𝑘𝑘
𝜕𝜕 𝜕𝜕 𝜕𝜕 �
∇ × 𝒂𝒂 = ��
𝜕𝜕𝜕𝜕 𝜕𝜕𝜕𝜕 𝜕𝜕𝜕𝜕 �
2 2 2
𝑥𝑥 𝑦𝑦 𝑧𝑧 𝑦𝑦 2 𝑧𝑧 2 𝑥𝑥 2 𝑧𝑧 2

𝜕𝜕 2 2 𝜕𝜕 𝜕𝜕 𝜕𝜕 𝜕𝜕 𝜕𝜕 2 2 2
= � (𝑥𝑥 𝑧𝑧 ) − (𝑦𝑦 2 𝑧𝑧 2 )� 𝚤𝚤⃗ − � (𝑥𝑥 2 𝑧𝑧 2 ) − (𝑥𝑥 2 𝑦𝑦 2 𝑧𝑧 2 )� 𝚥𝚥⃗ + � (𝑦𝑦 2 𝑧𝑧 2 ) − �⃗
(𝑥𝑥 𝑦𝑦 𝑧𝑧 )� 𝑘𝑘
𝜕𝜕𝜕𝜕 𝜕𝜕𝜕𝜕 𝜕𝜕𝜕𝜕 𝜕𝜕𝜕𝜕 𝜕𝜕𝜕𝜕 𝜕𝜕𝜕𝜕

�⃗ �
= −2�𝑦𝑦 2 𝑧𝑧𝚤𝚤⃗ + (𝑥𝑥𝑥𝑥 2 − 𝑥𝑥 2 𝑦𝑦 2 𝑧𝑧)𝚥𝚥⃗ + 𝑥𝑥 2 𝑦𝑦𝑧𝑧 2 𝑘𝑘

2.3.3.1 Theorem involving Curl

The theorem involving curl of vectors is better known as Stoke’s theorem. If we consider a surface 𝑆𝑆 in
ℝ3 that has a closed non-intersecting boundary, 𝐶𝐶, the topology of, say, one half of a tennis ball. That is,
“if we move along C and fall to our left, we hit the side of the surface where the normal vectors are sticking
out”. Stoke’s theorem states that for a vector field 𝑭𝑭 within which the surface is situated is given by:

� 𝑭𝑭 ∙ 𝑑𝑑𝒓𝒓 = � (∇ × 𝑭𝑭) ∙ 𝑛𝑛�⃗ 𝑑𝑑𝑑𝑑 (58)

𝐶𝐶 𝑆𝑆

The theorem can be useful in either direction: sometimes the line integral is easier than the surface integral,
and sometimes vice-versa.

Example 12: Evaluate the line integral of the function 𝑭𝑭(𝑥𝑥, 𝑦𝑦, 𝑧𝑧) = 〈𝑥𝑥 2 𝑦𝑦 3 , 𝑒𝑒 𝑥𝑥𝑥𝑥+𝑧𝑧 , 𝑥𝑥 + 𝑧𝑧 2 〉 around a
circle 𝑥𝑥 2 + 𝑦𝑦 2 = 1 in the plane 𝑦𝑦 = 0, oriented counterclock-wise as viewed from the positive
𝑦𝑦 −direction.

Solution: Whenever we want to integrate a vector field around a closed curve, and it looks like the
computation might be messy, think of applying Stoke’s Theorem. The circle 𝐶𝐶 in question is the positively-

43
oriented boundary of the disc 𝑆𝑆 given by 𝑥𝑥 2 + 𝑦𝑦 2 ≤ 1, 𝑦𝑦 = 0, with the unit normal vector 𝑛𝑛�⃗ pointing in
the positive 𝑦𝑦 −direction. That is 𝑛𝑛�⃗ = 𝚥𝚥⃗ = 〈0, 1, 0〉.

Stoke’s Theorem tells us that:

� 𝑭𝑭 ∙ 𝑑𝑑𝒓𝒓 = � (∇ × 𝑭𝑭) ∙ 𝑛𝑛�⃗ 𝑑𝑑𝑑𝑑

𝐶𝐶 𝑆𝑆

Evaluating the curl of 𝑭𝑭, we get:

𝚤𝚤⃗ 𝚥𝚥⃗ �⃗
𝑘𝑘
𝜕𝜕 𝜕𝜕 𝜕𝜕 �
∇ × 𝑭𝑭 = ��
𝜕𝜕𝜕𝜕 𝜕𝜕𝜕𝜕 𝜕𝜕𝜕𝜕 �
𝑥𝑥 2 𝑦𝑦 3 𝑒𝑒 𝑥𝑥𝑥𝑥+𝑧𝑧 𝑥𝑥 + 𝑧𝑧 2

�⃗ �
= �−𝑒𝑒 𝑥𝑥𝑥𝑥+𝑧𝑧 𝚤𝚤⃗ − 𝚥𝚥⃗ + (𝑦𝑦𝑒𝑒 𝑥𝑥𝑥𝑥+𝑧𝑧 − 3𝑥𝑥 2 𝑦𝑦 2 )𝑘𝑘

�⃗ � ∙ (0, 1, 0)
(∇ × 𝑭𝑭) ∙ 𝑛𝑛�⃗ = �−𝑒𝑒 𝑥𝑥𝑥𝑥+𝑧𝑧 𝚤𝚤⃗ − 𝚥𝚥⃗ + (𝑦𝑦𝑒𝑒 𝑥𝑥𝑥𝑥+𝑧𝑧 − 3𝑥𝑥 2 𝑦𝑦 2 )𝑘𝑘

= −1

� 𝑭𝑭 ∙ 𝑑𝑑𝒓𝒓 = � (∇ × 𝑭𝑭) ∙ 𝑛𝑛�⃗ 𝑑𝑑𝑑𝑑

𝐶𝐶 𝑆𝑆

= � −1 𝑑𝑑𝑑𝑑
𝑆𝑆

= −𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎(𝑆𝑆)

= −𝜋𝜋

2.4 Repeated Vector Operations – The Laplacian

So far, note the following:

i. grad must operate on a scalar field and gives a vector field in return
ii. div operates on a vector field and gives a scalar field in return, and,
iii. curl operates on a vector field and gives a vector field in return
44
In addition to the vector relations involving del (∇) mentioned above, there are six other combinations in
which del appears twice. The most important one which involves a scalar is:

𝒅𝒅𝒅𝒅𝒅𝒅 𝒈𝒈𝒈𝒈𝒈𝒈𝒈𝒈 𝜑𝜑 = ∇ ∙ ∇φ = ∇2 𝜑𝜑 (59)

where 𝜑𝜑(𝑥𝑥, 𝑦𝑦, 𝑧𝑧) that is a scalar point function. The operator ∇2 = ∇ ∙ ∇, is also known as the Laplacian,
takes a particularly simple form in Cartesian coordinates, which are:

2
𝜕𝜕 2 𝜕𝜕 2 𝜕𝜕 2
∇ = + + (60)
𝜕𝜕𝑥𝑥 2 𝜕𝜕𝑦𝑦 2 𝜕𝜕𝑧𝑧 2

When applied to a vector, it yields a vector, which is given in Cartesian coordinates:

2
𝜕𝜕 2 𝑨𝑨 𝜕𝜕 2 𝑨𝑨 𝜕𝜕 2 𝑨𝑨
∇ 𝑨𝑨 = + + (61)
𝜕𝜕𝑥𝑥 2 𝜕𝜕𝑦𝑦 2 𝜕𝜕𝑧𝑧 2

The cross product of two dels operating on a scalar function yields

𝚤𝚤⃗ 𝚥𝚥⃗ �⃗
𝑘𝑘
� 𝜕𝜕 𝜕𝜕 𝜕𝜕 �
∇ × ∇φ = 𝒄𝒄𝒄𝒄𝒄𝒄𝒄𝒄 𝒈𝒈𝒈𝒈𝒈𝒈𝒈𝒈 φ = 𝜕𝜕𝜕𝜕 𝜕𝜕𝜕𝜕 𝜕𝜕𝜕𝜕 = 0 (62)
�𝜕𝜕𝜕𝜕 𝜕𝜕𝜕𝜕 𝜕𝜕𝜕𝜕�
𝜕𝜕𝜕𝜕 𝜕𝜕𝜕𝜕 𝜕𝜕𝜕𝜕

If ∇ × 𝑨𝑨 = 0 for any vector 𝑨𝑨, then 𝑨𝑨 = ∇𝜑𝜑. In this case, 𝑨𝑨 is irrotational.

Similarly,

∇ ∙ ∇ × 𝑨𝑨 = 𝒅𝒅𝒅𝒅𝒅𝒅 𝒄𝒄𝒄𝒄𝒄𝒄𝒄𝒄 𝑨𝑨 = 0 (63)

Finally, a useful expansion is given by:

∇ × (∇ × 𝑨𝑨) = 𝒄𝒄𝒄𝒄𝒄𝒄𝒄𝒄 𝒄𝒄𝒄𝒄𝒄𝒄𝒄𝒄 𝑨𝑨 = ∇(∇ ∙ 𝑨𝑨) − ∇2 𝑨𝑨 (64)

Other forms for other coordinate systems for ∇2 are as follows:

1. Spherical polar coordinates:

45
 2
1 𝜕𝜕 2 𝜕𝜕 1 𝜕𝜕 𝜕𝜕 1 𝜕𝜕 2
∇ = 2 𝑟𝑟 + �sin 𝜃𝜃 � + 2 2 (65)
𝑟𝑟 𝜕𝜕𝜕𝜕 𝜕𝜕𝜕𝜕 𝑟𝑟 2 sin 𝜃𝜃 𝜕𝜕𝜕𝜕 𝜕𝜕𝜕𝜕 𝑟𝑟 sin 𝜃𝜃 𝜕𝜕𝜙𝜙 2

2. Two-dimensional polar coordinates:

2
𝜕𝜕 2 1 𝜕𝜕 1 𝜕𝜕 2
∇ = + + (66)
𝜕𝜕𝑟𝑟 2 𝑟𝑟 𝜕𝜕𝜕𝜕 𝑟𝑟 2 𝜕𝜕𝜃𝜃 2

3. Cylindrical coordinates:

𝜕𝜕 2 1 𝜕𝜕 1 𝜕𝜕 2 𝜕𝜕 2
∇2 = + + + (67)
𝜕𝜕𝑟𝑟 2 𝑟𝑟 𝜕𝜕𝜕𝜕 𝑟𝑟 2 𝜕𝜕𝜃𝜃 2 𝜕𝜕𝑧𝑧 2

Several other useful relations are summarised below:

DEL OPERATOR RELATIONS

Let 𝜑𝜑 and 𝜓𝜓 be scalar fields and 𝑨𝑨 and 𝑩𝑩 be vector fields

Sum of fields ∇(𝜑𝜑 + 𝜓𝜓) = ∇𝜑𝜑 + ∇𝜓𝜓

∇ ∙ (𝑨𝑨 + 𝑩𝑩) = ∇ ∙ 𝑨𝑨 + ∇ ∙ 𝑩𝑩

∇ × (𝑨𝑨 + 𝑩𝑩) = ∇ × 𝑨𝑨 + ∇ × 𝑩𝑩

Product of fields ∇(𝜑𝜑𝜑𝜑) = 𝜑𝜑(∇𝜓𝜓) + 𝜓𝜓(∇𝜑𝜑)

∇ ∙ (𝜑𝜑𝑨𝑨) = 𝜑𝜑(∇ ∙ 𝑨𝑨) + (∇𝜑𝜑) ∙ 𝑨𝑨

∇ × (𝜑𝜑𝑨𝑨) = 𝜑𝜑(∇ × 𝑨𝑨) + (∇𝜑𝜑) × 𝑨𝑨

∇ ∙ (𝑨𝑨 × 𝑩𝑩) = 𝑩𝑩 ∙ (∇ × 𝑨𝑨) − 𝑨𝑨 ∙ (∇ × 𝑩𝑩)

∇ × (𝑨𝑨 × 𝑩𝑩) = 𝑨𝑨 ∙ (∇ ∙ 𝑩𝑩) + (𝑩𝑩 ∙ ∇)𝑨𝑨 − 𝑩𝑩(∇ ∙ 𝑨𝑨) − (𝑨𝑨 ∙ ∇)𝑩𝑩

∇(𝑨𝑨 ∙ 𝑩𝑩) = 𝑨𝑨 × (∇ × 𝑩𝑩) − 𝑩𝑩(∇ ∙ 𝑨𝑨) + (𝑩𝑩 ∙ ∇)𝑨𝑨 − (𝑨𝑨 ∙ ∇)𝑩𝑩

Laplacian ∇ ∙ (∇𝜑𝜑) = ∇2 𝜑𝜑

∇ × (∇ × 𝑨𝑨) = ∇(∇ ∙ 𝑨𝑨) − ∇2 𝑨𝑨

46
 �⃗ , 𝑩𝑩 = 𝑥𝑥 2 𝚤𝚤⃗ + 𝑦𝑦𝑦𝑦𝚥𝚥⃗ − 𝑥𝑥𝑥𝑥𝑘𝑘
Example 13: If 𝑨𝑨 = 2𝑦𝑦𝑦𝑦𝚤𝚤⃗ − 𝑥𝑥 2 𝑦𝑦𝚥𝚥⃗ + 𝑥𝑥𝑧𝑧 2 𝑘𝑘 �⃗ and 𝜙𝜙 = 2𝑥𝑥 2 𝑦𝑦𝑧𝑧 3 , find
(a) (𝑨𝑨 ∙ ∇)𝜙𝜙
(b) 𝑨𝑨 ∙ ∇𝜙𝜙
(c) 𝑩𝑩 × ∇𝜙𝜙
(d) ∇2 𝜙𝜙

Solution:

(a)
𝜕𝜕 𝜕𝜕 𝜕𝜕
�⃗ � ∙ �
(𝑨𝑨 ∙ ∇)𝜙𝜙 = ��2𝑦𝑦𝑦𝑦𝚤𝚤⃗ − 𝑥𝑥 2 𝑦𝑦𝚥𝚥⃗ + 𝑥𝑥𝑧𝑧 2 𝑘𝑘 𝚤𝚤⃗ + 𝚥𝚥⃗ + 𝑘𝑘 �⃗ �� 𝜙𝜙
𝜕𝜕𝜕𝜕 𝜕𝜕𝜕𝜕 𝜕𝜕𝜕𝜕

𝜕𝜕 𝜕𝜕 𝜕𝜕
= �2𝑦𝑦𝑦𝑦 − 𝑥𝑥 2 𝑦𝑦 + 𝑥𝑥𝑧𝑧 2 � 2𝑥𝑥 2 𝑦𝑦𝑧𝑧 3
𝜕𝜕𝜕𝜕 𝜕𝜕𝜕𝜕 𝜕𝜕𝜕𝜕

𝜕𝜕 𝜕𝜕 𝜕𝜕
= 2𝑦𝑦𝑦𝑦 (2𝑥𝑥 2 𝑦𝑦𝑧𝑧 3 ) − 𝑥𝑥 2 𝑦𝑦 (2𝑥𝑥 2 𝑦𝑦𝑧𝑧 3 ) + 𝑥𝑥𝑧𝑧 2 (2𝑥𝑥 2 𝑦𝑦𝑧𝑧 3 )
𝜕𝜕𝜕𝜕 𝜕𝜕𝜕𝜕 𝜕𝜕𝜕𝜕

= 2𝑦𝑦𝑦𝑦(4𝑥𝑥𝑦𝑦𝑧𝑧 3 ) − 𝑥𝑥 2 𝑦𝑦(2𝑥𝑥 2 𝑧𝑧 3 ) + 𝑥𝑥𝑧𝑧 2 (6𝑥𝑥 2 𝑦𝑦𝑧𝑧 2 )

= 8𝑥𝑥𝑦𝑦 2 𝑧𝑧 4 − 2𝑥𝑥 4 𝑦𝑦𝑧𝑧 3 + 6𝑥𝑥 3 𝑦𝑦𝑧𝑧 4

(b)
𝜕𝜕 𝜕𝜕 𝜕𝜕
∇𝜙𝜙 = (2𝑥𝑥 2 𝑦𝑦𝑧𝑧 3 )𝚤𝚤⃗ + �⃗
(2𝑥𝑥 2 𝑦𝑦𝑧𝑧 3 )𝚥𝚥⃗ + (2𝑥𝑥 2 𝑦𝑦𝑧𝑧 3 )𝑘𝑘
𝜕𝜕𝜕𝜕 𝜕𝜕𝜕𝜕 𝜕𝜕𝜕𝜕

�⃗
= 4𝑥𝑥𝑥𝑥𝑧𝑧 3 𝚤𝚤⃗ + 2𝑥𝑥 2 𝑧𝑧 3 𝚥𝚥⃗ + 6𝑥𝑥 2 𝑦𝑦𝑧𝑧 2 𝑘𝑘
Therefore

�⃗ � ∙ �4𝑥𝑥𝑥𝑥𝑧𝑧 3 𝚤𝚤⃗ + 2𝑥𝑥 2 𝑧𝑧 3 𝚥𝚥⃗ + 6𝑥𝑥 2 𝑦𝑦𝑧𝑧 2 𝑘𝑘

𝑨𝑨 ∙ ∇𝜙𝜙 = �2𝑦𝑦𝑦𝑦𝚤𝚤⃗ − 𝑥𝑥 2 𝑦𝑦𝚥𝚥⃗ + 𝑥𝑥𝑧𝑧 2 𝑘𝑘 �⃗ �

= 8𝑥𝑥𝑦𝑦 2 𝑧𝑧 4 − 2𝑥𝑥 4 𝑦𝑦𝑧𝑧 3 + 6𝑥𝑥 3 𝑦𝑦𝑧𝑧 4

(c)
�⃗ , therefore:
∇𝜙𝜙 = 4𝑥𝑥𝑥𝑥𝑧𝑧 3 𝚤𝚤⃗ + 2𝑥𝑥 2 𝑧𝑧 3 𝚥𝚥⃗ + 6𝑥𝑥 2 𝑦𝑦𝑧𝑧 2 𝑘𝑘

47
 𝚤𝚤⃗ 𝚥𝚥⃗ �⃗
𝑘𝑘
𝑩𝑩 × ∇𝜙𝜙 = � 𝑥𝑥 2 𝑦𝑦𝑦𝑦 −𝑥𝑥𝑥𝑥 �
4𝑥𝑥𝑥𝑥𝑧𝑧 3 2𝑥𝑥 2 𝑧𝑧 3 6𝑥𝑥 2 𝑦𝑦𝑧𝑧 2

�⃗
= (6𝑥𝑥 2 𝑦𝑦 2 𝑧𝑧 3 + 2𝑥𝑥 3 𝑦𝑦𝑧𝑧 3 )𝚤𝚤⃗ + (−4𝑥𝑥 2 𝑦𝑦 2 𝑧𝑧 3 − 6𝑥𝑥 4 𝑦𝑦𝑧𝑧 2 )𝚥𝚥⃗ + (2𝑥𝑥 4 𝑧𝑧 3 − 4𝑥𝑥𝑦𝑦 2 𝑧𝑧 4 )𝑘𝑘

(d)
2 𝜕𝜕 2 𝜕𝜕 2 𝜕𝜕 2
∇ 𝜙𝜙 = 2 (2𝑥𝑥 𝑦𝑦𝑧𝑧 ) + 2 (2𝑥𝑥 𝑦𝑦𝑧𝑧 ) + 2 (2𝑥𝑥 2 𝑦𝑦𝑧𝑧 3 )
2 3 2 3
𝜕𝜕𝜕𝜕 𝜕𝜕𝜕𝜕 𝜕𝜕𝜕𝜕

= 4𝑦𝑦𝑧𝑧 3 + 0 + 12𝑥𝑥 2 𝑦𝑦𝑦𝑦

48
3. Linear Algebra, Matrices & Eigenvectors
In many practical systems, there naturally arises a set of quantities that can conveniently be represented
as a certain dimensional array, referred to as matrix. If matrices were simply a way of representing array
of numbers, then they would have only a marginal utility as a means of visualising data. However, a whole
branch of mathematics has evolved, involving manipulation of matrices, which has become a powerful
tool for the solution f many problems.

For example, consider the set of 𝑛𝑛 linear equations with 𝑛𝑛 unknowns

𝑎𝑎11 𝑌𝑌1 + 𝑎𝑎12 𝑌𝑌2 + ⋯ + 𝑎𝑎1𝑛𝑛 𝑌𝑌𝑛𝑛 = 0

𝑎𝑎21 𝑌𝑌1 + 𝑎𝑎22 𝑌𝑌2 + ⋯ + 𝑎𝑎2𝑛𝑛 𝑌𝑌𝑛𝑛 = 0
(68)
∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙∙
𝑎𝑎𝑛𝑛1 𝑌𝑌1 + 𝑎𝑎𝑛𝑛2 𝑌𝑌2 + ⋯ + 𝑎𝑎𝑛𝑛𝑛𝑛 𝑌𝑌𝑛𝑛 = 0

The necessary and sufficient condition for the set to have a non-trivial solution (other than 𝑌𝑌1 = 𝑌𝑌2 = ⋯ =
𝑌𝑌𝑛𝑛 = 0) is that the determinant of the array of coefficients is zero: 𝑑𝑑𝑑𝑑𝑑𝑑(𝐴𝐴) = 0.

3.1 Basic definitions and notation

A matrix is an array of numbers with 𝑚𝑚 rows and 𝑛𝑛 columns. The (i, j)th element is the element found in
row 𝑖𝑖 and column 𝑗𝑗.

For example, have a look at the matrix below. Tis matrix has 𝑚𝑚 = 2 rows, 𝑛𝑛 = 3 column, and therefore
the matrix order is 2 × 3. The (i, j)th element is 𝑎𝑎𝑖𝑖𝑖𝑖

𝑎𝑎11 𝑎𝑎12 𝑎𝑎13

𝐴𝐴 = �𝑎𝑎 𝑎𝑎22 𝑎𝑎23 � (69)
21

Matrices may be categorized based on the properties of its elements. Some basic definitions include:

1. The transpose of matrix 𝐴𝐴 (or 𝐴𝐴𝑇𝑇 ) is formed by interchanging element 𝑎𝑎𝑖𝑖𝑖𝑖 with element 𝑎𝑎𝑗𝑗𝑗𝑗 .
Therefore:

𝐴𝐴𝑇𝑇 = �𝑎𝑎𝑗𝑗𝑗𝑗 �, (𝐴𝐴 + 𝐵𝐵)𝑇𝑇 = 𝐴𝐴𝑇𝑇 + 𝐵𝐵 𝑇𝑇 , (𝐴𝐴𝐴𝐴)𝑇𝑇 = 𝐴𝐴𝑇𝑇 𝐵𝐵 𝑇𝑇 (70)

A symmetric matrix is equals to its transpose, 𝐴𝐴 = 𝐴𝐴𝑇𝑇 .

49
2. Diagonal matrix is a square matrix (𝑚𝑚 = 𝑛𝑛) that has it’s only non-zero elements along the
leading diagonal. For example:

𝑎𝑎11 0 0
𝑑𝑑𝑑𝑑𝑑𝑑𝑑𝑑 𝐴𝐴 = � 0 𝑎𝑎22 0 �
0 0 𝑎𝑎33

Diagonal can also be written for a list of matrices as:

𝑑𝑑𝑑𝑑𝑑𝑑𝑑𝑑 (𝑎𝑎11 , 𝑎𝑎22 , ⋯ 𝑎𝑎𝑛𝑛𝑛𝑛 )

Which denotes the block diagonal matrix with elements 𝑎𝑎11 , 𝑎𝑎22 , ⋯ 𝑎𝑎𝑛𝑛𝑛𝑛 along te diagonal and
zeros elsewhere. A matrix is formed in this way is sometimes called a direct sum of
𝑎𝑎11 , 𝑎𝑎22 , ⋯ 𝑎𝑎𝑛𝑛𝑛𝑛 and the operation is denoted by ⨁:

𝑎𝑎11 ⨁ ⋯ ⨁ 𝑎𝑎𝑛𝑛𝑛𝑛 = 𝑑𝑑𝑑𝑑𝑑𝑑𝑑𝑑 (𝑎𝑎11 , 𝑎𝑎22 , ⋯ 𝑎𝑎𝑛𝑛𝑛𝑛 )

3. In a square matrix of order n, the diagonal containing elements 𝑎𝑎11 , 𝑎𝑎22 , ⋯ 𝑎𝑎𝑛𝑛𝑛𝑛 is called the
principle or leading diagonal. The sum of elements in this diagonal is called the trace of 𝑛𝑛 × 𝑛𝑛
square matrix A, hence:

𝑇𝑇𝑇𝑇𝑇𝑇𝑇𝑇𝑇𝑇 (𝐴𝐴) = 𝑇𝑇𝑇𝑇 (𝐴𝐴) = � 𝑎𝑎𝑖𝑖𝑖𝑖 (71)

𝑖𝑖

We can also define a few more notations for Trace as:

𝑇𝑇𝑇𝑇(𝐴𝐴) = 𝑇𝑇𝑇𝑇(𝐴𝐴𝑇𝑇 ), 𝑇𝑇𝑇𝑇(𝑐𝑐𝑐𝑐) = 𝑐𝑐𝑐𝑐𝑐𝑐(𝐴𝐴), 𝑇𝑇𝑟𝑟(𝐴𝐴 + 𝐵𝐵) = 𝑇𝑇𝑇𝑇(𝐴𝐴) + 𝑇𝑇𝑇𝑇(𝐵𝐵) (72)

4. Determinant of a square 𝑛𝑛 × 𝑛𝑛 matrix 𝐴𝐴 is denoted as det(𝐴𝐴) or |𝐴𝐴|. It is determined by:

𝑛𝑛

|𝐴𝐴| = � 𝑎𝑎𝑖𝑖𝑖𝑖 𝑎𝑎(𝑖𝑖𝑖𝑖) (73)

𝑗𝑗=1
where:
𝑎𝑎(𝑖𝑖𝑖𝑖) = (−1)𝑖𝑖+𝑗𝑗 �𝐴𝐴(𝑖𝑖)(𝑗𝑗) � (74)

with �𝐴𝐴(𝑖𝑖)(𝑗𝑗) � denoting the submatrix that is formed from 𝐴𝐴 by removing the 𝑖𝑖th row and the 𝑗𝑗th
column.

Determinant of a matrix can also be defined as the following:

50
 |𝐴𝐴𝐴𝐴| = |𝐴𝐴||𝐵𝐵|, |𝐴𝐴| = |𝐴𝐴𝑇𝑇 |, |𝑐𝑐𝑐𝑐| = 𝑐𝑐 𝑛𝑛 |𝐴𝐴| (75)

5. Adjugate of a 𝑛𝑛 × 𝑛𝑛 matrix 𝐴𝐴 is defined as an 𝑛𝑛 × 𝑛𝑛 matrix of the cofactors of the elements of

the transposed matrix. Therefore we can write Adjugate of 𝑛𝑛 × 𝑛𝑛 matrix 𝐴𝐴 as:

𝑇𝑇
𝑎𝑎𝑎𝑎𝑎𝑎(𝐴𝐴) = �𝑎𝑎(𝑗𝑗𝑗𝑗) � = �𝑎𝑎(𝑖𝑖𝑖𝑖) � (76)

Adjugate has an interesting property:

𝐴𝐴 𝑎𝑎𝑎𝑎𝑎𝑎(𝐴𝐴) = 𝑎𝑎𝑎𝑎𝑎𝑎(𝐴𝐴)𝐴𝐴 = |𝐴𝐴|𝐼𝐼 (77)

3.2 Multiplication of matrices and multiplication of vectors and matrices

3.2.1 Matrix multiplication

If we let 𝐴𝐴 be order 𝑚𝑚 × 𝑛𝑛 and 𝐵𝐵 of order 𝑛𝑛 × 𝑝𝑝. Then the product of two matrices 𝐴𝐴 and 𝐵𝐵 is

𝐶𝐶 = 𝐴𝐴𝐴𝐴 (78)

or
𝑛𝑛

𝑐𝑐𝑖𝑖𝑖𝑖 = � 𝑎𝑎𝑖𝑖𝑖𝑖 𝑏𝑏𝑘𝑘𝑘𝑘 (79)

𝑘𝑘=1

where the resulting matrix 𝐶𝐶 is in the order of 𝑚𝑚 × 𝑝𝑝.

Square matrices obey the laws expressed as below:

Associative: 𝐴𝐴(𝐵𝐵𝐵𝐵) = (𝐴𝐴𝐴𝐴)𝐶𝐶 (80)

Distributive: (𝐴𝐴 + 𝐵𝐵)𝐶𝐶 = 𝐴𝐴𝐴𝐴 + 𝐵𝐵𝐵𝐵, (𝐵𝐵 + 𝐶𝐶)𝐴𝐴 = 𝐵𝐵𝐵𝐵 + 𝐶𝐶𝐶𝐶 (81)

Matrix Polynomials

Polynomials in square matrices are similar to the more familiar polynomials in scalars. Let us consider:

𝑝𝑝(𝐴𝐴) = 𝑏𝑏0 𝐼𝐼 + 𝑏𝑏1 𝐴𝐴 + ⋯ 𝑏𝑏𝑘𝑘 𝐴𝐴𝑘𝑘 (82)

The value of this polynomial is a matrix. The theory of polynomials in general holds, we have the useful
factorizations of monomials:

51
For any positive integer k,
𝐼𝐼 − 𝐴𝐴𝑘𝑘 = ((𝐼𝐼 − 𝐴𝐴)(𝐼𝐼 + 𝐴𝐴 + ⋯ 𝐴𝐴𝑘𝑘−1 ) (83)

For an odd positive integer k,

𝐼𝐼 + 𝐴𝐴𝑘𝑘 = ((𝐼𝐼 + 𝐴𝐴)(𝐼𝐼 − 𝐴𝐴 + ⋯ 𝐴𝐴𝑘𝑘−1 ) (84)

3.2.2 Traces and determinants of square Cayley products

The useful property of the trace for the matrix 𝐴𝐴 and 𝐵𝐵 that are conformable for the multiplication 𝐴𝐴𝐴𝐴 and
𝐵𝐵𝐵𝐵 is

𝑇𝑇𝑇𝑇(𝐴𝐴𝐴𝐴) = 𝑇𝑇𝑇𝑇 (𝐵𝐵𝐵𝐵) (85)

This is obvious from the definitions of matrix multiplication and the trace. Due to associativity of matrix
multiplications, equation (18) can be further extended to:

𝑇𝑇𝑇𝑇(𝐴𝐴𝐴𝐴𝐴𝐴) = 𝑇𝑇𝑇𝑇 (𝐵𝐵𝐵𝐵𝐵𝐵) = 𝑇𝑇𝑇𝑇(𝐶𝐶𝐶𝐶𝐶𝐶) (86)

If 𝐴𝐴 and 𝐵𝐵 are square matrices conformable for multiplication, then an important property of the
determinant is

|𝐴𝐴𝐴𝐴| = |𝐴𝐴||𝐵𝐵| (87)

Or we can write the equation as:

𝐴𝐴 0
�� �� = |𝐴𝐴||𝐵𝐵| (88)
−𝐼𝐼 𝐵𝐵

3.2.3 The Kronecker product

The Kronecker multiplication, denoted by ⨂, is not commutative, rather it is associative. Therefore,

𝐴𝐴 ⨂ 𝐵𝐵 may not equal to 𝐵𝐵 ⨂ 𝐴𝐴. Let us have a 𝑚𝑚 × 𝑚𝑚 matrix 𝐴𝐴 and an 𝑛𝑛 × 𝑛𝑛 matrix 𝐵𝐵. We can then
form an 𝑚𝑚𝑚𝑚 × 𝑚𝑚𝑚𝑚 matrix 𝐶𝐶 by defining the direct product as:

52
 𝑎𝑎 𝐵𝐵 𝑎𝑎12 𝐵𝐵 ⋯ 𝑎𝑎1𝑚𝑚 𝐵𝐵
⎡ 11 ⎤
⋯ 𝑎𝑎2𝑚𝑚 𝐵𝐵 ⎥
𝐶𝐶 = 𝐴𝐴 ⨂ 𝐵𝐵 = ⎢ 𝑎𝑎21 𝐵𝐵 𝑎𝑎22 𝐵𝐵 (89)
⎢ ⋮ ⋮ ⋮ ⎥
⎣𝑎𝑎𝑚𝑚1 𝐵𝐵 𝑎𝑎𝑚𝑚2 𝐵𝐵 ⋯ 𝑎𝑎 𝑚𝑚𝑚𝑚 𝐵𝐵⎦

To be more specific, let 𝐴𝐴 and 𝐵𝐵 be a 2 × 2 matrices

𝑎𝑎11 𝑎𝑎12 𝑏𝑏11 𝑏𝑏12

𝐴𝐴 = �𝑎𝑎 𝑎𝑎22 � 𝐵𝐵 = � �
21 𝑏𝑏21 𝑏𝑏22

The Kronecker product matrix 𝐶𝐶 is the 4 × 4 matrix

𝑎𝑎11 𝑏𝑏11 𝑎𝑎11 𝑏𝑏12 𝑎𝑎12 𝑏𝑏11 𝑎𝑎12 𝑏𝑏12

𝑎𝑎 𝑏𝑏 𝑎𝑎11 𝑏𝑏22 𝑎𝑎12 𝑏𝑏21 𝑎𝑎12 𝑏𝑏22
𝐶𝐶 = 𝐴𝐴 ⨂ 𝐵𝐵 = � 11 21 �
𝑎𝑎21 𝑏𝑏11 𝑎𝑎21 𝑏𝑏12 𝑎𝑎22 𝑏𝑏11 𝑎𝑎22 𝑏𝑏12
𝑎𝑎21 𝑏𝑏21 𝑎𝑎21 𝑏𝑏22 𝑎𝑎22 𝑏𝑏21 𝑎𝑎22 𝑏𝑏22

The determinant of Kronecker product of two square matrices 𝑚𝑚 × 𝑚𝑚 matrix 𝐴𝐴 and an 𝑛𝑛 × 𝑛𝑛 matrix 𝐵𝐵
has a simple relationship to the determinant of the individual matrices. Hence:

|𝐴𝐴 ⨂ 𝐵𝐵| = |𝐴𝐴|𝑚𝑚 |𝐵𝐵|𝑛𝑛 (90)

Assuming the matrices are conformable for the indicated operations, some additional properties of
Kronecker products are as follows:

(𝑎𝑎𝑎𝑎) ⨂ (𝑏𝑏𝑏𝑏) = 𝑎𝑎𝑎𝑎(𝐴𝐴 ⨂ 𝐵𝐵) = (𝑎𝑎𝑎𝑎𝑎𝑎) ⨂ 𝐵𝐵 = 𝐴𝐴 ⨂ (𝑎𝑎𝑎𝑎𝑎𝑎) (91)

where 𝑎𝑎 and 𝑏𝑏 are scalars.

(𝐴𝐴 + 𝐵𝐵) ⨂ (𝐶𝐶) = 𝐴𝐴 ⨂ 𝐶𝐶 + 𝐵𝐵 ⨂ 𝐶𝐶 (92)

(𝐴𝐴 ⨂ 𝐵𝐵 ) ⨂ 𝐶𝐶 = 𝐴𝐴 ⨂ (𝐵𝐵 ⨂ 𝐶𝐶) (93)

(𝐴𝐴 ⨂ 𝐵𝐵 ) 𝑇𝑇 = 𝐴𝐴𝑇𝑇 ⨂ 𝐵𝐵 𝑇𝑇 (94)

(𝐴𝐴 ⨂ 𝐵𝐵 )(𝐶𝐶 ⨂ 𝐷𝐷 ) = 𝐴𝐴𝐴𝐴 ⨂ 𝐵𝐵𝐵𝐵 (95)

53
3.3 Matrix Rank and the Inverse of a full rank matrix
The linear dependence or independence of the vectors forming the rows or columns of a matrix is an
important characteristic of the matrix. The maximum number of linearly independent vectors is called the
rank of the matrix, 𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟 (𝐴𝐴). Multiplication by a non-zero scalar does not change the linear dependence
of vectors. Therefore, for the scalar 𝑎𝑎 with 𝑎𝑎 ≠ 0, we have

𝑟𝑟𝑟𝑟𝑛𝑛𝑛𝑛 (𝑎𝑎𝑎𝑎) = 𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟(𝐴𝐴) (96)

For a 𝑛𝑛 × 𝑚𝑚 matrix 𝐴𝐴,

𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟 (𝐴𝐴) ≤ min(𝑛𝑛, 𝑚𝑚) (97)

Example 1: Find the rank of the matrix 𝐴𝐴 below:

1 2 1
𝐴𝐴 = �−2 −3 1�
3 5 0

Solution: First we understand that this is a 3 × 3 matrix. If w elook closey we can see that the first two
rows are linearly independent. However, the third row is dependent on the first and second rows where
𝑅𝑅𝑅𝑅𝑅𝑅 1 − 𝑅𝑅𝑅𝑅𝑅𝑅 2 = 𝑅𝑅𝑅𝑅𝑅𝑅 3. Therefore, rank of matrix 𝐴𝐴 is 2.

3.3.1 Full Rank matrices

If the rank of a matrix is the same as its smaller dimension, we say the matrix is of full rank. In the case
of non-square matrix, we say the matrix is of full row rank or full column rank just to emphasis which one
is of the smaller number. A matrix is a full row rank when each row is linearly independent, while ma
matrix is a full column rank when each column s linearly independent. For a square matrix, however, the
matrix is a full rank when all rows and columns are linearly independent and that the determinant of the
matrix is not zero.

Rank of product of two matrices is less than or equals to the lesser rank of the two, or:

𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟 (𝐴𝐴𝐴𝐴) ≤ min�𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟(𝐴𝐴), 𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟(𝐵𝐵)� (98)

Rank of sum of two matrices is less than or equals to the sum of their ranks, or:

𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟 (𝐴𝐴 + 𝐵𝐵) ≤ 𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟 (𝐴𝐴) + 𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟 (𝐵𝐵) (99)

54
From equation (99), we can also write:

|𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟 (𝐴𝐴) − 𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟 (𝐵𝐵)| ≤ 𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟 (𝐴𝐴 + 𝐵𝐵) (100)

3.3.2 Solutions of linear equations

An application of vectors and matrices involve systems of linear equations:

𝑎𝑎11 𝑥𝑥1 + ⋯ + 𝑎𝑎1𝑚𝑚 𝑥𝑥𝑚𝑚 = 𝑏𝑏1

⋮ ⋮ ⋮ (101)
𝑎𝑎𝑛𝑛1 𝑥𝑥1 + ⋯ + 𝑎𝑎𝑛𝑛𝑛𝑛 𝑥𝑥𝑚𝑚 = 𝑏𝑏𝑛𝑛

Or
𝐴𝐴𝑥𝑥 = 𝑏𝑏 (102)

In this system, 𝐴𝐴 is called the coefficient matrix. The 𝑥𝑥 that satisfied this system of equation is then called
the solution to the system. For a given 𝐴𝐴 and 𝑏𝑏, a solution may or may not exist. A system for which a
solution exist, is said to be consistent; otherwise, it is inconsistent. A linear system 𝐴𝐴𝑛𝑛𝑛𝑛𝑛𝑛 𝑥𝑥 = 𝑏𝑏 is
consistent if and only if:

𝑅𝑅𝑅𝑅𝑅𝑅𝑅𝑅([𝐴𝐴|𝑏𝑏]) = 𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟 (𝐴𝐴) (103)

Namely, the space spanned by the columns of 𝐴𝐴 is the same as that spanned by the columns of 𝐴𝐴 and the
vector 𝑏𝑏; therefore, 𝑏𝑏 must be a linear combination of the columns of 𝐴𝐴. A special case that yields equation
(103) for any 𝑏𝑏 is:

𝑅𝑅𝑅𝑅𝑅𝑅𝑅𝑅(𝐴𝐴𝑛𝑛𝑛𝑛𝑛𝑛 ) = 𝑛𝑛 (104)

And so if 𝐴𝐴 is of full row rank, the system is consistent regardless of the value of 𝑏𝑏. In this case, of course,
the number of rows of 𝐴𝐴 must not be greater than the number of columns. A square system in which 𝐴𝐴 is
non-singular is clearly consistent, and the solution is given by:

𝑥𝑥 = 𝐴𝐴−1 𝑏𝑏 (105)

3.3.3 Preservation of positive definiteness

A certain type of product of a full rank matrix and a positive definite matrix preserves not only the rank
but also the positive definiteness. If 𝐶𝐶 is an 𝑛𝑛 × 𝑛𝑛 and positive definite and 𝐴𝐴 = 𝑛𝑛 × 𝑚𝑚 of rank
𝑚𝑚 (𝑚𝑚 ≤ 𝑛𝑛), then 𝐴𝐴𝑇𝑇 𝐶𝐶𝐶𝐶 is positive definite. To understand this, let us assume the matrix C and A as

55
described. Letx be any m-vector such that 𝑥𝑥 ≠ 0 and ley 𝑦𝑦 = 𝐴𝐴𝐴𝐴. Because 𝐴𝐴 is a full column rank,
therefore 𝑦𝑦 ≠ 0, we then have:

𝑥𝑥 𝑇𝑇 (𝐴𝐴𝑇𝑇 𝐶𝐶𝐶𝐶) = (𝑥𝑥𝑥𝑥)𝑇𝑇 𝐶𝐶(𝐴𝐴𝐴𝐴) = 𝑦𝑦 𝑇𝑇 𝐶𝐶𝐶𝐶 > 0 (106)

Therefore, since 𝐴𝐴𝑇𝑇 𝐶𝐶𝐶𝐶 is symmetric:

1. If 𝐶𝐶 is positive definite and 𝐴𝐴 is of full column rank, then 𝐴𝐴𝑇𝑇 𝐶𝐶𝐶𝐶 is positive definite.

Furthermore, we then have the converse:

2. If 𝐴𝐴𝑇𝑇 𝐶𝐶𝐶𝐶 is positive definite, then 𝐴𝐴 is of full column rank.

For otherwise there exists an 𝑥𝑥 ≠ 0 such that 𝐴𝐴𝑥𝑥 = 0, and so 𝑥𝑥 𝑇𝑇 (𝐴𝐴𝑇𝑇 𝐶𝐶𝐶𝐶)𝑥𝑥 = 0.

3.3.4 A lower bound on the rank of a matrix product

Equation (98) gives an upper bound on the rank of the product of two matrices; where the rank cannot be
greater than the rank of either of the factors. Now, we will develop a lower bound of the rank of the product
of two matrices if one of them is square.

If 𝐴𝐴 is an 𝑛𝑛 × 𝑛𝑛 (square) and 𝐵𝐵 is a matrix with n rows, then:

𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟 (𝐴𝐴𝐴𝐴) ≥ 𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟 (𝐴𝐴) + 𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟 (𝐵𝐵) − 𝑛𝑛 (107)

3.3.5 Inverse of products and sums of matrices

The inverse of the Cayley product of two nonsingular matrices of the same size is particularly easy to
form. If 𝐴𝐴 and 𝐵𝐵 are square full rank matrices of the same size, then:

(𝐴𝐴𝐴𝐴)−1 = 𝐵𝐵 −1 𝐴𝐴−1 (108)

𝐴𝐴(𝐼𝐼 + 𝐴𝐴)−1 = (𝐼𝐼 + 𝐴𝐴−1 )−1 (109)

(𝐴𝐴 + 𝐵𝐵𝐵𝐵 )−1 𝐵𝐵 = 𝐴𝐴−1 𝐵𝐵(𝐼𝐼 + 𝐵𝐵 𝑇𝑇 𝐴𝐴−1 𝐵𝐵)−1 (110)

(𝐴𝐴−1 + 𝐵𝐵 −1 )−1 = 𝐴𝐴(𝐴𝐴 + 𝐵𝐵)−1 𝐵𝐵 (111)

56
 𝐴𝐴 − 𝐴𝐴(𝐴𝐴 + 𝐵𝐵 )−1 𝐴𝐴 = 𝐵𝐵 − 𝐵𝐵(𝐴𝐴 + 𝐵𝐵 )−1 𝐵𝐵 (112)

𝐴𝐴−1 + 𝐵𝐵 −1 = 𝐴𝐴−1 (𝐴𝐴 + 𝐵𝐵)𝐵𝐵 −1 (113)

(𝐼𝐼 + 𝐴𝐴𝐴𝐴 )−1 = 𝐼𝐼 − 𝐴𝐴(𝐼𝐼 + 𝐵𝐵𝐵𝐵)−1 𝐵𝐵 (114)

(𝐼𝐼 + 𝐴𝐴𝐴𝐴 )−1 𝐴𝐴 = 𝐴𝐴(𝐼𝐼 + 𝐵𝐵𝐵𝐵)−1 (115)

(𝐴𝐴 ⨂ 𝐵𝐵 )−1 = 𝐴𝐴−1 ⨂ 𝐵𝐵 −1 (116)

Note: When 𝐴𝐴 and/or 𝐵𝐵 are not full rank, the inverse may not exist.

3.4 Eigensystems

Suppose 𝐴𝐴 is an 𝑛𝑛 × 𝑛𝑛 matrix. The number 𝜆𝜆 is said to be an eigenvalue of 𝐴𝐴 if some non-zero vector 𝒙𝒙,
𝐴𝐴𝒙𝒙 = 𝜆𝜆𝒙𝒙. Any non-zero vector 𝒙𝒙 for which this equation holds is called an eigenvector for eigenvalue 𝜆𝜆
or an eigenvector of 𝐴𝐴 corresponding to eigenvalue 𝜆𝜆.

How to find eigenvalues and eigenvectors? To determine whether 𝜆𝜆 is an eigenvalue of 𝐴𝐴, we need to
determine whether there are any non-zero solutions to the matrix equation 𝐴𝐴𝒙𝒙 = 𝜆𝜆𝒙𝒙. To do this, we can
define the following:

(a) The eigenvalues of a symmetric matrix 𝐴𝐴 are the numbers 𝜆𝜆 that satisfy |𝐴𝐴 − 𝜆𝜆𝜆𝜆| = 0
(b) The eigenvectors of a symmetric matrix 𝐴𝐴 are the vectors 𝒙𝒙 that satisfy (𝐴𝐴 − 𝜆𝜆𝜆𝜆)𝒙𝒙 = 0

There are two theorems involved in the eigensystems and they are:

1. The eigenvalues of any real symmetric matrix are real.

2. The eigenvectors of any real symmetric matric corresponding to different eigenvalues are
orthogonal.

Example 2: Let 𝐴𝐴 be a square matrix as below. Find the eigenvalues and eigenvectors of matrix 𝐴𝐴

1 1
𝐴𝐴 = � �
2 2

Solution: To find the eigenvalues, we will need to find the determinant of |𝐴𝐴 − 𝜆𝜆𝜆𝜆| = 0, therefore:

1 1 1 0
|𝐴𝐴 − 𝜆𝜆𝜆𝜆| = �� � − 𝜆𝜆 � ��
2 2 0 1

57
 1 − 𝜆𝜆 1
=� �
2 2 − 𝜆𝜆

= (1 − 𝜆𝜆)(2 − 𝜆𝜆) − 2

= 𝜆𝜆2 − 3𝜆𝜆

So, the eigenvalues are the solutions of 𝜆𝜆2 − 3𝜆𝜆 = 0. We could simplify the equation is 𝜆𝜆(𝜆𝜆 − 3) = 0
with the solutions of 𝜆𝜆 = 0 and 𝜆𝜆 = 3. Hence the eigenvalues of 𝐴𝐴 are 0 and 3.

Now to find the eigenvectors for eigenvalue 0 we can solve the system (𝐴𝐴 − 0𝐼𝐼)𝒙𝒙 = 0, that is 𝐴𝐴𝒙𝒙 = 0, or

𝐴𝐴𝒙𝒙 = 0

1 1 𝑥𝑥1 0
� �� � = � �
2 2 𝑥𝑥2 0

We then have to solve for

𝑥𝑥1 + 𝑥𝑥2 = 0 , 2𝑥𝑥1 + 2𝑥𝑥2 = 0

Which gives 𝑥𝑥1 = −𝑥𝑥2 = −1. Therefore, the eigenvectors for eigenvalue 0 is:

−1
𝒙𝒙 = � �
1

Similarly, to find the eigenvector for eigenvalue 3, we will solve (𝐴𝐴 − 3𝐼𝐼)𝒙𝒙 = 0 which is:

(𝐴𝐴 − 3𝐼𝐼)𝒙𝒙 = 0

−2 1 𝑥𝑥1 0
� �� � = � �
2 −1 𝑥𝑥2 0

This is equivalent to the equations

−2𝑥𝑥1 + 𝑥𝑥2 = 0 , 2𝑥𝑥1 − 𝑥𝑥2 = 0

Which gives 𝑥𝑥2 = 2𝑥𝑥1 . If we choose 𝑥𝑥1 = 1, we then obtain the eigenvectors

1
𝒙𝒙 = � �
2

58
Example 3: Suppose that

4 0 4
𝐴𝐴 = �0 4 4�
4 4 8

Find the eigenvalues of 𝐴𝐴 and obtain one eigenvector for each eigenvalues.

Solutions: To find the eigenvalues, we will solve |𝐴𝐴 − 𝜆𝜆𝜆𝜆| = 0, so we can write as:

4 − 𝜆𝜆 0 4
|𝐴𝐴 − 𝜆𝜆𝜆𝜆| = � 0 4 − 𝜆𝜆 4 �
4 4 8 − 𝜆𝜆

4 − 𝜆𝜆 4 0 4 − 𝜆𝜆
= (4 − 𝜆𝜆) � � +4� �
4 8 − 𝜆𝜆 4 4

= (4 − 𝜆𝜆) �(4 − 𝜆𝜆)(8 − 𝜆𝜆) − 16� + 4�−4(4 − 𝜆𝜆)�

= (4 − 𝜆𝜆) �(4 − 𝜆𝜆)(8 − 𝜆𝜆) − 16� − 16(4 − 𝜆𝜆)

= (4 − 𝜆𝜆) �(4 − 𝜆𝜆)(8 − 𝜆𝜆) − 16 − 16�

= (4 − 𝜆𝜆) (32 − 12𝜆𝜆 + 𝜆𝜆2 − 32)

= (4 − 𝜆𝜆) (𝜆𝜆2 − 12𝜆𝜆)

= (4 − 𝜆𝜆) 𝜆𝜆 (𝜆𝜆 − 12)

Therefore, we can solve |𝐴𝐴 − 𝜆𝜆𝜆𝜆| = 0 and the eigenvalues are 4, 0, 12.

To find the eigenvectors for eigenvalue of 4, we solve the equation (𝐴𝐴 − 4𝐼𝐼)𝒙𝒙 = 0, that is,

(𝐴𝐴 − 4𝐼𝐼)𝒙𝒙 = 0

0 0 4 𝑥𝑥1 0
�0 𝑥𝑥
0 4 � � 2 � = �0 �
4 4 4 𝑥𝑥3 0

59
The equations we get out of the equation above are:

4𝑥𝑥3 = 0
4𝑥𝑥3 = 0
4𝑥𝑥1 + 4𝑥𝑥2 + 4𝑥𝑥3 = 0

Therefore, 𝑥𝑥3 = 0 and 𝑥𝑥2 = −𝑥𝑥1 . Choosing 𝑥𝑥1 = 1, we get the eigenvector

1
𝒙𝒙 = �−1�
0

Similar solution for 𝜆𝜆 = 0, the eigenvector is:

1
𝒙𝒙 = � 1 �
−1

And the solution for 𝜆𝜆 = 12, the eigenvector is:

1
𝒙𝒙 = �1�
2

3.5 Diagonalisation of symmetric matrices

A square matrix 𝑈𝑈 is said to be orthogonal if its reverse (if it exists) is equals to its transpose. Therefore:

𝑈𝑈 −1 = 𝑈𝑈 𝑇𝑇 or equivalently, 𝑈𝑈𝑈𝑈 𝑇𝑇 = 𝑈𝑈 𝑇𝑇 𝑈𝑈 = 𝐼𝐼 (117)

If 𝑈𝑈 is a real orthogonal matrix of order 𝑛𝑛 × 𝑛𝑛 and 𝐴𝐴 is a real matrix of the same order then 𝑈𝑈 𝑇𝑇 𝐴𝐴𝐴𝐴 is
called the orthogonal transform of 𝐴𝐴

Note: Since 𝑈𝑈 −1 = 𝑈𝑈 𝑇𝑇 for orthogonal U, the equality of 𝑈𝑈 𝑇𝑇 𝐴𝐴𝐴𝐴 = 𝐷𝐷 is the same as 𝑈𝑈 −1 𝐴𝐴𝐴𝐴 = 𝐷𝐷, the
diagonal entries of 𝐷𝐷 are the eigenvalues of 𝐴𝐴, and the columns of 𝑈𝑈 are the corresponding eigenvectors.

The theorems involving diagonalization of a symmetric matrix are as follows:

60
 1. If 𝐴𝐴 is a symmetric matrix in the order of 𝑛𝑛 × 𝑛𝑛 then it is possible to find an orthogonal matrix
𝑈𝑈 of the same order such that the orthogonal transform of 𝐴𝐴 with respect to 𝑈𝑈 is diagonal and the
diagonal elements of the transform are the eigenvalues of 𝐴𝐴.
2. Cayley-Hamilton Theorem: A real square matrix satisfies its own characteristic equation (i.e. its
own eigenvalue equation).

𝐴𝐴𝑛𝑛 + 𝑎𝑎𝑛𝑛−1 𝐴𝐴𝑛𝑛−1 + 𝑎𝑎𝑛𝑛−2 𝐴𝐴𝑛𝑛−2 + ⋯ + 𝑎𝑎1 𝐴𝐴 + 𝑎𝑎0 𝐼𝐼 = 0

Where

𝑎𝑎0 = (−1)𝑛𝑛 |𝐴𝐴| , 𝑎𝑎𝑛𝑛−1 = (−1)𝑛𝑛−1 𝑡𝑡𝑡𝑡(𝐴𝐴)

3. Trace Theorem: The sum of eigenvalues of matrix 𝐴𝐴 is equals to the sum of the diagonal elements
of 𝐴𝐴 and is defined as 𝑇𝑇𝑇𝑇(𝐴𝐴).
4. Determinant Theorem: The product of eigenvalues of 𝐴𝐴 is equals to the determinant of 𝐴𝐴.

Example 4: If we worked on the same matrix as in example 3 before, find the orthogonal matrix U, and
shows that 𝑈𝑈 𝑇𝑇 𝐴𝐴𝐴𝐴 = 𝐷𝐷:

4 0 4
𝐴𝐴 = �0 4 4�
4 4 8

Solution: As we already observed, matrix A is symmetric, and we have calculated the three distinct
eigenvalues 4, 0, 12 (in that order) and the eigenvectors associated with them are:

1 1 1
�−1� , �1� , �1�
0 −1 2

Now these eigenvectors are not f length 1. For example, the first eigenvector has a length of
�12 + (−1)2 + 02 = √2. So, if we divide each row by √2., we will indeed obtain eigenvector of length
1.

1/√2
�−1/√2�
0

We can similarly normalize the other two vectors and therefore we obtain:

61
 1/√3 1/√6
� 1/√3 � , �1/√6�
−1/√3 2/√6

Now we can form the matrix 𝑈𝑈 whose columns are these normalized eigenvectors:

1/√2 1/√3 1/√6

𝑈𝑈 = �−1/√2 1/√3 1/√6�
0 −1/√3 2/√6

Therefore, U is orthogonal and 𝑈𝑈 𝑇𝑇 𝐴𝐴𝐴𝐴 = 𝐷𝐷 = diag(4, 0, 12).

3.6 Matrix Factorization

Matrices can be factorized in a variety of ways as a product of matrices with different properties. These
different factorizations, or decompositions, reveal different aspects of matrix algebra and are useful in
different computational arenas. There are four types of factorizations namely Similarity Transform, LU
decomposition, QR decomposition and Singular Value decomposition. We will look into each
factorization method more closely.

3.6.1 Similarity Transform

Two square matrices A and B are said to be similar if an invertible matrix P can be found for which 𝐴𝐴 =
𝑃𝑃𝑃𝑃𝑃𝑃 −1. There are two types of similarity transforms:

a) Similarity to diagonal matrix.

Systems of differential equations sometimes can be uncoupled by diagonalizing a matrix, obtaining

the similarity transformation 𝐴𝐴 = 𝑃𝑃𝑃𝑃𝑃𝑃−1 , where 𝑛𝑛 columns of 𝑃𝑃 are the 𝑛𝑛 eigenvectors of 𝐴𝐴, and
𝐷𝐷 is the diagonal matrix and its entries are the corresponding eigenvalues of 𝐴𝐴.

b) Similarity to a Jordan canonical form.

However, the most general form is 𝐴𝐴 = 𝑃𝑃𝑃𝑃𝑃𝑃−1, where 𝐽𝐽 is the Jordan matrix rather than diagonal
matrix 𝐷𝐷. The Jordan matrix is a diagonal matrix with some additional 1’s on the superdiagonal,
the one above the main diagonal. For some matrices, the Jordan matrix is as close to
diagonalization as can be achieved.

62
A square matrix is similar to either a diagonal matrix or to a Jordan matrix. In either event, the eigenvalues
of 𝐴𝐴 appear on the diagonal of 𝐷𝐷 or 𝐽𝐽. A square symmetric matrix is orthogonally similar to a diagonal
matrix.

3.6.2 LU decomposition

LU decomposition can be obtained as a by-product of Gaussian elimination. The row reductions that yield
the upper triangular factor U also yields the lower triangular factor L. This decomposition is an efficient
way to solve systems of the form AX=Y, where the vector Y could be any one a number of right-hand
sides. In fact, Doolitte, Crout and Cholesky variations of the decomposition are important algorithms for
the numerical solution of systems of linear equations.

There are at least five different versions of LU decompositions:

1. Doolittle, 𝐿𝐿1 𝑈𝑈, 1’s on the main diagonal of 𝐿𝐿.

2. Crout, 𝐿𝐿𝐿𝐿1 , 1’s on the main diagonal of 𝑈𝑈.
3. LDU, 𝐿𝐿1 𝐷𝐷𝑈𝑈1 , 1’s on the main diagonals of 𝐿𝐿 and 𝑈𝑈 and 𝐷𝐷 is a diagonal matrix.
4. Gauss, 𝐿𝐿1 𝐷𝐷𝐿𝐿𝑇𝑇1 , 𝐴𝐴 is symmetric, 1’s on main diagonal of L, 𝐷𝐷 is a diagonal matrix.
5. Cholesky, 𝑅𝑅𝑅𝑅 𝑇𝑇 , 𝐴𝐴 is symmetric, positive definite, 𝑅𝑅 = 𝐿𝐿1 𝐷𝐷, with 𝐷𝐷 is a diagonal matrix.

Example 5: Find the LU decomposition of matrix 𝐴𝐴 below:

1 2 4
𝐴𝐴 = �3 8 14�
2 6 13

Solution: We can write the equation as 𝐴𝐴 = 𝐿𝐿𝐿𝐿

Using Dolittle: we know that

1 0 0 𝑈𝑈11 𝑈𝑈12 𝑈𝑈13

𝐿𝐿 = �𝐿𝐿21 1 0� and 𝑈𝑈 = � 0 𝑈𝑈22 𝑈𝑈23 �
𝐿𝐿31 𝐿𝐿32 1 0 0 𝑈𝑈33

Multiplying out the LU and setting the answer equals to A gives us:

𝑈𝑈11 𝑈𝑈12 𝑈𝑈13 1 2 4

�𝐿𝐿21 𝑈𝑈11 𝐿𝐿21 𝑈𝑈12 + 𝑈𝑈22 𝐿𝐿21 𝑈𝑈13 + 𝑈𝑈23 � = �3 8 14�
𝐿𝐿31 𝑈𝑈11 𝐿𝐿31 𝑈𝑈12 + 𝐿𝐿32 𝑈𝑈22 𝐿𝐿31 𝑈𝑈13 + 𝐿𝐿32 𝑈𝑈23 + 𝑈𝑈33 2 6 13

Now we can find the entries in L and U. From the top, we can see that:

63
𝑈𝑈11 = 1 , 𝑈𝑈12 = 2 , 𝑈𝑈13 = 4

Now if we consider the second row:

𝐿𝐿21 𝑈𝑈11 = 3 , 𝐿𝐿21 = 3

𝐿𝐿21 𝑈𝑈12 + 𝑈𝑈22 = 8 , 𝑈𝑈22 = 2

𝐿𝐿21 𝑈𝑈13 + 𝑈𝑈23 = 14 , 𝑈𝑈23 = 2

This pattern continues on the last row:

𝐿𝐿31 𝑈𝑈11 = 2 , 𝐿𝐿31 = 2

𝐿𝐿31 𝑈𝑈12 + 𝐿𝐿32 𝑈𝑈22 = 6 𝐿𝐿32 = 1

𝐿𝐿31 𝑈𝑈13 + 𝐿𝐿32 𝑈𝑈23 + 𝑈𝑈33 = 13 , 𝑈𝑈33 = 3

If we put all the values in the matrix, we have shown that:

1 2 4 1 0 0 1 2 4
𝐴𝐴 = �3 8 14� = �3 1 0� �0 2 2�
2 6 13 2 1 1 0 0 3

3.6.3 QR decomposition

The QR decomposition factors a matrix into a product of an orthogonal matrix 𝑄𝑄 (i.e. 𝑄𝑄 𝑇𝑇 𝑄𝑄 = 𝐼𝐼) and an
upper triangular matrix 𝑅𝑅. It is important ingredient of powerful numeric methods for finding eigenvalues
and for solving the least-squares problems.

The factor 𝑄𝑄 and 𝑅𝑅 for every real matrix are unique once the otherwise arbitrary signs on the diagonal of
𝑅𝑅 are fixed. Modern computational algorithms for finding eigenvalues numerically use some version of
the QR algorithm. Assume, for 𝐴𝐴 = 𝐴𝐴0 , do QR decomposition iteratively:

𝐴𝐴0 = 𝑄𝑄𝑜𝑜 𝑅𝑅0 , 𝐴𝐴1 = 𝑅𝑅0 𝑄𝑄0 = 𝑄𝑄1 𝑅𝑅1 , 𝐴𝐴2 = 𝑅𝑅1 𝑄𝑄1 = 𝑄𝑄2 𝑅𝑅2, (118)

If 𝐴𝐴 is real and no two eigenvalues hav equal magnitude, that is:

64
 0 < |𝜆𝜆𝑛𝑛 | < ⋯ < |𝜆𝜆2 | < |𝜆𝜆1 |

Then the sequence of matrix 𝐴𝐴𝑘𝑘 converges to an upper triangular matrix with eigenvalues of 𝐴𝐴0 on the
main diagonal. If, in addition, 𝐴𝐴 is symmetric, then 𝐴𝐴𝑘𝑘 converges to a diagonal matrix.

There are several methods for actually computing the QR decomposition. One of such method is the Gram-
Schmidt process.

Gram-Schmidt Process

In Gram-Schmidt process, we assume vectors as columns of matrix A. That is:

𝐴𝐴 = [𝒂𝒂1 |𝒂𝒂2 | ⋯ |𝒂𝒂𝑛𝑛 ]

Then, we can set:

𝒖𝒖1
𝒖𝒖1 = 𝒂𝒂1 , 𝒆𝒆1 =
‖𝒖𝒖1 ‖

𝒖𝒖2
𝒖𝒖2 = 𝒂𝒂2 − (𝒂𝒂2 ∙ 𝒆𝒆1 )𝒆𝒆1 , 𝒆𝒆2 =
‖𝒖𝒖2 ‖

𝒖𝒖𝑘𝑘+1
𝒖𝒖𝑘𝑘+1 = 𝒂𝒂𝑘𝑘+1 − (𝒂𝒂𝑘𝑘+1 ∙ 𝒆𝒆1 )𝒆𝒆1 − ⋯ − (𝒂𝒂𝑘𝑘+1 ∙ 𝒆𝒆𝑘𝑘 )𝒆𝒆𝑘𝑘 , 𝒆𝒆𝑘𝑘+1 =
‖𝒖𝒖𝑘𝑘+1 ‖

Note that ‖ ∙ ‖ is the 𝐿𝐿2 norm.

The resulting QR factorization is then:

𝒂𝒂1 ∙ 𝒆𝒆1 𝒂𝒂2 ∙ 𝒆𝒆1 ⋯ 𝒂𝒂𝑛𝑛 ∙ 𝒆𝒆1

0 𝒂𝒂2 ∙ 𝒆𝒆𝟐𝟐 … 𝒂𝒂𝑛𝑛 ∙ 𝒆𝒆2
𝐴𝐴 = [𝒂𝒂1 |𝒂𝒂2 | ⋯ |𝒂𝒂𝑛𝑛 ] = [𝒆𝒆1 |𝒆𝒆2 | ⋯ |𝒆𝒆𝑛𝑛 ] � ⋮ ⋮ ⋱ ⋮ � = 𝑄𝑄𝑄𝑄
0 0 ⋯ 𝒂𝒂𝑛𝑛 ∙ 𝒆𝒆𝒏𝒏

Example 6: Find the QR factorization of the matrix 𝐴𝐴 below:

1 1 0
𝐴𝐴 = �1 0 1�
0 1 1

Solution: First, let us write the vectors

65
𝒂𝒂1 = (1, 1, 0)𝑇𝑇 , 𝒂𝒂2 = (1, 0, 1)𝑇𝑇 , 𝒂𝒂3 = (0, 1, 1)𝑇𝑇

Note that all the vectors considered above and below and column vectors. Performing the Gram-Schmidt
procedure, we obtain:

𝒖𝒖1 = 𝒂𝒂1 = (1, 1, 0)𝑇𝑇

𝑇𝑇
𝒖𝒖1 1 𝑇𝑇
1 1
𝒆𝒆1 = = (1, 1, 0) = � , , 0�
‖𝒖𝒖1 ‖ √2 √2 √2

𝑇𝑇
𝑇𝑇
1 1
1 1 1
𝒖𝒖2 = 𝒂𝒂2 − (𝒂𝒂2 ∙ 𝒆𝒆1 )𝒆𝒆1 = (1, 0, 1)𝑇𝑇 −
� , , 0� = � , − , 1�
√2 √2 √2 2 2
𝑇𝑇 𝑇𝑇
𝒖𝒖2 1 1 1 1 1 2
𝒆𝒆2 = = � , − , 1� = � , − , �
‖𝒖𝒖2 ‖ �3/2 2 2 √6 √6 √6

𝒖𝒖3 = 𝒂𝒂3 − (𝒂𝒂3 ∙ 𝒆𝒆1 )𝒆𝒆1 − (𝒂𝒂3 ∙ 𝒆𝒆2 )𝒆𝒆2

𝑇𝑇
𝑇𝑇
1 1 1 1 1 1 2 𝑇𝑇 1 1 1 𝑇𝑇
= (0, 1, 1) − � , , 0� − � ,− , � = �− , , �
√2 √2 √2 √6 √6 √6 √6 √3 √3 √3
𝒖𝒖 1 1 1 𝑇𝑇
𝒆𝒆3 = 3 = �− , , �
‖𝒖𝒖3 ‖ √3 √3 √3

Therefore:

1 1 1
⎡ − ⎤
⎢√2 √6 √3⎥
⎢ 1 1 1 ⎥
𝑄𝑄 = [𝒆𝒆1 |𝒆𝒆2 | ⋯ |𝒆𝒆𝑛𝑛 ] = ⎢ − ⎥
⎢√2 √6 √3 ⎥
⎢0 2 1 ⎥
⎣ √6 √3 ⎦

2 1 1
⎡ ⎤
𝒂𝒂1 ∙ 𝒆𝒆1 𝒂𝒂2 ∙ 𝒆𝒆1 𝒂𝒂1 ∙ 𝒆𝒆3 ⎢ √2 √2 √2 ⎥
⎢ 3 1⎥
𝑅𝑅 = � 0 𝒂𝒂2 ∙ 𝒆𝒆2 𝒂𝒂2 ∙ 𝒆𝒆𝟑𝟑 � = ⎢ 0 ⎥
0 0 𝒂𝒂3 ∙ 𝒆𝒆𝟑𝟑 ⎢ √2 √6⎥
⎢0 2⎥
0
⎣ √3⎦

66
3.6.4 Singular Value decomposition

The singular value decomposition factors a matrix as a product of three factors, two being orthogonal
matrices (𝑈𝑈 and 𝑉𝑉) and one being diagonal (𝐷𝐷). The equation for Singular Value decomposition can be
written as:

𝐴𝐴 = 𝑈𝑈𝑈𝑈𝑉𝑉 𝑇𝑇 (119)

The columns in one orthogonal factor are left singular vectors, and the columns in the other orthogonal
factor are the right singular vectors. The matrix itself can be represented in outer product form in terms of
the left and right singular vectors. One use of this representation is in digital image processing.

Example 7: Factorize the matrix 𝐴𝐴 below using Singular Value decomposition method.

3 1 1
𝐴𝐴 = � �
−1 3 1

Solution: We’ll need to solve this problem one step at a time.

Step 1: Find orthogonal matrix 𝑈𝑈

3 1 1
𝐴𝐴 = � �
−1 3 1
3 −1
𝐴𝐴𝑇𝑇 = �1 3 �
1 1
Therefore,
3 −1
𝑇𝑇 3 1 1
𝐴𝐴𝐴𝐴 =� � �1 3 �
−1 3 1
1 1
11 1
=� �
1 11

If we can find the eigenvalues and eigenvectors of matrix 𝐴𝐴𝐴𝐴𝑇𝑇 , we get two eigenvalues of 𝜆𝜆 = 12 and
𝜆𝜆 = 10 (in that order). The eigenvectors for these eigenvalues then will be:

1 1
� �, � �
1 −1

So, the matrix now becomes:

1 1
� �
1 −1

67
Note that we will place the eigenvector for the highest eigenvalue 𝜆𝜆 = 12 in the first column of the matrix
above.

Now we need to change the matrix into orthogonal. To do this, we can use the Gram-Schmidt process as
discussed in Example 6 by using 𝒖𝒖1 = (1, 1)𝑇𝑇 and 𝒖𝒖2 = (1, −1)𝑇𝑇

So, the orthogonal matrix 𝑈𝑈 obtained is:

1 1
⎡ ⎤
𝑈𝑈 = ⎢ √2 √2 ⎥
⎢1 − 1⎥
⎣√2 √2⎦

Step 2: Find the orthogonal matrix 𝑉𝑉

𝑉𝑉 = 𝐴𝐴𝑇𝑇 𝐴𝐴
3 −1
3 1 1
𝐴𝐴𝑇𝑇 𝐴𝐴 = �1 3 � � �
−1 3 1
1 1
10 0 2
𝐴𝐴𝑇𝑇 𝐴𝐴 = � 0 10 4�
2 4 2

Similarly, we need to find the eigenvalues and eigenvectors of matrix 𝐴𝐴𝑇𝑇 𝐴𝐴. By solving for eigenvalues,
we obtained 3 eigenvalues of 𝜆𝜆 = 0, 𝜆𝜆 = 10 and 𝜆𝜆 = 12. The eigenvectors for these eigenvalues then will
be (in the order of eigenvalues):

1 2 1
� 2 �, �−1� , �2�
−5 0 1

So, the matrix now becomes:

1 2 1
�2 −1 2 �
1 0 −5

Note that we will place the eigenvector for the highest eigenvalue 𝜆𝜆 = 12 in the first column, followed by
𝜆𝜆 = 10 and finally 𝜆𝜆 = 0 of the matrix above.

68
Now we need to change the matrix into orthogonal. To do this, we can use the Gram-Schmidt process as
discussed in Example 6 by using 𝒖𝒖1 = (1,2,1)𝑇𝑇 , 𝒖𝒖2 = (2, −1, 0)𝑇𝑇 and 𝒖𝒖3 = (1, 2, −5)𝑇𝑇

So, the orthogonal matrix 𝑉𝑉 obtained is:

1 2 1
⎡ ⎤
⎢√6 √5 √30 ⎥
⎢2 1 2 ⎥
𝑉𝑉 = ⎢ − ⎥
⎢√6 √5 √30 ⎥
⎢1 0 −
5 ⎥
⎣√6 √30⎦

But we need the transpose of 𝑉𝑉 or 𝑉𝑉 𝑇𝑇 , therefore:

1 2 1
⎡ ⎤
⎢ √6 √6 √6 ⎥
⎢ 2 1 ⎥
𝑉𝑉 𝑇𝑇 =⎢ − 0 ⎥
⎢ √5 √5 ⎥
⎢ 1 2
−
5 ⎥
⎣√30 √30 √30⎦

Step 3: Now we need to get the Diagonal matrix 𝐷𝐷

The diagonal matrix 𝐷𝐷 is obtained by taking the square root of the non-zero eigenvalues and place them
diagonally on the same order of the original matrix 𝐴𝐴. Therefore, the diagonal matrix 𝐷𝐷 is:

0 0�
𝐷𝐷 = �√12
0 √10 0

Step 4: Finally, place all the matrices in the order of Equation (119), and we obtained:

𝐴𝐴 = 𝑈𝑈𝑈𝑈𝑉𝑉 𝑇𝑇
1 2 1
⎡ ⎤
1 1 ⎢ √6 √6 √6 ⎥
⎡ ⎤
�
3 1 1
� = ⎢√2 √2 ⎥ �√12 0 0 ⎢ 2 −
1
0 ⎥
⎥
�⎢
−1 3 1 ⎢1 −
1⎥ 0 √10 0 ⎢ √5 √5 ⎥
⎣√2 √2⎦ ⎢ 1 2
−
5 ⎥
⎣√30 √30 √30⎦

69
3.7 Solution of linear systems
These are methods for finding solutions of a set of linear equations which may be written in the matrix
form 𝐴𝐴𝒙𝒙 = 𝑏𝑏 where 𝒙𝒙 are 𝑛𝑛 unknown vectors. There are three methods involved and they will be discussed
separately below.

3.7.1 Direct methods

Direct methods of solving linear systems all use some form of matrix factorization. The LU factorization
is the most commonly used method to solve a linear system.

For certain patterned matrices, other direct methods may be more efficient. If a given matrix initially has
a large number of zeros, it is important to preserve the zeros in the same positions in the matrices that
result from operations on the given matrix. This helps to avoid unnecessary computations. The iterative
methods discussed in the next section are often more useful for sparse matrices.

Another important consideration is how easily an algorithm lends itself to implementation on advanced
computer architectures. Many of the algorithms for linear algebra can be vectorised easily. It is now
becoming more important to be able to parallelise the algorithms.

3.7.2 Iterative methods

3.7.2.1 The Jacobi method

The Jacobi’s method is the simplest iterative method for solving a (square) linear system. Let us start with
𝐴𝐴𝒙𝒙 = 𝑏𝑏. A can be decomposed into a diagonal component 𝐷𝐷 and the remainder 𝑅𝑅 = (𝐿𝐿 + 𝑈𝑈). The solution
is then obtained iteratively by:

𝒙𝒙𝑘𝑘+1 = 𝐷𝐷−1 (𝑏𝑏 − 𝑅𝑅𝑥𝑥 𝑘𝑘 ) (120)

Each element is given by:

1
𝑥𝑥𝑖𝑖𝑘𝑘+1 = �𝑏𝑏𝑖𝑖 − � 𝑎𝑎𝑖𝑖𝑖𝑖 𝑥𝑥𝑗𝑗𝑘𝑘 � , 𝑖𝑖 = 1,2, ⋯ , 𝑛𝑛 (121)
𝑎𝑎𝑖𝑖𝑖𝑖
𝑗𝑗≠𝑖𝑖
Note:

1. The method works well if the matrix A is diagonal dominant.

2. The matrix must verify 𝑎𝑎𝑖𝑖𝑖𝑖 ≠ 0

70
To understand this better in a matrix form, let us consider a 3 × 3 system of 𝐴𝐴𝒙𝒙 = 𝑏𝑏

𝐴𝐴11 𝐴𝐴12 𝐴𝐴13 𝑥𝑥 𝑏𝑏1

�𝐴𝐴21 𝐴𝐴22 𝐴𝐴23 � �𝑦𝑦� = �𝑏𝑏2 � (122)
𝐴𝐴31 𝐴𝐴32 𝐴𝐴33 𝑧𝑧 𝑏𝑏3

Solving for the first row for 𝑥𝑥, second row for 𝑦𝑦 and third row for 𝑧𝑧, and we obtain:

(𝑏𝑏1 − 𝐴𝐴12 𝑦𝑦 − 𝐴𝐴13 𝑧𝑧)

𝑥𝑥 =
𝐴𝐴11

(𝑏𝑏2 − 𝐴𝐴21 𝑥𝑥 − 𝐴𝐴23 𝑧𝑧)

𝑦𝑦 =
𝐴𝐴22

(𝑏𝑏3 − 𝐴𝐴31 𝑥𝑥 − 𝐴𝐴32 𝑦𝑦)

𝑧𝑧 =
𝐴𝐴33

Each iteration of the Jacobi method takes the form of:

(𝑏𝑏1 − 𝐴𝐴12 𝑦𝑦𝑛𝑛 − 𝐴𝐴13 𝑧𝑧𝑛𝑛 )

𝑥𝑥𝑛𝑛+1 =
𝐴𝐴11

(𝑏𝑏2 − 𝐴𝐴21 𝑥𝑥𝑛𝑛 − 𝐴𝐴23 𝑧𝑧𝑛𝑛 )

𝑦𝑦𝑛𝑛+1 =
𝐴𝐴22

(𝑏𝑏3 − 𝐴𝐴31 𝑥𝑥𝑛𝑛 − 𝐴𝐴32 𝑦𝑦𝑛𝑛 )

𝑧𝑧𝑛𝑛+1 =
𝐴𝐴33

We can re-write the equations above such that:

𝐴𝐴11 𝑥𝑥𝑛𝑛+1 = 𝑏𝑏1 − 𝐴𝐴12 𝑦𝑦𝑛𝑛 − 𝐴𝐴13 𝑧𝑧𝑛𝑛

𝐴𝐴22 𝑦𝑦𝑛𝑛+1 = 𝑏𝑏2 − 𝐴𝐴21 𝑥𝑥𝑛𝑛 − 𝐴𝐴23 𝑧𝑧𝑛𝑛

𝐴𝐴33 𝑧𝑧𝑛𝑛+1 = 𝑏𝑏3 − 𝐴𝐴31 𝑥𝑥𝑛𝑛 − 𝐴𝐴32 𝑦𝑦𝑛𝑛

Or in matrix form:

71
 𝐴𝐴11 0 0 𝑥𝑥𝑛𝑛+1 0 𝐴𝐴12 𝐴𝐴13 𝑥𝑥𝑛𝑛 𝑏𝑏1
� 0 𝐴𝐴22 𝑦𝑦
0 � � 𝑛𝑛+1 � = − �𝐴𝐴21 0 𝑦𝑦
𝐴𝐴23 � � 𝑛𝑛 � + �𝑏𝑏2 � (123)
0 0 𝐴𝐴33 𝑧𝑧𝑛𝑛+1 𝐴𝐴31 𝐴𝐴32 0 𝑧𝑧𝑛𝑛 𝑏𝑏3

Note that equation (123) is the same as equation (120).

Example 8: Let us consider the linear system below:

4 2 3 𝑥𝑥 8
� 3 −5 2� �𝑦𝑦� = �−14�
−2 3 8 𝑧𝑧 27

Note that there are no zeros on the leading diagonal, so we can solve the first row for 𝑥𝑥, second row for 𝑦𝑦
and third row for 𝑧𝑧, and we obtain:

(8 − 2𝑦𝑦 − 3𝑧𝑧)
𝑥𝑥 =
4

(−14 − 3𝑥𝑥 − 2𝑧𝑧)

𝑦𝑦 =
−5

(27 + 2𝑥𝑥 − 3𝑦𝑦)

𝑧𝑧 =
8

The arrangements above is the key to the method. At each stage of the process, new values of 𝑥𝑥, 𝑦𝑦 and 𝑧𝑧
will be obtained by substituting the previous values into the expressions. We then will apply a simple
iterative scheme of:

(8 − 2𝑦𝑦𝑛𝑛 − 3𝑧𝑧𝑛𝑛 )
𝑥𝑥𝑛𝑛+1 =
4

(−14 − 3𝑥𝑥𝑛𝑛 − 2𝑧𝑧𝑛𝑛 )

𝑦𝑦𝑛𝑛+1 =
−5

(27 + 2𝑥𝑥𝑛𝑛 − 3𝑦𝑦𝑛𝑛 )

𝑧𝑧𝑛𝑛+1 =
8

Let us begin the iterative process. We can start with 𝑥𝑥0 = 𝑦𝑦0 = 𝑧𝑧0 = 0, as the initial approximation as this
will make the iteration easy:

First iteration will then be:

72
 (8 − 2(0) − 3(0))
𝑥𝑥1 = =2
4

(−14 − 3(0) − 2(0))

𝑦𝑦1 = = 2.8
−5

(27 + 2(0) − 3(0))

𝑧𝑧1 = = 3.375
8

Second iteration gives:

(8 − 2(2.8) − 3(3.375))
𝑥𝑥2 = = −1.931
4

(−14 − 3(2) − 2(3.375))

𝑦𝑦2 = = 5.350
−5

(27 + 2(2) − 3(2.8))

𝑧𝑧2 = = 2.825
8

Third iteration gives:

(8 − 2(5.350) − 3(2.825))
𝑥𝑥3 = = −2.794
4

(−14 − 3(−1.931) − 2(2.825))

𝑦𝑦2 = = 2.771
−5

(27 + 2(−1.931) − 3(5.350))

𝑧𝑧2 = = 0.886
8

Eventually, the numbers will converge, and if you run this in a computer, you will end up after the 39th
iteration, we obtain the solution of 𝑥𝑥, 𝑦𝑦 and 𝑧𝑧 to be:

𝑥𝑥 −1
�𝑦𝑦� = � 3 �
𝑧𝑧 2

3.7.2.2 The Gauss-Seidel method

The Gauss-Seidel method is a variant of the Jacobi method that usually improves the rate of convergence.
This method implements the strategy of “always using the latest available value of a particular variable”.
In this method, we identify three matrices: a diagonal matrix 𝐷𝐷, a lower triangular matrix 𝐿𝐿 with 0s on the
diagonal and an upper triangular 𝑈𝑈 with 0s on its diagonal.

73
 (𝐷𝐷 + 𝐿𝐿)𝑥𝑥 = 𝑏𝑏 − 𝑈𝑈𝑈𝑈 (124)

We can write this entire sequence of Gauss-Seidel iterations in terms of these three fixed matrices:

𝑥𝑥 (𝑘𝑘+1) = (𝐷𝐷 + 𝐿𝐿)−1 �−𝑈𝑈𝑈𝑈 (𝑘𝑘) + 𝑏𝑏� (125)

To understand this better in a matrix form, let us consider the same 3 × 3 system of 𝐴𝐴𝒙𝒙 = 𝑏𝑏

𝐴𝐴11 𝐴𝐴12 𝐴𝐴13 𝑥𝑥 𝑏𝑏1

�𝐴𝐴21 𝐴𝐴22 𝐴𝐴23 � �𝑦𝑦� = �𝑏𝑏2 �
𝐴𝐴31 𝐴𝐴32 𝐴𝐴33 𝑧𝑧 𝑏𝑏3

Solving for the first row for 𝑥𝑥, second row for 𝑦𝑦 and third row for 𝑧𝑧, and we obtain:

(𝑏𝑏1 − 𝐴𝐴12 𝑦𝑦 − 𝐴𝐴13 𝑧𝑧)

𝑥𝑥 =
𝐴𝐴11

(𝑏𝑏2 − 𝐴𝐴21 𝑥𝑥 − 𝐴𝐴23 𝑧𝑧)

𝑦𝑦 =
𝐴𝐴22

(𝑏𝑏3 − 𝐴𝐴31 𝑥𝑥 − 𝐴𝐴32 𝑦𝑦)

𝑧𝑧 =
𝐴𝐴33

Each iteration of the Gauss-Seidel method takes the form of:

(𝑏𝑏1 − 𝐴𝐴12 𝑦𝑦𝑛𝑛 − 𝐴𝐴13 𝑧𝑧𝑛𝑛 )

𝑥𝑥𝑛𝑛+1 =
𝐴𝐴11

(𝑏𝑏2 − 𝐴𝐴21 𝑥𝑥𝑛𝑛+1 − 𝐴𝐴23 𝑧𝑧𝑛𝑛 )

𝑦𝑦𝑛𝑛+1 =
𝐴𝐴22

(𝑏𝑏3 − 𝐴𝐴31 𝑥𝑥𝑛𝑛+1 − 𝐴𝐴32 𝑦𝑦𝑛𝑛+1 )

𝑧𝑧𝑛𝑛+1 =
𝐴𝐴33

We can re-write the equations above such that:

𝐴𝐴11 𝑥𝑥𝑛𝑛+1 = 𝑏𝑏1 − 𝐴𝐴12 𝑦𝑦𝑛𝑛 − 𝐴𝐴13 𝑧𝑧𝑛𝑛

𝐴𝐴22 𝑦𝑦𝑛𝑛+1 = 𝑏𝑏2 − 𝐴𝐴21 𝑥𝑥𝑛𝑛+1 − 𝐴𝐴23 𝑧𝑧𝑛𝑛

74
 𝐴𝐴33 𝑧𝑧𝑛𝑛+1 = 𝑏𝑏3 − 𝐴𝐴31 𝑥𝑥𝑛𝑛+1 − 𝐴𝐴32 𝑦𝑦𝑛𝑛+1

Or in matrix form:

𝐴𝐴11 0 0 𝑥𝑥𝑛𝑛+1 0 0 0 𝑥𝑥𝑛𝑛+1 0 𝐴𝐴12 𝐴𝐴13 𝑥𝑥𝑛𝑛 𝑏𝑏1

� 0 𝐴𝐴22 𝑦𝑦
0 � � 𝑛𝑛+1 � = − �𝐴𝐴21 0 𝑦𝑦
0� � 𝑛𝑛+1 � − �0 0 𝑦𝑦
𝐴𝐴23 � � 𝑛𝑛 � + �𝑏𝑏2 � (126)
0 0 𝐴𝐴33 𝑧𝑧𝑛𝑛+1 𝐴𝐴31 𝐴𝐴32 0 𝑧𝑧𝑛𝑛+1 0 0 0 𝑧𝑧𝑛𝑛 𝑏𝑏3

Note that equation (126) is the same as equation (125).

Example 9: Let us consider the same linear system as in Example 8:

4 2 3 𝑥𝑥 8
� 3 −5 2� �𝑦𝑦� = �−14�
−2 3 8 𝑧𝑧 27

Using the Gauss-Seidel iteration, remember at each stage of the process, the latest values of 𝑥𝑥, 𝑦𝑦 and 𝑧𝑧 will
be obtained by substituting the previous values into the expressions. We then will apply a simple iterative
scheme of:

(8 − 2𝑦𝑦𝑛𝑛 − 3𝑧𝑧𝑛𝑛 )
𝑥𝑥𝑛𝑛+1 =
4

(−14 − 3𝑥𝑥𝑛𝑛+1 − 2𝑧𝑧𝑛𝑛 )

𝑦𝑦𝑛𝑛+1 =
−5

(27 + 2𝑥𝑥𝑛𝑛+1 − 3𝑦𝑦𝑛𝑛+1 )

𝑧𝑧𝑛𝑛+1 =
8

Let us begin the iterative process. We can start with 𝑥𝑥0 = 𝑦𝑦0 = 𝑧𝑧0 = 0, as the initial approximation as this
will make the iteration easy:

First iteration will then be:

(8 − 2(0) − 3(0))
𝑥𝑥1 = =2
4

(−14 − 3(2) − 2(0))

𝑦𝑦1 = =4
−5

(27 + 2(2) − 3(4))

𝑧𝑧1 = = 2.375
8

75
Which is already different to before.

Second iteration gives:

(8 − 2(4) − 3(2.375))
𝑥𝑥2 = = −1.781
4

(−14 − 3(−1.781) − 2(2.375))

𝑦𝑦2 = = 2.681
−5

(27 + 2(−1.781) − 3(2.681))

𝑧𝑧2 = = 1.924
8

Third iteration gives:

(8 − 2(2.681) − 3(1.924))
𝑥𝑥3 = = −0.784
4

(−14 − 3(−0.784) − 2(1.924))

𝑦𝑦2 = = 3.099
−5

(27 + 2(−0.784) − 3(3.099))

𝑧𝑧2 = = 2.017
8

If we carry on with the iteration, we will obtain the solution of 𝑥𝑥, 𝑦𝑦 and 𝑧𝑧 after the 9th iteration as:

𝑥𝑥 −1
�𝑦𝑦� = � 3 �
𝑧𝑧 2

3.7.2.3 The conjugate gradient method

The conjugate gradient method gives:

𝑥𝑥 (𝑘𝑘+1) = 𝑥𝑥 (𝑘𝑘) + 𝛼𝛼 (𝑘𝑘) 𝑝𝑝(𝑘𝑘) (127)

where 𝑝𝑝(𝑘𝑘) is a vector giving direction of the movement.

3.7.2.4 Multigrid methods

Iterative methods have important applications in solving differential equations. The solution of differential
equations by a finite difference discretisation involves the formation of a grid. The solution process may
begin with a fairly coarse grid on which a solution is obtained. Then a finer grid is formed, and the solution

76
is interpolated from the coarser grid to the finer grid to be used as a starting point for a solution over the
finer grid. The process is then continued through finer and finer grids. If all of the coarser grids are used
throughout the process, the technique is a multigrid method. There are many variations of exactly how to
do this. Multigrid methods are useful solution techniques for differential equations.

77