Chapter7 Numerical Analysis

http://www.amaderforum.
com
CHAPTER 7. NUMERICAL ANALYSIS

Numerical Analysis
Calculus and Fundamentals
The Solution of Nonlinear Equations f(x) = 0
The Solution of Linear Systems AX = B
Interpolation and Polynomial Approximation
Curve Fitting
Numerical Differentiation
Numerical Integration
Solution of Differential Equations
Solution of Partial Differential Equations
Eigenvalues and Eigenvectors
Numerical Optimization
http://www.amaderforum.com
Calculus and Fundamentals
Big "O" Truncation Error
Complex Numbers
Complex Functions
Big "O" Truncation Error

The 0th Order of Approximation
Clearly, the sequences
and
are both converging to zero.In
addition, it should be observed that the first sequence is converging to
zero more rapidly than the second sequence.In the coming modules
some special terminology and notation will be used to describe how
rapidly a sequence is converging.
Definition 1.The function
is said to be big Oh of
,denoted
,if there exist constants
and
such that
whenever
The big Oh notation provides a useful way of describing the rate of

growth of a function in terms of well-known elementary functions (
, etc.).The rate of convergence of sequences can be
described in a similar manner.
Definition 2.Let
and
said to be of order big Oh of
andNsuch that
whenever
be two sequences.The sequence

is
, denoted
,if there exist
Often a function
is approximated by a function
and the error
bound is known to be
.This leads to the following definition.
Definition 3.Assume that
is approximated by the function
and
that there exist a real constant
and a positive integer n so that
for sufficiently small h.
We say thatapproximateswith order of approximationandwrite
.
When this relation is rewritten in the form
that the notation
stands in place of the error bound
,we see
.The
following results show how to apply the definition to simple
combinations of two functions.
Theorem (Big "O" Remainders for Series Approximations).

Assume that
and
,and
(i)
(ii)
(iii)
.Then
,
,
provided that
It is instructive to consider
to be the
degree Taylor polynomial
approximation of
;then the remainder term is simply designated
,which stands for the presence of omitted terms starting with
the power
.The remainder term converges to zero with the same
rapidity that
converges to zero as h approaches zero, as expressed
in the relationship
for sufficiently smallh.Hence the notation

stands in place of the
quantity
, whereMis a constant or behaves like a constant.
Theorem (Taylor polynomial).

Assume that the function
continuous on
.Ifboth
and its derivatives

and
lie in the interval
,and
,
is the n-th degree Taylor polynomial expansion of
polynomial of degree nis
and
about
.The Taylor
are all
then
.
The integral form of the remainder is
,
and Lagrange's formula for the remainder is
where depends on and lies somewhere between
The following example illustrates the theorems above.The

computations use the addition properties
(i)
(ii)
(iii)
,
where
where
,
.
Order of Convergence of a Sequence

Numerical approximations are often arrived at by computing a sequence of
approximations that get closer and closer to the answer desired. The definition of big Oh
for sequences was given in definition 2, and the definition of order of
convergence for a sequence is analogous to that given for functions in
Definition 3.
Definition 4.Suppose that
and
is a sequence with
.We say that

converges to x with the order of
convergence
,if there exists a constant
such that
for n sufficiently large.
This is indicated by writing
or
with order of convergence
Example.Let
convergence
and
.
;then
with a rate of
Solution.
This follows immediately from the relation
for all n.
The Origin of Complex Numbers

Complex Numbers
Overview
Get ready for a treat. You're about to begin studying some of the most beautiful ideas in
mathematics. They are ideas with surprises. They evolved over several centuries, yet they
greatly simplify extremely difficult computations, making some as easy as sliding a hot
knife through butter. They also have applications in a variety of areas, ranging from fluid
flow, to electric circuits, to the mysterious quantum world. Generally, they are described
as belonging to the area of mathematics known as complex analysis.
Section 1.1The Origin of Complex Numbers

Complex analysis can roughly be thought of as the subject that applies the theory of
calculus to imaginary numbers. But what exactly are imaginary numbers? Usually,
students learn about them in high school with introductory remarks from their teachers
along the following lines: "We can't take the square root of a negative number. But let's
pretend we can and begin by using the symbol
." Rules are then learned for
doing arithmetic with these numbers. At some level the rules make sense. If
, it
stands to reason that
. However, it is not uncommon for students to wonder
whether they are really doing magic rather than mathematics.
If you ever felt that way, congratulate yourself! You're in the company of some of the
great mathematicians from the sixteenth through the nineteenth centuries. They, too, were
perplexed by the notion of roots of negative numbers. Our purpose in this section is to
highlight some of the episodes in the very colorful history of how thinking about
imaginary numbers developed. We intend to show you that, contrary to popular belief,
there is really nothing imaginary about "imaginary numbers." They are just as real as
"real numbers."
Our story begins in 1545. In that year the Italian mathematician Girolamo Cardano
published Ars Magna (The Great Art), a 40-chapter masterpiece in which he gave for the
first time an algebraic solution to the general cubic equation
.
Cardano did not have at his disposal the power of today's algebraic notation, and he
tended to think of cubes or squares as geometric objects rather than algebraic
quantities.Essentially, however, his solution began with the substiution
move transforms
into the cubic equation
squared term, which is called a depressed cubic and can be written as
.This
without a
.
You need not worry about the computational details, but the coefficients are
and
Exploration.
To illustrate, begin with

.The equation then becomes
and substitute
,
which simplifies to
Exploration.
If Cardano could get any value of x that solved a depressed cubic, he could easily get a
corresponding solution to
from the identity
. Happily,
Cardano knew how to solve a depressed cubic. The technique had been communicated to
him by Niccolo Fontana who, unfortunately, came to be known as Tartaglia(the
stammerer) due to a speaking disorder. The procedure was also independently discovered
some 30 years earlier by Scipione del Ferro of Bologna. Ferro and Tartaglia showed that
one of the solutions to the depressed cubic equation is
.
Although Cardano would not have reasoned in the following way, today we can take this
value for x and use it to factor the depressed cubic into a linear and quadratic term. The
remaining roots can then be found with the quadratic formula.
For example, to solve
se the substitution
to get
apply the "Ferro-Tartaglia" formula with
,u
,which is a depressed cubic equation.Next,
and
to get
.S
ince
is a root,
must be a factor of
.Dividing
into
gives
,which yields the remaining (duplicate) roots of
.The solutions to
are obtained by recalling
, which yields the three roots
and
.
Exploration.
So, by using Tartaglia's work and a clever transformation technique,

Cardano was able to crack what had seemed to be the impossible task
of solving the general cubic equation.Surprisingly, this development
played a significant role in helping to establish the legitimacy of
imaginary numbers.Roots of negative numbers, of course, had come
up earlier in the simplest of quadratic equations, such as
.The
solutions we know today as
, however, were easy for
mathematicians to ignore.In Cardano's time, negative numbers were
still being treated with some suspicion, as it was difficult to conceive of
any physical reality corresponding to them. Taking square roots of
such quantities was surely all the more ludicrous. Nevertheless,
Cardano made some genuine attempts to deal with
.
Unfortunately, his geometric thinking made it hard to make much
headway. At one point he commented that the process of arithmetic
that deals with quantities such as
"involves mental tortures and is
truly sophisticated." At another point he concluded that the process is
"as refined as it is useless." Many mathematicians held this view, but
finally there was a breakthrough.
In his 1572 treatise L'Algebra, Rafael Bombelli showed that roots of
negative numbers have great utility indeed. Consider the depressed
cubic
.Using
and
in the "Ferro-Tartaglia" formula
for the depressed cubic, we compute
, or in a
somewhat different form,
.
Simplifying this expression would have been very difficult if Bombelli
had not come up with what he called a "wild thought."He suspected
that if the original depressed cubic had real solutions, then the two
parts of x in the preceding equation could be written as
and
for some real numbers u and v.That is, Bombeli believed
and
,which would mean
and
.Then, using the wellknown algebraic identity
, and (letting
and
),and assuming that roots of negative numbers obey the rules
of algebra, he obtained
.
By equating like parts, Bombelli reasoned that
and
.Perhaps thinking even more wildly, Bombelli then
supposed that u and v were integers.The only integer factors of 2 are
2 and 1, so the equation
led Bombelli to conclude that
and
.From this conclusion it follows that
, or
.
Amazingly,
and
solve the second equation
, so
Bombelli declared the values for u and v to be u=2 and v=1,
respectively.
Since
, we clearly have
Bombelli showed that
.Similarly,
. But this means that

,
which was a proverbial bombshell.Prior to Bombelli, mathematicians

could easily scoff at imaginary numbers when they arose as solutions
to quadratic equations.With cubic equations, they no longer had this
luxury. That
was a correct solution to the equation
was
indisputable, as it could be checked easily.However, to arrive at this
very real solution, mathematicians had to take a detour through the
uncharted territory of "imaginary numbers."Thus, whatever else might
have been said about these numbers (which, today, we call complex
numbers), their utility could no longer be ignored.
Exploration.
Enter the solution for the cubic equation.
Use the Ferro-Tartaglia formula

a
nd verify thatzit is a solution.
Admittedly, Bombelli's technique applies only to a few specialized

cases, and lots of work remained to be done even if Bombelli's results
could be extended.After all, today we represent real numbers
geometrically on the number line.What possible representation could
complex numbers have?In 1673 John Wallis made a stab at a
geometric picture of complex numbers that comes close to what we
use today.He was interested in representing solutions to general
quadratic equations, which we can write as
to make the
following discussion easier to follow.When we use the quadratic
formula with this equation, we get
and
Wallis imagined these solutions as displacements to the left and right

from the point
. He saw each displacement, whose value is
,
as the length of the sides of the right triangles shown in Figure 1.1.
The points P and P represent the solutions to our equation, which is
clearly correct if
.But how should we picture P and P when
negative roots arise (i.e., when
)?Wallis reasoned that, with
negative roots, b would be less than c, so the lines of length b in
Figure 1.1 would no longer be able to reach all the way to the x
axis.Instead, they would stop somewhere above it, as Figure 1.2
shows.Wallis argued that P and P should represent the geometric
locations of the solutions
and
when
.He evidently thought that, because b is shorter than c, it could no
longer be the hypotenuse of the right triangle as it had been earlier.
The side of length c would now have to take that role.
Wallis's method has the undesirable consequence that

is
represented by the same point as is
. Nevertheless, this
interpretation helped set the stage for thinking of complex numbers as
"points on the plane." By 1732, the great Swiss
mathematicianLeonhard Euler(pronounced "oiler") adopted this view
concerning the n solutions to the equation
.You will learn shortly
that these solutions can be expressed as
for various
values of ;Euler thought of them as being located at the vertices of a
regular polygon in the plane. Euler was also the first to use the symbol
for
. Today, this notation is still the most popular, although some
electrical engineers prefer the symbol instead so that they can use
to represent current.
Is it possible to modify slightly Wallis's picture of complex numbers so
that it is consistent with the representation used today?To help you
answer this question, refer to the article by Alec Norton and Benjamin
Lotto, "Complex Roots Made Visible," The College Mathematics Journal,
15(3), June 1984, pp. 248--249, Jstor.
Two additional mathematicians deserve mention. The Frenchman
Augustin-Louis Cauchy (1789--1857) formulated many of the classic
theorems that are now part of the corpus of complex analysis.The
German Carl Friedrich Gauss (1777--1855) reinforced the utility of
complex numbers by using them in his several proofs of the
fundamental theorem of algebra (see Chapter 6).In an 1831 paper, he
produced a clear geometric representation of x+iy by identifying it
with the point (x, y) in the coordinate plane. He also described how to
perform arithmetic operations with these new numbers.
It would be a mistake, however, to conclude that in 1831 complex
numbers were transformed into legitimacy. In that same year the
prolific logician Augustus De Morgan commented in his book, On the
Study and Difficulties of Mathematics, "We have shown the symbol
to be void of meaning, or rather self-contradictory and absurd.
Nevertheless, by means of such symbols, a part of algebra is
established which is of great utility."
There are, indeed, genuine logical problems associated with complex
numbers. For example, with real numbers
so long as both
sides of the equation are defined. Applying this identity to complex
numbers leads to 1=1=((-1)(-1))=(-1)(-1)=-1.Plausible
answers to these problems can be given, however, and you will learn
how to resolve this apparent contradiction in Section 2.4. De Morgan's
remark illustrates that many factors are needed to persuade
mathematicians to adopt new theories. In this case, as always, a firm
logical foundation was crucial, but so, too, was a willingness to modify
some ideas concerning certain well-established properties of numbers.
As time passed, mathematicians gradually refined their thinking, and
by the end of the nineteenth century complex numbers were firmly
entrenched. Thus, as it is with many new mathematical or scientific
innovations, the theory of complex numbers evolved by way of a very
intricate process. But what is the theory that Tartaglia, Ferro, Cardano,
Bombelli, Wallis, Euler, Cauchy, Gauss, and so many others helped
produce? That is, how do we now think of complex numbers? We
explore this question in the remainder of this chapter.
Complex Functions and Linear Mappings
Complex Functions
Overview
The last chapter developed a basic theory of complex numbers. For the next few chapters
we turn our attention to functions of complex numbers. They are defined in a similar way
to functions of real numbers that you studied in calculus; the only difference is that they
operate on complex numbers rather than real numbers. This chapter focuses primarily on
very basic functions, their representations, and properties associated with functions such
as limits and continuity. You will learn some interesting applications as well as some
exciting new ideas.
2.1 Functions and Linear Mappings

A complex-valued function f of the complex variable z is a rule
that assigns to each complex number z in a set D one and
only one complex number w.We write
and call w the image
of z under f.A simple example of a complex-valued function
is given by the formula
.The set D is called the
domain of f, and the set of all images
is called
the range of f.When the context is obvious, we omit the
phrase complex-valued, and simply refer to a function f, or
to a complex function f.
We can define the domain to be any set that makes sense for
a given rule, so for
,we could have the entire
complex plane for the domain D, or we might artificially
restrict the domain to some set such as
.Determining the range for a function defined by a formula
is not always easy, but we will see plenty of examples later
on.In some contexts functions are referred to as mappings or
transformations.
In Section 1.6, we used the term domain to indicate a
connected open set.When speaking about the domain of a
function, however, we mean only the set of points on which
the function is defined.This distinction is worth noting,
and context will make clear the use intended.
Just as z can be expressed by its real and imaginary parts,
,we write
,where u and v are the real and
imaginary parts of w, respectively.Doing so gives us the
representation
.
Because u and v depend on x and y, they can be considered to
be real-valued functions of the real variables x and y; that
is,
and
Combining these ideas, we often write a complex function f

in the form
.
Figure 2.1 illustrates the notion of a function (mapping)
using these symbols.
Figure 2.1The mapping
There are two methods for defining a complex function in

Mathematica.
We now give several examples that illustrate how to express

a complex function.
Example 2.1.Write
in the for
Solution.Using the binomial formula, we obtain
so that
Example 2.2.Express the function

form
.
in the
Solution.Using the elementary properties of complex numbers,

it follows that
so that
Examples 2.1 and 2.2 show how to find u(x,y) and v(x,y) when
a rule for computing f is given. Conversely, if u(x,y) and
v(x,y) are two real-valued functions of the real
variables x and y, they determine a complex-valued function
,and we can use the formulas
and
to find a formula for f involving the variables z and
Example 2.3.Express
variables
.
by a formula involving the
Solution.Calculation reveals that
Using
in the expression of a complex function f may be
convenient.It gives us the polar representation
,
where U and V are real functions of the real variables r and
.
Remark.For a given function f, the functions u and v defined
above are different from those used previously in
which used Cartesian coordinates
instead of polar coordinates.
Example 2.4.Express
in both Cartesian and polar form.
Solution.For the Cartesian form, a simple calculation gives
so that
For the polar form, we get v
so that
Remark.Once we have defined u and v for a function f in

Cartesian form, we must use different symbols if we want to
express f in polar form. As is clear here, the functions u
and U are quite different, as are v and V.Of course, if we
are working only in one context, we can use any symbols we
choose.
For a given function f, the functions u and v defined here
are different from those defined by equation (2-1), because
equation (2-1) involves Cartesian coordinates and equation
(2-2) involves polar coordinates.
Example 2.5.Express
in polar form.
Solution. We obtain
so that
We now look at the geometric interpretation of a complex
function.If D is the domain of real-valued functions u(x,y)
and v(x,y), the equations
and
describe a transformation (or mapping) from D in the xy
plane into the uv plane, also called the w plane. Therefore,
we can also consider the function
to be a transformation (or mapping) from the set D in the z

plane onto the range R in the w plane.This idea was
illustrated in Figure 2.1. In the following paragraphs we
present some additional key ideas. They are staples for any
kind of function, and you should memorize all the terms in
bold.
If A is a subset of the domain D of f, the set
is called the image of the set A, and f is said to map A
onto B.The image of a single point is a single point, and
the image of the entire domain, D, is the range, R.The
mapping
is said to be from A into S if the image of A
is contained in S.Mathematicians use the notation
to
indicate that a function maps A into S.Figure 2.2
illustrates a function f whose domain is D and whose range
is R.The shaded areas depict that the function maps A onto
B.The function also maps A into R, and, of course, it maps D
onto R.
Figure 2.2
maps A onto B;
maps A into R.
The inverse image of a point w is the set of all points z in

D such that
.The inverse image of a point may be one
point, several points, or nothing at all.If the last case
occurs then the point w is not in the range of f.For
example, if
,the inverse image of the point
is
the single point , because
,and is the only
point that maps to
.In the case of
,the inverse
image of the point
is the set
.You will learn in
Chapter 5 that, if
,the inverse image of the point 0
is the empty set---there is no complex number z such that
.
The inverse image of a set of points, S, is the collection
of all points in the domain that map into S.If f maps D onto
R it is possible for the inverse image of R to be function
as well, but the original function must have a special
property: a function f is said to be one-to-one if it maps
distinct points
onto distinct points
.Many times
an easy way to prove that a function f is one-to-one is to
suppose
,and from this assumption deduce that
must equal
.Thus,
is one-to-one because if
,then
.Dividing both sides of the last
equation by gives
.Figure 2.3 illustrates the idea of a
one-to-one function: distinct points get mapped to distinct
points.
Figure 2.3A functionw = f(z)that is one-to-one.

The function
is not one-to-one because

,but
.Figure 2.4 depicts this situation: at least
two different points get mapped to the same point.
Figure 2.4A function that is not one-to-one.

In the exercises we ask you to demonstrate that one-to-one
functions give rise to inverses that are functions.Loosely
speaking, if
maps the set A one-to-one and onto the set
B, then for each w in B there exists exactly one point z in
AA such that
.For any such value of z we can take the
equation
and "solve" for z as a function of w.Doing so
produces an inverse function
where the following
equations hold:
Conversely, if
and
are functions that map A into
B and B into A, respectively, and the above hold, then f
maps the set A one-to-one and onto the set B.
Further, if f is a one-to-one mapping from D onto T and if A
is a subset of D, then f is a one-to-one mapping from A onto
its image B.We can also show that, if
is a one-to-one
mapping from A onto B and
is a one-to-one mapping from
B onto S, then the composite mapping
is a one-to-one
mapping from A onto S.
We usually indicate the inverse of by the symbol
.If the
domains of and
are A and B respectively, then we write
for all
,and
for all
Also, for
iff
iff
and
.
,and
.
Example 2.7. Show that the function

maps the line
the xy plane onto the line
in the w plane.
Solution.Method 1:With
.We let
and get
where
in
,we want to describe
is the notation for "if and only if."Note what this result says:
. The image of A under f, therefore, is the set
.
Method 2:We write

and note that the
transformation can be given by the equations
.Because A is
described by
,we can substitute
into the
equation
to obtain
,which we can rewrite as
.If you
use this method, be sure to pay careful attention to domains and
ranges.
We now look at some elementary mappings.If we let
fixed complex constant, the transformation
denote a
is a one-to-one mapping of the z-plane onto the w-plane and is called

a translation.This transformation can be visualized as a rigid
translation whereby the pointzis displaced through the vector
to
its new position
.The inverse mapping is given by
and shows thatTis a one-to-one mapping from the z-plane onto the wplane. The effect of a translation is depicted in Figure 2.5.
Figure 2.5The translation
If we let be a fixed real number, then for
,the transformation

a rotation.It can be visualized as a rigid rotation whereby the pointzis
rotated about the origin through an angle to its new position
.If
we use polar coordinates and designate
in the w-plane, then the
inverse mapping is
.
This analysis shows thatRis a one-to-one mapping from the z-plane
onto the w-plane.The effect of rotation is depicted in Figure 2.6.
Figure 2.6The rotation
Example 2.8.The ellipse centered at the origin with a horizontal major

axis of 4 units and vertical minor axis of 2 units can be represented by
the parametric equation
,for
Suppose we wanted to rotate the ellipse by an angle of

radians and
shift the center of the ellipse 2 units to the right and 1 unit up. Using
complex arithmetic, we can easily generate a parametric equation
r(t) that does so:
for
.Figure 2.7 shows parametric plots of these ellipses.
Figure 2.7(a)Plot of the original ellipse(b)Plot of the rotated ellipse
If we let
be a fixed positive real number, then the transformation

a magnification.If
,it has the effect of stretching the distance
between points by the factorK.If
,then it reduces the distance
between points by the factorK.The inverse transformation is given by
and shows thatSis a one-to-one mapping from the z-plane onto the wplane.The effect of magnification is shown in Figure 2.8.
Figure 2.8The magnification
Finally, if we let
and
then the transformation
.
, where
is a positive real number,

a linear transformation.It can be considered as the composition of a
rotation, a magnification, and a translation.It has the effect of rotating
the plane though an angle given by
,followed by a magnification
by the factor
, followed by a translation by the vector
.The
inverse mapping is given by
and shows that L is a oneto-one mapping from the z-plane onto the w-plane.
Example 2.9.Show that the linear transformation
the right half plane
onto the upper half plane
Solution.Method 1:Let
.To describe
for z to get
Thus
maps
.
,we solve
.We have the following
,which is the same as saying
Method 2: When we write
in Cartesian form as
,
we see that the transformation can be given by the equations

.Substituting
in the inequality
gives
,which is the upper half-plane
.
Method 3:The effect of the transformation
plane through the angle
and
,or
is a rotation of the
(when z is multiplied by
) followed by a
translation by the vector
.The first operation yields the set
.The second shifts this set up 1 unit, resulting in the set
illustrate this result in Figure 2.9.
Figure 2.9The linear transformation
.We
Translations and rotations preserve angles.First, magnifications rescale

distance by a factor K, so it follows that triangles are mapped onto
similar triangles, preserving angles.Then, because a linear
transformation can be considered to be a composition of a rotation, a
magnification, and a translation, it follows that linear transformations
preserve angles.Consequently, any geometric object is mapped onto
an object that is similar to the original object; hence linear
transformations can be called similarity mappings.
Note. The usage of the phrase "linear transformation" in a "complex
analysis course" is different than that the usage in "linear algebra
courses".
Example 2.10.Show that the image of the open disk

under the linear transformation
is the open disk
Solution.The inverse transformation is
the range of f as B, then
.
,so if we designate
Hence the disk with center

onto the disk with center
and radius 1 is mapped one-to-one and

and radius 5 as shown in Figure 2.10.
Figure 2.10The mapping
Example 2.11.Show that the image of the right half plane

under the linear transformation
is the half plane
.
Solution.The inverse transformation is given by
,
which we write as
.
Substituting
intoe
gives
.Figure 2.11 illustrates the mapping.
,which simplifies
Figure 2.11The the linear transformation
Fixed Point Iteration
Bisection Method
False Position or Regula Falsi Method
Newton-Raphson Method
Secant Method
Muller's Method
Aitken's Method & Steffensen's Acceleration
Accelerated &Modified Newton-Raphson
Improved Newton Method
Halley's Method
Horner's Method
Lin-Bairstow Method
Brent's Method
Graeffe's Method
Nonlinear Systems
Broyden's Method
Fixed Point Iteration

A fundamental principle in computer science is iteration.As the name suggests, a process
is repeated until an answer is achieved. Iterative techniques are used to find roots of
equations, solutions of linear and nonlinear systems of equations, and solutions of
differential equations.
A rule or function
for computing successive terms is needed,
together with a starting value .Then a sequence of values
is
obtained using the iterative rule
.The sequence has the
pattern
(starting value)
What can we learn from an unending sequence of numbers?If the

numbers tend to a limit, we suspect that it is the answer.
Finding Fixed Points

Definition ( FixedPoint ). A fixed point of a function
such that
.
Caution.A fixed point is not a root of the equation
solution of the equation
.
Geometrically, the fixed points of a function
intersection of the curve
and the line
is a number
,it is a
are the point(s) of

.
Definition (Fixed Point Iteration). The iteration

is called fixed point iteration.
for
Theorem (For a converging sequence). Assume that

is a
continuous function and that
is a sequence generated by fixed
point iteration.
If
,then
is a fixed point of
The following two theorems establish conditions for the existence of a

fixed point and the convergence of the fixed-point iteration process to
a fixed point.
Theorem (First Fixed Point Theorem). Assume that

, i. e.
is continuous on
.
Then we have the following conclusions.
(i).If the range of the mapping
satisfies
for all
,
then has a fixed point in
.
(ii).Furthermore, suppose that
is defined over
and that a
positive constant
exists with
for all
,then has a unique fixed point in
.
Theorem (Second Fixed Point Theorem). Assume that the
following hypothesis hold true.
(a) is a fixed point of a function ,
(b)
,
(c) is a positive constant,
(d)
, and
(e)
for all
.
Then we have the following conclusions.
(i).If
for all
,then the iteration
will
converge to the
unique fixed point
.In this case, is said to be an attractive
fixed point.
(ii).If
for all
,then the iteration
will not
converge to .
In this case, is said to be a repelling fixed point and the iteration
exhibits local divergence.
Remark 1.It is assumed that
in statement (ii).
Remark 2.Because is continuous on an interval containing

permissible to use the simpler criterion
and
in (i) and (ii), respectively.
, it is
Corollary. Assume that satisfies hypothesis (a)-(e)of the previous

theorem.Bounds for the error involved when using to approximate
are given by
for
,
and
for
Graphical Interpretation of Fixed-point Iteration

Since we seek a fixed point to
,it is necessary that the graph of
the curve
and the line
intersect at the point
.
The following animations illustrate two types iteration: monotone and
oscillating.
Algorithm (Fixed Point Iteration).To find a solution to the equation
by starting with and iterating
.
Mathematica Subroutine (Fixed Point Iteration).
Example.Use fixed point iteration to find the fixed point(s) for the
function
Solution.
Plot the function and determine graphically that there are two
solutions to the equation
.
Will iteration prove successful as a computational tool to numerically

find these two fixed points.
Use fixed point iteration to find a numerical approximation.

First, do the iteration one step at a time.Type each of the following
commands in a separate cell and execute them one at a time.
Now use the FixedPointIteration subroutine to perform the

computations.
Are 7 iterations enough to locate the fixed point?

How many decimal places agreement do you see to the equation
?
If it is not sufficient for your purposes you must request more
iterations!
Comparison 1. Compare with Mathematica's built in "FindRoot"

subroutine for numerically finding solutions.
Did Mathematica get all of the digits?Why?

There are options to every built in Mathematica subroutine. For
FindRoot it is:
Options[FindRoot]
{AccuracyGoal->Automatic,Compiled->True,DampingFactor->1,
Jacobian->Automatic,MaxIterations->15,WorkingPrecision->16}
It is the purpose of numerical analysis to study the last two items
"MaxIterations,""WorkingPrecision."
Comparison 2. Compare with Mathematica's built in "Solve"
subroutine for symbolically finding solutions.
Mathematica's answers are "mathematically perfect."

Now investigate fixed point iteration near the other fixed point,
Now use the FixedPointIteration subroutine to perform the
computations.
Remark. You need to look carefully at the above output and

determine what it means !Did it converge ?
The distinguishing property for determining convergence is the size of
.
If is near the fixed pointpand
converge top.
then the iteration will
If is near the fixed pointpand

converge top.
then the iteration will not
The distinguishing property for determining convergence is the size of

.
If is near the fixed point
and
then the iteration will converge to
in the neighborhood,
.
If is near the fixed point

and
then the iteration will not converge to
in the neighborhood,
.
The Bisection Method

Background. The bisection method is one of the bracketing methods for finding
roots of equations.
Implementation.Given a function f(x) and an interval which might contain a root,
perform a predetermined number of iterations using the bisection method.
Limitations.Investigate the result of applying the bisection method over an interval
where there is a discontinuity.Apply the bisection method for a function using an interval
where there are distinct roots.Apply the bisection method over a "large" interval.
Theorem (Bisection Theorem). Assume that
and that there exists a
number
such that
.
If
have opposite signs, and
represents the sequence of midpoints
generated by the bisection process, then
for
and the sequence
That is,
,
converges to the zero
Mathematica Subroutine (Bisection Method).
Example .Find all the real solutions to the cubic equation

Solution.
Plot the function.
There appears to be only one real root which lies in the interval [1,2].
Call the Bisection subroutine on the interval [1,2] using 10 iterations
After 10 iterations, the interval has been reduced to [a,b] where
The root lies somewhere in the interval [a,b] the width of which is
The reported root is alleged to be
The accuracy we can guarantee is one half of the interval width.
Is this the desired accuracy you want ?If not, more iterations are
required.
Compare our result with Mathematica's built in root finder.
Question.Why is Mathematica's answer different ?

How many bisections would it take to reduce the interval width to
?
Remember. The bisection method can only be used to find a real root
in an interval [a,b] in which f[x] changes sign.
Reduce the volume of printout.

After you have debugged you program and it is working properly, delete the unnecessary
print statements.
Concise Program for the Bisection Method

Now test the example to see if it still works. Use the last case in
Example 1 given above and compare with the previous results.
Reducing the Computational Load for the Bisection Method

The following program uses fewer computations in the bisection
method and is the traditional way to do it.Can you determine how
many fewer functional evaluations are used ?
Various Scenarios and Animations for the Bisection Method.
The Regula Falsi Method

Background.
The Regula Falsi method is one of the bracketing methods for finding roots of equations.
Implementation.Given a function f(x) and an interval which might contain a root,
perform a predetermined number of iterations using the Regula Falsi method.
Limitations.Investigate the result of applying the Regula Falsi method over an
interval where there is a discontinuity.Apply the Regula Falsi method for a function using
an interval where there are distinct roots.Apply the Regula Falsi method over a "large"
interval.
Theorem (Regula Falsi Theorem). Assume that

exists a number
such that
.
If
and that there
represents the sequence of points generated by the Regula Falsi process, then the
sequence
.
That is,
Mathematica Subroutine (Regula Falsi Method).
Example.Find all the real solutions to the cubic equation

Solution.
Plot the function.
There appears to be only one real root which lies in the interval [1,2].
Call the Regula Falsi subroutine on the interval [1,2] using 10

iterations
k
After 10 iterations, the interval has been reduced to [a,b] where
The root lies somewhere in the interval [a,b] width of which is

The reported root is alleged to be
The estimate of "how things are going" is the distance betweencand

the nearest endpoint to the interval.
Is this the desired accuracy you want ?If not, more iterations are
required.
k
Compare our result with Mathematica's built in root finder.
Remember. The Regula Falsi method can only be used to find a real root in an
interval [a,b] in which f[x] changes sign.

After you have debugged you program and it is working properly, delete the unnecessary
print statements.
Concise Program for the Regula Falsi
Now test the example to see if it still works. Use the last case in
Example 1 given above and compare with the previous results.
Reducing the Computational Load for the Regula Falsi Method

The following program uses fewer computations in the Regula Falsi
method and is the traditional way to do it.Can you determine how
many fewer functional evaluations are used ?
Various Scenarios and Animations for Regula Falsi Method .
Newton&'s Method
If
are continuous near a root , then this extra
information regarding the nature of
can be used to develop
algorithms that will produce sequences
that converge faster to
than either the bisection or false position method. The NewtonRaphson (or simply Newton's) method is one of the most useful and
best known algorithms that relies on the continuity of
.The method is attributed to Sir Isaac Newton (1643-1727) and Joseph
Raphson (1648-1715).
Theorem ( Newton-Raphson Theorem ).Assume that
and
there exists a number
, where
.If
, then there
exists a
such that the sequence
defined by the iteration
for
will converge to
for any initial approximation
Algorithm ( Newton-Raphson Iteration ).To find a root of

given an initial approximation using the iteration
for
Mathematica Subroutine (Newton-Raphson Iteration).
Example.Use Newton's method to find the three roots of the cubic polynomial
.
Determine the Newton-Raphson iteration formula
details of the computations for the starting value
Solution.
Graph the function.
that is used.Show
.
How many real roots are there ?Really !
The Newton-Raphson iteration formulag[x]is
Starting with
, Use the Newton-Raphson method to find a
numerical approximation to the root.First, do the iteration one step at
a time.Type each of the following commands in a separate cell and
execute them one at a time.
From the second graph we see that there are two other real roots, use
the starting values0.0and1.4to find them.
First, use the starting value
.
Then use the starting value
Compare our result with Mathematica's built in numerical root finder.
This can also be done with Mathematica's built in symbolic solve

procedure.
Definition (Order of a Root)Assume thatf(x)and its derivatives

are defined and continuous on an interval aboutx = p.We say
thatf(x) = 0has a root of ordermatx = pif and only if
.
A root of orderm = 1is often called a simple root, and ifm > 1it is called a
multiple root.A root of orderm = 2is sometimes called a double root, and so
on.The next result will illuminate these concepts.
Definition (Order of Convergence)Assume that converges top,and set

.If two positive constants
exist, and
then the sequence is said to converge topwith order of convergence R.The

numberAis called the asymptotic error constant.The cases
are given
specialconsideration.
(i)If
(ii)If
, the convergence of
is called linear.
is called quadratic.
Theorem (Convergence Rate for Newton-Raphson

Iteration)Assume that Newton-Raphson iteration produces a sequence
converges to the rootpof the function
.
that
Ifpis a simple root, then convergence is quadratic and

fo
rksufficiently large.
Ifpis a multiple root of orderm,then convergence is linear and
forksufficiently large.

After you have debugged you program and it is working properly,
delete the unnecessary print statements
a
nd
Concise Program for the Newton-Raphson Method
Now test this subroutine using the function in Example 1.
Error Checking in the Newton-Raphson Method

Error checking can be added to the Newton-Raphson method.Here we
have added a third parameter to the subroutine which estimate the
accuracy of the numerical solution.
The following subroutine call uses a maximum of 20 iterations, just to

make sure enough iterations are performed.However, it will terminate
when the difference between consecutive iterations is less than
.By
interrogatingkafterward we can see how many iterations were actually
performed.
Various Scenarios
The Secant Method

The Newton-Raphson algorithm requires two functions evaluations per
iteration,
and
.Historically, the calculation of a derivative
could involve considerable effort.But, with modern computer algebra
software packages such as Mathematica, this has become less of an
issue.Moreover, many functions have non-elementary forms (integrals,
sums, discrete solution to an I.V.P.), and it is desirable to have a
method for finding a root that does not depend on the computation of
a derivative. The secant method does not need a formula for the
derivative and it can be coded so that only one new function evaluation
is required per iteration.
The formula for the secant method is the same one that was used in
the regula falsi method, except that the logical decisions regarding
how to define each succeeding term are different.
Theorem (Secant Method).

Assume that
and there exists a number
, where
.If
, then there exists a
for
will converge to
for certain initial approximations
Algorithm ( Secant Method ).Find a root of

approximations
using the iteration
for
.
given two initial
Mathematica Subroutine (Secant Method).
Example.Use the secant method to find the three roots of the cubic
polynomial
.
Determine the secant iteration formula
that is used.
Show details of the computations for the starting value
Solution.
Enter the function.
The secant iteration formula
is
Hopefully, the iteration

Graph the function
will converge to a root of
There are three real root.

Starting with the values
Use the secant method to find a numerical approximation to the root.

First, do the iteration one step at a time.
Type each of the following commands in a separate cell and execute
them one at a time.
Now use the subroutine.
From the second graph we see that there are two other real roots.
Use the starting values
Use the starting values
Compare our result with Mathematica's built in numerical root finder.
Let's see how good they are.
Mathematica can obtain better numerical answers, but the number of

iterations needs to be increased.
Mathematica can also solve for the roots symbolically.
The answers can be manipulated into real expressions.
The answers can be expressed in decimal form.
These answers are in agreement with the ones we found with the
secant method.

After you have debugged you program and it is working properly,
delete the unnecessary print statements
and
and
Concise Program for the Secant Method
Now test this subroutine using the function in Example 1.
Error Checking in the Secant Method

Error checking can be added to the secant method.Here we have
added a third parameter to the subroutine which estimate the
accuracy of the numerical solution.
The following subroutine call uses a maximum of 20 iterations, just to

make sure enough iterations are performed.
However, it will terminate when the difference between consecutive
iterations is less than
.
By interrogatingkafterward we can see how many iterations were
actually performed.
Various Scenarios and Animations for the Secant Method.
Muller's Method
Background
Muller's method is a generalization of the secant method, in the
sense that it does not require the derivative of the function. It
is an iterative method that requires three starting points
,
, and
.A parabola is constructed that
passes through the three points; then the quadratic formula is
used to find a root of the quadratic for the next
approximation.It has been proved that near a simple root
Muller's method converges faster than the secant method and
almost as fast as Newton's method.The method can be used to
find real or complex zeros of a function and can be
programmed to use complex arithmetic.
Mathematica Subroutine (Muller's Method).
Example.Use Newton's method and Muller's method to find numerical
approximations to the multiple root
of the function
.
Show details of the computations for the starting value
.Compare the number
of iterations for the two methods.
Solution.
Graph the function.
Starting with
, use the Newton-Raphson method to find a
numerical approximation to the root.
We can use Mathematica's Solve procedure to determine some of the

roots.
For Newton's method, how far away is the eighth iteration

root
?
Note. The last iteration is actually stored in .
Starting with
,
, and
from the
, use Muller's method to find a
For Muller's method, how far away is the eighth iteration from the root
?
Note. The last iteration is actually stored in .
This is closer than

method.
which was obtained with Newton's
We are done.
Aside.Compare with Mathematica's built in routine.
Mathematica's answer is not so good, need to adjust the number of
iterations and the working precision.
Aitken's
Process and Steffensen's Acceleration Method
Background for Aitken's Process
We have seen that Newtons method converges slowly at a multiple
root and the sequence of iterates
exhibits linear convergence.A
technique called Aitken's process can be used to speed up
convergence of any sequence that is linearly convergent.In order to
proceed, we will need a definition.
Definition (Forward Difference).Given the sequence
the forward difference
by
for
Higher powers
for
,define
are defined recursively by

.
Whenk=2we have the useful formula

which simplifies to be:
for
Theorem (Aitken's Acceleration).Assume that the sequence

converges linearly to the limit p and that
for all
. If there exists
a real number A with
such that
then the sequence
defined by
converges to p faster than

.
,in the sense that
Steffensen's Acceleration
When Aitken's process is combined with the fixed point iteration in
Newtons method, the result is called Steffensen's acceleration.Starting
with , two steps of Newton's method are used to compute
and
compute
previous steps.
,then Aitken's
process is used to
.To continue the iteration set
and repeat the
Mathematica Subroutine (Steffensen's Method).
Example.Use Newton's method to construct a linearly convergent

sequence
which converges slowly to the multiple root
of
.
Then use the Aitken process to construct
which converges faster
to the root
.
Solution.
Graph the function.
Starting with
, use the Newton-Raphson method to find a
Since we know the root is

iteration.
,we can determine the error for each
Newton's method is converging linearly (or slowly), the error at each

step is being reduced by approximately one-half.Let us apply Aitken's
acceleration process to a sequence
of iterations generated by
Newton's method.
Again, we can determine the error for each term.
The sequence
converges top.
is converging topfaster than the sequence
The Accelerated and Modified Newton Methods

Background
Newton's method is commonly used to find the root of an equation.If the root is simple
then the extension to Halley's method will increase the order of convergence from
quadratic to cubic.At a multiple root it can be speed up with the accelerated NewtonRaphson and modified Newton-Raphson methods.
Theorem (Newton-Raphson Theorem ).Assume that
and
, where
.If
, then there
exists a
for
will converge to

are defined and continuous on an interval aboutx =
p.We say thatf(x) = 0has a root of ordermatx = pif and only if
.
A root of orderm = 1is often called a simple root, and ifm > 1it is
called a multiple root.A root of orderm = 2is sometimes called a
double root, and so on.The next result will illuminate these concepts.
Definition (Order of Convergence)Assume that converges
top,and set
exist,
and
then the sequence is said to converge topwith order of convergence

R.The numberAis called the asymptotic error constant.The cases
are given specialconsideration.
(i)If
(ii)If
is called linear.
Theorem (Convergence Rate for Newton-Raphson
Iteration)Assume that Newton-Raphson iteration produces a
sequence
that converges to the rootpof the function
.
Algorithm (Newton-Raphson Iteration).To find a root of
given an initial approximation using the iteration
for
Theorem (Acceleration of Newton-Raphson Iteration)Given that

the Newton-Raphson algorithm produces a sequence that converges
linearly to the root
of order
.Then the accelerated NewtonRaphson formula
fo
r
will produce a sequence
that converges quadratically top.
Mathematica Subroutine (Accelerated Newton-Raphson

Iteration).
Example.Use the accelerated Newton's method to find the double root

,of the cubic polynomial
.Use the starting value
Solution.
Graph the function.

Since thedouble root

has orderdouble root
Newton-Raphson iteration formulag[x]is
,the accelerated
Investigate quadratic convergence at the double root

starting value
,using the
First, do the iteration one step at a time.

Type each of the following commands in a separate cell and execute
them one at a time.
Notice that convergence is much faster than the standard NewtonRaphson iteration.
At the double root
we can explore the ratio
Therefore, the accelerated Newton--Raphson iteration is converging

quadratically.
More Background
Iff(x)has a root of multiplicitymatx=p, thenf(x)can be exssed in the
form
where
.In this situation, the functionh(x)is defined by
and it is easy to prove thath(x)has a simple root atx=p.When Newton's

method is applied for finding the rootx=pofh(x)we obtain the following
result.
Theorem (Modified Newton-Raphson Iteration)Given that the
Newton-Raphson algorithm produces a sequence that converges
linearly to the root
of multiplicity
.Then the modified Newtonhttp://www.amaderforum.com
Raphson formula
which can be simplified as

fo
r
will produce a sequence
that converges quadratically top.
Mathematica Subroutine (Modified Newton-Raphson Iteration).
MATHEMATICAL EXPOSITION
An Improved Newton's Method
by
John H. Mathews
The AMATYC Review, Vol. 10, No. 2, Spring, 1989, pp. 9-14.
Introduction
Newton's method is used to locate roots of the equation
Newton-Raphson iteration formula is:
.The
(1)
Given a starting value
,the sequence
(2)
provided that
is computed using:
for
.
If the value is chosen close enough to the root p, then the sequence
generated in (2) will converge to the root p.Sometimes the speed at
which
converges is fast (quadratic) and at other times it is slow
(linear).To distinguish these two cases we make the following
definitions.
Definition 1 (Order of Convergence)Assume that
top,and set
and
converges
exist,

(3)If
, and
then the convergence of

(4)If
, and
then the convergence of
is called linear.
The mathematical characteristic for determining which case occurs is

the "multiplicity" of the root p.
Definition 2 (Order of a Root)If
(5)
can be factored as
wheremis a positive integer,
and
is continuous at
of ordermat
.
and
,then we say that
has a root
A root of order
is often called a simple root, and if
it is called a
multiple root.A root of order
is sometimes called a double root, and
so on.
Theorem 1 (Convergence Rate for Newton-Raphson
Iteration)Assume that Newton-Raphson iteration (2) produces a
sequence
.
(6)Ifpis a simple root, then convergence is quadratic and
(7)Ifpis a multiple root of orderm,then convergence is linear and
There are two common ways to use Theorem 1 and gain quadratic
convergence at multiple roots. We shall call these methods A and B
(see Mathews, 1987, p. 72 and Ralston & Rabinowitz, 1978, pp. 353356).
Method A.Accelerated Newton-Raphson
Suppose thatpis a root of order
Raphson formula is:
(8)
.Then the accelerated Newton-
Let the starting value

iteratively;
be close top, and compute the sequence
(9)
for
Then the sequence generated by (9) will converge quadratically top.

Mathematica Subroutine (Accelerated Newton-Raphson
Iteration).
On the other hand, if
function
Using
then one can show that the
has a simple root at

in place of
in formula (1) yields Method B.
Method B.Modified Newton-Raphson

Suppose thatpis a root of order
Raphson formula is:
.Then the modified Newton-
(10)
Let the starting value
iteratively;
be close top, and compute the sequence
(11)
for
Then the sequence generated by (11) converges quadratically top.
Mathematica Subroutine (Modified Newton-Raphson Iteration).
Limitations of Methods A and B

Method A has the disadvantage that the ordermof the root must be
known a priori.Determiningmis often laborious because some type of
mathematical analysis must be used.It is usually found by looking at
the values of the higher derivatives of
.That is ,
has a root of
order mat
if and only if
(12)
.
Dodes (1978, pp. 81-82) has observed that in practical problems it is
unlikely that we will know the multiplicity.However, a constantmshould
be used in (8) to speed up convergence, and it should be chosen small
enough so that
does not shoot to the wrong side ofp.Rice (1983,
pp. 232-233) has suggested a way to empirically findm.If is a good
approximation topand , and somewhat distant from thenmcan be
determined by the calculation:
.
Method B has a disadvantage, it involves three functions
.
Again, the laborious task of finding the formula for
could detract
from using Method B.Furthermore, Ralston and Rabinowitz (1978, pp.
353-356) have observed that
will have poles at points where the
zeros of
are not roots of
.Hence,
may not be a continuous
function.
The New Method C.Adaptive Newton-Raphson
The adaptive Newton-Raphson method incorporates a linear search
method with formula (8).Starting with ,the following values are
computed:
(13)
for
.
Our task is to determine the valuemto use in formula (13), because it

is not known a priori.First, we take the derivative of
in formula (5),
and obtain:
(14)
When (5) and (14) are substituted into formula (1) we have
w
hich can be simplified to get
.
This enables us to rewrite (13) as
(15)
.
We shall assume that the starting value
is close enough topso that
(16)
,
where
The iterates
in (15) satisfy the following:
(17)
for
If we subtractpfrom both sides of (17) then
and the result after simplification is:

(18)
Since
,the iterates get closer topasjgoes from
manifest by the inequalities:
(19)
The values
,which is
.
are shown in Figure 1.
Notice that if the iteration (15) was continued for

then
could be larger than
.This is proven by
using the derivatives in (12) and the Taylor polynomial approximation
of degreemfor
expanded about
:
(20)
Figure 1.The values

using formula (15)
near a "double root"p(of order
If is closer topthan
hence we have:
for the "linear search" obtained by

).Notice that
then (19) and (20) imply that
(21)
.
Therefore, the way to computationally determinemis to successively
compute the values using formula (13) for
until we
arrive at
.
The New Adaptive Newton-Raphson Algorithm
Start with
,then we determine the next approximation
as follows:
Observe that the above iteration involves a linear search in either the
interval
when
or in the interval
when
.In the
algorithm, the value
is stored so that unnecessary
computations are avoided.After the point
has been found, it
should replace
and the process is repeated.
Mathematica Subroutine (Adaptive Newton-Raphson Iteration).
Example.Use the value

finding the double root
and compare Methods A,B and C for

of the equation
.
Solution.
First, we will compute the iterations for each method, and afterward a
table comparing the methods is given.For the method C, all the
iterations in the linear search are included.
Using formula (2), the standard Newton-Raphson method.
Using formula (9) Method A, the accelerated Newton-Raphson method

withm=2.
Using formula (11) Method B, the modified Newton-Raphson method.
Using formula (13) Method C, the adaptive Newton-Raphson method.
The details for obtaining
are:
Since
we reject and set
and continue the iteration using formula
(13).The subroutine makes all these choices automatically.
Summary of the above results.
Observe in this example that the standard Newton-Raphson method

converges linearly and methods A and B converge quadratically.The
reader can use formulas (12) to verify that
is the order of the root
.The new method C is almost as fast as methods A and B.
The goal of this investigation is to show how the adaptive NewtonRaphson method is superior to the standard Newton-Raphson method,
because of the limitations of Methods A and B.
Why is it difficult to locate a multiple root.Because the function values themselves are
essentially "noise" when you get close to a multiple root.
One way to tell is to graph their difference.
The formula
should be considered the "true value."
So one should be suspect of the computation
for values
ofxis nearx=1.
Behavior at a Triple Root

When the function has a triple root, then one more iteration for the
linear search in (13) is necessary.The situation is shown in Figure 2.
Figure 2. The values

using formula (15)
near a "triple root"p(of order
).Notice that
.
for the "linear search" obtained by
Halley's Method
Background
are defined and continuous on an interval about
.We say that
has a root of ordermat
if and only if
.
A root of order
is often called a simple root, and if
it is called a
multiple root.A root of order
.is sometimes called a double root,
and so on.The next result will illuminate these concepts.
Definition (Order of Convergence)Assume that converges
top,and set
exist,
and

(i)If
is called linear.
(ii)If
(ii)If
is called cubic.
Theorem ( Newton-Raphson Iteration ).

Assume that
and there exists a number
, where
.If
for
will converge to
Furthermore, if
convergence
is a simple root, then
will have order of
,i.e.
Theorem (Convergence Rate for Newton-Raphson Iteration)

Assume that Newton-Raphson iteration produces a sequence
.
Halley's Method
The Newton-Raphson iteration function is
(1)
It is possible to speed up convergence significantly when the root is

simple.A popular method is attributed to Edmond Halley (1656-1742)
and uses the iteration function:
(2)
The term in brackets shows where Newton-Raphson iteration function

is changed.
Theorem ( Halley's Iteration ).Assume that
and there
exists a number
, where
.If
for
will converge to
Furthermore, if
is a simple root, then
convergence
,i.e.
will have order of

.
Square Roots
The function
where
can be used with (1) and (2) to
produce iteration formulas for finding
.If it is used in (1), the result
is the familiar Newton-Raphson formula for finding square roots:
(3)
When it is used in (2) the resulting Halley formula is:
(4)or
This latter formula is a third-order method for computing

.Because
of the rapid convergence of the sequences generated by (3) and (4),
the iteration usually converges to machine accuracy in a few
iterations.Multiple precision arithmetic is needed to demonstrate the
distinction between second and third order convergence.The software
Mathematica has extended precision arithmetic which is useful for
exploring these ideas.
Example.Consider the function
, which has a root at
.
(a).Use the Newton-Raphson formula to find the root.Use the starting
value
(b).Use Halley's formula to find the root.Use the starting value
Solution (a).
Form the Newton-Raphson iteration functiong(x).
We start the iteration with

and carry 100 digits in the computations,
by telling Mathematica the precision of by issuing the command p[0]
=N[2,100].Next, a short program is written to compute the first seven
terms in the iteration:
Since the root is known to be exactly

we can have Mathematica
list the error
at each step in the iteration:
Looking at the error, we see that the number of accurate digits is

doubling at each step in the computations, hence convergence is
proceeding quadratically.
Verify the convergence rate.At the simple root
Therefore, the Newton-Raphson iteration is converging quadratically.
Solution 1 (b).
Now we will investigate Halley's iteration for finding square roots.
Form the Halley iteration functionh(x).
We start the iteration with

and carry 100 digits in the computations,
by telling Mathematica the precision of by issuing the command p[0]
=N[2,100].Next, a short program is written to compute the first five
terms in the iteration:
Since the root is known to be exactly

we can have Mathematica
list the error
at each step in the iteration:
Looking at the error, we see that the number of accurate digits is

tripling at each step in the computations, hence convergence is
proceeding cubically.
We can conclude that Halley's method is faster than Newton's method.
Verify the convergence rate.At the simple root
.
Therefore, the Halley iteration is converging cubically.
Horner's Method
Evaluation of a Polynomial
Let the polynomial
form
of degree n have coefficients
.Then
has the familiar
Horner's method (or synthetic division) is a technique for evaluating

polynomials.It can be thought of as nested multiplication.For example,
the fifth-degree polynomial
can be written in the "nested multiplication" form

.
Theorem (Horner's Method for Polynomial Evaluation)Assume
that
(1)
and
is a number for which
is to be evaluated.Then
computed recursively as follows.
(2)Set
and
,
for
Then
Moreover, the coefficients

(3)
and
(4)
can be
can be used to construct
and
where
is the quotient polynomial of degreen-1and
remainder.
is the
Example.Use synthetic division (Horner's method) to find
polynomial
.
for the
Example.
Use synthetic division (Horner's method) to find
.
for the polynomial
Solution.
This example is for pedagogical purposes.Our eventual goal is to use a vector for the
coefficients, but for now we will enter them separately into each coefficient.First we will
check out the nested multiplication idea.
Now we will use the recursive formulas to compute the sequence
Moreover, let us verify the formulas for the quotient and remainder
,
where
is the quotient polynomial of degreen-1and
is the remainder.
Heuristics
In the days when "hand computations" were necessary, the Horner
tableau (or table) was used.The coefficients
of the polynomial are
entered on the first row in descending order, the second row is
reserved for the intermediate computation step(
)and the bottom
row contains the coefficients and
.
Lemma (Horner's Method for Derivatives)Assume that
and
is a number for which
and
are to be evaluated.We
have already seen that
can be computed recursively as follows.
,and
for
The quotient polynomial

remainder
form the relation
and
.
We can compute
(i)
can be computed recursively as follows.
,and
for
The quotient polynomial

and remainder
form the relation
(ii)
The Horner tableau (or table) was used for computing the coefficients
is given below.
Using vector coefficients

As mentioned above, it is efficient to store the coefficients
of a
polynomial
of degree n in the vector
.Notice that this is a shift of the index for
and the polynomial
is
written in the form
.
Given the value
coefficients
,the recursive formulas for computing the

and
of
and
, have the new form
for
for
Then
Newton-Horner method
Assume that
is a polynomial of degree
and there exists a
number
, where
.If
such
that the sequence
defined by the Newton-Raphson iteration
formula
for
will converge torfor any initial approximation
.The
recursive formulas in the Lemma can be adapted to compute
and
and the resulting Newton-Horner
iteration formula looks like
for
Algorithm (Newton-Horner Iteration).To find a root of
an initial approximation using the iteration
for
Mathematica Subroutine (Newton-Horner Iteration).
given
Lemma (Horner's Method for Higher Derivatives)Assume that the

coefficients
of a polynomial
of degree n are stored in the
first row of the matrix
.Then the polynomial
can written in
the form
.
Given the value
is
,the subroutine for computing all the derivatives
and
for
Polynomial Deflation
Given the polynomial
in example 5,the iteration
will converge to the root

of
.The Mathematica command
NewtonHorner[3.0,6] produces the above sequence, then the quotient polynomial
is constructed with the command
The root stored in the computer is located in the variabler1.
The coefficients ofQ[x]printed above have been rounded off.Actually there is a little bit
of round off error in the coefficients forming
,we will have to dig them out to look at
them.
Now we have a computer approximation for the factorization
We should carry out one more step in the iteration using the
commandNewtonHorner[3.0,7]and get a more accurate calculation for the
coefficients of
.When this is done the result will be:
If the other roots of

are to be found, then they must be the roots of the quotient
polynomial
.The polynomial
is referred to as the deflated polynomial,
because its degree is one less than the degree of
.For this example it is possible to
factor
as the product of two quadratic polynomials
.Therefore,
has the factorization
,
and the five roots of
are
.
This can be determined by using Mathematica and the command Factor.
This still leaves some unanswered questions that we will answer in other modules.The
quadratic factors can be determined using the Lin-Bairstow method.Or if one prefers
complex arithmetic, then Newton's method can be used.For example, starting with the
imaginary number
Newton's method will create a complex sequence
converging to the complex root
of
.
However, starting with purely imaginary number

sequence.
will create a divergent
For cases involving complex numbers the reader should look at the Lin-Bairstow and the
Fundamental Theorem of Algebra modules.
Getting Real Roots
The following example illustrates polynomial deflation and shows that the order in which
the roots are located could be important.In light of example 6 we know that better
calculations are made for evaluating
whenxis small.The Newton-Horner subroutine
is modified to terminate early if
evaluates close to zero (when a root is located).
Mathematica Subroutine (Newton-Horner Iteration).
The Lin-Bairstow Method

Quadratic Synthetic Division
Let the polynomial
form
of degree n have coefficients
.Then
has the familiar
(1)
Let
be a fixed quadratic term.Then
can be expressed as
(2)
where
.Here
,
is the remainder when
is divided by
is a polynomial of degree
and can be represented by
(3)
.
If we set
and
,then
(4)
,
where
(5)
and equation (4) can be written
(6)
.
The terms in (6) can be expanded so that
is represented in powers ofx.
(7)
The numbers
coefficients
(8)Set
are found by comparing the coefficients of in equations (1) and (7).The

of
andare computed recursively.
,and
,and then
for
Example.Use quadratic synthetic division to divide

by
.
Example.
Use quadratic synthetic division to divide
.
by
Solution.
First, construct the polynomial
Second, given
construct the polynomials
Third, verify that
We are done.
Aside.We can have Mathematica compute the quotient and remainder using the built in
proceduresPolynomialQuotientandPolynomialRemainder.This is just for fun!
Heuristics
In the days when "hand computations" were necessary, the quadratic synthetic division
tableau (or table) was used.The coefficients
of the polynomial are entered on the
first row in descending order, the second and third rows are reserved for the intermediate
computation steps(
and
)and the bottom row contains the coefficients ,
and
.
Using vector coefficients

As mentioned above, it is efficient to store the coefficients
of a polynomial
of degree n in the vector
.Notice that this is a shift of the
index for
and the polynomial
is written in the form
.
Given the quadratic
,the quotient
and
.
and remainder
are
The recursive formulas for computing the coefficients
of
are
,and
,and then
for
The Lin-Bairstow Method

We now build on the previous idea and develop the Lin-Bairstow's method for
finding a quadratic factor
of
.Suppose that we start with the initial guess
(9)
and that
can be expressed as
.
Whenuandvare small, the quadratic (9)is close to a factor of

values
so that
.We want to find new
(10)
is closer to a factor of
than the quadratic (9).
Observe thatuandvare functions ofrands, that is

,and
.
The new values
satisfy the relations
,and
.
The differentials of the functionsuandvare used to produce the approximations
and
The new values
are to satisfy
,and
.
When the quantities
are small, we replace the above approximations with
equations and obtain the linear system:
(11)
All we need to do is find the values of the partial derivatives
and
and then use Cramer's rule to compute
announce that the values of the partial derivatives are
where the coefficients

are built upon the coefficients
calculated recursively using the formulas
(12)Set
,
.Let us
given in (8) and are
,and
,and then
for
The formulas in (12) use the coefficients
in (8).Since
,and
,and
the linear system in (11)can be written as
Cramer's rule can be used to solve this linear system.The required determinants are
,
and the new values
,and
are computed using the formulas
,
and
.
The iterative process is continued until good approximations torandshave been found.If
the initial guesses
are chosen small, the iteration does not tend to wander for a
long time before converging.When
,the larger powers ofxcan be neglected in
equation (1) and we have the approximation
.
Hence the initial guesses for
.
could be
and
,provided that
If hand calculations are done, then the quadratic synthetic division tableau can be
extended to form an easy way to calculate the coefficients
.
Bairstow's method is a special case of Newton's method in two dimensions.

Algorithm (Lin-Bairstow Iteration).To find a quadratic factor of
given an initial approximation
.
Mathematica Subroutine (Lin-Bairstow Iteration).
Brent's Method
Background
We start by reviewing the secant method and then extend it to the inverse quadratic
interpolation method.Mueller's proposed a method using successive bisection and inverse
quadratic interpolation which was extended by Brent's and others.It uses a combination of
the bisection, regula falsi, and inverse quadratic interpolation methods.
Theorem ( Secant Method ).Assume that
and there exists
a number
, where
.If
for
will converge to
.
Algorithm ( Secant Method ).Find a root of

approximations
using the iteration
given two initial
for
.
Theorem (Inverse Quadratic Method).Assume that

and
, where
.If
, then there
exists a
for
will converge to

.
Algorithm ( Inverse Quadratic Method ).Find a root of

three initial approximations
using the iteration
for
Mathematica Subroutine (Inverse Quadratic Method).
given
The computation of
is
seen to require 12 function evaluations (because occurs 13
times).This number can be reduced to 3 function evaluations per
iteration by using the following "algebraic trick."
Algorithm ( Inverse Quadratic Method ).Find a root of
given
three initial approximations
iteration.When the code in the
above subroutine is executed the computation of
is
seen to require 13 function evaluations.(because occurs 13
times).The number of function evaluations can by using the following
scheme.
for
Mathematica Subroutine (Inverse Quadratic Method).Efficient

version that uses only 3 function evaluations per iteration.
More Background
We will now review some root bracketing methods.The regula falsi
method usually converge faster than the bisection method
bisection.However, examples can be found when the bisection method
converges faster.To speed things up, Brent included inverse quadratic
interpolation and his method combines the root bracketing methods:
bisection, regula falsi; and inverse quadratic interpolation methods.
Theorem ( Bisection Method ). Assume that
and that there
exists a number
such that
.If
have opposite
signs, and
represents the sequence of midpoints generated by the
bisection process, then
fo
r
and the sequence

That is,
Mathematica Subroutine (Bisection Method).

Theorem ( Regula Falsi Method ). Assume that

such that
.
If
and that
represents the sequence of points generated by the Regula Falsi

process, then the sequence
.
That is,
Mathematica Subroutine (Regula Falsi Method).
Brent's Method
The secant method and inverse quadratic interpolation method can be
used to find a root
of the function
.Combining these
methods and making variations which include inverse interpolation
have been presented by A. van Wijngaarden, J. A. Zonneveld and E. W.
Dijkstra (1963), J. H. Wilkinson (1967), G. Peters and J. H. Wilkinson
(1969), T. J. Dekker (1969) and were improved by R. P. Brent (1971).
Brent's method can be used to find a root
provided that
have opposite signs.At a typical step we have three points
such that
,and the pointamay coincide with the
pointc.The points
change during the algorithm, and the root
always lies in either
or
.The valuebis the best approximation to
the root andais the previous value ofb.The method uses a combination
of three methods: bisection, regula falsi, and inverse quadratic
interpolation.It is difficult to see how each of these ideas are
incorporated into the subroutine.To assist with locating the lines that
must be used in logical sequence some of the lines have been color
coded.But some lines are used in more than one method so look
carefully at the subroutine and the portions listed below.
The Brent subroutine will find the root of the function
in the
interval
to within a tolerance
where is a positive
tolerance and
.
Algorithm (Brent's Method).Find a rootpof
given initial
bracketing interval
where
must have opposite signs.At
a typical step we have three points
such that
,andamay coincide withc.The points
change during the
algorithm, and the root always lies in either
or
.The valuebis
the best approximation to the root andais the previous value ofb.The
iteration uses a combination oftechniques:
(i)The Bisection Method
,or
(ii)Regula Falsi Method
,or
(iii) Quadratic Interpolation
Mathematica Subroutine (Brent's Method).
Graeffe's Method
Background
We use the following important result from the study of the theory of equations.
Theorem ( Vieta's Formulas ).Consider a polynomial of
roots
of degreenwith
.
Let be the elementary symmetric function or symmetric polynomial
for the variables
,
...
then
.
Moreover, we have the important identities relating the coefficients of
for
Separated Roots
If the roots of
are widely separated in magnitude, then they can
be approximated using ratios of the coefficients of
.This is the
heart of the Graeffe method.
Theorem (Separated Real Roots).If

is a
polynomial with real roots that are widely separated in magnitude
then
for
Graeffe Root Squaring Method

The root-finding method was popular in the 19th and 20th centuries.It
was invented independently by Karl Heinrich Grffe (1799-1873),
Germinal Pierre Dandelin (1794-1847), andNikolai Ivanovich
Lobachevsky (1792-1856)(See the article Dandelin, Lobacevskii, or
Graeffeby Alston S. Householder,The American Mathematical Monthly,
Vol. 66, No. 6. (Jun. - Jul., 1959), pp. 464-466, Jstor. )Graeffe's
method has the shortcoming that it proceeds to calculations where the
exponents exceed the maximum allowed by floating-point arithmetic
computation of most software packages.The extended precision in
Mathematica is adequate to investigate the method.
A necessary condition for theseparated root theorem produce a good
approximation
,is that the roots must be widely separated in
magnitude, we would need the separation to be more than
for
.The heart of the Graeffe method is to
start with "mildly" separated roots and construct a related polynomial
with sufficiently widely separated roots.This leads into the topic of root
squaring.
Theorem (Root Squaring).Given the polynomial
factored form
with roots
.Then
of degree n in
is defined by
.
is a polynomial of degree n with roots
The Goal
If the roots of
are real and distinct then successive root squaring
will generate a sequence of polynomials
,where each
polynomial
has degreen.The roots of
are
,and ifvis large
enough, then the roots of

will be widely separated.The roots
of
are all positive.The roots of
can be obtained by taking a
root
evaluating
, where the appropriate sign can be determined by

.The goal is to separate roots!
Theorem ( Graeffe's Method ).Given the polynomial

n with real distinct roots
.Define the sequence
of degree
as follows:
is a polynomial of degree n with roots

for
.Furthermore, the roots of
are approximated by
for
Where the appropriate sign can be determined by evaluating
Algorithm (Graeffe's Method).To find all the roots ofthe polynomial

of degree n which has real distinct roots
.Use the Graeffe
iteration
.
Mathematica Subroutine (Graeffe's Method)).
Extending Graeffe's Method

The literature on Graeffe's method contains a myriad of rules for
treating cases other than distinct, separated real roots.The rules
involve detailed study of the behavior of the coefficients of
,
which are to be listed in rows, and the coefficients of the powers of x
in columns.Hutchinson lists 11 rules for special cases, and his list was
later refined by Cronvich.There are special cases for distinct real roots,
double roots, triple roots, one pair of imaginary roots, two pairs of
imaginary roots, a pair of imaginary roots whose modulus is equal to
the absolute value of a real root, etc.It is not our purpose to study
these cases and leave them for the reader to investigate.We will look
at two of the easier cases which give a glimpse of what might happen.
Repeated Real Roots
The standard Graeffe iteration given in the Mathematica subroutine is
robust enough to treat the case of repeated real roots.However,
knowing that a double root appears is essential information that will be
used.If is a root of orderj,then
and the magnitude of
the repeated roots are given by the following computation.After v
iterations the polynomial
is constructed
The magnitude of the multiple root

formula
of orderj,is computed with the
.
The Efficient Graeffe Subroutine
It can be observed that the functions
are never used in Graeffe's
method, only their coefficients.So it is an unnecessary step to form the
polynomials.The following streamlined version of the subroutine uses
only the coefficients.Also, this version can be used with decimal entries
for the coefficients, where the previous version will not.
Mathematica Subroutine (Graeffe's Method)).
Fixed Point Iteration and Newton's Method in 2D and

3D
Background
Iterative techniques will now be introduced that extend the fixed point
and Newton methods for finding a root of an equation.We desire to
have a method for finding a solution for the system of nonlinear
equations
(1)
.
Each equation in (1) implicitly defines a curve in the plane and we
want to find their points of intersection.Our first method will be be
fixed point iteration and the second one will be Newton's method.
Definition (Jacobian Matrix).Assume that
are
functions of the independent variables
, then their Jacobian
matrix
is
.
Similarly, if
independent variables
are functions of the

, then their Jacobian matrix
is
.
Generalized Differential
For a function of several variables, the differential is used to show how changes of the
independent variables affect the change in the dependent variables. Suppose that we have
Suppose that the values of these functions in are known at the point
and we
wish to predict their value at a nearby point
.Let
,denote differential
changes in the dependent variables and , and
denote differential changes in
the independent variables. These changes obey the relationships
,
,
.
If vector notation is used, then this can be compactly written by using
the Jacobian matrix. The function changes aredFand the changes in
the variables are denoteddX.
or
.
Convergence near a Fixed Point
In solving for solutions of a system of equations, iteration is used.We
now turn to the study of sufficient conditions which will guarantee
convergence.
Definition (Fixed Point).A fixed point for the system of two
equations
,
.
is a point
such that
.Similarly, in three dimensions a
fixed point for the system of three equations
,
,
.
is a point
such that
.
Definition (Fixed Point Iteration).For a system of two equations,

fixed point iteration is
,and
,
Similarly, for a system of three equations, fixed point iteration is
,and
,
Theorem (Fixed-Point Iteration).Assume that all the functions and
their first partial derivatives are continuous on a region that contains
the fixed point
or
, respectively.If the starting point is
chosen sufficiently close to the fixed point, then one of the following
cases apply.
Case (i)Two dimensions.If
is sufficiently close to
and if
,
,
then fixed point iteration will converge to the fixed point
Case (ii)Three dimensions.If
and if
+
+
< 1,
+
+
< 1,
+
+
< 1,
If these conditions are not met, the iteration might diverge, which is
usually the case.
ProofNonlinear SystemsNonlinear Systems
Algorithm (Fixed Point Iteration for Non-Linear Systems) In two
dimensions, solve the non-linear fixed point system
given one initial approximation
, and generating a sequence

which
converges to the solution
,i.e.
Algorithm (Fixed Point Iteration for Non-Linear Systems)In three

dimensions, solve the non-linear fixed point system

which
converges to the solution
,i.e.
Algorithm (Fixed Point Iteration for Non-Linear Systems)Solve

the non-linear fixed point system
,
the solution ,
which converges to
.
Remark. First we give a subroutine for performing fixed point iteration.
Computer ProgramsNonlinear SystemsNonlinear Systems
Mathematica Subroutine (Fixed Point Iteration in nDimensions).
Example 1.Use fixed point iteration to find a solution to the nonlinear system
Example 1. Use fixed point iteration to find a solution to the nonlinear system
Solution 1.
First, enter the coordinate functions
Mathematica.
, and vector function into
Second, graph the curves

using Mathematica.
The points of intersection are the solutions we seek.
How many points of intersection are there ?
Use fixed point iteration to find a numerical approximation to the solution near
.
A fixed point satisfies the equation

, check it out.
. Our last approximation is stored in
Do you think that iteration produced the solution ? Why ?
Accuracy is determined by the tolerance and number of iterations. How accurate was the
solution "really"?
Since our tolerance was only
, the accuracy is what we should expect.
Use fixed point iteration to attempt finding a numerical approximation to the solution
near
.
But this time attempt only 10 iterations !
A fixed point satisfies the equation
, check it out.
Did iteration find the desired solution near
. Our last approximation is stored in
? Why ?
Use Mathematica to find that solution near

.
This time use the procedure NSolve (if you use Solve you might be unhappy).
Where did all those complex number come from ? Thought we were looking at curves in
the plane ?
Try solving the first equation for x then substituting one of the solutions in for x is the
second equation.
So we can solve for y, in a single polynomial equation of degree 12 which is known to

have 12 roots, some of which are complex.
For our course, we don't want the complex solutions because we usually work with real
numbers and our iteration schemes produces more real numbers.
However, you should be convinced that the second solution we seek is
{x -> 1.5560974843220556, y -> -0.5756507696188449}
Do you think we should have started closer to the solution and that fixed point iteration
should then work ?
If so, try the following iteration.
Is iteration converging to the desired solution {1.5560974843220556,

-0.5756507696188449} ?
Our goal is to obtain this second solution.
Theoretically, if we know the fixed point and look at the partial derivatives of the
equations then the row sum should be small.
Case (i) Two dimensions. If
,
and if
,
Consider the first inequality for our problem.
The sum of the absolute values of the partial derivatives is too big, so we should not have
expected that iteration would converge.
Newton's Method for Nonlinear Systems
We now outline the derivation of Newtons method in two
dimensions.Newtons method can easily be extended to higher
dimensions.Consider the system
(1)
which can be considered a transformation from thexy-plane to the uvplane.We are interested in the behavior of this transformation near the
point
whose image is the point
.If the two functions have
continuous partial derivatives, then the differential can be used to
write a system of linear approximations that is valid near the point
:
then substitute the changes

fo
r
,respectively.Then we will have
o
r
(2)
.
Consider the system (1) with u and v set equal to zero,
(3)
Suppose we are trying to find the solution
, then we can apply (2) and write
and we start iteration at the nearby point
.
Since
,
this becomes
,
or
.
When we solve this latter equation for
we get
and the next approximation
is
Theorem (Newton-Raphson Method for 2-dimensional

Systems).To solve the non-linear system
,
which converges to the solution

,i.e.
.
Suppose that
.
has been obtained, use the following steps to obtain
Step 1.Evaluate the function
Step 2.Evaluate the Jacobian
Step 3.Solve the linear system

fo
r
Step 4.Compute the next approximation
Mathematica Subroutine (Newton's Method for Systems in nDimensions).
Broyden's Method
Background
Newton's method can be used to solve systems of nonlinear equations.
Theorem (Newton-Raphson Method for 2-dimensional
Systems).To solve the non-linear system
,given one initial
approximation ,and generating a sequence
which converges to the solution
,i.e.
.Suppose that has been obtained, use the following steps to obtain
.
.
Step 2.Evaluate the Jacobian
.
fo
r
Mathematica Subroutine (Newton's Method for Systems in nDimensions).
More Background
A drawback of Newton's method is the requirement that the Jacobian matrix be
computed, which requires the calculation of partial derivatives per iteration.One
obvious improvement is to use a finite difference approximation for the Jacobian
matrix.For two dimensions this is
where h is small.Notice that this will require
function evaluations.
Mathematica Subroutine (PseudoNewton's Method for Systems

in n-Dimensions).
Broyden's Method
Calculation of the Jacobian matrix requires partial derivative
evaluations and the approximate Jacobian matrix requires
function
evaluations.Hence, a more computationally efficient method is
desired.Broyden's method is a least change secant update method or
quasi-Newton method.Recall the one-dimensional case of Newton's
method for solving
. The secant method replaces the derivative
with the difference quotient
The extension to higher dimensions is made by using the basic

property of the Jacobian
to write the equation
.
Broyden's method starts initially with the Jacobian matrix
.Then in successive iterations uses an update the approximate
Jacobian with the matrix :
fo
r
This equation is known as the quasi-Newton condition or secant

condition.Recall that two initial two values
are necessary to start
the secant method.In practice we are given one starting value and
compute the Jacobian
and use Newton's method to get the
first approximation .The following result gives the details of Broyden's
method.
Theorem (Broyden's Method for 2-dimensional Systems).To
solve the non-linear system
,given one initial approximation
,and generating a sequence
which converges to the solution ,i.e.
.
Step 0.Compute the initial Jacobian matrix
.
Use it to compute the first approximation using Newton's method
.
For
.Suppose that
obtain
.
has been obtained, use the following steps to

.
Step 2.Update the approximate Jacobian using
and

for
Mathematica Subroutine (Broyden's Method for Systems in nDimensions).
Improving Broyden's Method

Matrix inversion requires
calculations.Thus we seek a way to avoid
this time consuming step each time an iteration is performed.A matrix
inversion formula of Sherman and Morrison can facilitate in making a
more efficient algorithm.Except for
, no matrix inverse or solution
of a linear system computation is performed in the general step.This is
efficient when n is large.
Theorem ( Sherman-Morrison Formula ).If is a nonsingular
matrix and
are vectors such that
,then
or
.
Theorem (Broyden's Method for n-dimensional Systems).To
solve the non-linear system
,given one initial approximation
,and generating a sequence
which converges to the solution ,i.e.
.Compute the initial Jacobian matrix
.
Use it to compute the first approximation using Newton's method
.
For
.Suppose that
obtain
.
has been obtained, use the following steps to
Step 2.Update the approximate Jacobian using
,and
.
Step 3.Compute
using the Sherman-Morrison formula

.
Mathematica Subroutine (Improved Broyden's Method for
Systems in n-Dimensions).
The Solution of Linear Systems AX = B
Triangular Systems and Back Substitution
Gauss-Jordan Elimination and Pivoting
Tri-Diagonal Matrices
Inverse Matrix
LU Factorization
Cholesky, Doolittle and Crout Factorizations
Jacobi and Gauss-Seidel Iteration
Successive Over Relaxation - SOR
Pivoting Methods
Iterative Refinement
Row Reduced Echelon Form
Homogeneous Linear Systems
Kirchoff's Law
Leontief Model
Linear Programming-Simplex Method
Forward Substitution and Back Substitution
Background
We will now develop the back-substitution algorithm , which is
useful for solving a linear system of equations that has an uppertriangular coefficient matrix.
Definition ( Upper-Triangular Matrix ). An
matrix
called upper-triangular provided that the elements satisfy
whenever
.
If A is an upper-triangular matrix, then
triangular system of linear equations.
is
is said to be an upper-
(1)
Theorem ( Back Substitution ). Suppose that

triangular system with the form given above in (1). If
then there exists a unique solution.
is an upperfor
The back substitution algorithm. To solve the upper-triangular

system
by the method of back-substitution. Proceed with the
method only if all the diagonal elements are nonzero. First compute
and then use the rule
for
Or, use the "generalized rule"
for
where the "understood convention" is that

is an "empty
summation" because the lower index of summation is greater than the
upper index of summation.
Remark. The loop control structure will permit us to use one
formula.
Mathematica Subroutine (Back Substitution).
Pedagogical version for "printing all the details."
Lower-triangular systems.
We will now develop the lower-substitution algorithm, which is
useful for solving a linear system of equations that has a lowertriangular coefficient matrix.
Definition ( Lower-Triangular Matrix ). An
matrix
called lower-triangular provided that the elements satisfy
whenever
.
If A is an lower-triangular matrix, then
triangular system of linear equations.
(2)
is
is said to be a lower-
Theorem (Forward Substitution). Suppose that
triangular system with the form given above in (2). If
then there exists a unique solution.
is an lowerfor
The forward substitution algorithm. To solve the lower-triangular

system
by the method of forward-substitution. Proceed with the
method only if all the diagonal elements are nonzero. First compute
and then use the rule
for
Remark. The loop control structure will permit us to use one formula
for
Mathematica Subroutine (Forward Substitution).
The Newton Interpolation Polynomial.

The following result is an alternate representation for a polynomial
which is useful in the area of interpolation.
Definition ( Newton Polynomial ). The following expression is
called a Newton polynomial of degree n.
or
If n+1 points
are given, then the following equations can
be used to solve for the n+1 coefficients
.
or
for
k=1, 2,..., n+1.
This system of equations is lower-triangular.
...
Gauss-Jordan Elimination
In this module we develop a algorithm for solving a general linear

system of equations
consisting of n equations and n unknowns
where it is assumed that the system has a unique solution. The
method is attributed Johann Carl Friedrich Gauss (1777-1855) and
Wilhelm Jordan (1842 to 1899). The following theorem states the
sufficient conditions for the existence and uniqueness of solutions of a
linear system
.
Theorem ( Unique Solutions ) Assume that
following statements are equivalent.
(i) Given any
solution.
(ii) The matrix
matrix
, the linear system
is nonsingular (i.e.,
(iii) The system of equations

(iv) The determinant of
is an
matrix. The
has a unique
exists).
has the unique solution
is nonzero, i.e.
It is convenient to store all the coefficients of the linear system

in one array of dimension
. The coefficients of are stored
in column
of the array (i.e.
). Row contains all the
coefficients necessary to represent the
equation in the linear
system. The augmented matrix is denoted
and the linear
system is represented as follows:
The system
, with augmented matrix , can be solved by
performing row operations on . The variables are placeholders for
the coefficients and cam be omitted until the end of the computation.
Theorem ( Elementary Row Operations ). The following operations
applied to the augmented matrix yield an equivalent linear system.
(i) Interchanges:
(ii) Scaling:
The order of two rows can be interchanged.
Multiplying a row by a nonzero constant.
(iii) Replacement: Row r can be replaced by the sum of that tow

and a nonzero multiple of any other row;
that is:
.
It is common practice to implement (iii) by replacing a row with the
difference of that row and a multiple of another row.
Definition ( Pivot Element ). The number

in the coefficient
matrix that is used to eliminate
where
, is called
the
pivot element, and the
row is called the pivot row.
Theorem ( Gaussian Elimination with Back Substitution ).

Assume that is an
nonsingular matrix. There exists a unique
system
that is equivalent to the given system
, where is an
upper-triangular matrix with
for
. After
are
constructed, back substitution can be used to solve
for .
Algorithm I. (Limited Gauss-Jordan Elimination). Construct the
solution to the linear system
by using Gauss-Jordan elimination
under the assumption that row interchanges are not needed. The
following subroutine uses row operations to eliminate
in
column p for
.
Mathematica Subroutine (Limited Gauss-Jordan Elimination).
Provide for row interchanges in the Gauss-Jordan subroutine.

Add the following loop that will interchange rows and perform partial
pivoting.
To make these changes, copy your subroutine GaussJordan and place

a copy below. Then you can copy the above lines by selecting them
and then use the "Edit" and "Copy" menus. The improved GaussJordan subroutine should look like this (blue is for placement
information only). Or just use the active Mathematica code given
below.
Algorithm II. (Complete Gauss-Jordan Elimination). Construct

the solution to the linear system
by using Gauss-Jordan
elimination. Provision is made for row interchanges if they are
needed.
Mathematica Subroutine (Complete Gauss-Jordan Elimination).
Use the subroutine "GaussJordan" to find the inverse of a

matrix.
Theorem ( Inverse Matrix ) Assume that is an
nonsingular
matrix. Form the augmented matrix
of dimension
. Use Gauss-Jordan elimination to reduce the matrix so that the
identity
is in the first columns. Then the inverse
is located in
columns
.
Algorithm III. (Concise Gauss-Jordan

Elimination).nbsp; Construct the solution to the linear system
by using Gauss-Jordan elimination. The print statements are for
pedagogical purposes and are not needed.
Mathematica Subroutine (Concise Gauss-Jordan Elimination).
Remark. The Gauss-Jordan elimination method is the "heuristic"

scheme found in most linear algebra textbooks. The line of code
divides each entry in the pivot row by its leading coefficient

. Is
this step necessary? A more computationally efficient algorithm will be
studied which uses upper-triangularization followed by back
substitution. The partial pivoting strategy will also be employed, which
reduces propagated error and instability.
Application to Polynomial Interpolation

Consider a polynomial of degree n=5 that passes through the six
points
;
.
For each point
is used to an equation
, which in turn
are used to write a system of six equations in six unknowns
=
=
=
=
=
=
The above system can be written in matrix form MC = B
Solve this linear system for the coefficients

the interpolating polynomial
.
and then construct
Tri-Diagonal Linear Systems
Background
In applications, it is often the case that systems of equations arise
where the coefficient matrix has a special structure. Sparse matrices
which contain a majority of zeros occur are often encountered. It is
usually more efficient to solve these systems using a taylor-made
algorithm which takes advantage of the special structure. Important
examples are band matrices, and the most common cases are the
tridiagonal matrices.
Definition ( Tridiagonal Matrix ). An nn matrix A is called a

tridiagonal matrix if
whenever
. A tridiagonal
matrix has the for
(1)
Theorem Suppose that the tri-diagonal linear system

has
and
for
. If
, and
, an
d
, then there exists
a unique solution.
The enumeration scheme for the elements
in (1) do not take
advantage of the overwhelming number of zeros. If the two subscripts
were retained the computer would reserve storage for
elements. Since practical applications involve large values of
,
it is important to devise a more efficient way to label only those
elements that will be used and conceptualize the existence of all those
off diagonal zeros. The idea that is often used is to call elements of
the main diagonal
, subdiagonal elements
are directly
below the main diagonal, and the superdiagonal elements
are
directly above the main diagonal. Taking advantage of the single
subscripts on
we can reduce the storage requirement to
merely
elements. This bizarre way of looking at a tridiagonal
matrix is
.
Observe that A does not have the usual alphabetical connection with
its new elements named
, and that the length of the
subdiagonal and superdiagonal is n-1.
Definition (tri-diagonal system). If A is tridiagonal, then a

tridiagonal system is
(2)
A tri-diagonal linear system
can be written in the form
(3)
=
=
=
=
=
Goals
To understand the "Gaussian elimination process."
To understand solutions to"large matrices" that are contain mostly zero
elements, called "sparse matrices."
To be aware of the uses for advanced topics in numerical analysis:
(i) Used in the construction of cubic splines,
(ii) Used in the Finite difference solution of boundary value
problems,
(iii) Used in the solution of parabolic P.D.E.'s.
Algorithm (tridiagonal linear system). Solve a tri-diagonal

system
given in (2-3).
Below diagonal elements:
Main diagonal elements:

Above diagonal elements:
Column vector elements:
Mathematica Subroutine
The Matrix Inverse
Background
Theorem ( Inverse Matrix ) Assume that is an
nonsingular
matrix. Form the augmented matrix
of dimension
. Use Gauss-Jordan elimination to reduce the matrix so that the
identity
is in the first columns. Then the inverse
is located in
columns
. The augmented matrix
looks like:
We can use the previously developed Gauss-Jordan subroutine to find

the inverse of a matrix.
Algorithm (Complete Gauss-Jordan Elimination). Construct the

solution to the linear system
by using Gauss-Jordan
elimination. Provision is made for row interchanges if they are
needed.
Definition ( Hilbert Matrix ). The elements of the Hilbert matrix

of order n are
for
and
The Inverse Hilbert Matrix

The formula for the elements of the inverse Hilbert matrix
order n is known to be
which can be expressed using binomial coefficients

.
of
When exact computations are needed these formulas should be used
instead of using a subroutine or built in procedure for computing the
inverse of
.
Application to Continuous Least Squares Approximation

The continuous least squares approximation to a function
on
the interval [0,1] for the set of functions
can solved by
using the normal equations
(1)
for
Where the inner product is

system (1) for the coefficients
function
.
. Solve the linear
and construct the approximation
Definition ( Gram Matrix ). The Gram matrix G is a matrix of inner

products where the elements are
.
The case when the set of functions is
will produce the Hilbert matrix. Since we require the computation to
be as exact as possible and an exact formula is known for the inverse
of the Hilbert matrix, this is an example where an inverse matrix
comes in handy.
PA = LU Factorization with Pivoting
Background
Definition ( LU-Factorization ). The nonsingular matrix A has an
LU-factorization if it can be expressed as the product of a lowertriangular matrix L and an upper triangular matrix U:
.
When this is possible we say that A has an LU-decomposition. It turns
out that this factorization (when it exists) is not unique. If L has 1's
on it's diagonal, then it is called a Doolittle factorization. If U has 1's
on its diagonal, then it is called a Crout factorization. When
, it
is called a Cholesky decomposition. In this module we will develop an
algorithm that produces a Doolittle factorization.
Theorem ( LU Factorization with NO pivoting). If row
interchanges arenot needed to solve the linear system AX = B, then A
has the LU factorization (illustrated with 44 matrices).
Remark 1. This is not a linear system.

Remark 2. The easy solution uses row vectors and is a modification
of limited Gauss Jordan elimination.
Remark 3. A sufficient condition for the factorization to exist is that
all principal minors of A are nonsingular.
Mathematica Subroutine (Limited Gauss-Jordan Elimination).
Mathematica Subroutine (LandU).
Theorem (A = LU; Factorization with NO Pivoting). Assume that

A has a Doolittle, Crout or Cholesky factorization. The solution X to
the linear system
, is found in three steps:
1. Construct the matrices
, if possible.
2. Solve
for
using forward substitution.
3. Solve
for
using back substitution.
The above theorem assumes that there are no row

interchanges. We have seen in Example 3 an example of a
nonsingular matrix A could not be directly factored as A = LU. If row
interchanges are permitted then a factorization of a "permuted matrix"
will be obtained. A permutation of the first n positive
integers
. is an arrangement
of these integers
in a definite order. For example,
is a permutation of the five
integers
. The standard base vectors
for
are used in the next definition.
Definition ( Permutation Matrix). An nn permutation matrix P is

a matrix with precisely one entry whose value is "1" in each column
and row, and all of whose other entries are "0." The rows of P are a
permutation of the rows of the identity matrix and P can be written
as
.
The elements of
have the form
Theorem. Suppose that

is a permutation
matrix. The product PA is a new matrix whose rows consists of the
rows of A rearranged in the new order
.
For example, the permutation matrix

rows 1 and 2 and also interchange rows 3 and 4.
will interchange
Theorem. If A is a nonsingular matrix, then there exists a

permutation matrix P so that PA has an LU-factorization
PA = LU.
Theorem (PA = LU; Factorization with Pivoting). Given that A is

nonsingular. The solution X to the linear system
, is found in
four steps:
1.
2.
3.
4.
Construct the matrices

.
Compute the column vector
.
Solve
for
Solve
for
Mathematica Subroutine (LUfactor).

Remark. The matrix A is a global variable and elements are
changed when the LUfactor subroutine is executed. Hence, it is
important to save a copy of A in the variable A0.
Use one of the following two versions of the LUfactor subroutine and
the SolveLU subroutine. The first version of LUfactor uses parallel
programming and "row operations" instead of loops.
This is the second version of LUfactor and it uses more loops and
traditional programming.
Mathematica Subroutine (LUfactor).
Use the subroutine SolveLU which is similar to the forward

substitution and back substitution subroutines.
Mathematica Subroutine (SolveLU).
Remark. Everything has been carefully set up so that L, U,
and P can all be studied.
Application to Polynomial Curve Fitting

Theorem ( Least-Squares Polynomial Curve Fitting ). Given
the
data points
, the least squares
polynomial of degree m of the form
that fits the n data points is obtained by solving the following linear
system
for the m+1 coefficients

referred to as the "normal equations".
. These equations are
Cholesky, Doolittle and Crout Factorization
Background
Definition ( LU-Factorization ). The nonsingular matrix A has an
LU-factorization if it can be expressed as the product of a lowertriangular matrix L and an upper triangular matrix U:
.
When this is possible we say that A has an LU-decomposition. It turns
out that this factorization (when it exists) is not unique. If L has 1's
on it's diagonal, then it is called a Doolittle factorization. If U has 1's
on its diagonal, then it is called a Crout factorization. When
(or
), it is called a Cholesky decomposition.

A has a Doolittle, Crout or Cholesky factorization. The solution X to
the linear system
, if possible.
2. Solve
for
3. Solve
for
Theorem (A = LU; Doolittle Factorization). Assume that A has a

Doolittle factorization A = LU.
The solution X to the linear system

, if possible.
2. Solve
for
3. Solve
for
For curiosity, the reader might be interested in other methods of

computing L and U.
Theorem (A = LU; Crout Factorization). Assume that A has a
Crout factorization A = LU.

, if possible.
2. Solve
for
3. Solve
for
Mathematica Subroutine (Doolittle).
Mathematica Subroutine (Crout).
Mathematica Subroutine (Forward Elimination).
Mathematica Subroutine (Back Substitution).

Theorem ( Cholesky Factorization ). If A is real, symmetric and

positive definite matrix, then it has a Cholesky factorization
where U an upper triangular matrix.

Remark. Observe that
is a lower triangular matrix, so that A =
LU. Hence we could also write Cholesky factorization
where L a lower triangular matrix.
Theorem (A = LU; Cholesky Factorization). Assume that A has a

Cholesky factorization
, where
.
Or if you prefer to write the Cholesky factorization as

.
, where

, if possible.
2. Solve
for
3. Solve
for
The following Cholesky subroutine can be used when the

matrix A is real, symmetric and positive definite.
Observe that the loop starting with For[j=k,j<=n,j++, is not
necessary and that U is computed by forming the transpose of L.
Mathematica Subroutine (Cholesky factorization).
Application to Polynomial Curve Fitting

the
data points
, the least squares
polynomial of degree m of the form
that fits the n data points is obtained by solving the following linear
system

Application to Continuous Least Squares Approximation

The continuous least squares approximation to a function
on
the interval [0,1] for the set of functions
can solved by
using the normal equations
(1)
for
Where the inner product is

system (1) for the coefficients
function
.
. Solve the linear
and construct the approximation
Definition ( Gram Matrix ). The Gram matrix G is a matrix of inner

products where the elements are
.
The case when the set of functions is
will produce the Hilbert matrix. Since we require the computation to
be as exact as possible and an exact formula is known for the inverse
of the Hilbert matrix, this is an example where an inverse matrix
comes in handy.
Jacobi and Gauss-Seidel Iteration
Background
Iterative schemes require time to achieve sufficient accuracy and
are reserved for large systems of equations where there are a majority
of zero elements in the matrix. Often times the algorithms are taylormade to take advantage of the special structure such as band
matrices. Practical uses include applications in circuit analysis,
boundary value problems and partial differential equations.
Iteration is a popular technique finding roots of
equations. Generalization of fixed point iteration can be applied to
systems of linear equations to produce accurate results. The method
Jacobi iteration is attributed to Carl Jacobi (1804-1851) and GaussSeidel iteration is attributed to Johann Carl Friedrich Gauss (17771855) and Philipp Ludwig von Seidel (1821-1896).
Consider that the nn square matrix A is split into three parts, the
main diagonal D, below diagonal L and above diagonal U. We have A
= D - L - U.
Definition (Diagonally Dominant). The matrix

diagonally dominant if
for
is strictly
Theorem ( Jacobi Iteration ). The solution to the linear

system
can be obtained starting with
, and using iteration
scheme
where
and
If
is carefully chosen a sequence
is generated which converges
to the solution P, i.e.
.
A sufficient condition for the method to be applicable is that A is
strictly diagonally dominant or diagonally dominant and irreducible.
Theorem ( Gauss-Seidel Iteration ). The solution to the linear

system
scheme
where
and
If
.
Mathematica Subroutine (Jacobi Iteration).
Mathematica Subroutine (Gauss-Seidel Iteration).
Warning.
Iteration does not always converge. A sufficient condition for
iteration to Jacobi iteration to converge is that A is strictly
diagonally dominant. The following subroutine will check to see if a
matrix is strictly diagonally dominant. It should be used before any
call to Jacobi iteration or Gauss-Seidel iteration is made. There exists
a counter-example for which Jacobi iteration converges and GaussSeidel iteration does not converge. The "true" sufficient condition for
Jacobi iteration to converge is that the "spectral radius" of
is less than 1, where
is the diagonal of . That is, the magnitude
of the largest eigenvalue of M must be less than 1. This condition
seems harsh because numerical computation of eigenvalues is an
advanced topic compared to solution of a linear system.
More efficient subroutines

A tolerance can be supplied to either the Jacobi or Gauss-Seidel
method which will permit it to exit the loop if convergence has been
achieved.
Subroutines using matrix commands

In the Jacobi subroutine we can use fix point iteration as suggested
by the theory.
Successive Over Relaxation - SOR Method
Background
Suppose that iteration is used to solve the linear system
,
and that
is an approximate solution. We call
the
residual vector, and if
is a good approximation then
. A
method based on reducing the norm of the residual will produce a
sequence
that converges faster. The successive over relaxation SOR method introduces a parameter
which speeds up
convergence. The SOR method can be used in the numerical solution
of certain partial differential equations.
Consider that the nn square matrix A is split into three parts, the
main diagonal D, below diagonal L and above diagonal U. We have A
= D - L - U.
Definition (Diagonally Dominant). The matrix

diagonally dominant if
for
is strictly
Theorem ( Jacobi Iteration ). The solution to the linear

system
scheme
where
and
If
.
Theorem ( Gauss-Seidel Iteration ). The solution to the linear

system
scheme
where
and
If
.
Theorem ( SOR Iteration ). Given a value of the parameter

(chosen in the interval
), the solution to the linear
system
scheme
where
an
d
If
.
Remark. A theorem of Kahan states that the SOR method will
converge only if
is chosen in the interval
.
Remark. When we choose
Gauss-Seidel method.
the SOR method reduces to the
Mathematica Subroutine (Successive Over Relaxation).
Pivoting Methods
Background
In the
Gauss-Jordan module we saw an algorithm for solving a general linear
system of equations
consisting of n equations and n unknowns
where it is assumed that the system has a unique solution. The
method is attributed
Johann Carl Friedrich Gauss (1777-1855) and
Wilhelm Jordan (1842 to 1899). The following theorem states the
sufficient conditions for the existence and uniqueness of solutions of a
linear system
.
Theorem ( Unique Solutions ) Assume that

following statements are equivalent.
(i) Given any
solution.
(ii) The matrix

nonsingular (i.e.,
matrix
is an
, the linear system
matrix. The
has a unique
is
exists).
(iii) The system of equations
(iv) The determinant of
has the unique solution
is nonzero, i.e.
It is convenient to store all the coefficients of the linear system

in one array of dimension
. The coefficients of are stored
in column
of the array (i.e.
). Row contains all the
coefficients necessary to represent the
equation in the linear
system. The augmented matrix is denoted
and the linear
system is represented as follows:
The system
, with augmented matrix , can be solved by
performing row operations on . The variables are placeholders for
the coefficients and cam be omitted until the end of the computation.
Theorem (Elementary Row Operations ). The following operations

applied to the augmented matrix yield an equivalent linear system.
(i) Interchanges:
(ii)Scaling:
The order of two rows can be interchanged.

Multiplying a row by a nonzero constant.
(iii) Replacement: Row r can be replaced by the sum of that tow

and a nonzero multiple of any other row;
that is:
.
It is common practice to implement
(iii) by replacing a row with the difference of that row and a multiple of
another row.
Definition ( Pivot Element ). The number

in the coefficient
matrix that is used to eliminate
where
, is called
the
pivot element, and the
row is called the pivot row.
Theorem ( Gaussian Elimination with Back Substitution ).
Assume that is an
nonsingular matrix. There exists a unique
system
that is equivalent to the given system
, where is an
upper-triangular matrix with
for
. After
constructed, back substitution can be used to solve
for
are
. ng
Pivoting Strategies
There are numerous pivoting strategies discussed in the
literature. We mention only a few to give an indication of the
possibilities.
(i) No Pivoting. No pivoting means no row interchanges. It can be
done only if Gaussian elimination never run into zeros on the
diagonal. Since division by zero is a fatal error we usually avoid this
pivoting strategy.
Pivoting to Avoid
If
, then row p cannot be used to eliminate the elements in
column p below the main diagonal. It is necessary to find row k,
where
and k > p, and then interchange row p and row k so that
a nonzero pivot element is obtained. This process is called pivoting,
and the criterion for deciding which row to choose is called a pivoting
strategy. The first idea that comes to mind is the following one.
(ii) Trivial Pivoting. The trivial pivoting strategy is as
follows. If
, do not switch rows. If
, locate the first row
below p in which
and then switch rows k and p. This will result
in a new element
, which is a nonzero pivot element.
Pivoting to Reduce Error
Because the computer uses fixed-precision arithmetic, it is possible
that a small error will be introduced each time that an arithmetic
operation is performed. The following example illustrates how use of
the trivial pivoting strategy in Gaussian elimination can lead to
significant error in the solution of a linear system of equations.
(iii) Partial Pivoting. The partial pivoting strategy is as
follows. If
, locate row u below
p in which
and
and then switch rows u and
p. This will result in a new element
, which is a nonzero pivot
element.
Remark. Only row permutations are permitted. The strategy is to
switch the largest entry in the pivot column to the diagonal.
(iv) Scaled Partial Pivoting. At the start of the procedure we
compute scale factors for each row of the matrix as follows:
for
.
The scale factors are interchanged with their corresponding row in the
elimination steps. The scaled partial pivoting strategy is as
follows. If
p in which
and
and then switch rows u
and p. This will result in a new element
, which is a nonzero
pivot element.
Remark. Only row permutations are permitted. The strategy is to
switch the largest scaled entry in the pivot column to the diagonal.
(v) Total Pivoting. The total pivoting strategy is as
follows. If
p and column v to the right of p in which
and
and then: first switch rows u and p and second switch column v
and p. This will result in a new element
, which is a nonzero
pivot element. This is also called "complete pivoting" or "maximal
pivoting."
Remark. Both row and column permutations are permitted. The
strategy is to switch the largest entry in the part of the matrix that we
have not yet processed to the diagonal.
Iterative Refinement - Residual Correction
Background
In the
LU-Factorization module we developed the "LU-solve" method which
will now be used as the computational engine in the iterative
refinement - residual correction method.
Definition ( LU-Factorization ). The matrix A has an LUfactorization if it can be expressed as the product of a lower-triangular
matrix L and an upper triangular matrix U
.
A has a LU-factorization. The solution X to the linear system
,
is found in three steps:
, if possible.
2. Solve
for
3. Solve
for
The above theorem assumes that there are no row interchanges. If

row interchanges are permitted then a factorization of a "permuted
matrix" will be obtained. A permutation of the first n positive
integers
. is an arrangement
of these integers
in a definite order. The standard base vectors
for
are used in the next definition.
Definition ( Permutation Matrix ). An nn permutation matrix P is

a matrix with precisely one entry whose value is "1" in each column
and row, and all of whose other entries are "0." The rows of P are a
permutation of the rows of the identity matrix and P can be written
as
. The elements of
have the form
Theorem. Suppose that

is a permutation
matrix. The product PA is a new matrix whose rows consists of the
rows of A rearranged in the new order
.
Exploration
Theorem. If A is a nonsingular matrix, then there exists a
permutation matrix P so that PA has an LU-factorization
P A = L U.
Theorem (PA = LU; Factorization with Pivoting). Given that A is
nonsingular. The solution X to the linear system
, is found in
four steps:
1.
2.
3.
4.
Construct the matrices

.
Compute the column vector
.
Solve
for
Solve
for
The "LU-solve" Method

The following pair of subroutines are the heart of the computational
engine for the computational engine in the iterative refinement residual correction method. When we refer to a "LU-solve step" it
means use LUfactor followed by SolveLU. The next example will
review these concepts, and more explanation is in the
Mathematica Subroutine (LUfactor). Matrix A is a global variable

and elements are changed when the LUfactor is executed. We save a
copy of A in A0.
Mathematica Subroutine (SolveLU). The subroutine SolveLU

which is similar to the
back substitution subroutine.
Iterative Refinement - Residual Correction Method

This is a method for improving the accuracy of a solution produced
using LU-factorization. To start the process, the factorization
is computed only once, and we have
(1)
Use the LU solver to construct

to
, i.e.
which is an approximate solution
Form the error in the computation
(2)
, this is called the residual
Now subtract
, obtaining
substitution
from
or
. Use the
and this equation becomes
.
Use the LU solver to compute

(3)
as follows
.
Then the improvement
is made
(4)
The process can be iterated to obtain a sequence
to
converging
For
Algorithm (Iterative Refinement). Compute the

factorization
is computed once. The notation
means that is the computed solution for the equation
Set
For
Mathematica Subroutine (Iterative Refinement). The following

subroutine will perform the
step of iterative refinement. Input
and output
. The local variables in the subroutine are labeled as
the first step. The subroutine call should be
, it is
assumed that
is already a global variable and will create and
return
. Notice that the computational part of the subroutine
consists of only three lines:
The remainder of the subroutine consists of print statements.
Remark 1. Historically, iterative improvement was devised to use a

limited amount of "
double precision" arithmetic to try to refine the computed solution
of a linear system
. We see an inherent difficulty in this
process: if we cannot solve
exactly, we cannot expect to
solve
exactly! The signal feature of iterative improvement is
to calculate the residuals
in double precision and
everything else in single precision.
FORTRAN is well suited for defining the type of a variable to be either
single precision or
double precision. A resource for this technique is found in, Section 2.5
Iterative Improvement of a Solution to Linear Equations pp. 47-50, in
the book by William H. Press, Saul A. Teukolsky, William T. Vetterling,
Brian P. Flannery: Numerical Recipes in Fortran 77, Cambridge
University Press, 1992, Cambridge, UK.
Remark 2. The examples we present are for pedagogical purposes
and because this module does not use FORTRAN, we cannot illustrate
the full purpose of iterative improvement. Also, because the
convergence will be rapid, we only perform two iterations and to not
need a subroutine for the computations.
Row Reduced Echelon Form
Background
An important technique for solving a system of linear
equations
is to form the augmented matrix
reduce
to reduced row echelon form.
and
Definition (Reduced Row Echelon Form). A matrix is said to be in

row-reduced echelon form provided that
(i) In each row that does not consist of all zero elements, the first
non-zero element in this row is a 1. (called. a "leading 1).
(ii) In each column that contains a leading 1 of some row, all other
elements of this column are zeros.
(iii) In any two successive rows with non-zero elements, the leading
1 of the the lower row occurs farther to the right than the leading 1 of
the higher row.
(iv) If there are any rows contains only zero elements then they are
grouped together at the bottom.
Theorem (Reduced Row Echelon Form).

form of a matrix is unique.
The reduced row echelon
Definition (Rank). The number of nonzero rows in the reduced row

echelon form of a matrix
is called the rank of
and is denoted
by
.
Theorem. Consider the m n linear system

is the augmented matrix.
, where
(i)
If
(ii) If
of solution.
(iii) If
solution.
then the system has a unique solution.

then the system has an infinite number
then the system is inconsistent and has no
Free Variables
When the linear system is underdetermined, we needed to
introduce free variables in the proper location. The following
subroutine will rearrange the equations and introduce free variables in
the location they are needed. Then all that is needed to do is find the
row reduced echelon form a second time. This is done at the end of
the next example.
Mathematica Subroutine (Under Determined Equations).
Homogeneous Linear Systems
Background
We have seen how Gaussian elimination can be used to obtain the
reduced row echelon form of a matrix and the solution of a linear
system
.
Now we consider the special case of solving a homogeneous linear
system where the column vector is
.
(1)
which can be written in matrix form

.
At first glance it seems that the solution to (1) should be
, and
certainly this is the case. Since it was easy to come up with,
, it is called the "trivial solution," and if
,
then it is the unique solution. What makes the solution to (1)
interesting is the case when
and if this happens there is an
infinite number of solutions.
An important technique for solving a homogeneous system of linear
equations
is to form the augmented matrix
and
reduce
to reduced row echelon form.
Definition (Reduced Row Echelon Form). A matrix is said to be in
row-reduced echelon form provided that
(i) In each row that does not consist of all zero elements, the first
non-zero element in this row is a 1. (called. a "leading 1).
(ii) In each column that contains a leading 1 of some row, all other
elements of this column are zeros.
(iii) In any two successive rows with non-zero elements, the leading
1 of the the lower row occurs farther to the right than the leading 1 of
the higher row.
(iv) If there are any rows contains only zero elements then they are
grouped together at the bottom.
Definition (Rank). The number of nonzero rows in the reduced row

echelon form of a matrix
is called the rank of
and is denoted
by
.
Lemma.
, where
Theorem.
(i)
If
(ii) If
solution.
Consider the m n homogeneous linear system

is the augmented matrix then
Consider the n n homogeneous linear system
then the system has the unique solution
then the system has an infinite number of
When the matrix , is an n n, square matrix, then the

determinant can be used to express these ideas.
Theorem.
(i)
If
Consider the n n homogeneous linear system

then the system has the unique solution
.
.
(ii) If
solution.
then the system has an infinite number of
Free Variables
When the linear system is underdetermined, we needed to
introduce free variables in the proper location. The following
subroutine will rearrange the equations and introduce free variables in
the location they are needed. Then all that is needed to do is find the
row reduced echelon form a second time. This is done at the end of
the next example.
Mathematica Subroutine (Under Determined Equations).
Application to Chemistry
Propane is used to reduce automobile emissions. Consider the chemical equation
which relates how propane molecules (
) combine with oxygen atoms ( ) to form
carbon dioxide (
) and water (
):
When a chemist wants to "balance this equation," whole numbers

must
be found so that the number of atoms of carbon (C), hydrogen (H) and oxygen (O) on the
left match their respective number on the right. To balance the equation, write three
equations which keep track of the number of carbon, hydrogen and oxygen atoms,
respectively.
Kirchoff's Law
Background
Solution of linear systems can be applied to resistor network
circuits. Kirchoff's voltage law says that the sum of the voltage drops
around any closed loop in the network must equal zero. A closed loop
has the obvious definition: starting at a node, trace a path through the
circuit that returns you to the original starting node.
Network #1
Consider the network consisting of six resistors and two battery,
shown in the figure below.
There are two closed loops. When Kirchoff's voltage law is applied, we
obtain the following linear system of equations.
Network #2
Consider the network consisting of nine resistors and one battery,
There are three loops. When Kirchoff's voltage law is applied, we

Network #3
Consider the network consisting of six resistors and two batteries,
There are three loops. When Kirchoff's voltage law is applied, we

The Leontief Economic Model
Introduction
The mathematics model for the economy of a country or a region is
based on the various sectors of this economy. The Leontief model is
addresses this problem. Assume that each industry in the economy
has two types of demands: an external demand (from outside the
system) and an internal demand (demand placed on one industry by
another in the same system), the Leontief model is a system of linear
equations. The Leontief model was invented in the 30's by Wassily
Leontief who developed an economic model of the United States
economy by dividing it into 500 economic sectors. Wassily Leontief
received the economics Nobel Prize on October 18, 1973.
The Closed Leontief Model
Assume that an economy consists of n interdependent industries
(or sectors)
. Each industry will consume some of the
goods produced by the other industries, including itself (for example, a
power-generating plant uses some of its own power for
production). An economy is called
closed if it satisfies its own needs; that is, no goods leave or enter the
system. We make the following conventions:
(i)
is the production level of industry
(ii)
is the number of units produced by industry
necessary to produce one unit by industry ,
that is
(iii)
is the number of units produced by industry
consumed by industry
,
(iv)
produced by industry
and
is the total number of units

.
Since the economy is closed, the total production for industry

it total consumption and we have the equations
equals
for
If the economy is balanced, the total production of each industry must

be equal to its total consumption. This results in the linear system:

.
The matrix
is called the input-output matrix, and
production vector.
is the
The Open Leontief Model

The closed Leontief model describes the case when no goods leave
or enter the economy. But often times, an economy has to satisfy an
outside demand. In this case, let
be the demand from the
outside industry. Let
and
be as in the closed model, then
for
This results in the linear system:

,
where
and
are the same as in the closed model and
demand vector.
is the
One way to solve this linear system is
.
Of course, the matrix
must be invertible, which might not be
always the case. If, in addition,
has nonnegative entries, then
the components of the vector
are nonnegative and therefore they
are acceptable as solutions for this model. We say in this case that the
matrix
is productive.
Linear Programming - The Simplex Method
Background for Linear Programming
Linear programming is an area of linear algebra in which the goal is to

maximize or minimize a linear function
of
variables
on a region
whose boundary is defined by linear
inequalities and equations. In this context, when we speak of a "linear
function of variables" we mean that
has the form
.
The region
is a convex polytope. If all the vertices of
are
known, then the maximum of
will occur at one of these
vertices.
The solution can be constructed using the simplex method and is
attributed to George Dantzig (1914 - ) who was born in Portland,
Oregon. The simplex method starts at the origin and follows a path
along the edges of the polytope to the vertex where the maximum
occurs. The history of the development of the simplex method has
been summarized in the article:
An Interview with George B. Dantzig: The Father of Linear
Programming by Donald J. Albers; Constance Reid; in The College
Mathematics Journal, Vol. 17, No. 4. (Sep., 1986), pp. 292-314,
Jstor.
Definition (Convex Polytope). In two dimensions a convex

polytope is a region that is the intersection of a finite set of half-planes
(the general idea of a convex polygon). In three dimensions a convex
polytope is solid region that is the intersection of a finite set of halfspaces (the generalized convex polyhedron). The generalization in n
dimensions is called a polytope.
Standard Form of the Linear Programming Problem
The standard form of the linear programming problem is to
maximize
of
variables
.
(1)
Maximize
where
(2)
Subject to the m constraints
where
(3)
for
for
With the primary constraints
for
The coefficients
and
can be any real number. It is often
the case that
, but the cases
or
can occur.
Setting up the Extended Simplex Tableau

For computational purposes, we construct a tableau. The first rows
consist of the coefficients matrix
, the identity matrix
and
the column vector
. In the
-row of the tableau the first
elements are the coefficients
, which are called the coefficients
of the augmented objective equation. The remainder of the bottom
row is filled in with zeros. An extra column on the right will be used in
the decision process in solving for the variables.
Column
The non-negative variable

is called a slack variable and is the
amount that the linear combination
falls short
of the bound
. It's purpose is to change an inequality to an
equation, i.e. we have
(4)
for rows
in the tableau.
The goal of the simplex method is to exchange some of the columns

of 1's and 0's of the slack variables into columns of 1's and 0's of the
decision variables.
A explanation of this tableau is given in the link below.
Simplex Method Details
Algorithm (Simplex Method). To find the minimum of

the convex polytope .
(i) Use non-negative slack variables
system of equations and the initial tableau.
and form a
(ii) Determine the exchange variable, the pivot row and pivotal
element.
The exchange variable is chosen in the pivot column
where
is the smallest negative coefficient.
over
The pivot row

is chosen where the minimum ratio
for all rows with
.
The pivot element is
.
occurs
(iii) Perform row operations to zero out elements in the pivotal column
above and below the pivot row .
(iv) Repeat steps (ii) and (iii) until there are no negative
coefficients
in the bottom row.
Mathematica Subroutines (Simplex Method). To find the

minimum of
over the convex polytope .
Activate the following four cells.
Warning. The above subroutines are for pedagogical purposes to

illustrate the simplex method. They will not work for certain cases
when
, and in cases where more than one coefficient
is set
equal to zero in one step of the iteration. For complicated problems
one of the Mathematica subroutines: ConstrainedMin,
ConstrainedMax, or LinearProgramming should be used.
Interpolation and Polynomial Approximation
Maclaurin and Taylor Series
Lagrange Polynomial Interpolation and Approximation
Newton Interpolation Polynomial
Hermite Polynomial Interpolation
Cubic Splines
B-Splines
Bzier Curves
Chebyshev Approximation Polynomial
Pade Approximation
Rational Approximation
Aitken's and Neville's Interpolation
Legendre Polynomials
The Tangent Parabola
Catenary
Maclaurin and Taylor Polynomials
Background.
When a Taylor series is truncated to a finite number of terms the
result is a Taylor polynomial. A Taylor series expanded about
, is
called a Maclarin series. These Taylor (and Maclaurin) polynomials are
used to numerically approximate functions. We attribute much of the
founding theory to Brook Taylor (1685-1731), Colin Maclaurin (16981746) and Joseph-Louis Lagrange (1736-1813).
Theorem (Taylor Polynomial Approximation ). Assume

that
, then
,
where
we write
is a polynomial that can be used to approximate
, and
.
The remainder term
has the form

,
for some value

that lies between
. The formula
referred to as the Lagrange form of the remainder.
Corollary 1. Assume that

polynomial of degree
for
for
, and that the Taylor

is
, then
.
is
Corollary 2. Assume that

for
, and that the Taylor

is
, then
,
where
Lagrange Polynomials
Background.
We have seen how to expand a function
in a Maclaurin polynomial about
involving the powers and a Taylor polynomial about
involving the powers
. The Lagrange polynomial of degree passes through the
points
for
and were investigated by the mathematician Joseph-Louis
Lagrange (1736-1813).
Theorem ( Lagrange Polynomial ). Assume that

and
for
are distinct values. Then
,
where
and we write
.
The Lagrange polynomial goes through the
for
The remainder term
points
, i.e.
.
has the form
,
for some value
that lies in the interval
The cubic curve in the figure below illustrates a Lagrange polynomial of degree n = 3,
which passes through the four points
for
.
Theorem. (Error Bounds for Lagrange Interpolation, Equally

Spaced Nodes) Assume that
defined on
, which contains the equally
spaced nodes
. Additionally, assume that
and the derivatives
of
up to the order
are continuous and bounded on the special
subintervals
,
,
,
, and
, respectively; that is,
,
for
. The error terms corresponding to these three cases have the following
useful bounds on their magnitude
(i).
is valid for
(ii).
is valid for
(iii).
is valid for
(iv).
is valid for
(v).
is valid for
,
.
Algorithm ( Lagrange Polynomial ). To construct the Lagrange

polynomial
of degree , based on the

points
Lagrange coefficient polynomials
for
for
for degree
. The
are:
You can use the first Mathematica subroutine that does things in the
"traditional way" or you are welcome to use the second subroutine that
illustrates "Object Oriented Programming."
Mathematica Subroutine (Lagrange Polynomial). Traditional
programming.
The above algorithm is sufficient for understanding and/or constructing

the Lagrange polynomial.
Object Oriented Programming. Welcome to the brave new world of
"Object Oriented Programming." Use the following Mathematica
subroutine which is "programmed" using the "mathematical
objects"
. Templates for the objects are located by
going to "File" then select "Palettes", then select "BasicInput."
Mathematica Subroutine (Lagrange Polynomial). Object
oriented programming.
Mathematica Subroutine (Lagrange Polynomial). Compact
object oriented programming.
The Newton Polynomial
Background.
We have seen how to expand a function
polynomial about
about
.
in a Maclaurin
and a Taylor polynomial
These polynomials have a single "center" . Polynomial interpolation

can be used to construct the polynomial of degree
that passes
through the n+1
points
, fo
r
. If multiple "centers"
are used, then the
result is the so called Newton polynomial. We attribute much of the
founding theory to Sir Isaac Newton (1643-1727).
Theorem ( Newton Polynomial ). Assume that

and
for
,
where
and we write
.
The Newton polynomial goes through the
for
The remainder term
.
has the form
points
, i.e.
,
for some value
. The
coefficients
are constructed using divided differences.
Definition. Divided Differences.

The divided differences for a function
are defined as follows:
The divided difference formulae are used to construct the divided

difference table:
The coefficient of the Newton polynomial

is
and it is the top element in the column of the i-th divided differences.
The Newton polynomial of degree
points
, for
that passes through the

is
The cubic curve in the figure below illustrates a Newton polynomial

of degree n = 3.
Theorem. (Error Bounds for Newton Interpolation, Equally

Spaced Nodes) Assume that
defined on
, which contains the equally
spaced nodes
. Additionally, assume that
and the
derivatives of
up to the order
are continuous and bounded
on the special subintervals
,
,
,
, and
,
respectively; that is,
,
for
. The error terms corresponding to these three cases
have the following useful bounds on their magnitude
(i).
is valid for
(ii).
(iii).
(iv).
(v).
is valid for
is valid for
is valid for
is valid for
,
.
Algorithm ( Newton Interpolation Polynomial ). To construct

and evaluate the Newton polynomial of degree
that passes
through the n+1 points
, for
where
and
Remark 1. Newton polynomials are created "recursively."
.
Remark 2. Mathematica's arrays are lists and the subscripts must
start with 1 instead of 0.
Mathematica Subroutine (Newton Polynomial).
Hermite Polynomial Interpolation
Background for the Hermite Polynomial
The cubic Hermite polynomial p(x) has the interpolative

properties
and
both the
function values and their derivatives are known at the endpoints of the
interval
. Hermite polynomials were studied by the French
Mathematician Charles Hermite (1822-1901), and are referred to as a
"clamped cubic," where "clamped" refers to the slope at the endpoints
being fixed. This situation is illustrated in the figure below.
Theorem (Cubic Hermite Polynomial). If

the interval
, there exists a unique cubic
polynomial
such that
,
,
,
.
is continuous on
Remark. The cubic Hermite polynomial is a generalization of both the

Taylor polynomial and Lagrange polynomial, and it is referred to as an
"osculating polynomial." Hermite polynomials can be generalized to
higher degrees by requiring that the use of more nodes
and the extension to agreement at higher derivatives
for
and
. The details are found in
advanced texts on numerical analysis
More Background. The Clamped Cubic Spline

A clamped cubic spline is obtained by forming a piecewise cubic
function which passes through the given set of knots
with the condition the function values, their
derivatives and second derivatives of adjacent cubics agree at the
interior nodes. The endpoint conditions are
,
where
are given.
More Background. The Natural Cubic Spline

A natural cubic spline is obtained by forming a piecewise cubic
function which passes through the given set of knots
with the condition the function values, their
derivatives and second derivatives of adjacent cubics agree at the
interior nodes. The endpoint conditions are
.
The natural cubic spline is said to be "a relaxed curve."
Cubic Splines
Cubic Spline Interpolant

Definition (Cubic Spline). Suppose that
where
. The function
are n+1 points,

is called a
cubic spline if there exists n cubic polynomials

with
coefficients
that satisfy the properties:
I.
for
.
II.
for
.
The spline passes through each data point.
III.
for
.
The spline forms a continuous function over [a,b].
IV.
for
The spline forms a smooth function.
IV.
for
The second derivative is continuous.
Lemma (Natural Spline). There exists a unique cubic spline with

the free boundary conditions
and
.
Remark. The natural spline is the curve obtained by forcing a flexible

elastic rod through the points but letting the slope at the ends be free
to equilibrate to the position that minimizes the oscillatory behavior of
the curve. It is useful for fitting a curve to experimental data that is
significant to several significant digits.
Program (Natural Cubic Spline). To construct and evaluate the

cubic spline interpolant
for the
data
points
, using the free boundary
conditions
and
.
Mathematica Subroutine (Natural Cubic Spline).
Remark. There are five popular types of splines: natural spline, clamped
spline, extrapolated spline, parabolically terminated spline, endpoint
curvature adjusted spline.
When Mathematica constructs a cubic spline it uses the "natural cubic spline."
Clamped Spline.
Lemma (Clamped Spline). There exists a unique cubic spline with

the first derivative boundary conditions
and
.
A property of clamped cubic splines.

A practical feature of splines is the minimum of the oscillatory
behavior they possess. Consequently, among all functions f(x) which
are twice continuously differentiable on [a,b] and interpolate a given
set data points
, the cubic spline has "less wiggle." The
next result explains this phenomenon.
Theorem (Minimum property of clamped cubic splines). Assume
that
and
is the unique clamped cubic spline
interpolant for
which passes through
and satisfies
the clamped end conditions
and
. Then
.
Program (Clamped Cubic Spline). To construct and evaluate the

cubic spline interpolant S(x) for the n+1 data
points
,
using the first derivative boundary conditions
and
B-Splines
Cubic spline construction using the B-spline function.

Under special circumstances a basis set of splines
can be
used to form a cubic B-spline function. This concept makes the
construction of a spline very easy, it is just at linear combination:
All we need to do is solve for the

coefficients
. And to
make things even more appealing, the linear system to be solved has
a tri-diagonal "appearance":
Caution must prevail when solving this underdetermined system

of
equations in
unknowns. Two end conditions must be
supplied for constructing the coefficients
and
. These end
conditions are specially crafted to form either a natural cubic spline or
a clamped cubic spline.
How can such an elegant construction possible ? It's simple, you
must have a uniform grid of points
on the interval
. The
uniform spacing is
for
and the interpolation nodes to be used are

. The equally spaced abscissa's are
The corresponding ordinates are

and the data points
are
. They are often times referred to as the knots because
this is where we join the piecewise cubics, like pieces of string
"knotted" together to form a larger piece of string. If this is your
situation, then the B-spline construction is for you.
Caveat. If you have unequally spaced points, then this is construction
does not apply and construction of the cubic spline require a more
cumbersome algorithm because each piecewise cubic will need to be
individually crafted in order to meet all the conditions for a cubic
spline.
The basic B-spline function.

Construction of cubic B-spline interpolation can be accomplished by
first considering the following basic function.
The function
is a piecewise continuous on the interval
, it is
zero elsewhere. In advanced courses this simple concept is
glamorized by saying that
is a function with "compact
support." That is, it is supported (or non-zero) only on a small set.
The graph of the function
G
lobal`B
Verify that
is a cubic spline.
Each part of
is piecewise cubic.
Are the functions
,
and
continuous for all
?
Since
is composed of the piecewise functions
,
,
,
,
,
, all that is necessary is to see if they join up properly
at the nodes
. However, this will take 15 computations
to verify. This is where Mathematica comes in handy. Follow the link
below if you are interested in the proof.
The above proof that
is a cubic spline used the formulas
,
,
,
,
.
It is the analytic way to do things and illustrates "precise
mathematical" reasoning.
If you trust graphs, then just look at the graphs of

,
, and
and try to determine if they are continuous.
However, there might be problems lurking about. If you seek the
mathematical truth you should look at the next cell link.
Let's translate the B-spline over to the node

step size .
and use the uniform
Now form the linear combination for the spline.
At each of the nodes
for
computation will reveal that
If the B-spline is to go through the points

then the following equations must hold true
for
for
For the natural cubic spline, we want the second derivatives to be zero
at the left endpoint
.
Therefore we must have
Computation will reveal that
To construct the natural cubic spline, we must have

.
We can solve this equation for the spline coefficient
For the natural cubic spline, we want the second derivatives to be zero
at the right endpoint
.
Therefore we must have
Computation will reveal that
To construct the natural cubic spline, we must have

.
We can solve this equation for the spline coefficient
The above construction shows how to calculate all the coefficients

.
Method I. B-spline construction using equations.

Illustration using 7 knots.
The following example uses n = 6. There are n+3 = 9 equations to
solve and n+1 = 7 data points or knots. First set up the 9 equations
to be solved.
The Bzier Curve
Background.
The Bzier Curve is the original computer generated "French Curve"

and it's discovery is attributed to the French engineer, Pierre
Bzier. Let us first review ideas concerning the cubic polynomial of
degree 3 passes through the 4 points
for
and is
shown in Figure 1.
Figure 1.
Background for Hermite Interpolating Polynomial.

The cubic Hermite polynomial p(x) has the interpolative
properties
and
both the
function values and their derivatives are known at the endpoints of the
interval
. The Hermite polynomial is referred to as a "clamped
cubic," where "clamped" refers to the slope at the endpoints being
fixed. This situation is illustrated in Figure 2.
Figure 2.
The Bzier Curve

The Bzier curve was originally used by Pierre Bzier for CAD/CAM
operations at Renault motor car company. Bzier curves are the basis
of the entire Adobe PostScript drawing model which is used in the
software products: Adobe Illustrator, Macromedia Freehand and
Fontographer.
Construction of the Bzier Curve

A cubic Bzier curve is defined by four points. Two of the points are
endpoints to the curve,
is the starting point and
is the
destination point. The points
are control points or
"handles." A parametric formula for the curve is
interval
where
are cubic equations defined in the
as shown in Figure 3.
Figure 3.
Theorem (Bzier Curve). The parametric equations for

for the Bzier
curve
are given by
,
and
for
More background
If a control point is chosen farther away from a endpoint, (but in the
same direction), then the Bzier curve will more closely approximate
the tangent line near that endpoint.
Bernstein polynomials.
Definition (Bernstein Polynomials). The Bernstein polynomials of
degree are
, for
, where
Construction of the Bzier Curve using Bernstein polynomials.

The coordinate functions
and
for the Bzier curve we are
using can written as a linear combinations of the Bernstein polynomials
:
,
and
.
Verify that the two functions
and
are the same.
Similarly, the two functions

and
are the same.
To verify we have the correct end conditions, we can evaluate
and its derivative at
and
, and see if it has the required
properties.
Recall that this is what we wanted in the first construction.
Justification for the factor 3.

Since we want the two constructions of the Bzier curve to be the
same, it is common practice to use the term
in the
development of the parametric equations. Similarly the
term
is justified.
The construction of a Bzier curve using Bernstein polynomials is
more appealing mathematically because the coefficients in the linear
combination are just the coordinates of the given four points.
Mathematica Subroutine (Bzier Curve). Construct the Bzier

given endpoints
and
and control points
.
For illustration purposes the variables pts, ctr and lin are used to help
form dots and control lines for the graph we will draw.
This is for pedagogical purposes, and usually they would not be
necessary.
Chebyshev Polynomials
Background for the Chebyshev approximation polynomial.

We now turn our attention to polynomial interpolation
for f(x) over [-1,1] based on the
nodes
.
Both the Lagrange and Newton polynomials satisfy
,
where the remainder term
has the form
.
and
is the polynomial of degree
given by
Using the relationship
our task is to determine how to select the set of nodes
that
minimizes
. Research investigating the minimum error
in polynomial interpolation is attributed to the Russian mathematician
Pafnuty Lvovich Chebyshev (1821-1894).
Table of Chebyshev Polynomials.
=
=
=
=
=
=
=
=
Recursive Relationship.
The Chebyshev polynomials can be generated recursively in the
following way. First, set
and use the recurrence relation

.
Relation to trigonometric functions.

The signal property of Chebyshev polynomials is the trigonometric
representation on [-1,1].
Consider the following expansion using the Mathematica command
"FunctionExpand."
These celebrated Chebyshev polynomials are readily available in

Mathematica and called under the reserved name "ChebyshevT[n,x]."
Roots of the Chebyshev polynomials
The roots of
are
.
These will be the nodes for polynomial approximation of degree n.
The Minimax Problem

An upper bound for the absolute value of the remainder term,
, is the product
of
and
. To minimize the factor
, Chebyshev discovered that
must be chosen so
that
, which is stated in the following result.
Theorem (Minimax Property). Assume that n is fixed. Among

all possible choices for Q(x) and thus among all possible choices for
the distinct nodes
in [-1,1],
the polynomial
property
is the unique choice which has the
Moreover,
.
Rule of Thumb.
The "best a priori choice" of interpolation nodes for the interval [1,1] are the n+1 nodes that are zeros of the Chebyshev
polynomial
.
Here is a visual analysis of equally spaced nodes verses Chebyshev
nodes on [-1,1], and their affect on the magnitude of Q(x) in the
remainder term
.
Transforming the Interval for Interpolation

Sometimes it is necessary to take a problem stated on an
interval [a,b] and reformulate the problem on the
interval [c,d] where the solution is known. If the
approximation
to
is to be obtained on the
interval [a,b], then we change the variable so that the problem is
reformulated on [-1,1]:
or
where
on [-1,1] are
,
. The required Chebyshev nodes of
,
and the interpolating nodes
change of variable
on [a,b] are obtained using the

for
Theorem (Lagrange-Chebyshev Approximation). Assume

that
is the Lagrange polynomial that is based on the Chebyshev
interpolating nodes
on [a,b] mentioned above.
If
then
holds for all
Algorithm ( Lagrange-Chebyshev Approximation ). The

Chebyshev approximation polynomial
of degree
for f(x) over [-1,1] can be written as a sum of
:
.
The coefficients
are computed with the formulas
,
and
for
where
for
Mathematica Subroutine ( Chebyshev Polynomial

Approximation ).
The first subroutine "Chebyshev" uses recursion to construct the
Chebyshev Approximation Polynomial.
The following subroutine "ChebyshevPoly" version uses

Mathematica's built-in functions to construct the Chebyshev
approximation polynomial.
In the construction, vector operations are used to assist the
computations, since, similar terms occur in the summation for each of
the coefficients.
Both give the same results.
Mathematica Subroutine ( Chebyshev Approximation ).

The following subroutine "ChebyshevPoly" version uses
Mathematica's built-in functions to construct the Chebyshev
approximation polynomial.
In the construction, vector operations are used to assist the
computations, since, similar terms occur in the summation for each of
the coefficients.
Both give the same results.
First, generate some Chebyshev polynomials with Mathematica's built

in function.
Pad Approximation
Background.
A Pad rational approximation to f(x) on [a,b] is the quotient of two
polynomials
and
of degrees n and m, respectively. We use
the notation
to denote this quotient:
.
We attribute much of the founding theory to Henri Eugne Pad (18631953).
Theorem ( Pad Approximation ). Assume that

Maclaurin polynomial expansion of degree at least
,
where
and
respectively.
are polynomials of degree n and m,
, and that
. Then
Rational Approximation
Background
A rational approximation to f(x) on the interval [a,b] is obtained by
forming the quotient of two polynomials
and
of degrees n
and m, respectively. We use the notation
to denote this
quotient:
.
The polynomials used in the construction are:
and
.
The Pade approximation is a form of rational approximation which is
analogous to the Taylorapproximation because it is based on the
derivatives of f(x) at x=0. The Pad approximation is very accurate
near the center of expansion. However, the error increases as one
moves away from the center of expansion. More accurate rational
approximations are obtained if "interpolation nodes" are used, and we
permit the error to be spread out more evenly over the entire
interval. The process is similar to Lagrange and Chebyshev
interpolation.
Aitken's and Neville's Interpolation Methods
Background
We will now study the iterated interpolation methods of Aitken and
Neville. Alexander Craig Aitken (1895-1967) was one of New
Zealand's prominent mathematicians. Eric Harold Neville (1889-1961)
was a mathematics professor at University of Reading, Berkshire,
U.K. The algorithms we seek are remarkably similar:
To assist with understanding these algorithms we must study

iterated polynomial interpolation.
Existence and Uniqueness

Theorem (Polynomial Existence and Uniqueness). Given a
set n+1 of distinct nodes
(where
whenever
). There is a unique polynomial of degree
that passes through the n+1 points
for
, i.e.
.
Iterated Interpolation
We now discuss some heuristic methods of constructing
interpolation polynomials recursively. The methods of Aitken and
Neville are examples of how iteration is used to construct a sequence
of polynomial approximating of increasing order.
Definition (Selected Interpolation). Given the function
that is to be approximated, and the set of nodes:
.
For any subset of k nodes
the polynomial that agrees with f[x] at the points
is denoted
.
The polynomial
of degree k-1 agrees with f[x] at these
knots
for
Successive Interpolation
Consider polynomial interpolation based on equally spaced nodes
If all
nodes are used then a loose claim is that the interpolating
polynomial
will have order of approximation
. Usually there is an abundance of nodes (think 50, 100,...) and the
degree of the interpolating polynomial is small (think 2, 3, 4, 5 or
6). Polynomials of smaller degree
are of practical value:
The accuracy of interpolation increases with the degree of the

polynomial. Since the polynomial constructions are unique the
following theorem applies for the Lagrange Polynomial, Newton
polynomial and the polynomials constructed with both Aitken's method
and Neville's method too.
Theorem (Remainder Term). Assume that

and
for
, where
, for example, the Lagrange
polynomial
,
and we write
.
The Lagrange polynomial goes through the
for
points
, i.e.
The remainder term
has the form
,
for some value
The Main Results

In practice the subset of
nodes
is not chosen at random
over the larger set
. Instead, the nodes are clustered near a
specific value of x at which the function f[x] is to be approximated
by
. Often times it is a sequential subset of
nodes
with
. It is desired to have the ability to
use permutations of the list
when constructing the
interpolating polynomial. It is also useful to use a sequence of
polynomial approximations with increasing degree. The following
theorem gives the recursive step for generating such a sequence.
Theorem (Recursive Polynomial Construction). Given the

function
that is to be approximated, and the set of
distinct nodes
For any pair of nodes

polynomials:
which agrees with

nodes
, suppose that we have constructed the
at the
,
which agrees with

nodes
at the
.
Then
is formed by making a combination of the above two polynomials
,
or
,
and it agrees with
nodes
at all the
.
Remark. Other equivalent ways to write
are
,
or
.
The Interpolation Tableau

The methods of Aitken and Neville use recursive polynomial construction. The
following tables indicate how the two constructions are made. The extra column at the
right (the values
) have been included to assist with performing hand
calculations. This extra column is not necessary for hand computations. It will be
revealed that the diagonals are equivalent.
Neville's Method. In each new elements is computed using the element in the
{same row, preceding column} and {preceding row, preceding column}.
Table 1. Neville's Method

for
.
Aitken's Method. In each new elements is computed using the
element in the {same row, preceding column} and {top row,
preceding column}.
Table 2. Aitken's Method

for
.
Exploration Aitken and Neville Methods Scroll down to this exploration.
Recursive Programming
Aitken's and Neville's polynomials can be programmed recursively
with the following subroutines.
Computer Programs Aitken's and Neville's Interpolation Methods
Mathematica Subroutine (Neville Polynomials).
Mathematica Subroutine (Aitken Polynomials).
The Rearranged Nodes

Aitken's and Neville's methods agree on the diagonal, and one
usually tests for convergence of these values. If
is to be
approximated at
then one improvement that can be made is to
rearrange the nodes so that they are "closer to " in the sense that is
explained in the following result.
Theorem(Rearrangement of Nodes).
, and the set of
distinct
Given the function
nodes
.
If
is to be approximated at the point
be the rearrangement of
such that
sequence. Then the diagonal terms
will converge to
nodes.
, then let
is an increasing
faster than any other rearrangement of the
The Improved Interpolation Tableau

The tables for Aitken's and Neville's methods can be stored in a
two-dimensional array and do not need long subscript lists as shown in
the following tables.
Neville's Method. In each new elements is computed using the

element in the {same row, preceding column} and {preceding row,
preceding column}.
Table 3. Neville's
Method
for
.
Aitken's Method. In each new elements is computed using the
element in the {same row, preceding column} and {top row,
preceding column}.
Table 4. Aitken's
Method
for
Mathematica Subroutine (Neville Interpolation).
Mathematica Subroutine (Aitken Interpolation).
The Improved Subroutines using Matrices

The subroutines for Aitken's and Neville's methods can be modified
to use matrices, this is the final improvement.
Algorithm (Neville Interpolation).
the Neville interpolation at
is
Given the nodes

where we compute:
Mathematica Subroutine (Neville Interpolation).
Algorithm (Aitken Interpolation).

the Aitken interpolation at
is
Given the nodes

where we compute:
Mathematica Subroutine (Aitken Interpolation).
Legendre Polynomials
Background
We have seen how Newton polynomials and Lagrange polynomials
are used to approximate
on an interval
. The constructions
were based on a discrete set of interpolation points in the interval. We
will now consider "least squares" approximations where the polynomial
is "close" to the function throughout the interval
. Our final
construction will use Legendre polynomials that were first studied by
the French mathematician Adrien-Marie Legendre (1752-1833).
Given a set of interpolation points
t
he Newton polynomial and Lagrange polynomial are algebraically
equivalent, and they are equivalent to the polynomial constructed with
Mathematica's built in subroutine "InterpolatingPolynomial." The
subroutine "Fit" can be used to construct the discrete least squares fit
polynomial.
Definition (Discrete Least Squares Approximation) Given a

function
on
and
and interpolation points
equally spaced nodes

. The
degree
polynomial
is the discrete least squares interpolation
fit provided that the coefficients
of
minimize the sum
Theorem (Discrete Least Squares Approximation) The

polynomial
satisfies the
for
equations
These equations can be simplified to obtain the normal equations for

finding the coefficients
for
Remark. This is the degenerate case of a least squares fit (i.e. if there
were
data points we would have used
instead of
).
Information on polynomial curve fitting can be found in the module
Least Squares Polynomials.
The following example shows that if n+1 points are used to find the
discrete least squares approximation polynomial of degree n , then it is
the same as the Newton (and Lagrange) interpolation polynomial that
passes through the n+1 points.
Continuous Least Squares Approximation

Another method for approximating
on an interval
is to
find a polynomial
with a small average error over the entire
interval. This can be accomplished by integrating the square of the
difference
over
. The following derivation is done on
an arbitrary interval
, but we will soon see that it is
advantageous to use the interval
.
Definition (ContinuousLeast Squares Approximation) Given a
function
on
. The nth degree polynomial
is
the continuous least squares fit for
provided that the coefficients
minimize the integral
.
Theorem (Continuous Least Squares Approximation) The
polynomial
satisfies the
equations
for
These equations can be simplified to obtain the normal equations for

finding the coefficients
for
Orthogonal Polynomials
To start we need some background regarding an the inner product.
Definition ( Inner Product ). Consider the vector space of functions
whose domain is the interval
. We define the inner product of two
functions
as follows
.
Mathematica Function ( Inner Product ). To compute the inner
product of two real functions over
.
Remark. The inner product is a continuous analog to the ordinary dot

product that is studied in linear algebra. If the integral is zero then
are said to be orthogonal to each other on
. All the
functions we use are assumed to be square-integrable, i. e.
.
Basis Functions
A basis for a vector space V of functions is a set of linear independent
functions
which has the property that any
can be
written uniquely as a linear combination
.
Fact. The set
power series.
is a basis for the set
Definition (Orthogonal Basis) The set

an orthogonal basis on
provided that
when
of all polynomials and

is said to be
and
when
In the special case when

orthonormal basis.
for
we say that
is an
Theorem ( Gram-Schmidt Orthogonalization ). Given

we can construct a set of orthogonal polynomials
interval
as follows:
Use the inner product
over the
, and define
Remark. A set of orthonormal polynomials over the interval

is
Remark. When these polynomials are constructed over the interval

and normalized so that
they are called the Legendre
polynomials, and form a basis for the set of polynomials and power
series over the interval
.
Corollary 1. The set of orthogonal polynomials
is a basis for
the set V of all polynomials and power series over the interval
.
Corollary 2. The set of Legendre polynomials
is a
basis for the set V of all polynomials and power series over the interval
.
Proof Legendre Polynomials
An Alternate Recursive Formula

Another way to recursively define the Legendre polynomials is
Efficient Computations
We now present the efficient way to compute the continuous least
squares approximation. It has an additional feature that each
successive term increases the degree of approximation. Hence, an
increasing sequence of of approximations can obtained recursively:
Theorem (Legendre Series Approximation) The Legendre series
approximation of order for a function
over
is given by
where
is the
Legendre polynomial and
The Shifted Legendre Polynomials

The "shifted Legendre polynomials
are orthogonal on
where
are the Legendre polynomials on
,
.
Exploration.
Theorem (Shifted Legendre Series Interpolation) The shifted

Legendre series approximation of order for a function
over
is given by
where
is the
shifted Legendre polynomial and
The Tangent Parabola
by
John H. Mathews
Russell W. Howell The AMATYC Review, Vol. 23, No. 1, Fall 2001, pp.
25-32.
Background
The limit of "the secant line" is "the tangent line." We recast this in
the context of polynomial approximation. The secant line is the
Newton polynomial
, and the tangent line as the Taylor
polynomial
, where each has degree
.
The Newton polynomial
, has the form:
(i)
The coefficients and are determined by forcing

to pass
through two points
and
. Here we have used
the notation
for the second point.
Using the equation
and the two points produces a
lower-triangular linear system of equations:
which simplifies to be
(ii)
which is easily solved using forward elimination:
, and
get
. Substitute
and
into equation (i) and
(iii)
As h approaches 0 in equation (iii), the limit of the difference
quotient
Newton Polynomial
is the derivative
and the limit of the
is seen to be the Taylor polynomial
:
.
This background is the motivation to investigate the idea of polynomial

approximations and their limits.
The secant line which is used to approximate
it is based on
two points
and
. What if we used three
points? Then we could determine a polynomial of degree
, which
could also be used to approximate
. Thus, we have the
concept of "the secant parabola" with interpolation points
,
, and
.
Figure 1. The secant parabola approximating

, at
using
When the interpolation points are moved closer to the middle point
the "the secant parabola" approaches a limiting position.
, at
Figure 2. The secant parabola approximating

using
The limiting position of the secant parabola is called the "tangent

parabola." The following figure shows this case when h goes to 0 and
all three nodes coincide.
Figure 3. The tangent parabola approximating

, at
where
.
Can you guess what the tangent parabola will turn out to be? We will
reveal this pleasant surprise at the end of the article.
The Secant Parabola
A precise discussion of "the secant parabola" is now
presented. Recall that a polynomial of degree
expanded
about
can be written in the form
(1)

are to be determined. Since
is
to be the interpolating polynomial for
, it must pass through
the three points
,
, and
. Using
the first point
we obtain the relation
,
which implies that
.
Proceeding, we solve for the two coefficients
substitution
in equation (1) and writing
(2)
by first making the
Then make substitutions for the two points

, and
, respectively, in equation (2) and obtain two relations
and
.
Simplification produces the following two equations which will be used
to solve for
.
,
(3)
.
Subtract the first equation in (3) from the second and then divide
by 2h and get
.
Add the equations in (3), subtract
and then divide by
and get
.
The Secant Parabola Formula
The secant parabola for
which passes through
, and
involves the variable x and
parameters
and has the form
(4)
.
A Numerical Experiment
Formulas for the above graphs of the secant parabola
for
are constructed using formula (4). The
computations are centered at the point
using the step
sizes
.
Finding the Limit Numerically

The limit of the secant polynomials is found by evaluating formula
(4) using decreasing step sizes
.
The numerical results are summarized in Table 1.
Table 1. The secant parabola approximating

, at
where
Finding the Limit Symbolically

The entries in the table show that the coefficients of
tending to a limit as
. Thus the "tangent parabola" is
are
(5)
.
The first limit in (5) is well known, it is
.
The second limit in (5) is studied in numerical analysis, and is known
to be
, which can be verified by applying L'hopital's rule using
the variable h as follows
Therefore, we have shown the limit of the "secant parabolas" to be

(6)
Therefore, the "tangent parabola" in (5) is revealed to be the Taylor

For our example with
, and
, we have
The Lagrange Connection

In numerical analysis, the Lagrange interpolation polynomial is
constructed, and it can be shown to be equivalent to the formula in
(4), however the hand computations are messy. If a computer algebra
system, such as Mathematica is used, then it is easy to verify that the
two forms are equivalent. First, enter the formula for the Lagrange
polynomial
Then enter formula (4)
The above two formulas can be expanded and shown to be equal.

Therefore, formula (4) is equivalent to Lagrange interpolation, hence
the Lagrange form of the remainder applies too.
The Remainder Term

In numerical analysis, the remainder term for a Lagrange
interpolation polynomial of degree
, is known to be
,
where
depends on
and lies somewhere between
When we take the limit of

we get
as
.
it is plain to see that
,
which is the remainder term for the Taylor polynomial of degree
. This cinches the fact that the limit of the secant polynomial is the
tangent polynomial.
Conclusion
The purpose of this article has been to show that the Taylor
polynomial is the limiting case of a sequence of interpolating
polynomials. The development has been to first show graphical
convergence, which is quite rapid. This can be illustrated in the
classroom by using graphical calculators or with computer software
such as Mathematica or Maple. Then a selected set of interpolating
polynomials is tabulated, which is a new twist to the idea of limit, it
involves the concept of convergence of a sequence of
functions. Finally, the power of calculus is illustrated by discovering
that the limiting coefficients are
and
. Then one
recognizes that the "tangent polynomial" is a Taylor polynomial
approximation. Moreover, we have motivated the "what if" exploration
by showing what happens to "the secant parabola" with interpolation
points
,
, and
when the points
"collide" at the single point
. Thus the mystery behind the
Taylor polynomial being based on a single point is revealed. It is
hoped that teachers reading this article will gain insight to how to use
technology in teaching mathematics. Higher degree polynomials have
been investigated by the authors in the article "Investigation of
Tangent Polynomials with a Computer Algebra System ", and some of
the ideas are given below.
The Tangent Cubic Polynomial

A natural question to ask now is: "What about polynomial
approximation of higher degrees?" Exploration of the Newton
polynomials involves complicated symbolic manipulations and is prone
to error when carried out with hand computations. These derivations
can become instructive and enjoyable when they are performed with
computer algebra software. Let
be the Newton polynomial that
passes through the four points
for
. It may be
shown that the Taylor polynomial
is the limit of
as
. We shall use the power of Mathematica to assist us with this
derivation. Begin by setting
equal to the general form of a
Newton polynomial of degree n by issuing the following Mathematica
commands:
Now form the set of four equations that force the polynomial to pass
through the four equally-spaced points. Notice that this is a lowertriangular system of linear equations.
Then solve this lower triangular linear system, and construct the
polynomial
, and store it as the function
.
Finally, compute the limit to verify that our conjecture was correct:
Eureka! The limiting case of

as
is the Taylor polynomial
. Observe that the option
must be used in
Mathematica's limit procedure. This is a mathematicians way to tell
the computer that
is "sufficiently differentiable."
Tangent Polynomials of Higher Degree

Consider the seven points
for
. We
can construct the Newton polynomial of degree 6 and take the limit to
obtain the Taylor polynomial.
Now form the set of seven equations that force the polynomial to pass
through the four equally-spaced points.
Then solve this lower triangular system, and construct the

polynomial
, and store it as the function
.
Finally, compute the limit and see that it is the Taylor polynomial.
The Catenary
Background
A catenary is the curve formed by a flexible cable of uniform density
hanging from two points under its own weigh. Cables of suspension
bridges and attached to telephone poles hang in this shape. If the
lowest point of the catenary is at
, then the equation of the
catenary is
Approximated by a parabola
Notice that
is an even function. The following
computation shows that the first term in the Maclaurin series is

. For this reason it is often claimed that the shape of a hanging cable
is "approximated by a parabola."
Arc Length
The arc length of the curve
is found by using
the integrand
. The length of the
catenary over the interval [0,a] is given by the calculation
.
Catenary Fit
In order to find a catenary that has width
and height
all we need to observe that

is and even function
and that
goes through
and we also want it to go through
the point
. On first glance, we see that all we need to do is solve
the equation
for
. However, this is not possible to
do analytically with Mathematica, as we can find out by issuing the
following command.
Therefore, we must resort to using a numerical approximation for c

instead of a "formula." In our exploration, the graphs we used the
mysterious constant
, which was found by determining the
catenary that passes through the origin and the point
.
Conclusion
Either a hanging cable is in the shape of a parabola or a catenary,
let's look at the history of this controversy. The following paragraph is
from the "Concise Encyclopedia of Mathematics" by Eric W. Weisstein.
"The curve a hanging flexible wire or chain assumes when
supported at its ends and acted upon by a uniform gravitational
force. The word catenary is derived from the Latin word for
"chain''. In 1669, Jungius disproved Galileo's claim that the curve of
a chain hanging under gravity would be a Parabola ( MacTutor
Archive ). The curve is also called the Alysoid and Chainette. The
equation was obtained by Leibniz, Huygens, and Johann Bernoulli in
1691 in response to a challenge by Jakob Bernoulli." Other
mathematicians involved with the study of the catenary have been
Robert Adrain, James Stirling, and Leonhard Euler.
An article suitable for undergraduates to read is "The Catenary and
the Tractrix (in Classroom Notes)", Robert C. Yates, American
Mathematical Monthly, Vol. 66, No. 6. (Jun. - Jul., 1959), pp. 500-505.
The St. Louis Arch at the Jefferson National Expansion Memorial was
constructed in the shape of a catenary. Visit the National Park Service
web site arch history and architecture. Or, go directly to the web
page for the precise mathematical formula for the St. Louis arch
( catenary curve equation ).
The University of British Columbia Mathematics Department has an
amusing property of the catenary (Java animation). Which is part of
their "Living Mathematics Project" for "Constructing a new medium for the
communication of Mathematics''.
Curve Fitting
Least Squares Lines
Least Squares Polynomials
Nonlinear Curve Fitting
Logistic Curve
FFT and Trigonometric Polynomials
Conic Fit
Circle of Curvature
Least Squares Lines
Background
The formulas for linear least squares fitting were independently
derived by German mathematician Johann Carl Friedrich Gauss (17771855) and the French mathematician Adrien-Marie Legendre (17521833).
Theorem (Least Squares Line Fitting ). Given the
points
, the least squares line
fits the points has coefficients a and b given by:
data
and
.
Remark. The least squares line is often times called the line of
regression.
Mathematica Subroutine (Least Squares Line).
that
Philosophy. What comes first the chicken or the egg ? Which

coordinate is more sacred, the abscissas or the ordinates. We are
always free to choose which variable is independent when we graph a
line;
or
. When you realize that two different
"least squares lines" can be produced we are amazed. What should we
do ? Which line should we use ? You must decide a priori which
variable is independent and which is dependent and then proceed.
Exercise 3 asked you to think about the mathematics that is involved
with this "paradox."
Another "Fit"
Theorem (Power Fit). Given the
data
points
, the power curve
points has coefficients a given by:
that fits the
.
Remark. The case m = 1 is a line that passes through the origin.
Mathematica Subroutine (Power Curve).
Least Squares Polynomials

the
data points
, the least squares polynomial of
degree m of the form
that fits the n data points is obtained by solving the following linear system

One thing is certain, to find the least squares polynomial the above
linear system must be solved. There are various linear system solvers
that could be used for this task. However, since this is such an
important computation, most mathematical software programs have a
built-in subroutine for this purpose. In Mathematica it is called the "
Fit" procedure. Fit[data, funs, vars] finds a leastsquares fit to a list
of data as a linear combination of the functions funs of variables vars.
We will check the "closeness of fit" with the Root Mean Square or
RMS measure for the "error in the fit."
Mathematica Subroutine (Least Squares Parabola).
Caution for polynomial curve fitting.

Something goes radically wrong if the data is radically "NOT
polynomial." This phenomenon is called "polynomial wiggle." The
next example illustrates this concept.
Linear Least Squares

The linear least-squares problem is stated as follows. Suppose that
data points
and a set of
linearly independent
functions
are given. We want to fine
coefficients
so that the function
given by the linear combination
will minimize the sum of the squares of the errors

.
Theorem (Linear Least Squares). The solution to the linear least
squares problem is found by creating the matrix
whose elements
are
The coefficients
are found by solving the linear system
where
and
Nonlinear Curve Fitting
Data Linearization Method for Exponential Curve Fitting.

Fit the curve
to the data points
Taking the logarithm of both sides we obtain

, thus
.
Introduce the change of variable
equation becomes
. Then the previous
which is a linear equation in the variables X and Y.

Use the change of variables
and obtain
for
Fit the points
form
.
Comparing the equations
that
. Thus
on all the data points

.
with a "least squares line" of the
we see
are used to construct the coefficients which are then used to "fit the
curve"
to the given data points
in the xy-plane.
Data Linearization Method for a Power Function Curve Fitting.
Fit the curve
to the data points
Taking the logarithm of both sides we obtain

, thus
.
Introduce the change of variable
previous equation becomes
. Then the
which is a linear equation in the variables X and Y.

Use the change of variables
points and obtain
for
Fit the points
form
.
Comparing the equations
that
. Thus
on all the data

.
with a "least squares line" of the
we see
are used to construct the coefficients which are then used to "fit the
curve"
to the given data points
in the xy-plane.
Logistic Curve Fitting
Background for the Logistic Curve Fitting.

Fit the curve
points
to the data
.
Rearrange the terms
. Then take the logarithm of both sides:

.
Introduce the change of variables:

becomes
. The previous equation
which is now "linearized."

Use this change of variables on the data points
abscissa's but transformed ordinates.
Now you have transformed data points:
, i.e. same
.
Use the "Fit" procedure get Y = A X + B, which must match the form
, hence we must have
and a = A.
Remark. For the method of "data linearization" we must know the constant L in
advance. Since L is the "limiting population" for the "S" shaped logistic curve, a value
of L that is appropriate to the problem at hand can usually be obtained by guessing.
Example. Use the method of "data linearization" to find the logistic curve that fits the
data for the population of the U.S. for the years 1900-1990. Fit the
curve
to the census data for the population of the U.S.
Date
Populatlion
76094000
92407000
106461000
123076741
132122446
152271417
180671158
205052174
227224681
249464396
Fast Fourier Transform (FFT)
Definition ( Piecewise Continuous ). The function

is
piecewise continuous on the closed interval
, if there exists
values
with
such that f is continuous
in each of the open intervals
, for
and has lefthand and right-hand limits at each of the values
, for
.
Definition ( Fourier Series ).

and is piecewise continuous on
Series
for
is
If
is periodic with period

, then the Fourier
,
formulae:
are given by the so-called Euler's
,
and
.
Theorem (Fourier Expansion). Assume that

is the Fourier
Series for
. If
are piecewise continuous
on
, then
is convergent for all
.
The relation
continuous. If
holds for all

where
is
is a point of discontinuity of
, then
,
where
denote the left-hand and right-hand limits,
respectively. With this understanding, we have the Fourier Series
expansion:
Definition (Fourier Polynomial). If

is periodic with
period
and is piecewise continuous on
, then the Fourier
Polynomial
for
of degree m is
,
formulae:
are given by the so-called Euler's
,
and
.
Numerical Integration calculation for the Fourier trigonometric

polynomial.
Assume that
is periodic with period
and is piecewise
continuous on
, we shall construct the Fourier trigonometric
polynomial over [0,2L] of degree m.
,
There are n subintervals of equal width
. The coefficients are
based on
for
and
for
The construction is possible provided that
Remark. The sums can be viewed as numerical integration of Euler's

formulae when [0,2L] is divided into n subintervals. The trapezoidal
rule uses the weights
for both the
and
. Since f(x) has
period
, it follows that
. This permits us to use 1 for all
the weights and the summation index from
.
The Fast Fourier Transform for data.

The FFT is used to find the trigonometric polynomial when only data
points are given. We will demonstrate three ways to calculate the
FFT. The first method involves computing sums, similar to "numerical
integration," the second method involves "curve fitting," the third
method involves "complex numbers."
Computing the FFT with sums.

Given data points
for
periodic with period
[0,2L] of degree m.
where
and
over [0,2L] where
Also given that

, to that the data is
. We shall construct the FFT polynomial over
,
The abscissa's form n subintervals of equal width
on
. The coefficients are
for
based
and
for
The construction is possible provided that
.
.
Determinants and Conic Section Curves
Background.
Five points in the plane uniquely determine an equation for a conic
section. The implicit formula for a conic section is often mentioned in
textbooks, and the special cases for an ellipse, hyperbola, parabola,
circle are obtained by either setting some coefficients equal to zero or
making them the same value.
Implicit Equation for a Line.

The equation
points
of the line through the two

can be computed with the determinant
The Circle of Curvature: It's a Limit!
by
John H. Mathews
The AMATYC Review, Vol. 25, No. 1, Fall 2003, pp. 57-63.
Given the function

, the formula for the curvature (and
radius of curvature) is stated in all calculus textbooks
Definition ( Curvature ).
Definition ( Radius of Curvature ).
Definition ( Osculating Circle ). At the point {x,f[x]} on the

curve y = f[x], the osculating circle is tangent to the curve and has
radius r[x].
Finding Curvature at Any Point

For the above example the circle of curvature was easy to locate
because it's center lies on the y-axis. How do you locate the center if
the point of tangency is not the origin? To begin, we need the
concepts of tangent and normal vectors.
Tangent and Normal Vectors
Given the graph
, a vector tangent to the curve at the
point (x,f(x)) is
, which can be written as
. The unit tangent vector is
Definition (Unit Tangent ).

For vectors in
, a corresponding perpendicular vector called the
unit normal vector is given by
Lemma (Unit Normal)
A new construction of the Circle of Curvature
What determines a circle? A center and a radius. The formula
for the radius of curvature is well established. What idea could we use
to help understand the situation. We could use the fact that three
points determine a circle and see where this leads.
Derivation of the Radius of Curvature

The standard derivation of the formula for radius curvature involves
the rate of change of the unit tangent vector. This new derivation
starts with the collocation the collocation circle to go through the three
points
,
, and
on the
curve
. The limit as
is the osculating circle to the
curve
at the point
. The radius of curvature and
formulas for the location of its center are simultaneously derived. The
computer algebra software Mathematica is used to assist in finding the
limits.
Start with the equation
, of a circle. Then write
down three equations that force the collocation circle to go through the
three points
,
, and
on the
curve
. Enter the equations into Mathematica
Solve the equations for

and extract the formula for the radius of
the collocation circle. Since it depends on
we will store it as the
function
The formula looks bewildering and one may wonder if it is of any

value.
However, we can demonstrate the power of Mathematica and see if it
can take the limit.
The computer gets the correct formula, but leaves out all the human
understanding. The formula for
can be rewritten so that
human insight and inspiration is involved, but the computer cannot
think of the formulas we desire to see. So human intervention must
be called upon and the simplifications must be typed in by hand.
Notice that when we take the limit as
, the limiting value for each
of the radicals in the numerator is

. Mathematica is capable
of finding them, we illustrate this with the third one.
Therefore, the limit in the numerator is

. The difference
quotient in the denominator is recognized as the numerical
approximation formula for the second derivative, hence the is
.
We already knew that Mathematica knows the rules for taking limits of
functions when the formula is given. Now we know that it has the
"artificial intelligence" to rearrange quantities involving an arbitrary
function and can identify difference quotients for approximating
derivatives and find their appropriate limits. It takes quite a bit of
trust to let Mathematica do our thinking. From the steps we filled in,
we can gain trust for the computer's way of doing it.
The Osculating Circle

We now show that the limit of the collocation circle as
is the
osculating circle.
Now we want to find the center (a, b) of the osculating circle.
The abscissa for the center of the collocation circle is
Take the limit as

circle of curvature.
, to obtain the abscissa for the center of the
If we want to see "what's happening" in this limit, then we must

rearrange the formula for
. A little finesse permits us to write it
as follows
The numerators involve three difference quotients, all of which tend to

when
, and the difference quotient in the denominators
tends to
when
.
Thus we have established the formula
abscissa of center of the circle of curvature.
for the
The Abscissa the Easy Way
Subtract from
the radius of curvature times
The Ordinate for the Center of the Circle of Curvature

The ordinate for the center of the collocation circle is
If we want to see "what's happening" in this limit, then we must

rearrange the formula for
. A little finesse permits us to write it
as follows
The limit of
as
is merely
, and we have
already observed that the limit in the denominator is
and both
quotients in the numerator tend to
as
.
Thus we have established the formula
the ordinate of center of the circle of curvature.
for
The Ordinate of the Circle the Easy Way
Add to
The Osculating Circle

At the point
on the curve
, the center and radius
of the osculating circle are given by the limits calculated in the
preceding discussion.
Generalizations for 2D
In two dimensions, a curve can be expressed with the parametric
equations
and
. Similarly, the formulas for the radius
of curvature and center of curvature can be derived using limits. At
the point
the center and radius of the circle of
convergence is
Remark. The absolute value is necessary, otherwise the formula

would only work for a curve that is positively oriented.
The Abscissa the Easy Way
Subtract from
abscissa of the circle of curvature is
. The
The Ordinate the Easy Way

Add to
ordinate of the circle of curvature is
. The
Richardson Extrapolation
Derive Numerical Differentiation Formulae
Background.
Numerical differentiation formulas formulas can be derived by first constructing the
Lagrange interpolating polynomial
through three points, differentiating the
Lagrange polynomial, and finally evaluating
at the desired point. In this
module the truncation error will be investigated, but round off error from computer
arithmetic using computer numbers will be studied in another module.
Theorem (Three point rule for
derivative, based on three points is
). The central difference formula for the first
,
and the remainder term is
.
Together they make the equation
the truncation error bound is
where
notation for the error term for
, and
. This gives rise to the Big "O"

:
.

). The central difference formula for the second
,
.
the truncation error bound is
, and
where
for
This gives rise to the Big "O" notation or the error term
:
.
Project I.
Investigate the numerical differentiation
formula
and truncation error
bound
wher
e
. The truncation error is
investigated. The round off error from computer arithmetic using computer numbers will
be studied in another module.
Enter the three point formula for numerical differentiation.
Aside. From a mathematical standpoint, we expect that the limit of the difference
quotient is the derivative. Such is the case, check it out.
Example Consider the function

. Find the formula for the third derivative
, it will be used in our explorations for
the remainder term and the truncation error bound. Graph
. Find the
bound
. Look at it's graph and
estimate the value
, be sure to take the absolute value if necessary.
Solution
Project II.
Investigate the numerical differentiation formulae
and truncation error bound
where
. The truncation error is investigated. The round off error
from computer arithmetic using computer numbers will be studied in another module.
Enter the formula for numerical differentiation.
Aside. It looks like the formula is a second divided difference, i.e. the difference quotient
of two difference quotients. Such is the case.
Aside. From a mathematical standpoint, we expect that the limit of the second divided
difference is the second derivative. Such is the case.
Example. Consider the function

. Find the formula for the fourth
derivative
, it will be used in our explorations for the remainder term and the
truncation error bound. Graph
. Find the bound
. Look
at it's graph and estimate the value
Richardson Extrapolation
Numerical differentiation formulas can be derived by first constructing the Lagrange
interpolating polynomial
through three points, differentiating the Lagrange
polynomial, and finally evaluating
at the desired point. The truncation error is
be investigated, but round off error from computer arithmetic using computer numbers
will be studied in another lab.
,
.
error bound is
where
for
, and the truncation
. This gives rise to the Big "O" notation for the error term
:
.
Theorem (Five point rule for

derivative, based on five points is
,
.
error bound is
, and the truncation
where
for
. This gives rise to the Big "O" notation for the error term
:
.
Theorem (Richardson's Extrapolation for

). The central difference formula
for the first derivative, based on five points is a linear combination of
and
.
,
where
and
Richardson's Extrapolation.
Richardson's extrapolation relates the five point rule and the three point
rule,
, that was studied previously.

.
Enter the three point formula for numerical differentiation.
Project III.
Investigate the numerical differentiation
formulae
and error
bound
where
.
The truncation error is be investigated, but round off error from computer arithmetic
using computer numbers will be studied in another lab.
Enter the five point formula for numerical differentiation.

. Find the formula for the third
derivative
. Find the bound
. Look
. Find the formula for the third
derivative
. Find the bound
. Look
Solution
Project IV.
Investigate Richardson's extrapolation for numerical differentiation.
Example In general, show that
Solution
Derive Numerical Differentiation Formulae

The traditional "pencil and paper" derivations for numerical differentiation formulas for
and
are done independently as if there was no connection among the
derivations.This new approach gives a parallel development of the formulas.It requires
the solution of a "linear system" that includes symbolic quantities as coefficients and
constants. The power of a computer algebra system such as Mathematica is used to
elegantly solve the linear system for
and
.The extension to proofs of higher
order numerical differentiation formulas for the central, forward or backward differences
is easy to accomplish.The notation convention using square brackets "[ and ]" instead of
parenthesis is used to make all the formulas look compatible with the Mathematica input
and output that is used.
The Three Point Central Difference Formulas

Using three points
,
, and
we give a parallel development of the
numerical differentiation formulas for
and
.Start with the Taylor series for
expanded in powers of .Terms must be included so that the remainder term is
included.Using degree
will suffice.
The Mathematica notation for the remainder term is

.The term
is
removed from the series with the command Normal, and a Taylor polynomial of degree
is formed.However, we must remember that the accuracy for the Taylor polynomial
is
.
Construct two "equations" by setting the series equal to the function at
We have two equations in the two unknown
and
, and all the other
quantities ,
,
,
,
and
are considered as
constants.Now use the Mathematica command "Solve" to solve for
and
The result is not too easy to read, so we can use the commands
and
to
manipulate the formula.In the print statements we add"
"to remind us that we are
using truncated infinite series.The following commands will create a nicer looking
printout for the above solution.
Thus, we have derived the numerical differentiation formula for

and
and
the first term in the series expansion for the remainder which involves,
or
, respectively.Since the "numerical differentiation formulae" are "truncated" infinite
series, we know that the ellipses "
" means that there are infinitely many more term
which are not shown.
When we do not include the"
", we must evaluate the lowest order derivative in the
series for the remainder at the value
instead of
, then we can "chop off" the
infinite series at the term involving
or
, respectively.The Mathematica
commandReplaceAll is used to accomplish this task.
We have obtained the desired numerical differentiation formulas and their remainder
terms.We should mention that new in version 4.1 ofMathematica is symbolic recognition
of order of approximation for divided difference formulas.We can add the term to the
numerical differentiation formula term, and Mathematica will conclude that it is the
derivative plus .The following commands illustrate this feature of Mathematica.
Therefore, we have established the numerical differentiation formulas
and
And the corresponding formulas with the big "O" notation
are
and
Comparison with the Traditional Derivations

The above derivation differs only slightly from the traditional derivations which also start
with the two equations
(1)
When deriving the numerical differentiation formula for

the equations (1) are
subtracted and the terms involving
cancel and manipulations are used to solve for
and its truncation error term.
The traditional derivation of the numerical differentiation formula for
starts with
the same equations (1).But this time the equations are added and the terms involving
cancel and manipulations are used to solve for
and its truncation error
term.
There is no "big deal" made about the fact that the starting place is the same.We shall see
for the higher order formulas that using the same starting place will be the key to
successful computer derivations of numerical differentiation formulas.
The Five Point Central Difference Formulas

Using five points
,
,
,
, and
we can give a
parallel development of the numerical differentiation formulas for
,
,
and
.Start with the Taylor series for
expanded in powers of
.Terms must be included so that the remainder term is included.Using degree
will
suffice.Since the remainder term
, we will use Mathematica's Normal
command to remove it from the series, and form the Taylor polynomial of degree
.
Use five points and set up the "equations" by setting

for
, which is automated by using Mathematica's Table command.Although
the value
, will produce
which is the boolean value True, it will do no
harm when solving the set of equations and we will see that the generations of higher
order formulas which require more complicated sets of equations is more easily
automated with the use of a table.
We can consider that these are four equations in the four unknowns
,
,
and
, and all the other quantities ,
,
,
,
,
,
and
are constants.Now use Mathematica to solve
for
,
,
and
.Since this requires typing of four derivatives
to be solved, we will automate this process too, by using a table to construct the
"variables."
Amazingly, the algebra involved in solving the four equations will result in the
cancellation of
in the odd order derivatives and the term
will cancel in
the even order derivatives.This can be accomplished using Mathematica's Solve
procedure.
The output is not too clear to read, so we shall use the formula manipulation commands
and
to group the numerical differentiation portion, and then use the
ReplaceAll command to change
and
to
and
, then a final
use of Together will clean up the numerical differentiation formula part.All this will be
accomplished in the following three Mathematica commands.If the reader is curious
about exactly what is happening in each step, then the semi-colons can be deleted and the
results of each operation will be shown.
Notice that the formula for approximating

and
have truncation error terms
involving so they are numerical differentiation formulas of order
.But the
formula for approximating
and
have truncation error terms involving
so they are numerical differentiation formulas of order
.This is one of the many
surprises in the theory of numerical analysis.
Again, we can uses Mathematica's 4.1 ability to symbolic recognize the order of
approximation for divided difference formulas.If we add the term
or
to the
numerical differentiation formula then we will be surprised to learn that Mathematica
will tell us that it is the derivative plus
or
.The following commands illustrate
this point.
Therefore, we have established the numerical differentiation formulas
These formulas can be written with the big "O" notation
and
if desired.
Appendix. Subroutines for Generating Numerical Differentiation Formulae

The central difference formulae use an odd number of points
,and an even
number of
equations.Since we want to include the remainder terms, we need to use
series expansions of order
.The remainder term in these formulas all involve
even powers ofhand even and odd derivatives depending on the situation.Also, the
subroutine requires a replacement of the point where the remainder term is evaluated to
be
instead of
.
The Five Point Central Difference Formulae

These are to difficult to do by hand, so we use the computer.
Higher order formulas are easy to obtain with Mathematica.

Subroutines for Generating the Forward and Backward Difference Formulae
Once we have the idea for the central difference formula, all we need to do is adjust the
set of to be forward step sizes or backward step sizes and we can generate the forward
and backward differentiation formulas.
Explorations with the Higher Order Formulas
Higher order formulas are easy to obtain with Mathematica.
Numerical Integration
Riemann Sums
Midpoint Rule
Newton-Cotes Integration
Trapezoidal Rule for Numerical Integration
Simpson's Rule for Numerical Integration
Simpson's 3/8 Rule for Numerical Integration
Boole's Rule
Romberg Integration
Adaptive Simpson's Rule
Gauss-Legendre Quadrature
Cubic Spline Quadrature
Monte Carlo Pi
Monte Carlo Integration
2D Trapezoidal and Simpson Rules
Riemann Sums
Definition. Definite Integral as a Limit of a Riemann Sum. Let
be continuous
over the interval
,
and let
be a partition, then the definite integral is given
by
,
where
and the mesh size of the partition goes to zero in the "limit," i.e .
.
The following two Mathematica subroutines are used to illustrate this concept, which was
introduced in calculus.
The Left Riemann sum uses
in the definition.
Mathematica Subroutine (Left Riemann Sum).
The Right Riemann sum uses
in the definition.
Mathematica Subroutine (Right Riemann Sum).
The midpoint rule uses
in the definition.
Improvements can be made in two directions, the midpoint rule evaluates the function
at
, which is the midpoint of the subinterval
Riemann sum.
, i.e.
in the
Mathematica Subroutine (Midpoint Rule).
The Trapezoidal Rule is the average of the left Riemann sum and the right Riemann
sum.
Mathematica Subroutine (Trapezoidal Rule).
Example Let
over
. Use the left Riemann sum with n = 25, 50, and
100 to approximate the value of the integral.
Midpoint Rule
Theorem (Midpoint Rule) Consider
. The midpoint rule is
over
, where
.
This is an numerical approximation to the integral of
and we have the expression
over
.
The remainder term for the midpoint rule is
somewhere between
, and have the equality
, where lies
Composite Midpoint Rule

An intuitive method of finding the area under a curve y = f(x) is by
approximating that area with a series of rectangles that lie above the
intervals
. When several rectangles are used, we call it
the composite midpoint rule.
Theorem (Composite Midpoint Rule) Consider

. Suppose that the interval
is subdivided
into m subintervals
over
of equal width
the equally spaced nodes

for
composite midpoint rule for m subintervals is
by using
.
The
.
and we write
over
Remainder term for the Composite Midpoit Rule

Corollary (Midpoint Rule: Remainder term) Suppose that
subdivided into m subintervals
composite midpoint rule
of width
is
.
The
is an numerical approximation to the integral, and

.
Furthermore, if
error term
, then there exists a value c with a < c < b so that the

has the form
.
This is expressed using the "big
" notation
Remark. When the step size is reduced by a factor of

be reduced by approximately
.
the error term
Algorithm Composite Midpoint Rule. To approximate the integral
,
by sampling
, where
at the equally spaced points

.
for
should
Or you can use the traditional program.
Example Numerically approximate the integral

midpoint rule with m = 1, 2, 4, 8, and 16 subintervals.
by using the
Newton-Cotes Integration
Introduction to Quadrature
We now approach the subject of numerical integration. The goal is
to approximate the definite integral of f(x) over the
interval [a,b] by evaluating f(x) at a finite number
of sample points.
Definition (Quadrature Formula) Suppose that
. A formula of the form
(1)
with the property that
(2)
is called a numerical integration or quadrature formula. The
term E[f] is called the truncation error for integration. The values
are called the quadrature nodes and
are called the weights.
Depending on the application, the nodes

are chosen in
various ways. For the Trapezoidal Rule, Simpsons Rule, and Booles
Rule, the nodes are chosen to be equally spaced. For Gauss-Legendre
quadrature, the nodes are chosen to be zeros of certain Legendre
polynomials. When the integration formula is used to develop a
predictor formula for differential equations, all the nodes are chosen
less than b. For all applications, it is necessary to know something
about the accuracy of the numerical . This leads us to the next
definition.
Definition (Degree of Precision) The degree of precision of a

quadrature formula is the positive integer n such that
for all
polynomials
of degree
, but for which
for some
polynomial
of degree n+1. That is
when degree
and
when degree
The form of
can be anticipated by studying what happens
when f(x) is a polynomial. Consider the arbitrary polynomial
of degree i. If
, then
for
all x, and
for all x. Thus it is not surprising that
the general form for the truncation error term is
(3)
where K is a suitably chosen constant and n is the degree of

precision. The proof of this general result can be found in advanced
books on numerical integration. The derivation of quadrature formulas
is sometimes based on polynomial interpolation. Recall that there
exists a unique polynomial
of degree
, passing through
the m+1 equally spaced points
. When this polynomial
is used to approximate f(x) over [a,b], and then the integral
of f(x) is approximated by the integral of
, the resulting
formula is called a Newton-Cotes quadrature formula. When the
sample points
and
are used, it is called a closed NewtonCotes formula. The next result gives the formulas when approximating
polynomials of degree
are used.
Theorem (Closed Newton-Cotes Quadrature Formula) Assume

that
are equally spaced nodes and
. The first
four closed Newton-Cotes quadrature formulas:
(4) Trapezoidal Rule
(5) Simpsons Rule
(6) Simpson 3/8 Rule

(7) Booles Rule
Corollary (Newton-Cotes Precision) Assume that f(x) is
sufficiently differentiable; then E[f] for Newton-Cotes quadrature
involves an appropriate higher derivative.
(8) The trapezoidal rule has degree of precision n=1. If
then
.
(9) Simpsons rule has degree of precision n=3. If
, then
.
(10) Simpsons
rule has degree of precision n=3. If
, then
.
(11) Booles rule has degree of precision n=5. If
, then
.
, the equally
spaced quadrature nodes
,
,
,
, and
, and the corresponding function values
,
,
,
, and
. Apply the various
quadrature formulas (4) through (7).
Trapezoidal Rule
Simpsons 3/8 Rule
Simpsons Rule
Booles Rule
Trapezoidal Rule for Numerical Integration
Theorem (Trapezoidal Rule) Consider
. The trapezoidal rule is
, where
over
.
over
.
The remainder term for the trapezoidal rule is
lies somewhere between
, where
Composite Trapezoidal Rule

An intuitive method of finding the area under a curve y = f(x) is by
approximating that area with a series of trapezoids that lie above the
intervals
. When several trapezoids are used, we call it
the composite trapezoidal rule.
Theorem (Composite Trapezoidal Rule) Consider

is subdivided
into m subintervals
of equal width
the equally spaced nodes
for
composite trapezoidal rule for m subintervals is
over
by using
The
.
and we write
over
Remainder term for the Composite Trapezoidal Rule

Corollary (Trapezoidal Rule: Remainder term) Suppose that
is subdivided into m subintervals
of
width
The composite trapezoidal rule

.
Furthermore, if
error term
, then there exists a value c with a < c < b so that the

has the form
.
" notation

be reduced by approximately
the error term
Algorithm Composite Trapezoidal Rule. To approximate the integral
,
by sampling
, where
at the
equally spaced points
. Notice that
and
for
.
should
Or you can use the traditional program.

trapezoidal rule with m = 1, 2, 4, 8, and 16 subintervals.
by using the
Recursive Integration Rules

Theorem (Successive Trapezoidal Rules) Suppose that
and
the points
subdivide
into
subintervals equal
width
relationship
. The trapezoidal rules
obey the
Definition (Sequence of Trapezoidal
Rules) Define
with step size
define
size
, which is the trapezoidal rule

. Then for each
is the trapezoidal rule with step
Corollary (Recursive Trapezoidal Rule) Start

with
rules
. Then a sequence of trapezoidal

is generated by the recursive formula
for
where
The recursive trapezoidal rule is used for the Romberg integration algorithm.
Simpson's Rule for Numerical Integration
The numerical integration technique known as "Simpson's Rule" is credited to the
mathematician Thomas Simpson (1710-1761) of Leicestershire, England. His also
worked in the areas of numerical interpolation and probability theory.
Theorem (Simpson's Rule) Consider

, and
. Simpson's rule is
, where
over
.
over
.
The remainder term for Simpson's rule is
somewhere between
, where lies
Composite Simpson Rule

Our next method of finding the area under a curve
is by
approximating that curve with a series of parabolic segments that lie
above the intervals
. When several parabolas are used,
we call it the composite Simpson rule.
Theorem (Composite Simpson's Rule) Consider

is subdivided into
subintervals
of equal width
equally spaced sample points
for
composite Simpson's rule for
subintervals is
over
by using the
. The
.

we write
over
Remainder term for the Composite Simpson Rule

Corollary (Simpson's Rule: Remainder term)
is subdivided into
subintervals
. The composite Simpson's rule
Suppose that
of width
.
.
Furthermore, if
error term
, then there exists a value with

has the form
so that the
.
" notation

should be reduced by approximately
the remainder term
Algorithm Composite Simpson Rule. To approximate the integral
,
by sampling
for
at the
, where

. Notice that
and
and
Mathematica Subroutine (Simpson Rule). Traditional programming.
Mathematica Subroutine (Simpson Rule). Object oriented programming.

Simpson's rule with m = 1, 2, 4, and 8.
by using
Simpson's 3/8 Rule for Numerical Integration
The numerical integration technique known as "Simpson's 3/8 rule" is credited to the
mathematician Thomas Simpson (1710-1761) of Leicestershire, England. His also
worked in the areas of numerical interpolation and probability theory.
Theorem (Simpson's 3/8 Rule) Consider

over
,
, and
. Simpson's 3/8 rule is
, where
.
over
.
The remainder term for Simpson's 3/8 rule is
somewhere between
, where lies
Composite Simpson's 3/8 Rule

is by
approximating that curve with a series of cubic segments that lie
above the intervals
. When several cubics are used, we
call it the composite Simpson's 3/8 rule.
Theorem (Composite Simpson's 3/8 Rule) Consider

is subdivided into
over
subintervals
of equal width
by using the
for
. The
composite Simpson's 3/8 rule for
subintervals is
.
we write
over
and
Remainder term for the Composite Simpson's 3/8 Rule

Corollary (Simpson's 3/8 Rule: Remainder term)
is subdivided into
subintervals
. The composite Simpson's 3/8 rule
Suppose that
of width
.
.
Furthermore, if
error term

has the form
so that the
.
" notation

the remainder term
Algorithm Composite Simpson's 3/8 Rule. To approximate the integral
,
by sampling
for
at the
, where

. Notice that
and
Mathematica Subroutine (Simpson's 3/8 Rule). Object oriented programming.
Boole's Rule
Theorem (Boole's Rule) Consider
,
, and
. Boole's rule is
over
, where
.
over
.
The remainder term for Boole's rule is
, where lies somewhere between
Composite Boole's Rule

is by
approximating that curve with a series of quartic segments that lie
above the intervals
. When several parabolas are used,
we call it the composite Boole's rule.
Theorem (Composite Boole's Rule) Consider

over
that the interval
is subdivided into
subintervals
. Suppose
of
equal width
by using the equally spaced sample
points
for
. The composite Boole's rule for
subintervals is
.
we write
.
over
and
Remainder term for the Composite Boole's Rule

Corollary (Boole's Rule: Remainder term)
subdivided into
subintervals
composite Boole's rule
Suppose that
of width
. The
.
.
Furthermore, if
error term

has the form
so that the
.
" notation

the remainder term

.
Algorithm Composite Boole's Rule. To approximate the integral
,
by sampling
for
at the
, where

. Notice that
and
Mathematica Subroutine (Boole's Rule). Object oriented programming.
is
Romberg Integration
Background for Romberg Integration. To approximate the
integral
by generating a table of approximations, and using
as
the final answer.
The approximations
are stored in a special lower triangular matrix. The
elements
of the first column are computed using the sequential trapezoidal rule
based on
subintervals of
; then
is computed using Romberg's rule.
Elements of row j are
The algorithm is terminated when
.
.
The subroutine Romberg is "dynamic" in the following sense. At the start, we initialize
the array with the command
and it contains one row and one
element
, in which we place one element
. Next, the
increment command,
is used to make
, and the Append
command,
, is invoked which adds a second row to ,
which is initialized with zeros,
. Then the TrapRule subroutine is called to
perform the sequential trapezoidal rule and fill in the first entry
and Romberg's
rule is used to fill in the second entry
. And so it goes, the sequential trapezoidal
rule is used to fill in the first entry in succeeding rows and Romberg's rule fills in rest of
the entries in that row. The algorithm is terminated when
.
Mathematica Subroutine (Romberg Integration).
Adaptive Simpson's Rule
The adaptive Simpson's rule for quadrature uses the two subroutines "Simpson" and
"Adapt."The program is "recursive".There is no brake available if something goes wrong,
i.e. if a pathological "bad" function is thrown it's way it may proceed on a slippery path
of infinite recursion.
Mathematica Subroutines (Adaptive Simpson's Rule).
Execute the following Mathematica subroutine, which is the "long version" of the
subroutine we have been using previously.
Gauss-Legendre Quadrature
Background. Gauss-Legendre Quadrature.
We wish to find the area under the curve
. What method gives
the best answer if only two function evaluations are to be made? We have already seen
that the trapezoidal rule is a method for finding the area under the curve that uses two
function evaluations at the endpoints (-1,f[-1]), and (1,f[1]). But if the graph of y
= f[x] is concave, the error in approximation is the entire region that lies between the
curve and the line segment joining the points. If we are permitted to use the nodes
and
that lie inside the interval [-1,1], the line through the two points
and
crosses the curve, and the area under the line more closely approximates
the area under the curve. This method is attributed to Johann Carl Friedrich Gauss (17771855) and Adrien-Marie Legendre (1752-1833).
The line through

through
The line
.
The equation of the line through the two points

and
is
(1)
and the area of the trapezoid under this line is

(2)
Notice that the trapezoidal rule

choose
and
, and
is a special case of (2). When we

,
.
The trapezoidal rule is

,
and it exact for straight lines (i.e.
).
If the abscissas
and
=
the Gauss-Legendre 2 point quadrature rule
, and weights
are used, we have
,
which exact for cubic polynomials (i.e.
).
Theorem (Gauss-Legendre Quadrature). An approximation to the integral
is obtained by sampling
at the
unequally spaced abscissas
, where the corresponding weights are
.
The abscissa's and weights for Gauss-Legendre quadrature are often expressed in decimal
form.
n=2 Rule
where
n=3 Rule
where
n=4 Rule
where
n=5 Rule
where
Remark. For ease of reading the above list of rules has used the notation
instead of
and
, respectively.
and
Theorem (Error for Gauss-Legendre Quadrature). The error terms for the rules n = 2,
3, 4 and 5 can be expressed as follows:
n=2 Rule
n=3 Rule
n=4 Rule
n=5 Rule
Mathematica Subroutine (Gauss-Legendre Quadrature).
Cubic Spline Quadrature
Background for Cubic Spline.
Suppose that
. The function
polynomials
the properties:
are n+1 points, where

is called a cubic spline if there exists n cubic
with coefficients
that satisfy
I.
for
II.
for
.
The spline passes through each data point.
III.
for
.
The spline forms a continuous function over [a,b].
IV.
for
The spline forms a smooth function.
IV.
for
The second derivative is continuous.
Natural Spline. There exists a unique cubic spline with the free
boundary conditions
and
.
Cubic Spline Quadrature. Integrate the natural cubic spline over the
interval [a,b].
Algorithm Natural Cubic Spline. To construct and evaluate the cubic
spline interpolant
for the
data
points
, using the free boundary
conditions
and
. Then integrate the natural cubic
spline for a quadrature method.
Mathematica Subroutine (Natural Cubic Spline).

Execute the following large group of cells:
Monte Carlo Pi
Background
We start the familiar example of finding the area of a circle. Figure 1 below shows a
circle with radius
inscribed within a square. The area of the circle is
, and the area of the square is
. The ratio of the area of the circle to the
area of the square is
Figure 1.
If we could compute ratio, then we could multiple it by four to obtain the value
. One particularly simple way to do this is to pick lattice points in the square and count
how many of them lie inside the circle, see Figure 2. Suppose for example that the points
are selected, then there are 812 points inside the circle
and 212 points outside the circle and the percentage of points inside the circle
is
with the following calculation
. Then the area of the circle is approximated
Figure 2.
Monte Carlo Method for

Monte Carlo methods can be thought of as statistical simulation methods that utilize a
sequences of random numbers to perform the simulation. The name "Monte Carlo'' was
coined by Nicholas Constantine Metropolis (1915-1999) and inspired by Stanslaw Ulam
(1909-1986), because of the similarity of statistical simulation to games of chance, and
because Monte Carlo is a center for gambling and games of chance. In a typical process
one compute the number of points in a set A that lies inside box R. The ratio of the
number of points that fall inside A to the total number of points tried is equal to the ratio
of the two areas (or volume in 3 dimensions). The accuracy of the ratio depends on the
number of points used, with more points leading to a more accurate value.
A simple Monte Carlo simulation to approximate the value of
could involve
randomly selecting points
in the unit square and determining the
ratio
, where is number of points that satisfy
of sample size
there were
points satisfying
3. Using this data, we obtain
and
. In a typical simulation
, shown in Figure
Figure 3.
Every time a Monte Carlo simulation is made using the same sample size it will
come up with a slightly different value. The values converge very slowly of the
order
. This property is a consequence of the Central Limit Theorem.
Remark. Another interesting simulation for approximating
is known as Buffon's
Needle problem. The reader can find many references to it in books, articles and on the
internet.
Mathematica Subroutine (Monte Carlo Pi).
Area Under a Curve
Monte Carlo simulation can be used to approximate the area under a
curve
for
. First, we must determine the rectangular box
as follows.
where
Second, randomly pick points

in , where
independent uniformly distributed random variables over
respectively. Third, calculate the ratio as follows:
, where is number of points that lie in
containing
are chosen from

,
The area is computed using the approximation,

formula
, and we can use the
.
An "estimate" for the accuracy of the above computation is
Algorithm Monte Carlo Simulation. Given that

Carlo simulation to approximate the integral
for
. Use Monte
Mathematica Subroutine (Monte Carlo Simulation).
Caveat. The above subroutine is for pedagogical purposes. The standard
implementation of Monte Carlo integration is done by selecting points only in the domain
of the function, for a one dimensional integral we would use random numbers to select
points
in the interval
and then use the approximation
.
This idea will be developed in the module Monte Carlo integration.
Monte Carlo Integration
Background
We will be discussing how to approximate the value of an integral based on the
average of function values. The following concept is useful.
Theorem (Mean Value Theorem for Integrals). If
, then there exists a number , with
, such that
is continuous over
.
This can be written in the equivalent form:
.
Remark. This computation shows that the area under the curve is the base width
times the "average height"
.
Composite Midpoint Rule
An intuitive method of finding the area under a curve
is to
approximate that area with a series of rectangles that lie above the
intervals
.
Theorem (Composite Midpoint Rule). Consider
. Let the interval
be subdivided into
subintervals
of equal width
spaced nodes
rule for n subintervals is
for
. Form the equally

. The composite midpoint
.
This can be written in the equivalent form
,
where
over
Corollary (Remainder term for the Midpoint Rule) The composite
midpoint rule
the integral, and
is an numerical approximation to
.
Furthermore, if
error term
, then there exists a value

has the form
with
so that the
.
" notation
Algorithm Composite Midpoint Rule. To approximate the integral
,
by sampling
, where
at the
equally spaced points
for
Monte Carlo Method

Monte Carlo methods can be thought of as statistical simulation methods that utilize a
sequences of random numbers to perform the simulation. The name "Monte Carlo'' was
coined by Nicholas Constantine Metropolis (1915-1999) and inspired by Stanslaw Ulam
(1909-1986), because of the similarity of statistical simulation to games of chance, and
because Monte Carlo is a center for gambling and games of chance.
Approximation for an Integral

The Monte Carlo method can be used to numerically approximate the value of an
integral. For a function of one variable the steps are:
(i)
Pick n randomly distributed points
(ii)
Determine the average value of the function
in the interval
.
(iii)
Compute the approximation to the integral

.
(iv)
An estimate for the error is
where
Every time a Monte Carlo simulation is made using the same sample size it will come
up with a slightly different value. Larger values of
will produce more accurate
approximations. The values converge very slowly of the order
. This property
is a consequence of the Central Limit Theorem.
Mathematica Subroutine (Monte Carlo for 1 Dimensional Integrals).
The above subroutine is all we need to "do the math." The following subroutine presents
the results in a nice format.
Iterated Integrals in Higher Dimensions
Sometimes we are given integrals which cannot be done analytically, especially in
higher dimensions where the standard methods of discretization can become
computationally expensive. For example, the error in the composite midpoint rule (and
the composite trapezoidal rule) of an d-dimensional integral has the order of
convergence
. We can apply the inequality
when
to see that Monte-Carlo integration will usually converge faster for quintuple multiple
integrals and higher, i.e.
, etc.
Approximation for a Multiple Integral

The Monte Carlo method can be used to numerically approximate the value of a
multiple integrals. For a function of d variables the steps are:
(i) Pick n randomly distributed points
"volume"
(ii)
in the
.
Determine the average value of the function
.
(iii)
Compute the approximation to the integral
.
(iv)
An estimate for the error is
where
Mathematica Subroutine (Monte Carlo for 3 Dimensional Integrals).
2D Trapezoidal and Simpson Rules
Background
The trapezoidal rule and Simpson's rule for ordinary integrals can be extended to
multiple integrals.
Theorem (Trapezoidal 2D Rule) Consider

. Given that the interval
into subintervals
of equal width
for
assume that the interval
is subdivided into
subintervals
of equal width
spaced sample points
for
The composite Trapezoidal rule is
over the rectangle

is subdivided
by using the
. Also,
by using the equally

.
where
It can be shown that the error term is of the form
, that is
Remark. The Trapezoidal rule had the pattern of weights
the Trapezoidal 2D rule extends this pattern to a grid in the rectangle R.
Theorem (Simpson's 2D Rule) Consider

. Given that the interval
into
subintervals
of equal width
for
assume that the interval
is subdivided into
subintervals
of equal width
spaced sample points
for
The composite Simpson's rule is
and
over the rectangle

is subdivided
by using the
. Also,
by using the equally

.
where
It can be shown that the error term is of the form
, that is
.
Remark. Simpson's rule had the pattern of weights
and Simpson's 2D rule extends this pattern to a grid in the rectangle R.
Mathematica Subroutine (Trapezoidal 2D Rule). Object oriented programming.
Solution of Differential Equations
Euler's Method for ODE's
Taylor Series Method for ODE's
Runge-Kutta Method
Runge-Kutta-Fehlberg Method
Adams-Bashforth-Moulton Method
Milne-Simpson's Method
Shooting Methods for ODE's
Finite Difference Method for ODE's
Galerkin's Method
Painleve Property
Lotka-Volterra Model
Pendulum
Projectile Motion
Lorenz Attractor
van der Pol System
Harvesting Model
Frobenius Series Solution
Picard Iteration
Spring-Mass Systems
Euler's Method for ODE's
The first method we shall study for solving differential equations is called Euler's
method, it serves to illustrate the concepts involved in the advanced methods. It has
limited use because of the larger error that is accumulated with each successive
step.However, it is important to study Euler's method because the remainder term and
error analysis is easier to understand.
Theorem(Euler's Method)
Assume thatf(t,y)is continuous and satisfies a Lipschits condition in the
variabley,and consider theI. V. P. (initial value problem)
with
,over the interval
.
Euler's method uses the formulas
,and
for
as an approximate solution to the differential equation using the discrete set of points
.
Error analysis for Euler's Method
When we obtained the formula
for Euler's method, the neglected term for each step
has the form
.If this was the only error at each step, then at the end of
the interval
, after
steps have been made, the accumulated error would be
.
The error is more complicated, but this estimate predominates.
Theorem (Precision of Euler's Method)

Assume that
is the solution to the I.V.P.
with
.If
and
is the sequence of approximations generated by Euler's
method, then at each step, the local trunctaion error is of the order
,and the overall global truncation error
is of the order
,for
.
The error at the right end of the interval is called the final global error
.
Remark.The global truncation error
is used to study the behavior of the error for various step
sizes.It can be used to give us an idea of how much computing effort
must be done to obtain an accurate approximation.
Numerical methods used in this module.Use Euler's method and the modified Euler's
method. Construct numerical solutions of order
and
, respectively.The theory for the modified Euler method is not presented at this
time, we are to trust that its development is similar, but the order for the error is better
and is known to be
.
Algorithm (Euler's Method).To approximate the solution of the initial value
problem
with
over
at a discrete set of points using the formulas
,and
for
.
Mathematica Subroutine (Euler's Method).
Algorithm (Modified Euler's Method).To approximate the solution of the

initial value problem
with
over
,and
for
.
Mathematica Subroutine (Modified Euler's Method).
Taylor Series Method for ODE's
The Taylorseries method is of general applicability and it is the standard to which we can
compare the accuracy of the various other numerical methods for solving an I. V. P.It can
be devised to have any specified degree of accuracy.
Theorem(Taylor SeriesMethod of Order n)

Assume thatf(t,y)is continuous and satisfies a Lipschits condition in the variabley,and
consider theI. V. P. (initial value problem)
with
,over the interval
.
The Taylor series method uses the formulas
,and
for
where
is
evaluated at
,as an approximate solution to the differential equation using the discrete set of points
.
Theorem(Precisionof Taylor SeriesMethod of Order n)

Assume that
with
.If
and
is the sequence of approximations generated by the Taylor
series method of ordern, then at each step, the local truncation error is
of the order
,and the overall global truncation error
is of the order
,for
.
The error at the right end of the interval is called the final global error
.
Algorithm (Taylor Series Method).

To compute a numerical approximation for the solution of the initial value problem
with
over
,and
,for
where
is
evaluated at
.
Mathematica Subroutine (Taylor SeriesMethod of Order n=4).
Runge-Kutta Method
Theorem (Runge-Kutta Method of order 4) Assume that f(t,y) is
continuous and satisfies a Lipschits condition in the variable y, and consider the I. V. P.
(initial value problem)
with
, over the interval
.
The Runge-Kutta method uses the formulas
, and
for
where
.
Theorem (Precision of the Runge-Kutta Method of Order

4) Assume that
with
. If
and
is the sequence of approximations generated by the
Runge-Kutta method of order 4, then at each step, the local truncation
error is of the order
, and the overall global truncation error
is of the order
, for
.
The error at the right end of the interval is called the final global
error
.
Algorithm (Runge-Kutta Method). To compute a numerical
approximation for the solution of the initial value problem
with
over
at a discrete set of points using the formula
, for
where
,
,
, and
.
Mathematica Subroutine (Runge-Kutta Method of Order 4).
Runge-Kutta-Fehlberg Method
Runge-Kutta-Fehlberg Method (RKF45)
GATEGATEGATEGATEOne way to guarantee accuracy in the solution of
an I.V.P. is to solve the problem twice using step sizes h and
and compare answers at the mesh points corresponding to the
larger step size.GATEGATEBut this requires a significant amount of
computation for the smaller step size and must be repeated if it is
determined that the agreement is not good enough. The Runge-KuttaFehlberg method (denoted RKF45) is one way to try to resolve this
problem.GATEGATEIt has a procedure to determine if the proper step
size h is being used.GATEGATEAt each step, two different
approximations for the solution are made and compared.GATEGATEIf
the two answers are in close agreement, the approximation is
accepted. If the two answers do not agree to a specified accuracy, the
step size is reduced.GATEGATEIf the answers agree to more significant
digits than required, the step size is increased.
GATEGATEGATEGATEEach Runge-Kutta-Fehlberg step requires the use
of the following six values:
GATEGATEGATE
Then an approximation to the solution of the I.V.P. is made using a
Runge-Kutta method of order 4:
GATEGATE
And a better value for the solution is determined using a Runge-Kutta
method of order 5:
The optimal step size sh can be determined by multiplying the scalar s

times the current step size h. The scalar s is
where
is the specified error control tolerance.
Mathematica Subroutine (Runge-Kutta-Fehlberg Method
RFK45)GATEGATETo compute a numerical approximation for the
solution of the initial value problem
with
over
at a discrete set of points with an error controlGATEGATE
.GATEGATEStart with the initial pointGATEGATE
GATEGATEand generate the sequence of approximationsGATEGATE
.GATEGATEThe step size is automatically adjusted.GATEGATEThus the
number
is determined by the subroutine.GATEGATEAt each step the final answer is computed
using the formulaGATEGATEGATE
.GATEGATE
Remark.GATEGATEThe formula for adjusting the step size is a popular
choice for numerical analysis textbooks.GATEGATE
Remark.GATEGATESome textbooks are now promoting the use of the
RK order five formulaGATEGATE
GATEGATEas the accepted value instead ofGATEGATE
GATEGATEfrom the order four formula.GATEGATEThe following
subroutine includes these adjustments.GATEGATE
GATE
Mathematica Subroutine (Runge-Kutta-Fehlberg Method
RFK54)GATEGATETo compute a numerical approximation for the
with
over
at a discrete set of points with an error controlGATEGATE
.GATEGATEStart with the initial pointGATEGATE
GATEGATEand generate the sequence of approximationsGATEGATE
.GATEGATEThe step size is automatically adjusted.GATEGATEGATEThus
the number
is determined by the subroutine.GATEGATEAt each step the final answer is computed
using the formula
.GATEGATE
Remark.GATEGATEOne might question the wisdom of using the RKF54
instead of the RKF45, it is still being debated.GATEGATEIn some sense
the "guarantee" of accuracy is diminished.GATEGATEThe RKF54 is
given for you to explore.
Adams-Bashforth-Moulton Method
The methods of Euler, Heun, Taylor and Runge-Kutta are called single-step methods
because they use only the information from one previous point to compute the successive
point, that is, only the initial point
is used to compute
and in
general
is needed to compute
. After several points have been found it is
feasible to use several prior points in the calculation. The Adams-Bashforth-Moulton
method uses
in the calculation of
. This method is not selfstarting; four initial points
,
,
, and
must be given in
advance in order to generate the points
.
A desirable feature of a multistep method is that the local truncation error (L. T. E.) can
be determined and a correction term can be included, which improves the accuracy of the
answer at each step. Also, it is possible to determine if the step size is small enough to
obtain an accurate value for
, yet large enough so that unnecessary and timeconsuming calculations are eliminated. If the code for the subroutine is fine-tuned, then
the combination of a predictor and corrector requires only two function evaluations
of f(t,y) per step.
Theorem (Adams-Bashforth-MoultonMethod) Assume

that f(t,y) is continuous and satisfies a Lipschits condition in the variable y, and
consider the I. V. P. (initial value problem)
with
, over the interval
The Adams-Bashforth-Moulton method uses the formulas

the predictor
.
, and
, and
the corrector
for
as an approximate solution to the differential equation using the discrete set of
points
.
Remark. The Adams-Bashforth-Moulton method is not a self-starting method. Three
additional starting values
must be given. They are usually computed using
the Runge-Kutta method.
Theorem (Precision of Adams-BashforthMoultonMethod) Assume that
is the solution to the
I.V.P.
with
. If
and
is
the sequence of approximations generated by Adams-Bashforth-Moulton
method, then at each step, the local truncation error is of the order
is of the order
, for
error
.
Algorithm (Adams-Bashforth-Moulton Method). To approximate the solution of
the initial value problem
with
over
at a
discrete set of points using the formulas:
use the predictor
and the corrector
for
Mathematica Subroutine (Adams-Bashforth-Moulton Method).
Milne-Simpson's Method
The methods of Euler, Heun, Taylor and Runge-Kutta are called single-step methods
because they use only the information from one previous point to compute the successive
point, that is, only the initial point
is used to compute
and in general
is needed to compute
. After several points have been found it is feasible to use several prior points in the
calculation. The Milne-Simpson method uses
in the calculation of
. This method is not self-starting; four initial points
,
,
, and
must be given in advance in order to generate the points
.
A desirable feature of a multistep method is that the local truncation error (L. T. E.) can
be determined and a correction term can be included, which improves the accuracy of the
answer at each step. Also, it is possible to determine if the step size is small enough to
obtain an accurate value for
, yet large enough so that unnecessary and time-consuming calculations are
eliminated. If the code for the subroutine is fine-tuned, then the combination of
a predictor and corrector requires only two function evaluations of f(t,y) per step.
Theorem (Milne-Simpson's Method) Assume that f(t,y) is continuous and
satisfies a Lipschits condition in the variable y, and consider the I. V. P. (initial value
problem)
with
, over the interval
.
The Milne-Simpson method uses the formulas
, and
the predictor
, and
the corrector
for
.
Remark. The Milne-Simpson method is not a self-starting method. Three additional
starting values
must be given. They are usually computed using the Runge-Kutta method.
Theorem (Precision of the Milne-Simpson Method) Assume
that
with
. If
and
is the sequence of approximations generated by MilneSimpson method, then at each step, the local truncation error is of the order
is of the order
, for
.
error
Algorithm ((Milne-Simpson's Method). To approximate the

with
over
at a discrete set of points using the formulas:
use the predictor
,
and the corrector
for
.
Mathematica Subroutine (Milne-Simpson's Method).
Shooting Methods for ODE's
Background for Boundary Value Problems
Another type of differential equation has the form
(1)
for
with the boundary conditions
with
This is called a boundary value problem. The conditions that
guarantee that a solution to (1) exists should be checked before any
numerical scheme is applied; otherwise, a list of meaningless output
may be generated. The general conditions are stated in the following
theorem.
Theorem (Boundary Value Problem). Assume that

is continuous on the region
and that
and
are continuous on
. If there exists a constant
for which
satisfy
and
,
then the boundary value problem
with
has a unique solution

.
The notation
has been used to distinguish the third variable of the
function
. Finally, the special case of linear differential equations is
worthy of mention.
Corollary (Linear Boundary Value Problem). Assume that

in the theorem has the form
and that f and its partial derivatives
and
are continuous on
for which p(t) and q(t) satisfy
and
,
then the linear boundary value problem
with
.
We are all familiar with the differential equation
and its general solution
. The boundary conditions with
can only be solved if
. Unfortunately, because of this counter example, the "theory" which "guarantees" a
solution must be phrased with
. A careful reading of the "theory" reveals that this is a
sufficient condition and not a necessary condition. Indeed there are many problems that
can be solved with the "shooting method" , all we ask is to be cautious with its
implementation and take note that it might not apply sometimes.
Reduction to Two I.V.P.s: Linear Shooting Method

Finding the solution of a linear boundary problem is assisted by the
linear structure of the equation and the use of two special initial value
problems. Suppose that u(t) is the unique solution to the I.V.P.
with
.
Furthermore, suppose that v(t) is the unique solution to the I.V.P.
with
.
Then the linear combination
.
is a solution to
with
.
Program (Linear Shooting Method). To approximate the solution of
the boundary value problem
with
over the interval [a,b] by using the Runge-Kutta method of order
n=4.
The method involves solving a two systems of equations over
. First solve
with
,
and
.
Then solve
with
,
and
.
Finally, the desired solution x(t) is the linear combination
.
The subroutine Runge2D will be used to construct the two solutions
, and
.
Theory of computation.
What should the "theory" really say? "Existence theory" needs for numerical analysis
needs to be "computational theory." We really need to be guaranteed that two "linearly
independent" solutions u(t) and v(t) given above can be computed. In practice, if
then we need to compute
.
Background
Theorem (Boundary Value Problem). Assume that
is continuous on the region
and that
and
are continuous on
for which
satisfy
and
,
then the boundary value problem
with
.
The notation
has been used to distinguish the third variable of the
function
. Finally, the special case of linear differential equations is
worthy of mention.
Corollary (Linear Boundary Value Problem). Assume that

in the theorem has the form
and that f and its partial derivatives
and
are continuous on
for which p(t) and q(t) satisfy
and
,
then the linear boundary value problem
with
.
Finite-Difference Method
Methods involving difference quotient approximations for derivatives
can be used for solving certain second-order boundary value
problems. Consider the linear equation
(1)
over [a,b] with
. Form a partition of [a, b] using the points
, where
and
for
. The central-difference formulas discussed in Chapter 6
are used to approximate the derivatives
(2)
and
(3)
Use the notation

for the terms
on the right side of (2) and (3) and drop the two terms
. Also, use the notations
,
, and
this produces the difference equation
which is used to compute numerical approximations to the differential

equation (1). This is carried out by multiplying each side by
and then collecting terms involving
and arranging them in a system of linear equations:
for
, where
and
. This system has the familiar tridiagonal form.
We are all familiar with the differential equation
and its general solution
. The boundary conditions with
can only be solved if
. Unfortunately, because of this counter example, the "theory"
which "guarantees" a solution must be phrased with "
." A careful reading of the "theory" reveals that
this is a sufficient condition and not a necessary condition. Indeed
there are many problems that can be solved with the "shooting
method" , all we ask is to be cautious with its implementation and take
note that it might not apply sometimes.
Program (Finite-Difference Method). To approximate the solution
of the boundary value problem
with
and
over the interval
by using the finite difference method of order
. The mesh we use is
and the solution points are
.
Procedures.
(i) Construct the tri-diagonal matrix and vector.
(ii) Solve the system in (i).
(iii) Join the mesh points and vector from (ii) to form the solution
points.
Galerkin's Method
Background
To start we need some background regarding an the inner product.
Definition (Inner Product). Consider the vector space of real functions whose domain
is the closed interval
. We define the inner product of two functions
as follows
.
Remark. The inner product is a continuous infinite dimensional analog to the ordinary
dot product that is studied in linear algebra. If the inner product is zero then
are said to be orthogonal to each other on
. All the functions we use are assumed to be square-integrable, i. e.
.
Mathematica Function (Inner Product). To compute the inner product of two real
functions over
.
Lemma. If
for any function
, then
.
Basis for a Vector Space
A complete basis for a vector space V of functions is a set of linear independent
functions
which has the property that any
can be written uniquely as a linear combination
.
For example, if V the set of all polynomials and power series, then a complete basis is
.
Property. If
and
all
then
.
We mention these concepts without proof so as to provide a little background.
Weighted Residual Methods

A weighted residual method uses a finite number of functions
. Consider the differential equation
(1)
over the interval
.
The term
denotes a linear differential operator.
Multiplying (1) by any arbitrary weight function
and integrating over the interval
one obtains
(2)
for any arbitrary
.
Equations (1) and (2) are equivalent, because
is any arbitrary function.
We introduce a trial solution
to (1) of the form
(3)
,
and replace
with
on the left side of (1).
The residual is defined as follows
(4)
The goal is to construct
so that the integral of the residual will be zero for some choices of weight
functions. That is,
will partially satisfy (2) in the sense that
(5)
for some choices of
.
Galerkin's Method
One of the most important weighted residual methods was invented by the Russian
mathematician Boris Grigoryevich Galerkin (February 20, 1871 - July 12,
1945). Galerkin's method selects the weight function functions in a special way: they are
chosen from the basis functions, i.e.
. It is required that the following
equations hold true
(6)
for
.
To apply the method, all we need to do is solve these
equations for the coefficients
.
Galerkin's Method for solving an I. V. P.
Suppose we wish to solve the initial value problem
(i)
,
with
over the interval
.
We use the trial function
(ii)
.
There are
equations to solve
for
, i.e.
(iii)
for
.
Remark. For the solution of an I. V. P. we choose
.
Galerkin's Method for solving an a B. V. P.
Suppose we wish to solve a boundary value problem over the interval
,
(I)
,
with
We define
and use the trial function
(II)
.
There are
equations to solve
for
, i.e.
(III)
for
.
Remark. The functions
must all be chosen with the boundary properties
and
for
.
Painleve Property
Background
If you are tired of your Runge-Kutta solution "blowing up" at a
singularity, then this module could help remedy the situation.
Definition (Singularity). A singularity of
is a point
at which
"blows up" or is not defined.
Definition (Removable Singularity). A singular point
is called a removable singularity if
has a Taylor series expansion about
, i.e. if
has a representation of the form
valid for
.
A common situation is that
is not defined and we only need to define
in order to "remove the singularity."
Definition (Pole). A singular point
is called a pole if
has a series expansion about
which includes only a finite number of negative powers
with
, i.e. if
has a representation of the form
valid for
.
The leading coefficient must be non-zero,
, and we say that
has a pole of order
at
. When
we say it has a simple pole at
.
Remark. When you look at the graph
, a pole at
is a vertical asymptote at
.
Theorem (Poles and Zeros). If
has a pole at
then the function
has a removable singularity at
. If we define
then the equation
will have a root at
.
Definition (Logarithmic Singularity). A logarithmic singularity
involves a logarithmic branch point in the complex plane.
For example, the function
has a logarithmic singularitie at the point
.
Definition (Algebraic Branch Point). A algebraic branch point is a
singular point associated with a fractional power.
For example, the "multivalued function"
has algebraic branch point at
.
Restriction
The Painlev property excludes the occurance of logarighmic branch
points and algebraic branch points. The underlying solution must be
analytic except at isolated points where it has poles. It is not
necessary to dwell on the above definitions, but it is important to know
that we are restricting the type of singularities we want to allow.
Definition (Movable Singularity). If the singularities of a
differential equation depend on the initial conditions then they are said
to be movable singularities.
Definition 1. (Painlev Property) The second-order ordinary
differential equation
has the Painlev property if all movable singularities of
all solutions are poles.
Remark. We will take the liberty to extend this concept to first order
equations.
Definition 2. (Painlev Property) The first-order ordinary
differential equation
has the Painlev property if all movable singularities of all
solutions are poles.
Remark. Movable singularities depend on initial conditions and in
general it is difficult to predict their location. The following examples
have been chosen because the analytic solution can be found.
Computed Solution Curves for Differential Equations
An important problem in numerical analysis is to compute
approximate solutions of the differential equation
(1)
.
Under modest (and well known) assumptions on f, the "general
solution" of (1) consists of an infinite family of functions, each of which
may be distinguished by selection of an initial point
. Starting from this initial point, numerical methods attempt
to approximate the solution
on some specified interval
. Continuity of
does not ensure the continuity of
.
Suppose that
has an infinite discontinuity at
, that is
. Then the reciprocal
tends to zero as
, and
will have a removable singularity at
provided that we define
. We can use the change of variable
(2)
.
Now differentiate each side of (2) and get
Then substitute
from (1) and obtain
(3)
Differential equation (3) is equivalent to (1) in this sense: Given a

neighborhood N of
and a number
, equation (1) has a solution with
and
for all x in N if and only if equation (3) has a solution with
and
.
We call equation (3) the companion differential equation and write it
as
(4)
.
Numerical methods "track" a specific solution curve through the
starting point
. The success of using (4) for tracking the solution
near a singularity is the fact that
as
if and only if
as
. A numerical solution
to (4) can be computed over a small interval containing
, then (2) is used to determine a solution curve for (1) that lies on
both sides of the vertical asymptote
.
A procedure such as the Runge-Kutta method, uses a fixed step
size
and for each
an approximation
is computed for
. If
as
then the numerical method fails to follow the true solution
accurately because of the inherent numerical instability of computing a
"rise" as the product of a very large slope and very small "run" (a
computation which magnifies the error present in the value
). One way to reduce this error is to select a bound B and change
computational strategy as soon as a value
is computed for which
, that is, as soon as the possibility of a singularity is
"sensed." Then we stop using (1) and start with the point
as an initial value to the differential equation (4). Then
proceed to track the reciprocal
, which will not suffer from the difficulties created by steep
slopes.
The following strategy can be employed to extend any single-step
numerical method. We use equation (1) and the initial value
and compute
where
for
and
.
Then switch equations and use (4) with the initial value
and compute
where
for
and
.
Continue in a similar fashion and alternate between formula (1) and
formula (4) until
The decision process, for the "extended" Runge-Kutta method is:
IF
THEN
Perform one Runge-Kutta step using
to compute
,
ELSE
Set
and perform one Runge-Kutta step using
to compute
,
and keep track of
.
ENDIF
Before (4) is used for numerical computations, the formula for
should be simplified in advance so that the "
" or "
" computational problems do not occur.
Lotka-Volterra Model
Background
Math-Model (Lotka-Volterra Equations) The "Lotka-Volterra equations" refer to two
coupled differential equations
There is one critical point which occurs when

and it is
.
The Runge-Kutta method is used to numerically solve O.D.E.'s over an interval
.
Mathematica Subroutine (Runge-Kutta Method) To compute a
numerical approximation for the solution of the initial value problem
with
over the interval
,
where
,
,
, and
.
Extension to 2D. The Runge-Kutta method is easily extended to solve a system of D.E.'s
over the interval
.
Mathematica Subroutine (Runge-Kutta Method in 2D space) To
compute a numerical approximation for the solution of the initial value
problem
with
,
with
,
over the interval
at a discrete set of points.
Note. The Runge-Kutta method in 2D is a "vector form" of the one-dimensional method,

here the function f is replaced with F.
Pendulum
The second order D. E. approach
Math-Model (Nonlinear Pendulum) A simple pendulum consists of a point
mass m attached to a rod of negligible weight. The torque
is
(1)
where
denotes the the angle of the rod measured downward from a vertical axis. The
moment of inertia for the point mass is
where l is the length of the rod. The
torque can also be expressed as
, where
is the angular acceleration, using
Newton's second law, and the second derivative, this can be written as
(2)
Equating (1) and (2) results in the nonlinear D. E.

(3)
Math-Model (Linear Pendulum) Introductory courses discuss the pendulum with small
oscillations as an example of a simple harmonic oscillator. If the angle of oscillation
is small, use the approximation
in equation (3) and obtain the familiar
linear D. E. for simple harmonic motion:
(4)
Using the substitution
, the solution to (4) is known to be
(5)
which has period

becomes
(6)
. When the solution (5) is written with a phase shift, it

.
Mathematica Subroutine (Runge-Kutta Method for second order

D.E.'s) To compute a numerical approximation for the solution of the
with
over the interval
The systems of D. E.'s approach

The pendulum can also be explored using a phase curve in the phase plane. This
requires a method to solve systems of D. E.'s, and our choice will be the Runge-Kutta
method. Plotting several curves will enable us to make a phase portrait which help
understand some of the subtle features of the non-linear pendulum.
Mathematica Subroutine (Runge-Kutta Method in 2D space) To

compute a numerical approximation for the solution of the initial value
problem
with
with
over the interval
,
,

Projectile Motion
Background
In calculus, a model for projectile motion with no friction is considered, and a
"parabolic trajectory" is obtained. If the initial velocity is
and
is the initial angle
to the horizontal, then the parametric equations for the horizontal and vertical
components of the position vector are
(1)
and
(2)
Solve equation (1) for t and get

(2) and the result is
, then replace this value of t in equation
,
which is an equation of a parabola.
The time
required to reach the maximum height is found by solving
,
yields
,
and the maximum height is
.
The time till impact is found by solving
, which yields
and for this model,
. The range is found by calculating
,
:
.
For a fixed initial velocity
when
, the range
is a function of
, and is maximum
Numerical solution of second order D. E.'s
This module illustrates numerical solutions of a second order differential
equation. First, we consider the special case where the projectile is fired vertically along
the y-axis and has no horizontal motion, i. e. x(t)= 0. The effect of changing the
amount of air drag or air resistance is investigated. It is known that the drag force acting
on an object which moves very slowly through a viscous fluid is directly proportional to
the velocity of that object. However, there are examples, such as Millikan's oil drop
experiment, when the drag force is proportional to the square of the velocity. Further
investigations into the situation could involve the Reynolds number.
Math-Models (Projectile Motion I) The following mathematical models are are

considered.
(i).
No air resistance
.
, and
(ii).
Air resistance proportional to velocity
, and
.
(iii).
Air resistance proportional to the square of the velocity
the ascent, and
(iv).
, for
, for the descent, and
Air resistance proportional to the
power of the velocity
Mathematica Subroutine (Runge-Kutta Method for second order

D.E.'s) To compute a numerical approximation for the solution of the
with
over the interval
Lorenz Attractor
Lorenz Attractor
The Lorenz attractor is a set of differential equations which are popular in the field of
Chaos. The equations describe the flow of fluid in a box which is heated along the
bottom. This model was intended to simulate medium-scale atmospheric
convection. Lorenz simplified some of the Navier-Stokes equations in the area of fluid
dynamics and obtained three ordinary differential equations
,
,
.
The parameter p is the Prandtl number,
is the quotient of the Rayleigh
number and critical Rayleigh number and b is a geometric factor. Lorenz is attributed to
using the values
There are three critical points (0,0,0) corresponds to no convection, and the two
points
and
correspond to
steady convection.
The latter two points are to be stable, only if the following equation holds
.
Program (Runge-Kutta Method in 3D space) To compute a
with
with
with
over the interval
,
,
,
Mathematica Subroutine (Runge-Kutta Method in n-dimensions).
Rssler Attractor
In 1976 the Swiss mathematician Otto Rssler was studying oscillations in chemical
reactions and discovered another set of equations with attractor. They also are involved
with the study of Navier-Stokes equations.
,
,
.
Rssler is acclaimed to use the parameters where a = 0.2, b = 0.2, and c =
5.7. Screw chaos occurs when the parameter values are a = 0.343, b =
1.82 and c = 9.75. Rssler wanted to find a minimum system which would exhibit
chaos. This system of equations looks easier than the Lorenz system, but it is harder to
analyze.
The van der Pol Equation

The van der Pol equation is
,
where
is a constant.
When
the equation reduces to
, and has the familiar solution
. Usually the term
in equation (1) should be regarded as friction or resistance, and this is the
case when the coefficient
is positive. However, if the coefficient
is negative then we have the case of "negative resistance." In the age of
"vacuum tube" radios, the "tetrode vacuum tube" (cathode, grid, plate), was used for a
power amplifier and was known to exhibit "negative resistance." The mathematics is
amazing too, and van der Pol, Balthasar (1889-1959) is credited with developing
equation (1). The solution curves exhibits orbital stability.
The van der Pol equation can be written as a second order system
,
and
.
Any convenient numerical differential equation solver such as the
Runge-Kutta method can be used to compute the solutions.
Background. The Runge-Kutta method is used to numerically solve O.D.E.'s over

.
Extension to 2D. The Runge-Kutta method is easily extended to solve a system of D.E.'s
over the interval
.
Program (Runge-Kutta Method in 2D space) To compute a
with
,
with
,
over the interval

Harvesting Model
Background
The exponential model
is used to study uninhibited population growth and
solution is the exponential function
. When the term
is added we obtain the logistic differential equation which is used to model
inhibited population growth or bounded population growth. The logistic differential
equation is
.
One form of the solution is
.
The terms have been carefully determined so that the initial condition is
.
The limiting value L of y(t) is given by
.
The graph is the so called "S-shaped" curve. The choice of parameters
creates the curve shown below.
Harvesting a Logistic Population

When the harvesting term -k is incorporated into into bounded population model we
have
.
There are three solution forms for this differential equation, and they correspond to the
nature of the stationary solutions ( x(t) = c).
Definition(Stationary Points) The stationary points of the D. E.

are solutions where
and are the roots of the characteristic equation
.
The roots are known to be
, and the stationary solutions are
.
Remark. Since x(t) is a real function, there are no stationary solutions when
.
Case (i) If
there is one stationary solution
.
When
, the differential equation has the form
and the solution is
The solution with the initial condition

is
.
If
then
.
If
then function x(t) has a vertical asymptote at
and the population x(t) becomes extinct at some time

(where
), i. e.
.
Case (ii) If
there are two stationary solutions
and
.
When
and the solution is
The two real roots of the characteristic equation

, are
.
is
.
If
then
.
If
then the population x(t) becomes extinct at some time
, i. e.
.
Case (iii) If
there are no stationary solutions.
When
and the solution is

is
.
The function x(t) has a vertical asymptote at
so the population x(t) becomes extinct at some time
(where
.), i.e.
.
Numerical Solutions
If only the graph of the solution is required, and the formula is not needed, then an
efficient way to solve the differential equation is with a numerical method such as
Modified Euler's method or the Runge-Kutta method. The choice of method depends on
the accuracy required. If an accurate table of numerical values is required then the
Runge-Kutta method should be used.
Program (Modified Euler's Method) To compute a numerical
approximation for the solution of the initial value problem
with
over
.
Frobenius Series Solution
Background.
Consider the second order linear differential equation
(1)
.
Rewrite this equation in the form
, then use the substitutions
and
and rewrite the differential equation (1) in the form
(2)
.
Definition (Analytic). The functions

and
are analytic at
if they have Taylor series expansions with radius of convergence
and
, respectively. That is
which converges for

and
which converges for
Definition (Ordinary Point). If the functions
and
are analytic at
, then the point
is called an ordinary point of the differential equation
.
Otherwise, the point
is called a singular point.
Definition (Regular Singular Point). Assume that

is a singular point of (1) and that
and
are analytic at
.
They will have Maclaurin series expansions with radius of convergence
and
, respectively. That is
which converges for

and
which converges for

Then the point
is called a regular singular point of the differential equation (1).
Method of Frobenius.
This method is attributed to the german mathemematican Ferdinand Georg Frobenius
(1849-1917 ). Assume that
is regular singular point of the differential equation
.
A Frobenius series (generalized Laurent series) of the form
can be used to solve the differential equation. The parameter

must be chosen so that when the series is substituted into the D.E. the coefficient of the
smallest power of
is zero. This is called the indicial equation. Next, a recursive equation for the
coefficients is obtained by setting the coefficient of
equal to zero. Caveat: There are some instances when only one Frobenius solution
can be constructed.
Definition (Indicial Equation). The parameter
in the Frobenius series is a root of the indicial equation
.
Assuming that the singular point is
, we can calculate
as follows:
and
The Recursive Formulas.
For each root
of the indicial equation, recursive formulas are used to calculate the unknown
coefficients
. This is custom work because a numerical value for
is easier use.
Application of the Vibrating Drum

The two dimensional wave equation is
,
in rectangular coordinates it is
,
and in polar coordinates it is
.
Consider a drum head that a flexible circular membrane of radius
. Assume that it is struck in the center and this produces radial vibrations only where
the displacement depends only on time
and distance
from the center. Then
satisfies the D.E.
.
Surface equation for the vibrating drum.
The solution we are seeking in Example 7 is
where the boundary condition
requires that
, hence
. Therefore the fundamental solutions to the wave equation for
the drum head is
, for n = 1,2,3.
Picard Iteration
Introduction
The term "Picard iteration" occurs two places in undergraduate mathematics. In
numerical analysis it is used when discussing fixed point iteration for finding a numerical
approximation to the equation
. In differential equations, Picard iteration is a
constructive procedure for establishing the existence of a solution to a differential
equation
that passes through the point
.
The first type of Picard iteration uses computations to generate a "sequence of
numbers" which converges to a solution. We will not present this application, but
mention that it involves the traditional role of computer software as a "number
cruncher."
The goal of this article is to illustrate the second application of Picard iteration; i. e.
how to use a computer to efficiently generate a "sequence of functions" which converges
to a solution. We will see that computer software can perform the more sophisticated
task of "symbol cruncher." For years, see references [3 to 7].
Background
Most differential equations texts give a proof for the existence and uniqueness of the
solution to a first order differential equation. Then exercises are given for performing the
laborious details involved in the method of successive approximations. The concept
seems straightforward, just repeated integration, but students get bogged down with the
details. Now computers can do all the drudgery and we can get a better grasp on how the
process works.
Theorem 1 (Existence Theorem). If both
rectangle
and
solution to the initial value problem (I.V.P.)
are continuous on the

, then there exists a unique
(1)
for all values of x in some (smaller) interval
.
contained in
Picard's Method for D.E.'s

The method of successive approximations uses the equivalent integral equation for (1)
and an iterative method for constructing approximations to the solution. This is a
traditional way to prove (1) and appears in most all differential equations textbooks. It is
attributed to the French mathematician Charles Emile Picard (1856-1941).
Theorem 2 (Successive Approximations - Picard Iteration). The solution to the I.V.P

in (1) is found by constructing recursively a sequence
of functions
, and
(2)
.
Then the solution
(3)
to (1) is given by the limit:

.
Symbolic Analysis
Computer algebra systems are useful for performing the arduous task of repeated
integrations. To perform Picard iteration using Mathematica one must define
, and supply the initial value
, and the number of iterations n. The following
subroutine will compute the first
approximations.
Mathematica Subroutine (Picard Iteration).
Getting the Graphs
If we want to graph several solution, we will need to store them somewhere. The
following version of the Picard subroutine uses the vector . Since the subscript for the
elements of a vector start with
, the notation for the sequence of stored functions will
be
.
Mathematica Subroutine (Vector Form for Picard Iteration).
Extension to First Order Systems in 2D

Suppose that we want to solve the initial value problem for a system of two differential
equations
,
and
(7)
Picard iteration can be used to generate two sequences

and
which converge to the solutions
and
, respectively, see reference [2]. They
are defined recursively by
,
, and
(8)
The sequence of approximations will converge to the solution, i.e.
(9)
Spring-Mass Systems
Background
Consider the system of two masses
and two springs with no external force. Visualize a wall on the left and to the
right a spring , a mass, a spring and another mass. Assume that the spring constants are
. See Figure 1 below.
Figure 1. Coupled masses with spring attached to the wall at the left.
Assume that the masses slide on a frictionless surface and that the functions
denote the displacement from static equilibrium of the masses
, respectively. It can be shown by using Newton's second law and Hooke's law
that the system of D. E.'s for
is
Remark. The eigenfrequencies can be obtained by taking the square root of the
eigenvalues of the matrix
.
More Background
Consider the system of two masses
and three springs with no external force. Visualize a wall on the left and to the
right a spring , a mass, a spring, a mass, a spring and another wall. Assume that the
spring constants are
. See Figure 2 below.
Figure 2. Coupled masses with springs attached to walls at the left and right.
Assume that the masses slide on a frictionless surface and that the functions
denote the displacement from static equilibrium of the masses
, respectively. It can be shown by using Newton's second law and Hooke's law
that the system of D. E.'s for
is
Remark. The eigenfrequencies can be obtained by taking the square root of the
eigenvalues of the matrix
.
Solution of Partial Differential Equations
Finite Difference Method
Crank-Nicolson Method
Elliptic PDE's
Finite Difference Method
Background for Hyperbolic Equations

Wave Equation
As an example of a hyperbolic partial differential equation, we consider the wave
equation
for0 < x < aand0 < t < b,with the boundary conditions
The wave equation models the displacement u of a vibrating elastic

string with fixed ends atx=0andx=a.Although analytic solutions to the
wave equation can be obtained with Fourier series, we use the problem
as a prototype of a hyperbolic
equation.
Program (Finite-Difference Solution for the Wave Equation)

To approximate the solution of the wave equation
over the rectangle
with
,for
,and
,for
.
Crank-Nicolson Method
Background for Parabolic Equations

Heat Equation
As an example of parabolic partial differential equations, we consider the onedimensional heat equation
for0 < x < aand0 < t < b.
with the initial condition
fort = 0and
.
and the boundary conditions
forx = 0and
,
forx = aand
.
The heat equation models the temperature in an insulated rod with
ends held at constant temperatures
and
and the initial temperature distribution along the rod
beingf(x).Although analytic solutions to the heat equation can be
obtained with Fourier series, we use the problem as a prototype of a
parabolic equation for numerical solution.
Program (Forward-Difference method for the heat equation)

To approximate the solution of the heat equation
over the rectangle
with
,for
.and
,for
.
Elliptic PDE's
Background for Elliptic Equations

As examples of elliptic partial differential equations, we consider the Laplace
equation, Poisson equation, and Helmholtz equation. Recall that the
Laplacian of the functionu(x,y)is
.
With this notation, we can write the Laplace, Poisson, and Helmholtz
equations in the following forms:
It is often the case that the boundary values for the functionu(x,y)are
known at all points on the sides of a rectangular regionRin the plane.
In this case, each of these equations can be solved by the numerical
technique known as the finite-difference method.
Program Dirichlet Method for Laplace's equation)

To approximate the solution of the Laplace's equation
over the rectangle
with
,
,for
,and
,for
.It is assumed that
and integersnandmexist so that
.
Power method
Jacobi method
Householder Transformations
Compartment Model
Earthquake Model
Matrix Exponential
Faddeev-Leverrier Method
Hessenberg Factorization
Background
We will now review some ideas from linear algebra. Proofs of the theorems are either left
as exercises or can be found in any standard text on linear algebra.We know how to solve
n linear equations in n unknowns.It was assumed that the determinant of the matrix was
nonzero and hence that the solution was unique. In the case of a homogeneous system
AX = 0, the unique solution is the trivial solution X = 0.If
,there exist nontrivial
solutions to AX = 0. Suppose that
,and consider solutions to the homogeneous
linear
system
A homogeneous system of equations always has the trivial solution

.Gaussian elimination can be used to obtain the
reduced row echelon form which will be used to form a set of
relationships between the variables, and a non-trivial solution.
Example 1.Find the nontrivial solutions to the homogeneous system
Background for Eigenvalues and Eigenvectors

Definition (Linearly Independent).The vectors
to be linearly independent if the equation
are said
implies that
.If the vectors are not linearly
independent they are said to be linearly dependent.
Two vectors in
are linearly independent if and only if they are not
parallel.Three vectors in
are linearly independent if and only if they
do not lie in the same plane.
Definition (Linearly Dependent).The vectors
be linearly dependent if there exists a set of numbers
all zero, such that
are said to
not
Theorem.
The vectors
are linearly dependent if and only if at least one
of them is a linear combination of the others.
A desirable feature for a vector space is the ability to express each vector as s linear
combination of vectors chosen from a small subset of vectors.This motivates the next
definition.
Definition (Basis).Suppose that
is a set ofmvectors
in
.The setS i s called a basis for if for every vector
there exists
a unique set
of scalars
so thatXcan be expressed as the linear
combination
Theorem.In ,any set ofnlinearly independent vectors forms a basis

of .Each vector
is uniquely expressed as a linear combination of
the basis vectors.
Theorem.Let
be vectors in
(i)Ifm>n,then the vectors are linearly independent.

(ii)Ifm=n,then the vectors are linearly dependent if and only if
,where
.
Applications of mathematics sometimes encounter the following
questions:What are the singularities of
,where is a parameter?
What is the behavior of the sequence of vectors
?What are the
geometric features of a linear transformation?Solutions for problems in
many different disciplines, such as economics, engineering, and
physics, can involve ideas related to these equations. The theory of
eigenvalues and eigenvectors is powerful enough to help solve these
otherwise intractable problems.
LetAbe a square matrix of dimensionn nand letXbe a vector of
dimensionn.The productY = AXcan be viewed as a linear
transformation fromn-dimensional space into itself.We want to find
scalars for which there exists a nonzero vectorXsuch that
(1)
that is, the linear transformationT(X) = AXmapsXonto the multiple

.When this occurs, we callXan eigenvector that corresponds to the
eigenvalue , and together they form the eigenpair
forA.In general,
the scalar and vectorXcan involve complex numbers.For simplicity,
most of our illustrations will involve real calculations.However, the
techniques are easily extended to the complex case.The n n identity
matrixIcan be used to write equation (1) in the form
(2)
The significance of equation (2) is that the product of the matrix

and the nonzero vectorXis the zero vector!The theorem of
homogeneous linear system says that (2) has nontrivial solutions if
and only if the matrix
is singular, that is,
(3)
This determinant can be written in the form
(4)
Definition (Characteristic Polynomial).When the determinant in
(4) is expanded, it becomes a polynomial of degree n, which is called
the characteristic polynomial
(5)
There exist exactly n roots (not necessarily distinct) of a polynomial of
degree n.Each root can be substituted into equation (3) to obtain an
underdetermined system of equations that has a corresponding
nontrivial solution vector X.If is real, a real eigenvectorXcan be
constructed. For emphasis, we state the following definitions.
Definition (Eigenvalue).IfAis andn nreal matrix, then
itsneigenvalues
are the real and complex roots of the
characteristic polynomial
.
Definition (Eigenvector).If is an eigenvalue ofAand the nonzero
vectorVhas the property that
thenVis called an eigenvector ofAcorresponding to the eigenvalue

.Together, this eigenvalue and eigenvector V is called an eigenpair
.
The characteristic polynomial
can be factored in the form
where is called the multiplicity of the eigenvalue

multiplicities of all eigenvalues isn;that is,
. The sum of the
.
The next three results concern the existence of eigenvectors.
Theorem (Corresponding Eigenvectors).Suppose thatAis andn
nsquare matrix.
(a)For each distinct eigenvalue there exists at least one
eigenvectorVcorresponding to .
(b)If has multiplicityr,then there exist at mostrlinearly independent
eigenvectors
that correspond to .
Theorem (Linearly Independent Eigenvectors).Suppose thatAis
andn nsquare matrix.If the eigenvalues
are distinct and
are the k eigenpairs, then
linearly independent vectors.
is a set ofk
Theorem (Complete Set of Eigenvectors).Suppose thatAis andn

nsquare matrix.If the eigenvalues ofAare all distinct, then there
existnnearly independent eigenvectors
.
Finding eigenpairs by hand computations is usually done in the
following manner.The eigenvalue of multiplicityris substituted into the
equation
.
Then Gaussian elimination can be performed to obtain the row reduced
echelon form, which will involven-kequations innunknowns, where
.Hence there arekfree variables to choose. The free variables
can be selected in a judicious manner to produceklinearly independent
solution vectors
that correspond to .
Free Variables
When the linear system is underdetermined, we needed to introduce
free variables in the proper location. The following subroutine will
rearrange the equations and introduce free variables in the location
they are needed.Then all that is needed to do is find the row reduced
echelon form a second time.This is done at the end of the next
example.
Power method
Power Method
We now describe the power method for computing the dominant eigenpair.Its extension
to the inverse power method is practical for finding any eigenvalue provided that a good
initial approximation is known.Some schemes for finding eigenvalues use other methods
that converge fast, but have limited precision.The inverse power method is then invoked
to refine the numerical values and gain full precision.To discuss the situation, we will
need the following definitions.
DefinitionIf is an eigenvalue of Athat is larger in absolute value
than anyother eigenvalue, it is called the dominant eigenvalue. An
eigenvector corresponding to is called a dominant eigenvector.
DefinitionAn eigenvectorVis said to be normalized if the coordinate of
largest magnitude is equal to unity (i.e., the largest coordinate in the
vectorVis the number 1).
Remark.It is easy to normalize an eigenvector
forming a new vector
where
by
and
Theorem (Power Method)Assume that the nn matrixAhasndistinct

eigenvalues
and that they are ordered in decreasing
magnitude; that is,
.If is chosen
appropriately, then the sequences
generated recursively by
and
and
where
eigenvector
and
and
and eigenvalue
,will converge to the dominant

,respectively. That is,
Remark.If is an eigenvector and

vector must be chosen.
, then some other starting
Speed of Convergence
In the iteration in the theorem uses the equation
,
and the coefficient of
that is used to form
.Hence, the speed of convergence of
goes to zero in proportion to

to
is governed by the
terms
.Consequently, the rate of convergence is linear.Similarly,
the convergence of the sequence of constants
to is linear.The
Aitken method can be used for any linearly convergent sequence
to form a new sequence,
,
that converges faster. TheAitken can be adapted to speed up the
convergence of the power method.
Shifted-Inverse Power Method

We will now discuss the shifted inverse power method.It requires a good starting
approximation for an eigenvalue, and then iteration is used to obtain a precise
solution.Other procedures such as the QM and Givens method are used first to
obtain the starting approximations.Cases involving complex
eigenvalues, multiple eigenvalues, or the presence of two eigenvalues
with the same magnitude or approximately the same
magnitude will cause computational difficulties and require more
advanced methods.Our illustrations will focus on the case where the
eigenvalues are distinct.The shifted inverse power method is based on
the following three results (the proofs are left as exercises).
Theorem (Shifting Eigenvalues)Suppose that ,Vis an eigenpair
ofA.If is any constant, then
,Vis an eigenpair of the matrix
.
Theorem (Inverse Eigenvalues)Suppose that ,Vis an eigenpair
ofA.If
,then
Theorem (Shifted-Inverse Eigenvalues)Suppose that ,Vis an
eigenpair ofA.If
, then
Theorem (Shifted-Inverse Power Method)Assume that the nn

matrixAhas distinct eigenvalues
and consider the eigenvalue
. Then a constant can be chosen so that
is the dominant
eigenvalue of
.Furthermore, if is chosen appropriately, then
thesequences
and
generated recursively by
and
where
and
,will converge to the dominant
eigenpair , of the matrix
. Finally, the corresponding
eigenvalue for the matrixAis given by the calculation
Remark.For practical implementations of this Theorem, a linear

system solver is used to compute in each step by solving the linear
system
.
Mathematica Subroutine (Power Method).To compute the dominant value and its
associated eigenvector for the nn matrixA.It is assumed that the n eigenvalues have
the dominance property
.
Application to Markov Chains

In the study of Markov chains the elements of the transition matrix are the probabilities
of moving from any state to any other state.A Markov process can be described by a
square matrix whose entries are all positive and the column sums are all equal to 1.For
example, a 33 transition matrix looks like
where
vector is
and
.
The computation
redistributed in the next state.Similarly we see that
.The initial state
shows how the
is
shows how the
is redistributed in the
next state.
and
shows how the
is redistributed in the next
state.
Therefore, the distribution for the next state is
A recursive sequence is generated using the general rule

fork = 0, 1, 2, ... .
We desire to know the limiting distribution
.Since we will also have
we obtain the relation
From which it follows that
Therefore the limiting distributionPis the eigenvector corresponding to the dominant

eigenvalue
.The following subroutine reminds us of the iteration used in the power
method.
Jacobi method
Jacobis Method
Jacobis method is an easily understood algorithm for finding all eigenpairs for a
symmetric matrix. It is a reliable method that produces uniformly accurate answers for
the results. For matrices of order up to 1010, the algorithm is competitive with more
sophisticated ones. If speed is not a major consideration, it is quite acceptable for
matrices up to order 2020. A solution is guaranteed for all real symmetric matrices when
Jacobis method is used. This limitation is not severe since many practical problems of
applied mathematics and engineering involve symmetric matrices. From a theoretical
viewpoint, the method embodies techniques that are found in more sophisticated
algorithms. For instructive purposes, it is worthwhile to investigate the details of Jacobis
method.
Jacobi Series of Transformations

Start with the real symmetric matrix . Then construct the sequence of
orthogonal matrices
as follows:
and
forj = 1, 2, ....
It is possible to construct the sequence
so that
.
In practice we will stop when the off-diagonal elements are close to
zero. Then we will have
.
Remark.Current research by James W. Demmel and Kresimir Veselic (1992) indicate
that Jacobi's method is more accurate than QR.You can check out their research by
following the link in the list of internet resources.The abstract for their research follows
below.
Abstract.We show that Jacobi's method (with a proper stopping criterion) computes
small eigenvalues of symmetric positive definite matrices with a uniformly better relative
accuracy bound than QR, divide and conquer, traditional bisection, or any algorithm
which first involves tridiagonalizing the matrix. In fact, modulo an assumption based on
extensive numerical tests, we show that Jacobi's method is optimally accurate in the
following sense: if the matrix is such that small relative errors in its entries cause small
relative errors in its eigenvalues, Jacobi will compute them with nearly this accuracy. In
other words, as long as the initial matrix has small relative errors in each component,
even using infinite precision will not improve on Jacobi (modulo factors of
dimensionality). ...
Mathematica Subroutine (Jacobi iteration for eigenvectors).

To compute the full set of eigen-pairs
of then by nreal symmetric
matrixA.Jacobi iteration is used to find all eigenvalue-eigenvector pairs.
Householder's Method
Each transformation in Jacobi's method produced two zero off-diagonal
elements, but subsequent iterations might make them nonzero. Hence
many iterations are required to make the off-diagonal entries
sufficiently close to zero.
Suppose that A is a real symmetric matrix.Householder's method is
used to construct a similar symmetric tridiagonal matrix.Then the QR
method can be used to find all eigenvalues of the tridiagonal matrix.
We now develop the method attributed to Alston Scott Householder
(May 5, 1904 - July4, 1993) which is routinely taught in courses in
linear algebra, and numerical analysis. Several several zero offdiagonal elements are created with each iteration, and they remain
zero in subsequent iterations. We start by developing an important
step in the process.
Theorem (Householder Reflection)

If and are vectors with the same norm, there exists an orthogonal
symmetric matrix such that
,
where
and
.
Since
is both orthogonal and symmetric, it follows that

.
Remark.It should be observed that the effect of the mapping

to reflect through the line whose direction is , hence the name
Householder reflection.
is
Corollary (
Householder Matrix)Let be an
matrix, and
vector.If is an integer with
, we can construct a vector
matrix
so that
any
and
(1)
Suppose that is a symmetric
matrix. Then a sequence of
transformations of the form
will reduce to a symmetric tridiagonal
matrix.Let us visualize the process when
.The first transformation
is defined to be
,where is constructed by applying the above
Corollary with the vector being the first column of the matrix .The
general form of is
where the letter

transformation
stands for some element in .As a result, the

does not affect the element
of :
(2)
The element denoted is changed because of premultiplication by ,

and is changed because of postmultiplication by ; since is
symmetric, we have
.The changes to the elements denoted have
been affected by both premultiplication and postmultiplication.Also,
since is the first column of , equation (1) implies that
.
The second Householder transformation is applied to the matrix
defined in (2) and is denoted
,where is constructed by applying
the Corollary with the vector being the second column of the matrix
.The form of is
where stands for some element in .The

identity block in the
upper-left corner ensures that the partial tridiagonalization achieved in
the first step will not be altered by the second transformation
.The outcome of this transformation is
(3)
The elements and were affected by premultiplication and

postmultiplication by .Additional changes have been introduced to
the other elements by the transformation.The third Householder
transformation,
,is applied to the matrix defined in (3) where
the Corollary is used with being the third column of .The form of
is
Again, the
elements of
identity block ensures that

does not affect the
, which lie in the upper
corner, and we obtain
.
Thus it has taken three transformations to reduce
to tridiagonal form.
In general, for efficiency, the transformation

is not performed in
matrix form.The next result shows that it is more efficiently carried out
via some clever vector manipulations.
Theorem (Computation of One Householder Transformation)If
is a Householder matrix, the transformation
is accomplished as
follows.Let
Let
and compute
and
,
then
Reduction to Tridiagonal Form

Suppose that
is a symmetric
matrix.Start with
.
Construct the sequence
for
of Householder matrices, so that

,
where has zeros below the subdiagonal in columns

is a symmetric tridiagonal matrix that is similar to
called Householder's method.
.Then
.This process is
Compartment Model
The Compartment Model I.

The "compartment" model is used to describe the concentration of a dissolved substance
in several compartments in a system.For example, the "three-stage system" consists of
three tanks containing
gallons of solute.Pure water flows at the raterinto the
first tank and is mixed, then flows at the raterinto the second tank and is mixed, then
flows at the raterinto thethird tank and is mixed, and finally flows out of the third tank at
the rater.We define the constants
equation for the system is
.Then the differential
Example 1.Find the solution to the "three-stage system" where
with the initial conditions

.
Plot the solution curves for
for
The Compartment Model II.

Another "compartment" model is used to describe the concentration of a dissolved
substance in several compartments in a system.For example, the "closed" three-stage
system consists of three tanks containing
gallons of solute.The liquid flows at
the raterfrom the first tank and is mixed, then flows at the raterinto the second tank and is
mixed, then flows at the raterinto thethird tank and is mixed, and finally flows out of the
third tank and back into the first tank at the rate r.We define the constants
.Then the differential equation for the system is
Example 2.Find the solution to the "closed" three-stage system where

.
for
The Compartment Model III.

The "five-stage system" consists of five tanks containing
gallons of
solute.Pure water flows at the rate into the first tank and is mixed, then flows at the rate
into the second tank and is mixed, then flows at the rate into successive tanks and is
mixed, and finally flows out of the fifth tank at the rate .
We define the constants
,
differential equation for the system is
, and
.Then the
Remark. For a tractable example, if the size of the tanks is increasing then ordinary
eigenvectors are obtained to form the solution.Otherwise, the system might require
"generalized eigenvectors" which are important, but we do not want to digress in that
direction in this module.
Example 3.Find the solution to the "five-stage system" where

.
for
Earthquake Model
Background.
In the study of earthquake induced vibrations on multistory buildings, the free transverse
oscillations satisfy the equation
(1)
where the forces acting on the i-th floor are

.
Consider a building withn floors each of massmslugs and the horizontal restoring force
ofk tons/footbetween floors. Then this system reduces to the form
(2)
where
Example 1.Consider a building withn = 6floors each of massm = 1250 slugs (weight of
20 tons)
and the horizontal restoring force ofk = 10,000 lb/ft = 5 tons/footbetween floors.
Then
,and this system reduces to the form
Compute the eigenvalues of matrix

,
and the natural frequencies andperiodsPof oscillation of the building.
More Background.
A horizontal earthquake oscillation of amplitude of the form
will produce an
acceleration
, and the opposite internal force on each floor of the building is
.The resulting non-homogeneous system is
,where
Example 2.Solving the above non-homogeneous system for the coefficient

vectorvforX[t].
The vectorvis the solution to the equation
.
Use the earthquake amplitudee = 0.075 ft = 0.9 in.for this example.
Solve the linear system using the parameters
ande = 0.075.
Find the coefficient vectorvand the vectorX[t].Plot the vibrations of each floor.
Example 3.For the above non-homogeneous system the coefficient vectorvis the solution
to the equation
.
Plot the maximum amplitude of oscillation of the floors vs the parameter over the
interval
,
this graph should have six vertical asymptotes corresponding to each value in the table
above.
Then plot the maximum amplitude as a function of the periodpin seconds.
Matrix Exponential
Background for the Fundamental Matrix

We seek a solution of a homogeneous first order linear system of differential
equations.For illustration purposes we consider the
case:
First, write the system in vector and matrix form

.
Then, find the eigenvalues and eigenvectors of the matrix
eigenpairs ofAby
and
,denote the
Assumption.Assume that there are two linearly independent eigenvectors

which correspond to the eigenvalues
, respectively.Then two linearly
independent solution to
are
,and
.
Definition (Fundamental Matrix Solution)The fundamental matrix solution
formed by using the two column vectors
.
(1)
The general solution to
(2)
.
is the linear combination
.
It can be written in matrix form using the fundamental matrix solution

,is
as follows
.
Notation.When we introduce the notation
,
and
The fundamental matrix solution

(3)
or
(4)
can be written as
.
The initial condition

If we desire to have the initial condition
,then this produces the equation
.
The vector of constant
can be solved as follows
.
The solution with the prescribed initial conditions is
.
Observe that
where
following important definition
is the identity matrix.This leads us to make the
Definition (Matrix Exponential)If

is a fundamental matrix solution to
,then the matrix exponential is defined to be
.
Notation. This can be written as
(5)
or
(6)
,
.
Fact.For a
system, the initial condition is

,
and the solution with the initial condition
is
,
or
.
Theorem (Matrix Diagonalization)The eigen decomposition of a
is
square matrix A
,
which exists when A has a full set of eigenpairs
diagonal matrix
for
,and d is the
and
is the augmented matrix whose columns are the eigenvectors of A.
.
Matrix power
How do you compute the higher powers of a matrix ?For example, given
then
,
and
,etc.
The higher powers seem to be intractable!But if we have an eigen decomposition, then
we are permitted to write
and
in general
Fact.For a
matrix this is
which can be simplified
Theorem (Series Representation for the Matrix Exponential)The solution to

is given by the series
,which becomes
and has the simplified form

,
or
.
Let be ann nmatrix.The determination of eigenvalues and eigenvectors
requires the solution of
(1)
where is the eigenvalue corresponding to the eigenvector .The values
must satisfy the equation
(2)
Hence is a root of an nth degree polynomial

we write in the form
(3)
, which
.
The Faddeev-Leverrier algorithm is an efficient method for finding the

coefficients of the polynomial
.As an additional benefit, the
inverse matrix is obtained at no extra computational expense.
Recall that the trace of the matrix ,written
(4)
,is
The algorithm generates a sequence of matrices

traces to compute the coefficients of
,
(5)
Then the characteristic polynomial is given by

and uses their
(6)
In addition, the inverse matrix is given by

(7)
For a proof for equations (6) and (7) see Vera Nikolaevna Faddeeva,
"Computational Methods of Linear Algebra," (Translated From The
Russian By Curtis D. Benster), Dover Publications Inc. N.Y., Date
Published: 1959 ISBN: 0486604241.
Hessenberg Factorization
Background
If A is symmetric then Householder's method can be used to reduce it to a similar
symmetric tridiagonal matrix.If A is nonsymmetric then Householder's method can be
used to reduce it to a similar Hessenberg matrix.These are the preliminary steps that are
made before applying the QR method for finding eigenvalues of .
Definition(Hessenberg Matrix)
An
matrix with
Hessenberg matrix is
for
is called a Hessenberg matrix.The form of a
Definition(Unitary Matrix)
(i) For real matrices, a unitary matrix is a matrix for which
(ii)For complex matrices, a unitary matrix is a matrix for which
.
.
Theorem (Hessenberg Factorization of a Matrix)

There are two cases to consider.
(iii)Given a real matrix , there exists a unitary matrix and Hessenberg matrix
so that
.
(iv) Given a complex matrix , there exists a unitary matrix and Hessenberg matrix
so that
.
Theorem (Hessenberg Factorization of a Symmetric Matrix)Given a real symmetric
matrix , there exists a unitary matrix and tri-diagonal symmetric matrix so that
.
Remark.This is the case that is easiest to illustrate in a first course in numerical methods.
TheoremIf
then
eigenvectors of .
is similar to and the eigenvalues of
Remark.The eigenvectors of
are the same as the
are in general different from the eigenvectors of .
Program (Householder Reduction to Upper-Hessenberg Form).To reduce the

real
matrix to a Hessenberg matrix form by using
Householder transformations.The
following version of the program uses "loops" extensively and is more traditional in
programming structure.It also contains a print statement so that you can watch the
Householder transformations perform their "magic."
Disclaimer.The following subroutine is for pedagogical purposes only, it may not work
for all cases.
Program (Householder Reduction to Upper-Hessenberg Form).To reduce the

real
matrix to a Hessenberg form form by using
Householder transformations.The
following version of the program uses "objects" extensively and is more like "object
oriented programming."
Disclaimer.The following subroutine is for pedagogical purposes only, it may not work
for all cases.
Caveat.The above subroutines will not reduce an

complex matrix to Hessenberg
form.You should use Mathematica's built in procedureHessenbergDecompositionif
you use complex matrices.If you do use Mathematica's built in procedure you will need
to learn how software developers are permitted to dig into Mathematica's kernel and use
its internal mathematical subroutines (i.e. picking Mathematica's brain).
Golden Ratio Search
Fibonacci Search
Quadratic Interpolative Search
Nelder Mead Method
Powell's Method
Steepest Descent - Gradient Search
Newton's Search Method
Golden Ratio Search
Bracketing Search Methods
An approach for finding the minimum of
in a given interval is
to evaluate the function many times and search for a local
minimum. To reduce the number of function evaluations it is
important to have a good strategy for determining where
is to be
evaluated. Two efficient bracketing methods are the golden ratio and
Fibonacci searches. To use either bracketing method for finding the
minimum of
, a special condition must be met to ensure that
there is a proper minimum in the given interval.
Definition (Unimodal Function) The function
on
, if there exists a unique number
is decreasing on
is unimodal
such that
and
is increasing on
Golden Ratio Search

If
is known to be unimodal on
, then it is possible to
replace the interval with a subinterval on which
takes on its
minimum value. One approach is to select two interior points
. This results in
. The condition that
is unimodal
guarantees that the function values
and
are less
than
.
If
, then the minimum must occur in the subinterval

, and we replace b with d and continue the search in the new
subinterval
. If
, then the minimum must occur in the
subinterval
, and we replace a with c and continue the search in
the new subinterval
. These choices are shown in Figure 1
below.
If
, then squeeze from the right and use
the
If
, then squeeze from the left and use
the
new interval
and the four points
.
new
interval
and the four points
.
Figure 1. The decision process for the golden ratio
search.
The interior points c and d of the original interval

, must be
constructed so that the resulting intervals
, and
are
symmetrical in
. This requires that
, and produces
the two equations
(1)
and
(2)
where
,
,
(to preserve the ordering
).
We want the value of r to remain constant on each
subinterval. If r is chosen judicially then only one new
point e (shown in green in Figure 1) needs to be constructed for the
next iteration. Additionally, one of the old interior points (either c or
d) will be used as an interior point of the next subinterval, while the
other interior point (d or c) will become an endpoint of the next
subinterval in the iteration process. Thus, for each iteration only one
new point e will have to be constructed and only one new function
evaluation
, will have to be made. As a consequence, this means
that the value r must be chosen carefully to split the interval of
into subintervals which have the following ratios:
(3)
and
(4)
If
interval
and
and
and only one new function evaluation is to be made in the

, then we must have
.
Use the facts in (3) and (4) to rewrite this equation and then simplify.
,
,
,
.
Now the quadratic equation can be applied and we get
.
The value we seek is
"golden ratio." Similarly, if
that
and it is often referred to as the

, then it can be shown
Algorithm (Golden Ratio Search for a Minimum). To numerically
approximate the minimum of
on the interval
by using a
golden ratio search. Proceed with the method only if
is a
unimodal function on the interval
. Terminate the process
after max iterations have been completed.
Mathematica Subroutine (Golden Ratio Search for a Minimum).
The next example compares the secant method for root-finding with
the golden ratio search method.
Algorithm (Secant Method). Find a root of

two initial approximations
using the iteration
for
given
Fibonacci Search
Bracketing Search Methods
is to be
minimum of
on
is decreasing on
is unimodal
such that
and
is increasing on
Fibonacci Search
In the golden ratio search two function evaluations are made at the
first iteration and then only one function evaluation is made for each
subsequent iteration. The value of
remains constant on each
subinterval and the search is terminated at the
subinterval,
provided that
or
where
are the
predefined tolerances. The Fibonacci search method differs from the
golden ratio method in that the value of
is not constant on each
subinterval. Additionally, the number of subintervals (iterations) is
predetermined and based on the specified tolerances.
The Fibonacci search is based on the sequence of Fibonacci
numbers which are defined by the equations
for
Thus the Fibonacci numbers are
Assume we are given a function
that is unimodal on the
interval
. As in the golden ratio search a value
is
selected so that both of the interior points
will be used in the
next subinterval and there will be only one new function evaluation.
If
, then the minimum must occur in the subinterval

, and we replace
and
and continue the search in
the new subinterval
. If
, then the
minimum must occur in the subinterval
, and we replace
and
and continue the search in the new subinterval
. These choices are shown in Figure 1 below.
If
, then squeeze from the right and
, then squeeze from the left and
use the new interval
.
interval
.
If
use the new
Figure 1. The decision process for the Fibonacci ratio

search.
If
and only one new function evaluation is to be made
in the interval
subinterval
, then we select
for the
. We already have relabeled
and since
we will relabel it by
then we will have

(1)
The ratio
and
.
is chosen so that
and subtraction produces
(2)
And the ratio
is chosen so that
(3)
Now substitute (2) and (3) into (1) and get

(4)
Also the length of the interval

has been shrunk by the factor
. Thus
and using this in (4) produces
(5)
Cancel the common factor

(6)
in (5) and we now have

.
Solving (6) for
(7)
produces
.
Now we introduce the Fibonacci numbers

subscript
following
. In equation (7), substitute
for the
and get the
Reasoning inductively, it follows that the Fibonacci search can be

begun with
and
for
Note that the last step will be

,
thus no new points can be added at this stage (i.e. the algorithm
terminates). Therefore, the set of possible ratios is
.
There will be exactly n-2 steps in a Fibonacci search!
The
subinterval is obtained by reducing the length of
the
subinterval by a factor of
length of the last subinterval will be
. After
steps the
.
If the abscissa of the minimum is to be found with a tolerance of
, then we want
. It is necessary to
use n iterations, where n is the smallest integer such that
(8)
Note. Solving the above inequality requires either a trial and error
look at the sequence of Fibonacci numbers, or the deeper fact that the
Fibonacci numbers can be generated by the formula
.
Knowing this fact may be useful, but we still need to compute all the
Fibonacci numbers
in order to calculate the
ratios
The interior points

and
of the
found, as needed, using the formulas
(9)
(10)
subinterval
are
,
.
Note. The value of n used in formulas (9) and (10) is found using
inequality (8).
Algorithm (Fibonacci Search for a Minimum). To numerically
approximate the minimum of
on the interval
by using a
Fibonacci search. Proceed with the method only if
is a unimodal
function on the interval
. Terminate the process after n
iterations have been completed, where
Mathematica Subroutine (Fibonacci Search for a Minimum).
Each iteration requires the determination of two new interior points,
one from the previous iteration and the second from formula (9) or
(10). When
, the two interior points will be concurrent
in the middle of the interval. In following example, to distinguish the
last two interior points a small distinguishability constant, , is
introduced. Thus when
is used in formula (9) or (10), the
coefficients of
are
or
, respectively.
Quadratic Interpolative Search
Background for Search Methods
is to be
minimum of

on
is decreasing on
is unimodal
such that
and
is increasing on
Minimization Using Derivatives

Suppose that
is unimodal over
minimum at
. Also, assume that
in
. Let the starting value
lie in
the minimum point p lies to the right of
minimum point p lies to the left of
.
and has a unique

is defined at all points
. If
, then
. If
, then the
Background for Bracketing the Minimum

Our first task is to obtain three test values,
(1)
so that
(2)
,
.
Suppose that
; then
and the step size h should be
chosen positive. It is an easy task to find a value of h so that the
three points in (1) satisfy (2). Start with
(provided that
Case (i)
); if not, take
in formula (1)
, and so on.
If (2) is satisfied we are done.
Case (ii)
If
, then
.
We need to check points that lie farther to the right. Double the
step size and repeat the process.
Case (iii)
If
, we have jumped over p and h is too
large.
We need to check values closer to
. Reduce the step size by a
factor of
and repeat the process.
When
, the step size h should be chosen negative and then
cases similar to (i), (ii), and (iii) can be used.
Quadratic Approximation to Find p

Finally, we have three points (1) that satisfy (2). We will use quadratic
interpolation to find
, which is an approximation to p. The
Lagrange polynomial based on the nodes in (1) is
(3)
where
The derivative of
.
is
(4)
Solving
.
in the form
(5)
Multiply each term in (5) by
yields
.
and collect terms involving
This last quantity is easily solved for
.
The value
is a better approximation to p than
. Hence we can replace
with
and repeat the two processes
outlined above to determine a new h and a new
. Continue the
iteration until the desired accuracy is achieved. In this algorithm the
derivative of the objective function
was used implicitly in (4) to
locate the minimum of the interpolatory quadratic. The reader should
note that the subroutine makes no explicit use of the derivative.
Cubic Approximation to Find p

We now consider an approach that utilizes functional evaluations of
both
and
. An alternative approach that uses both
functional and derivative evaluations explicitly is to find the minimum
of a third-degree polynomial that interpolates the objective
function
at two points. Assume that
is unimodal and
differentiable on
, and has a unique minimum at
. Let
. Any good step size h can be used to start the iteration. The Mean
Value Theorem could be used to obtain
and
if
was just to the right of the minimum, then the slope
might be twice
which would mean that
we
do not know how much further to the right
lies, so we can
imagine that
is close to
and estimate h with the formula:
.
Thus
. The cubic approximating polynomial
is expanded
in a Taylor series about
(which is the abscissa of the
minimum). At the minimum we have
the form:
, and we write
(6)
and
in
(7)
The introduction of
in the denominators of (6) and (7) will
make further calculations less tiresome. It is required
that
,
,
, and
. To
find
we define:
(8)
and we must go through several intermediate calculations before we

end up with .
Use use (6) to obtain
Then use (8) to get
Then substitute
and we have
(9)
Use use (7) to obtain
Then use (8) to get
Then substitute
and we have
(10)
Finally, use (7) and write
Then use (8) to get

(11)
Now we will use the three nonlinear equations (9), 10), (11) listed
below in (12). The order of determining the variables will be
(the variable will be eliminated).
(12)
First, we will find

in (12) as follows:
which is accomplished by combining the equation
Straightforward simplification yields

is given by
(13)
, therefore
Second, we will eliminate

by combining the equation in (12) as
follows, multiply the first equation by
and add it to the third
equation
which can be rearranged in the form

Now the quadratic equation can be used to solve for
It will take a bit of effort to simplify this equation into its

computationally preferred form.
Hence,
(14)
Therefore, the value of
is found by substituting the calculated
value of
in (14) into the formula
. To continue the
iteration process, let
and replace and with and ,
respectively, in formulas (12), (13), and (14). The algorithm outlined
above is not a bracketing method. Thus determining stopping criteria
becomes more problematic. One technique would be to require that
, since
.
Nelder Mead Method
Nelder-Mead Method
The Nelder-Mead method is a simplex method for finding a local
minimum of a function of several variables. It's discovery is attributed
to J. A. Nelder and R. Mead. For two variables, a simplex is a triangle,
and the method is a pattern search that compares function values at
the three vertices of a triangle. The worst vertex, where
is
largest, is rejected and replaced with a new vertex. A new triangle is
formed and the search is continued. The process generates a
sequence of triangles (which might have different shapes), for which
the function values at the vertices get smaller and smaller. The size of
the triangles is reduced and the coordinates of the minimum point are
found.
The algorithm is stated using the term simplex (a generalized
triangle in n dimensions) and will find the minimum of a function of n
variables. It is effective and computationally compact.
Initial Triangle
Let
be the function that is to be minimized. To start, we are
given three vertices of a triangle:
, for
. The
function
is then evaluated at each of the three
points:
, for
. The subscripts are then reordered
so that
. We use the notation
(1)
, and
to help remember that is the best vertex,

and is the worst vertex.
is good (next to best),
Midpoint of the Good Side

The construction process uses the midpoint of the line segment
joining and . It is found by averaging the coordinates:
(2)
Reflection Using the Point

The function decreases as we move along the side of the triangle
from to , and it decreases as we move along the side from to
. Hence it is feasible that
takes on smaller values at points that
lie away from on the opposite side of the line between and . We
choose a test point that is obtained by reflecting the triangle
through the side
. To determine , we first find the midpoint of
the side
. Then draw the line segment from to and call its
length d. This last segment is extended a distance d through to
locate the point . The vector formula for is
(3)
Expansion Using the Point

If the function value at is smaller than the function value at ,
then we have moved in the correct direction toward the
minimum. Perhaps the minimum is just a bit farther than the point
. So we extend the line segment through and to the point . This
forms an expanded triangle
. The point is found by moving an
additional distance d along the line joining and . If the function
value at is less than the function value at , then we have found a
better vertex than . The vector formula for is
(4)
Contraction Using the Point

If the function values at and are the same, another point must
be tested. Perhaps the function is smaller at , but we cannot replace
with because we must have a triangle. Consider the two midpoints
and of the line segments
and
, respectively. The point with
the smaller function value is called , and the new triangle is
.
Note: The choice between and might seem inappropriate for the
two-dimensional case, but it is important in higher dimensions.
Shrink Toward
If the function value at is not less than the value at , the points
and must be shrunk toward . The point is replaced with , and
is replaced with , which is the midpoint of the line segment joining
with .
Logical Decisions for Each Step

A computationally efficient algorithm should perform function
evaluations only if needed. In each step, a new vertex is found, which
replaces . As soon as it is found, further investigation is not needed,
and the iteration step is completed. The logical details for twodimensional cases are given in the proof.
Algorithm (Nelder-Mead Search for a Minimum). To approximate

a local minimum of
, where
is a continuous
function of n real variables, and given the
initial starting
points
for
.
Remark. This is also known as the polytope method.
Mathematica Subroutine (Nelder-Mead Search for a
Minimum). To approximate a local minimum of
, where
is a continuous function of 2 real variables, and given
the initial starting points
for
.
Powell's Method
Background - The Taxi Cab Method
Let
be an initial guess at the location of the minimum of the
function
. Assume that the partial derivatives
of the function are not available. An intuitively appealing approach to
approximating a minimum of the function f is to generate the next
approximation
by proceeding successively to a minimum
of f along each of the n standard base vectors. This process can be
called the taxi-cab method and generates the sequence of points
.
Along each standard base vector
the
function f is a function of one variable, and the minimization
of f might be accomplished by the application of either the golden
ratio or Fibonacci searches if f is unimodal in this search direction.
The iteration is then repeated to generate a sequence of points
. Unfortunately, the method is, in general, inefficient due to the
geometry of multivariable functions. But the construction
is involved in the first step of Powell's method, and it is
worthwhile to see how it works. A typical sequence of points generated by the
taxi-cab method is shown in Figure 1 below.
Figure 1. A sequence of points in 2D generated by the taxi-cab method.
Algorithm (Taxi Cab Method for Finding a Minimum). To numerically
approximate a local minimum of
, where f is a continuous
function of n real variables and
by starting with one
point
and using the "taxi-cab" search along the directions of the
standard base vectors
.
Mathematica Subroutine (Taxi Cab Method for Finding a
Minimum). To numerically approximate a local minimum of
, start with and using the "taxi-cab" search.
Notice. To streamline the algorithm we use Mathematica's built in
procedure FindMinimum to perform the line searches along the base
vectors
. In practice alternate methods could be employed.
Powell's Method
The essence of Powell's method is to add two steps to the process
described in the preceding paragraph. The vector
represents,
in some sense, the average direction moved over the n intermediate
steps
in an iteration. Thus the point
is determined
to be the point at which the minimum of the function f occurs along
the vector
. As before, f is a function of one variable along this
vector and the minimization could be accomplished with an application
of the golden ratio or Fibonacci searches. Finally, since the
vector
was such a good direction, it replaces one of the
direction vectors for the next iteration. The iteration is then repeated
using the new set of direction vectors to generate a sequence of
points
. In one step of the iteration instead of a zig-zag path
the iteration follows a "dog-leg" path. The process is outlined below.
Let
function
be an initial guess at the location of the minimum of the
.
Let
for
be the set of standard
base vectors.
Initialize the vectors
for
and use their
transpose
to form the columns of the matrix U as follows:
.
Initialize the counter
(i)
Set
(ii) For
find the value of
and set
(iii) Set
set
(iv)
,
that minimizes
.
for
and
Increment the counter
(v) Find the value of

and set
(vi)
.
that minimizes
Repeat steps (i) through (v) until convergence is achieved.
A typical sequence of points generated by Powell's method is shown

in Figure 2 below.
Figure 2. A sequence of points in 2D generated by Powell's

method.
Algorithm (Powell's Method for Finding a Minimum). To
numerically approximate a local minimum of
, where f is a
continuous function of n real variables and
by
starting with one point
and using the "dog-leg" search along the
directions of the direction vectors
.
Mathematica Subroutine (Powell's Method for Finding a
Minimum). To numerically approximate a local minimum of
, start with and using the "taxi-cab" search.
Notice. To streamline the algorithm we use Mathematica's built in
procedure FindMinimum to perform the line searches along the
direction vectors
. In practice alternate methods could be
employed.
Steepest Descent - Gradient Search
Gradient and Newton's Methods
Now we turn to the minimization of a function
of n variables,
where
and the partial derivatives of
are
accessible.
Steepest Descent or Gradient Method

Definition (Gradient). Let
that
by
exists for
, is the vector
be a function of
such
. The gradient of
(1)
, denoted
Illustrations
Recall that the gradient vector in (1) points locally in the direction of
the greatest rate of increase of
. Hence
points locally in
the direction of greatest decrease
. Start at the point
and
search along the line through
in the direction
. You will arrive at a point
, where a local minimum occurs when
the point
is constrained to lie on the line
. Since partial
derivatives are accessible, the minimization process can be executed
using either the quadratic or cubic approximation method.
Next we compute
and move in the search
direction
. You will come to
, where a local
minimum occurs when
is constrained to lie on the line
. Iteration will produce a sequence,
, of points with the
property
If
then
will be a local minimum
Outline of the Gradient Method
Suppose that
(i)
(ii)
has been obtained.
Evaluate the gradient vector
Compute the search direction
(iii) Perform a single parameter minimization of

on the interval
, where b is large.
This will produce a value
where a local minimum for
. The relation
shows that this is a minimum for
along the search
line
.
(iv)
Construct the next point
(v) Perform the termination test for minimization, i.e.

are the function values
and
sufficiently close and the
distance
small enough ?
Repeat the process.
Algorithm (Steepest Descent or Gradient Method). To
, where f is a
by
and using the gradient method.
Program (Steepest Descent or Gradient Method). To numerically
approximate a local minimum of
, where f is a continuous
function of n real variables and
by starting with
one point
and using the gradient method. If partial derivatives are
accessible, the minimization process can be executed using either the
quadratic or cubic approximation method. For illustration purposes we
use the quadratic approximation method for finding a minimum.
Newton's Search Method
Newton's Method
The quadratic approximation method for finding a minimum of a
function of one variable generated a sequence of second degree
Lagrange polynomials, and used them to approximate where the
minimum is located. It was implicitly assumed that near the
minimum, the shape of the quadratics approximated the shape of the
objective function
. The resulting sequence of minimums of
the quadratics produced a sequence converging to the minimum of the
objective function
. Newton's search method extends this
process to functions of n independent variables:
. Starting at an initial point
, a sequence of second-degree
polynomials in n variables can be constructed recursively. If the
objective function is well-behaved and the initial point is near the
actual minimum, then the sequence of minimums of the quadratics will
converge to the minimum of the objective function. The process will
use both the first- and second-order partial derivatives of the objective
function. Recall that the gradient method used only the first partial
derivatives. It is to be expected that Newton's method will be more
efficient than the gradient method.
Background
of n variables,
where
are
accessible. Although the Newton search method will turn out to have
a familiar form. For illustration purposes we emphasize the two
dimensional case when
. The extension to n dimensions is
discussed in the hyperlink.
Definition (Gradient). Assume that

two variables,
is a function of
, and has partial derivatives
and
. The gradient of
vector function
, denoted by
, is the
Definition (Jacobian Matrix). Assume that

functions of two variables,
, their Jacobian
matrix
is
are
Definition (Hessian Matrix). Assume that

is a
function of two variables,
, and has partial derivatives up to
the order two. The Hessian matrix
is defined as follows:
.
Lemma 1. For
the Hessian matrix
matrix for the two functions
, i. e.
is the Jacobian
Lemma 2. If the second order partial derivatives of

continuous then the Hessian matrix
is symmetric.
are
Taylor Polynomial for f(x,y)

Assume that
is a function of two variables,
, and has partial derivatives up to the order two. There are two ways
to write the quadratic approximation to f(x,y), based on series and
matrices, respectfully. Assume that the point of expansion is
and use the notation
and
, then
.
The Taylor polynomial using series notation is
The Taylor polynomial using vector and matrix notation is
The latter can be written in the form

.
Using vector notations
and
it looks like
.
The above formula is also the expansion of

point
with
.
centered at the
Newton's Method for Finding a Minimum

of n variables,
where
are
accessible. Assume that the first and second partial derivatives
of
exist and are continuous in a region
containing the point
, and that there is a minimum at the point
. The quadratic polynomial approximation to
is
A minimum of
occurs where
Using the notation
and
symmetry of
, we write
The gradient
and the
is given by the calculation
Thus the expression for
can be written as
.
If
is close to the point
(where a minimum
of f occurs), then
is invertible the above equation can be
solved for
, and we have
.
This value of
can be used as the next approximation to
first step in Newton's method for finding a minimum
and is the
Lemma If column vectors are preferred over row vectors, then

is given by the computation
.
Remark. This is equivalent to a Newton-Raphson root finding
problem: Given the vector function
find the root of
the equation
. For this problem the NewtonRaphson formula is known to be
,
where our previous work with Newton-Raphson method used column
vectors
and
. Here we use
gives the relationship
and Lemma 1
.
Outline of the Newton Method for Finding a Minimum
Start with the approximation

Set
.
(i)
(ii)
(iii)
Evaluate the gradient vector

Compute the next point
to the minimum point
and Hessian matrix

.
Perform the termination test for minimization. Set
Repeat the process.

In equation (ii) the inverse of the Hessian matrix is used to solve
for
. It would be better to solve the system of linear equations
represented by equation (ii) with a linear system solver rather than a
matrix inversion. The reader should realize that the inverse is
primarily a theoretical tool and the computation and use of inverses is
inherently inefficient.
Algorithm (Newton's Method for Finding a Minimum). To

, where f is a
by
and using the Newton method search for a
minimum.
Program (Newton's Method for Finding a Minimum). To
, where f is a
by
and using the Newton method search for a
minimum. For illustration purposes we propose the following
subroutine for
variables.

Chapter7 Numerical Analysis

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Chapter7 Numerical Analysis

Uploaded by

Copyright:

Available Formats

http://www.amaderforum.

CHAPTER 7. NUMERICAL ANALYSIS

Calculus and Fundamentals

The Solution of Nonlinear Equations f(x) = 0

The Solution of Linear Systems AX = B

Interpolation and Polynomial Approximation

Solution of Differential Equations

Solution of Partial Differential Equations

Eigenvalues and Eigenvectors

Calculus and Fundamentals

Big "O" Truncation Error

Big "O" Truncation Error

The big Oh notation provides a useful way of describing the rate of

be two sequences.The sequence

Theorem (Big "O" Remainders for Series Approximations).

for sufficiently smallh.Hence the notation

Theorem (Taylor polynomial).

and its derivatives

where depends on and lies somewhere between

The following example illustrates the theorems above.The

Order of Convergence of a Sequence

.We say that

The Origin of Complex Numbers

Section 1.1The Origin of Complex Numbers

To illustrate, begin with

So, by using Tartaglia's work and a clever transformation technique,

,which would mean

Bombelli showed that

. But this means that

which was a proverbial bombshell.Prior to Bombelli, mathematicians

Enter the solution for the cubic equation.

Use the Ferro-Tartaglia formula

Admittedly, Bombelli's technique applies only to a few specialized

Wallis imagined these solutions as displacements to the left and right

Wallis's method has the undesirable consequence that

Complex Functions and Linear Mappings

2.1 Functions and Linear Mappings

Combining these ideas, we often write a complex function f

Figure 2.1The mapping

There are two methods for defining a complex function in

We now give several examples that illustrate how to express

Solution.Using the binomial formula, we obtain

Example 2.2.Express the function

Solution.Using the elementary properties of complex numbers,

by a formula involving the

Solution.Calculation reveals that

in both Cartesian and polar form.

Solution.For the Cartesian form, a simple calculation gives

Remark.Once we have defined u and v for a function f in

to be a transformation (or mapping) from the set D in the z

The inverse image of a point w is the set of all points z in

Figure 2.3A functionw = f(z)that is one-to-one.

is not one-to-one because

Figure 2.4A function that is not one-to-one.

Example 2.7. Show that the function

,we want to describe

Method 2:We write

is a one-to-one mapping of the z-plane onto the w-plane and is called

Figure 2.5The translation

If we let be a fixed real number, then for

is a one-to-one mapping of the z-plane onto the w-plane and is called

Figure 2.6The rotation

Example 2.8.The ellipse centered at the origin with a horizontal major