E1

251 Linear and Nonlinear

Op2miza2on

Chapter 2: Background Calculus

2.1.1 Sequence (definition)

A sequence xn : ! " is a mapping from non-negative integers to a set of real
1
number. Examples: xn = , xn = (1)n .
n
2.1.2 Limit of a sequence (definition)
A sequence {xn } is said to converge to the limit x if and only if, for each > 0,
there exists a positive integer N such that xn x < whenever n > N . We
write lim n xn = x.

2.1.3 Limit of a function

A function f (x) is said to have f * as the limit as x tends x * if and only if for
each > 0, there exists a > 0 such that f (x) f * < and x * x < .
We write lim xx* f (x) = f * .
2.1.4 Limit of a function in terms of sequences
A function f (x) is said to have f * as the limit as x tends x * if and only if for
every sequence {xn } with limit x * , the limit lim n f ( xn ) exists and is equal
to f * .
2.1.5 Continuity:
A function is continuous at x * if and only if it has a limit at x * .
3

2.2 Derivative
2.2.1 Derivative
A function f (x) is said to be differentiable x * if the following limit exists:
f (x * + h) f (x * )
lim h0
. The value of the limit is called the derivative at
h
x * denoted by f (1) (x * ). nth derivative is denoted as f (n) (x * ).
2.2.2 Affine approximation
A function f (x) that is differentiable in the interval [a,b] can be expressed as a
sum of derivative based affine approximation and an error term as follows:
f (x) = f (a) + f (1) (a)(x a) + r(x).
!###"###
$
A( x )

The error terms r(x) tends to zero as x tends a. This follows directly from the
definition of derivative.
4

2.3 Mean value theorems

2.3.1 Rolle's theorem
Let f (x) be function that is differentiable in the interval
[a,b] and let f (a) = f (b) = 0. Then there exists a point [a,b] such
that f (1) ( ) = 0.
f (x)
Tangent at has zero slope

x=a

x =

x=b

2.3.2 Mean value theorem

Let f (x) be function that is differentiable in the interval [a,b]. Then there exists
a point [a,b] such that f (b) = f (a) + f (1) ( )(b a).
f (x)

x=a

x =

x
x=b

Geometric interpretation:
Differentiability ensure that there is point ( , f ( )) where
the tangent to the function is parallel to the line connecting the points
(a, f (a)) and (b, f (b)).

2.4 Taylor's Formula

2.4.1 Formula with reminder term
For a function f (x) that is continuously differentiable m times in the interval
[a,b] we have
f (b) = f (a) + f (1) (a)(b a) +
where [a,b].

1 (2)
1 (m )
f (a)(b a)2 + ! +
f ( )(b a)m
2!
m!

2.4.2 Alternative form

For a function f (x) that is continuously differentiable m times in the interval
[a,b] we have
1
1 (m )
f (b) = f (a) + f (1) (a)(h) + f (2) (a)(h)2 + ! +
f (a + h)(h)m
2!
m!
where [0,1], h = b a.
7

2.5 Taylor's formula with order notation

2.5.1 The small-o notation
A function f(x) is said to be in the order of another function g(x) if
f(x)
limx0
=0. We denote this by f(x) = o(g(x)).
g(x)
Examples from Chong and Zak:
1) x 2 = o(x)

x3
2)
2
4
2x + 3x
3) x 3 = o(x 2 )
4) x = o(1)

= o(x)

5) If f (x) = o(1), then x m f (x) = o(x m ).

8

2.5.2 Order of the reminder term of Taylor's formula

We consider the term f (m ) (a + h) for [0,1]. Since f (m ) (x) is continuous in
the interval [a,a + h] lim h0 f (m ) (a + h) = f (m ) (a).
Hence f (m ) (a + h) = f (m ) (a) + o(1),
Substituting the second form of Taylor's formula gives
1 (2)
1 (m )
(1)
2
f (b) = f (a) + f (a)(h) + f (a)(h) + ! + f (a) + o(1) (h)m .
2!
n!
This is equivalent to
h 2 (2)
h m (m )
f (b) = f (a) + hf (a) +
f (a) + ! +
f (a) + o(h m )
2!
n!
(1)

2.6 Derivatives of multivariable functions

2.6.1 Basic notations
Mul$variable func$on nota$on:

A multivariable function is denoted as f (x) where x = x1

x2

xn

Gradient of a mul$-variable func$on

f (x) =
f (x)
f (x)
f (x)
x
x
x
1
2
n

f (x1 + h, x2 ,, xn ) f (x1 , x2 ,, xn )
f (x) = lim
h0
x1
h
Sometimes we write x f (x) for clarity.
Hessian

{ xx f (x)

=F(x)}i, j

2
=
f (x)
xi x j
10

2.6.2 The "line trace" of a multivariable function

The line trace of a muti-variable function f (x) at x along direction
d is the following single-variable function:
g[x,d ] ( ) = f (x + d).

2.6.3 Derivatives of g[x,d ] ( ) = f (x + d)

n

g[x,d
] ( ) = dk
k=1

f (x + d)
xk

2
g[x,d
f (x + d)
] ( ) = d j dk
x j xk
j=1 k=1
n

11

2.6.4 Affine approximation formula for multivariable function

Let f (x) be a multivariable function continuous in the line connecting the points
a and b, then we want find an affine approximation formula of the form:
f (x ) = f (a) + mT (x a) + r(x ), where x [a,b] and r(x ) is the reminder terms
that tends to zero as x tends to a.
The formula:
f (x ) = f (a) + (x a)T f (a) + r(x ).

12

2.7 Taylor's formula for multivariable function

Recap:
Let f (x) be an 2 times continously differentiable function
on an interval [ a,b ]. Then for h = b a, we have
h (1)
h 2 (2)
f (b) = f (a) + f (a) +
f (a + h)
1!
2!
for some [0,1].
Form I:
f (y) = f (x) + (y x)T f (x) + (y x)T F(x + (y x))(y x)
Form II:

f (y) = f (x) + (y x)T f (x) + (y x)T F(x)(y x) + o (y x2 )

13

2.8 Derivatives of composite functions

2.8.1
Let g(x) be function from ! n to ! (vector input-scalar output) and let f(t) be
function from ! to ! n (scalar input-vector output). Then consider the composite
T
d
function h(t) = g(f(t)). Its derivative is given by h(t) = x g(x) x=f (t ) f (t).
dt

2.8.2
Let g(x) be function from ! n to ! (vector input-scalar output) and let A
be and n m matrix. Then for the function h(y) = g(Ay), the gradient
is given by y h(y) = A T x g(x) x=Ay , and the Hessian is given by
yy h(y) = A T xx g(x)A x=Ay

14

2.9 Implicit function theorem

Let h(x) be a function from ! n to ! m (vector input-vector output) with m < n.
Let x 0 be a point satisfying h(x 0 ) = 0, and let there be a neighborhood around
x 0 such that for every x , h(x) is p times continuously differentiable and the
Jacobian J h ( x) is invertible. Then there exist a function from ! nm to ! m
g(y 0 )

denoted by g() such that x 0 =

for some y 0 ! nm and
y 0
g() is p times continuously differentiable in a neighborhood around y 0 .
h1 (x)
h1 (x)
"

xm
x1
Jacobian: J h ( x) =
#
$
#

hm (x) " hm (x)

xm
x1

15