You are on page 1of 41

MBE2036 Engineering Computing, part 8, version 2

Multidimensional Unconstrained
Optimization
Part 8

Jia Pan
Office: AC-1 Y6720
Email: jiapan@cityu.edu.hk

Department of Mechanical and Biomedical Engineering City University of Hong Kong


MBE2036 Engineering Computing, part 8, version 2

Analytical Method
For one dimensional unconstraint optimization
problem, the optimal point is the point where
the curve is flat
In mathematical terms, the first derivative f(x)=0
If second derivative f(x)<0, the point is a maximum
If second derivative f(x)>0, the point is a minimum
MBE2036 Engineering Computing, part 8, version 2

Analytical Method (cont)


For two dimensional problems, the optimal point
for a function f(x,y)is:
First partial derivatives should be zero i.e. and

If |H| >0 and then f(x,y) has a local minimum.

If |H| >0 and then f(x,y) has a local maximum.


If |H| <0, then f(x,y) has a saddle point.
MBE2036 Engineering Computing, part 8, version 2

Analytical Method (cont)


Hessian of f(x,y):

H= Eq 1

The determinant of Hessian:


= Eq2

The analytical approach for finding optimal points for two


dimensional functions can be extended to multi-dimensional
functions.
MBE2036 Engineering Computing, part 8, version 2

Example 1
Find the optimal point of f(x,y)=x2 y2
0 as x and y are
independent variables

Similarly,
MBE2036 Engineering Computing, part 8, version 2

f(x,y)=x2 y2
0.3

0.2

0.1

0
z

-0.1

-0.2

-0.3

-0.4
0.5

0
0.5

0
y
-0.5
-0.5
x
MBE2036 Engineering Computing, part 8, version 2

Example 1(cont)
=2

From Eq 2:
=

Therefore , (0,0) is a saddle point which is neither a maximum


or minimum
MBE2036 Engineering Computing, part 8, version 2

Example 2
Find the optimal point of f(x,y)=2xy +2x x2 2y2

2xy x2 2y2

=
2xy x2 2y2

= y Eq 4
MBE2036 Engineering Computing, part 8, version 2

f(x,y)=2xy +2x x2 2y2

10

-10

-20
Z

-30

-40

-50

-60
-2

-1

0
-2
1 -1
0
2
1
3 2
3
Y 4 4 X
MBE2036 Engineering Computing, part 8, version 2

f(x,y)=2xy +2x x2 2y2 0

-10

100 -20

0
Z

-100

-2
-30

-1
-2

0 -1
-40

1 0

1
2
2 -50

3
3
Y X
4 4
MBE2036 Engineering Computing, part 8, version 2

Example 2 (cont)
To find the optimal point:

4y =0 Eq 6

Solving Eq 5 and 6, we obtain the optimal point x=2, y =1


MBE2036 Engineering Computing, part 8, version 2

Example 2 (cont)
4y =0

=-2 4y

4y
2
MBE2036 Engineering Computing, part 8, version 2

Example 2 (cont)
=

As |H| > 0 and , function value f(2,1) is a


maximum.
The above example is pretty straight forward to find the
optimal point using first partial derivatives and Hessian.
However, this is not always that straight forward. So,
numerical method would be used instead!
MBE2036 Engineering Computing, part 8, version 2

Finite Difference Approximation


MBE2036 Engineering Computing, part 8, version 2

Gradient Methods
In any multi-dimensional optimization searching
algorithm, there are two key questions to be
asked when searching for an optimal solution.
1. The first one is about the direction of travel.
2. The second question is how long one should pursue
a solution in this direction before re-evaluating an
alternative direction.
MBE2036 Engineering Computing, part 8, version 2

Gradient Methods (cont)


The concept of gradients tell you what direction
to go, at least in the short term
One dimensional case:
Ideal but not practical method:
Start from any selected point of (x0, y0). Evaluate
gradient at (x0, y0).
Walk for a tiny distance along the gradient direction
till (x1, y1).
Re-evaluate gradient at (x1, y1) and repeat the
process.
MBE2036 Engineering Computing, part 8, version 2

Gradient Methods (cont)


Pros: always keep steepest direction and walk
shortest distance
Cons: not practical due to continuous re-
evaluation of the gradient.
Question: How to find the best (x1, y1)
MBE2036 Engineering Computing, part 8, version 2

Gradient Methods (cont)


The concept of gradients tells you what direction
to go, at least in the short term.
For the second question, if a re-evaluation
strategy is executed too often, it increases
computational costs. If it is executed too
infrequently, one may end up pursuing a direction
that may take the search away from the optimal
solution.
The steepest ascent algorithm proposes a simple
solution to the second question.
MBE2036 Engineering Computing, part 8, version 2

Steepest Ascent Method (or Steepest


Descent)
Start at an initial point
Maximum point
(xo,yo), determine the
direction of steepest
ascend, that is, the
gradient.
Then search along the
direction of the gradient
until f(x,y) stops
increasing i.e. becomes
level along your (xo,yo)
direction of travel.
MBE2036 Engineering Computing, part 8, version 2

Steepest Ascent Method (cont)


This stopping point Maximum point
becomes the starting
point where the
gradient is re-
evaluated and a new
direction followed.
This process is
repeated until the
(xo,yo)
peak is reached.

The inner the contour line the higher the value


MBE2036 Engineering Computing, part 8, version 2

Steepest Ascent Method (cont)

Steepest ascent method uses the gradient approach as


its choice for the best direction.
If f(x,y) is a two dimensional function, the gradient
vector tells us
What direction is the steepest ascent?
How much we will gain by taking that step?
MBE2036 Engineering Computing, part 8, version 2

Steepest Ascent Method (cont)


The gradient is defined as:
f f
del f f i j Eq 7
x y

Directional derivative
of f(x,y) at point x=a
and y=b
MBE2036 Engineering Computing, part 8, version 2

Steepest Ascent Method (cont)

For n dimensions

f
x ( x )
1
f ( x )
f ( x) x 2 Eq8

f
( x )
x n
MBE2036 Engineering Computing, part 8, version 2

Steepest Ascent Method (cont)


To transform a function
of x and y into a
function of h along the
gradient section:

f
x xo h Eq 9
x
f
y yo h Eq 10
y h is distance
along the h axis
MBE2036 Engineering Computing, part 8, version 2

Steepest Ascent Method (cont)


Assume:
MBE2036 Engineering Computing, part 8, version 2

Example 3
Using Steepest Ascent method for finding the maximum
point of f(x,y)=2xy +2x x2 2y2

Using initial guesses, x=-1 and y=1

2xy x2 2y2

2xy x2 2y2
Recall:
is the
= y Eq 4 max point
MBE2036 Engineering Computing, part 8, version 2

Example 3 (cont)
Using initial guesses, x=-1 and y=1

y= 2(1) 4(1)=6
Therefore the gradient vector is:
MBE2036 Engineering Computing, part 8, version 2

Example 3(cont)
To find the maximum, we could search along the
gradient direction, that is, along an h axis running
along the direction of the above gradient vector.

The function can be expressed along this axis as:


Using initial guesses, x=-1 and y=1

f(x,y)=2xy +2x x2 2y2


MBE2036 Engineering Computing, part 8, version 2

Example 3(cont)
Multiplying the terms, we develop a one-dimensional
function along the gradient as:

For finding the maximum point along the h axis:

Therefore
This means that If we travel along the h0-axis, g(h)
reaches maximum value when h=0.2
MBE2036 Engineering Computing, part 8, version 2

Example 3(cont)
Substitute h=0.2 into Eq 9 and Eq 10

The search moves from point 0 to point 1


The second iteration repeat the previous steps with the
new starting point of (0.2, -0.2)
MBE2036 Engineering Computing, part 8, version 2

Example 3(cont)
MBE2036 Engineering Computing, part 8, version 2

Example 3(cont)

y= 2(0.2) 4(0.2)=1.2
MBE2036 Engineering Computing, part 8, version 2

Example 3(cont)
f(x,y)=2xy +2x x2 2y2
Substituting x2, y2 values into the functions yields:

For finding the maximum point along the h1 axis:

Therefore
This means that If we travel along the h1-axis, g(h)
reaches maximum value when h=1
MBE2036 Engineering Computing, part 8, version 2

Example 3(cont)
= 0.2+1.2(1)=1.4

The search moves from point 1 to point 2


The third iteration repeat the previous steps with the
new starting point of (1.4, 1), and in so doing move
closer to the maximum. The approach can be repeated
until approach to zero for the any new
starting point.
MBE2036 Engineering Computing, part 8, version 2

Example 3(cont)
MBE2036 Engineering Computing, part 8, version 2

Example 3(cont)

y= 2(1.4) 4(1)=-1.2
MBE2036 Engineering Computing, part 8, version 2

Example 3(cont)
What can you observe?
MBE2036 Engineering Computing, part 8, version 2

Seems
Is this a coincidence?
MBE2036 Engineering Computing, part 8, version 2

Algorithm of Steepest Ascent


Step 0: Initial guess , set k = 0
Step 1: compute , if ,
then stop
Step 2: define
Step 3: solve and find
Step 4: , k -> k+1 go to
step 1
MBE2036 Engineering Computing, part 8, version 2

Other Methods
Newtons method
Taylors series

At the optimal point, , which means

Newtons method in high dimensional space:

Does not guarantee


Steepest descent: simple but slow; Newtons method:
complex but fast
MBE2036 Engineering Computing, part 8, version 2

Q&A

Thank you

Department of Mechanical and Biomedical Engineering City University of Hong Kong

You might also like