Deterministic methods

Types of
Optimization Models
Stochastic
(probabilistic
information on data)

Discrete, Integer
(S = Zn)

Linear
(f and g are linear)

Deterministic
(data are certain)

Continuous
(S = Rn)

Nonlinear
(f and g are nonlinear)

Unidimensional Search

Chapter 5

(1) If have a search direction, want to minimize in that direction by numerical methods

(2) Search Methods in General

2.1. Non Sequential Simultaneous evaluation of f at n points no good (unless on parallel computer).
2.2. Sequential One evaluation follows the other.

(3) Types of search that are better or best is often problem dependent. Some of the types are:

Chapter 5

a. Newton, Quasi-Newton, and Secant methods.

b. Region Elimination Methods (Fibonacci, Golden Section, etc.).
c. Polynomial Approximation (Quadratic Interpolation, etc.).
d. Random Search
(4) Most methods assume
(a) a unimodal function, (b) that the min is bracketed at the start and (c) also you start in a
direction that reduces f.

The Golden Ratio

In mathematics and the arts, two quantities are in the golden ratio if the
ratio between the sum of those quantities and the larger one is the same as
the ratio between the larger one and the smaller.
The golden ratio is a mathematical constant approximately 1.6180339887.
The golden ratio is also known as the most aesthetic ratio between the two
sides of a rectangle.
The golden ratio is often denoted by the Greek letter (phi).

Construction of the Golden Section

Firstly, divide a square such that it makes two precisely equal rectangles.

 Take the diagonal of the rectangle as the radius to contsruct a circle to

touch the next side of the square.
Then, extend the base of the square so that it touches the circle.

 When we complete the shape to a rectangle, we will realize that the

rectangle fits the optimum ratio of golden.
The base lenght of the rectangle (C) divided by the base lenght of the
square (A) equals the golden ratio.
C / A =A / B = 0.6180339 = The Golden Ratio

1+ 5
2

The Golden Spiral

After doing the substraction infinitely many times, if we draw a spiral

starting from the square of the smallest rectangle, (Sidelenght of the
square=Radius of the spiral) we will get a Golden Spiral. The Golden
spiral determines the structure and the shape of many organic and
inorganic assets.

Leonardo Da Vinci
Many artists who lived after Phidias have
used this proportion. Leonardo Da Vinci
called it the "divine proportion" and featured
it in many of his paintings, for example in the
famous "Mona Lisa". Drawing a rectangle
around her face. You will realize that the
measurements are in a golden proportion.
You can further explore this by subdividing
the rectangle formed by using her eyes as a
horizontal divider.

The Vitruvian Man

Leonardo
did
an
entire
exploration of the human body
and the ratios of the lengths of
various body parts. Vitruvian
Man illustrates that the human
body is proportioned according to
the Golden Ratio.

The Parthenon
Phi was named for the
Greek sculptor Phidias.
The exterior dimensions of
the Parthenon in Athens,
built in about 440BC, form a
perfect golden rectangle.

 The baselenght of Egyptian pyramids divided by the height of

them gives the golden ratio..

Golden Ratio in Human Hand and Arm

Look at your own hand:

You have ...
2 hands each of which has ...
5 fingers, each of which has ...
3 parts separated by ...
2 knuckles

The length of different parts

in your arm also fits the
golden ratio.

Golden Ratio in the Human Face

The dividence of every long line to the short line equals the golden ratio.
Lenght of the face / Wideness of the face
Lenght between the lips and eyebrows / Lenght of the nose,
Lenght of the face / Lenght between the jaw and eyebrows
Lenght of the mouth / Wideness of the nose,
Wideness of the nose / Distance between the holes of the nose,
Length between the pupils / Length between teh eyebrows.
All contain the golden ratio.

The Golden Spiral can be seen in the arrangement of seeds on

flower heads.

Golden Ratio In The Sea Shells

The shape of the inner and outer surfaces of the sea shells, and their
curves fit the golden ratio..

Golden Ratio In the Snowflakes

The ratio of the braches of a snowflake results in the golden ratio.

Golden Section
Origin of golden section:
I1

I1 I 2 I 3

I1 fI1 f 2 I1

f f 1 0 f1, 2
2

I2 = fI1

1 5

5 1
f
0.618034
2
Final interval:

I N f N I1

I2 = fI1

I3 = fI2

1 12 23 3 5 5
8 8
1 2 3 5
8

xU
xL xL

xL

xU

Golden-Section Search
divides intervals by
K = 1.6180

Derivative depending methods

f ( X ) X 2 X 2 X X X 2 X1 5
4
1

2
1

2
2

2
1

f
0 4 X 13 4 X 2 X 1 2 X 1 2
X 1

Chapter 4

f
0 2 X 12 2 X 2
X 2
min at (1,1)
12 X 12 4 X 2 2 4 X 1
H

4 X1
2

10 4
at (1,1), H
0

4 2

Chapter 5

To Bracket the Minimum

1. Compute f ( x ( 0) ) and f ( x 0 x)
If f ( x ( 0) x) f ( x ( 0 ) ), let x NEW xOLD
If f ( x ( 0) x) f ( x ( 0) ), let xNEW 2xOLD
2. Compute f ( x (1) ) f ( x ( 0) xNEW )
Continue doubling x until f ( x ( 0 ) x ( k ) ) f ( x ( 0) x ( k 1) )

3. Compute f ( x ( k 1) x ( k 2 ) )
1
(2 )
2

Chapter 5

You now have 4 equally spaced points x ( 3) , x , x ( 2 ) , x (1) .

Discard the point that enables you to keep a bracket
on the point giving the minimum f(x) (using the
closest points)

Newtons Method
Newtons method for an equation is

Chapter 5

f ( x) f ( x0 ) f ( x0 )( x x0 ) 0
f ( x0 )
( x x0 )
f ( x0 )

or

f ( x0 )
x x0
f ( x0 )

Application to Minimization
The necessary condition for f(x) to have a local minimum is f(x) = 0. Apply Newtons method.

x ( k 1)

(k )

f (x )
(k )
x
(k )

f (x )

Interpretation in terms of a quadratic function:

1
f ( x) f ( x ) f ( x )( x x ) f ( x ( k ) )( x x ( k ) ) 2
2
df(x)
Let
0
dx
Then f ( x ( k ) ) f ( x ( k ) )( x x ( k ) ) 0
( A)

Chapter 5

(k )

(k )

(k )

Examples
Minimize

f ( x) a0 a1 x a2 x 2
f ( x) a1 2a2 x
f ( x) 2a2

Chapter 5

x (1)

(0)
a

2
a
x
a
a
2
x (0) 1
x (0) 1 x (0) 1
2a2
2 a2
2 a2

Minimize

f ( x) x 4 x 2 1
f ( x) 4 x 3 2 x
f ( x) 12 x 2 2
x (1)

4
3
x
2x
x (0)
12 x 2 2

Start at x (0) 1, x (1) 1

Continue

3 2
0.100
12 2

General Strategy for Gradient methods

Chapter 6

(1) Calculate a search direction

(2) Select a step length in that direction to reduce f(x)

k 1

x s x x
k

Steepest Descent
Search Direction

s f ( x )
k

Method terminates at any stationary point.

f ( x) 0

Dont need to normalize

So procedure can stop at saddle point. Need to show

*

H (x )

is positive definite for a minimum.

Chapter 6

Step Length
How to pick
analytically
numerically

Fletcher Reeves Conjugate Gradient Method

Let

s f ( x )
0

s f ( x ) 1s
1

Chapter 6

s f ( x ) s
2

are chosen to make s H s

i

k 1

0 (conjugate directions)

f ( x

k 1

) f ( x ) 2f ( x )( x x )

f ( x

k 1

) f ( x ) H x H k s

k
1
k 1
k
s H f ( x ) f ( x ) k
T

k 1
k
1
(s ) f ( x ) f ( x ) H / k

Chapter 6

k T

Using definition of conjugate directions, (s )T Hs

f ( x

k 1

k 1

=0,

1
k 1
k
) f ( x ) H H f ( x ) k s 0
k

All cross products cancel out, giving the weighting factor:

k 1

k 1

T f ( x )f ( x )
k

k
k
T f ( x )f ( x )
s

k 1

f ( x

k 1

) k s

Chapter 6

Chapter 6

Chapter 6

Chapter 6

Chapter 6

2
2
Minimize f ( x1 3) 9( x2 5) using the method of conjugate gradients with x10 1 and x20 1 as an initial point.

1
In vector notation,x 0
1
f

4

72

x0

Chapter 6

For steepest descent,

s f
0

x0

4

72

Steepest Descent Step (1-D Search)

1
0 4
x ,
1
72
1

0 0.

The objective function can be expressed as a function of 0 as follows:

f ( 0 ) (4 0 2)2 9(72 0 4)2 .

Minimizing f(0), we obtain f = 3.1594 at 0 = 0.0555. Hence

1.223
x1

5.011

Calculate Weighting of Previous step

The new gradient can now be determined as
f

and 0 can be computed as

3.554

0.197

(3.554) 2 (0.197) 2

0.00244.
(4) 2 (72) 2
0

and

Chapter 6

Generate New (Conjugate) Search Direction

3.554
4 3.564
s1

0.00244

0.197
72 0.022
1.223
1 3.564
x2

0.022
5.011

One dimensional Search

Solving for 1 as before [i.e., expressing f(x1) as a function of 1 and minimizing with respect to 1] yields
f = 5.91 x 10-10 at 1 = 0.4986. Hence
X

3.0000

5.0000

which is the optimum (in 2 steps, which agrees with the theory).