JohnRasmussen, Insti t ut e of Mec hanical Engineeri ng, Aalborg Univ ersity, 2001

x x d
k k k k ( ) ( ) ( ) ( ) +
+
1

Engineering design
optimization
Lecture 7: Algori t hms for
multiple-di mension opti mi zation
PRepeti ti on of t he g ol den s ecti on alg orit h m
PUnconst r ai ned mi mi ni zati on
<The idea behind line search
<The steepest descent algorithm
<The conjugate gradient algori thm
<Newtonsmethod
PPe nal t y met hods
<The exteriormethod
<The interiormethod
Lecture 7
Mul tiple-dimension optimization
To read:
PArora: chapter [5.4 - 5.4.3] + [5.5 - 5.6.1[ + [5.9 - 5.9.1]
The golden section algorithm
- repetition
doubl e gol den( doubl e A, doubl e B) {
doubl e alf 1=0, alf 2= 0;
doubl e f alf 1= 0, f alf 2= 0;
doubl e G=0.618034; / * T he gol den s ect i on */
doubl e t ol;
/ * Mak e sure B > A */
if ( B< =A ) {
f pri ntf (st derr, " gol den: err or: I l l egal i nt er v al ends\ n" );
r et ur n A;
}
/ * I i nit i al ize v ar i abl es */
t ol=( B-A) *0. 0001;
alf 1 = A + ( 1- G) *( B- A);
alf 2 = B - ( 1- G) *( B-A);
f alf 1=f ( alf 1);
f alf 2=f ( alf 2);
The golden section algorithm
- using right hand interval
whil e (B- A > t ol ) {
// Us e t he l eft hand int er v al, if t he f unction v al ue at t he
// ri ght hand gol den poi nt is t he l ar ger
if (f alf 2>f alf 1) {

// Shift r e- usabl e r es ult s left
B = alf 2;
alf 2 = alf 1;
f alf 2 = f alf 1;

// Co mput e new alf 1 and f unction v al ue
alf 1 = A + (1-G) *( B- A);
f alf 1 = f ( alf 1);
}

x
f (x )
A B alf1 alf2
The golden section algorithm
- using left hand interval
// ot her wi s e, use t he ri ght hand i nt er v al
else {
// Shift r e- usabl e r es ult s left
A = alf 1;
alf 1 = alf 2;
f alf 1 = f alf 2;

// Co mput e new Al pha2 and f unction v al ue
alf 2 = B - ( 1- G) *( B-A);
f alf 2 = f ( alf 2);
}
} // Gol den sect i on loop
/ * Ret ur n t he mi dpoi nt of t he int erv al when it i s s mal l enough */
r et ur n ( alf 1+ alf 2)/ 2;
} /* gol den */
x
f (x )
A B alf1 alf2
PChoose a search direction,
d
(k)
PMinimize along the search
di rection (by golden section).
Step = "
(k)
d
(k)
.
PRepeat until convergence
Unconstrained minimization
- in multiple dimensions
x
x
1
2
JohnRasmussen, Insti t ut e of Mec hanical Engineeri ng, Aalborg Univ ersity, 2001
d
f
f
k ( )

d f x d
f x
f x
k k k k
k
k
k
( ) ( ) ( ) ( )
( )
( )
( )
( )
( )
( )
+

_
,

1
1
2
d f x d
f x
f x
k k k k
k
k
k
( ) ( ) ( ) ( )
( )
( )
( )
( )
( )
( )
+

_
,

1
1
2
The obviouschoice when
minimizing a function i sto
choose the path that goes
as much downhill as
possible. This algorithm i s
known assteepest
descent.
Some people call thi stype
of algorithm greedy.
In real life, being greedy i s
often only profitable in the
short term.
Thi sal so appliesin
optimization.
Choice of direction
Steepest descent.
x
x
1
2 It is possible to show mathematically, that
each new direction in steepest descent i s
perpendicular to the previousone.
Thi smeansthat the algori thm approaches
the optimum using only very few directions.
In 2-D, only two different directionsare used.
The steps in each di rection tend to get
smaller foreach iteration.
They may become so small that the
algorithm thinkstoo soon that it has
converged.
In any case, convergence can be very slow.
Steepest descent
Zig-zagging by nature
x
x
1
2
On problems with similar scales in the different variable directions, steepest
descent often workswell.
If the level curves are circular, then the optimum i sfound in the first chosen
di rection.
If the level curves are longi sh, then the algorithm typically requires many
iterations.
Steepest descent
- may work well or terribly
The conjugate gradient method can be seen
as a way of detecting and eliminating zig-
zagging. It al so hasmore subtle
mathematical explanations, but we dont
have to worry much about those.
The search direction i scomputed by the
formula:
The conjugate gradient method
Evening the zig-zags
x
x
1
2
Wek now t hat t he gr adi ent v ani s hes at t he
opti mum. T hi s means, t hat if t he pr ocess i s goi ng
wel l, t hen t he gr adi ent get s s mall er f or each
i ter ation. If t his is t r ue, t hen $
(k)
is a s mal l number,
and we do nt get much c orr ect ion f r om t he
st e epest desc ent met hod.
If t he gr adi ent does not g et s mall er, t hen we need
mor e c or rect ion, and t his is prec isely what we get.
The conjugate gradient method
Why it works
x
x
1
2
Gradient zero
Correction
Steepest descent
There is a very easy way of making sure that an optimization processstays
within a defined set of constraints: tax eventual violations.
Thi si sthe basis of the so-called penal ty methods.
They are al so called transformation methods because they replace the
opriginal constrained problem wi th an equivalent one wi thout constraints.
The transformed problem can then be solved by an unconstrained
algorithm.
Penalty methods
- a poor mans approach to constrained optimization
JohnRasmussen, Insti t ut e of Mec hanical Engineeri ng, Aalborg Univ ersity, 2001
( )
Minimize
subject to
is convert ed to
Minimize
f
g
f P
i
( )
( )
( ) ( ) ( )
x
x
x x g x

+
0

( ) [ ]
( ) ( ) max , ( ) x x x +

_
,

f r g
i
i
m
0
2
1
Consider the problem to the left.
We want to minimize F(x) provided
g
1
and g
2
are positive.
Golden section would solve thi s
right away, but forthe sake of the
argument, let usjust assume that
we cannot impose the constraints.
Instead, we can penalize them.
Penalty methods
- basic idea
So we replace F by a new function,
M, which i sconstructed so that it
increasesrapidly when a constraint
i sviolated.
Minimizing Mwill almost give us the
solution to the original problem.
There are two types of penalization:
- Exterior (a tax)
- Interior (capital puni shment)
Penalty methods
- penalization
Thi spenalty doesnot come into play until
a constraint hasbeen violated.
The severenessof the penal ty depends
on the penalty factor, r.
Small valuesof r will cause constraint
violations. Large values will make the
problem difficul t to solve because the
function getssharp kinks.
The acceptable r valuesare problem-
dependent. It i sa good idea to make the
functionsdimensionless.
Exterior penalty
- the mild form
Original problem: Linearobjective
function and two constraints in two
dimensions.
Exterior penalty
- examples
Optimum
Penalized problem, r = 0.05.
Notice that the optimum fall squite far
from the solution to the original
problem.
Exterior penalty
- example
Penalized problems, r
=0.1 and r = 1.0.
The optimum
approachesthe
solution to the original
problem but never
reachesit completely.
The level curvesget
sharper edgesand
the problem becomes
more di fficult to solve
numerically.
Exterior penalty
- examples
JohnRasmussen, Insti t ut e of Mec hanical Engineeri ng, Aalborg Univ ersity, 2001
( ) ( )
( )
x x
x
+

1
]
1

_
,

f r
g
i i
m
1
1
PA penalty term i sadded only after constraint violation
PThe objective function inside the feasible domain i sunaffected
PThe pseudo objective function is defined everywhere the original function
i s. We dont need a feasible point to get started.
PThe solution alwaysfall sslightly outside the feasible domain of the original
problem. Notice that the original problem may be undefined outside the
feasible domain.
PIncreasing the penalty brings the solution closerto the solution to the real
problem, but it al so makes the problem more difficult to solve numerically.
PIt handlesequality- as well asinequality constraints.
Exterior penalty
- properties
Thi spenalty i salwayspresent, and it
really kicks in when a constraint i s
approached.
The penalty goesto infini ty at the
constraint.
The severenessof the penalty depends
on the penalty factor, r.
Small valuesof r will cause the constraint
to kick in late but suddently aswe
approach a constraint.
The penalty i s-infinity (!) right outside
the constraint.
Interior penalty
- capital puni shment
PThe penalty i salwayspresent.
PThe pseudo objective function is undefined at the constraintsand goesto
infinity outside. If the algori thm happensto violate a constraint, then
chances are that it will never return to the feasible domain.
PWe need a feasible point to start the algorithm.
PThe solution alwaysfall sslightly inside the feasible domain of the original
problem. Thi smeansthat all solutionsare usable.
PIncreasing the penalty bringsthe solution closerto the solution to the real
problem, but it al so makes the problem more difficult to solve numericall y.
PIt handlesonly inequality constraints.
Interior penalty
- properties
PP enal t y met hods ar e ch eap and dir t y sol uti o ns t o
constr ai ne d opti mizati o n.
PThe y ar e pr o bl e m- de pen de nt an d may be di f fi cul t t o appl y.
PThe y ar e not sui t a bl e f or g ener al a ppli cati ons, b ut t he y may
suffi c e f or speci al pur poses.
PThe Aug ment ed Lag r ang ia n met hod is a f ur t her de vel op ment
of pen alt y met hods. I t i s o nl y slightl y mor e co mpli cat ed and
i t d oes awa y wi t h man y of t he pr obl e ms of i nt eri or and
e xt eri or pe nal ti es.
Penalty methods
- properties in general
PPi c k a st ar ti ng poi nt i n n di me nsi o ns
PCo mp ute t he val ue of t he o bj ect i ve
PCo mp ute t he gr a di ent b y fi ni t e di f f er ence
PCo mp ute a sear ch dir ecti on fr o m t he gr a di ent and possi bl y
t he gr a di ent of t he pr e vi ous st ep.
PP erf or m 1- D mi ni mi zati on i n t he ch ose n dir ecti on b y Gol de n
S ecti o n search
PRepe at t he l oop u ntil con ver g ence.
A conjugate gradient algorithm
- skeleton
A conjugate gradient algorithm
- skeleton
// conjug.cpp : Demo application for conjugate gradients
//
#include <stdio.h>
#include <math.h>
#include "conjug.h"
// --------------------------------------------------------------
// These variablesare global because it simplifiesthe code.
// Global variablesare not recommended forlarger programs
double x[MAXDES]; // Design variable vector
double s[MAXDES]; // Search direction
long nDV; // Numberof design variables
JohnRasmussen, Insti t ut e of Mec hanical Engineeri ng, Aalborg Univ ersity, 2001
A conjugate gradient algorithm
- skeleton
// --- -- --- -- --- -- --- -- --- -- --- -- --- -- --- -- --- -- --- -- --- -- --- -- --
// Mai n pr ogr a m
i nt mai n(i nt ar gc, c har * argv[])
{
// I nitial ize v ar i abl es t o somet hi ng
x[ 0] = 1. 0;
x[ 1] = - 1. 0;
// Cal l conj ugat e gr adi ent mi ni mizat i on
doubl e mi ni mum = CJGmi n();
// Pr i nt t he r es ult
pri ntf (" T he mi ni mum of f i s %lf\ n", mi ni mum);
pri ntf (" Loc at ed at poi nt ");
f or ( l ong j= 0; j< nDV; j++ ) pr i ntf( " %lf", x[ j]);
pri ntf ("\ n");
// end t he pr ogr am
retur n 0;
} // mai n
A conjugate gradient algorithm
- skeleton
doubl ef( doubl e *xx );
doubl ef al pha( doubl e al pha);
doubl e CJGmi n();
v oi d Gr adi ent( doubl e *xx, doubl e Fxx, doubl e *Gr ad);
v oi d Sear c hDir( l ong it nr, doubl e *Gr ad, doubl e *Ol dGr ad, doubl e *O l dss, doubl e *ss );
doubl eGol denSect i onSearc h( doubl e A, doubl e B, double *al pha);
doubl e nor m2( doubl e *v) ;
v oi d nor mal ize( doubl e *v);
#def i ne MAXDES 20 // Maxmi mum number of desi gn vari abl es
A conjugate gradient algorithm
- skeleton
// Thi si sthe objective function. Any function of several variables
// can be used
double f(double *xx) {
// Set the numberof design variableshere
nDV=2;
// The banana function
double ret = 10.0*xx[0]*xx[0]*xx[0]*xx[0] -20.0*xx[0]*xx[0]*xx[1]
+ 10.0*xx[1]*xx[1] + xx[0]*xx[0] - 2*xx[0] + 5.0;
// Return the result to caller
return ret;
} // f
A conjugate gradient algorithm
- skeleton
// --------------------------------------------------------------
// Thi si sa one-dimensional version of the objective function
// given by the parameter alpha
double falpha(double alpha) {
double xx[ MAXDES];
// Construct point from x di stance alpha along s
for (long j=0;j<nDV;j++) xx[j ] = x[j] + alpha*s[j];
// Call the original function
double ret = f(xx);
// Return the result to caller
return ret;
} // falpha
A conjugate gradient algorithm
- skeleton
// --- -- --- -- --- -- --- -- --- -- --- -- --- -- --- -- --- -- --- -- --- -- --- -- --
// T hi s f unct i on c omput es a s ear c h di r ect i on ss given a gr adi ent
// acc or di ng t o t he c onj ugat e gr adi ent met hod. I f t he input v ar i abl e
// it nr= = 0, t he searc h dir ection is j ust t he opposit e of t he gr adi ent.
// If it no> 0, it is modified acc or di ng t o t he pr evi ous s ear c h di r ect i on,
// Ol ds s.
v oi d Sear c hDir( l ong it nr, doubl e *Gr ad, doubl e *O l dG r ad, doubl e *Ol dss, doubl e *ss) {
// T odo: Comput e a searc h dir ect i on ss acc or di ng t o st eepest desc ent
// or c onj ugat e gr adi ents
// Nor mal i z e lengt h aft er modif i c ation
nor mal ize(s s);
r et ur n;
}
A conjugate gradient algorithm
- skeleton
// --------------------------------------------------------------
// Thi sfunction computesthe gradient of the objective function
// at the point xx by finite di fference. The current objective
// function value Fxx must be known
void Gradient(double *xx, double Fxx, double *Grad) {
double delta=1.0e-8;
for (long j=0;j<nDV;j++) {
xx[j] += delta; // Perturb the j 'th variable by delta
Grad[j] = (f(xx)-Fxx)/delta;
xx[j] -= delta; // Restore the original value
}
}
JohnRasmussen, Insti t ut e of Mec hanical Engineeri ng, Aalborg Univ ersity, 2001
A conjugate gradient algorithm
- skeleton
// -------- ------- -------- ------- -------- ------- -------- ------- --
//T hi sf unct ion mi ni mizes the objectiv ef unct ion by the conj ugate
// gr adient method.
double CJGmi n() {
doubl e al pha=1000.0;
doubl e B= 4. 0; // Se arch lengt h. F its t he banana f unct ion
long count=0;
doubl e df[MAXDES],OldGr ad[MAXDES],OldS[MAXDES];
// Compute c urrent objectiv e
doubl eF xx = f(x);
// Mai n l oop. Keep minimizi ng until t he change is small enough
whil e (alpha > 1. 0e- 4) {
// T odo: Pr ogr am the search direct ion -> 1-D mi ni mizat ion
// l oop here
c ount++;
}
pr intf("\ nCJGmi n: %d iter ations\ n",count);
ret urn Fxx;
}
PDo wnl oad t he skel et on C code fr o m
www. i me. auc. d k/~jr /l ect ur e not es. ht m
PIf you use C, st ar t pr ogr a mmi ng your o wn c onj ug ate
gr a di ent s meth od i nt o t he pr ogr a m
PIf you use any ot her l ang uag e, tr ansl at e t he co de by h and
a nd st ar t de vel opi ng t he f uncti on ( use t h e g ol den secti on
f uncti on you d evel ope d i n l ect ur e 6 as a su br outi ne)
PMa ke it eas y t o swi t ch bet wee n conj ug at e gr a di ents a nd
st e epest d esce nt.
PCo mpil e a nd t est wi t h dif f er ent 2-D f uncti ons. Tr y t he
ba nana f unct i on , Ar or a, e xa mpl e 5. 15. Tr y wi t h and wi t h out
conj ug ati ng th e gr adi ents.
Assignment
Program your own conjugate gradient algorithm