Introduction to Optimization Methods

Introduction to Non-Linear
Optimization

© 2008 Solutions 4U Sdn Bhd. All Rights Reserved

Fundamental of Optimization

Optimization in Process Plants

© 2008 Solutions 4U Sdn Bhd. All Rights Reserved

Fundamental of Optimization

Optimization Tree

Figure 1: Optimization tree.

© 2008 Solutions 4U Sdn Bhd. All Rights Reserved

 Fundamental of Optimization

What is Optimization?

 Optimization is an iterative process by which a desired solution

(max/min) of the problem can be found while satisfying all its
constraint or bounded conditions.

Figure 2: Optimum solution is found

while satisfying its constraint (derivative
must be zero at optimum).

 Optimization problem could be linear or non-linear.

 Non –linear optimization is accomplished by numerical ‘Search
Methods’.
 Search methods are used iteratively before a solution is achieved.
 The search procedure is termed as algorithm.
© 2008 Solutions 4U Sdn Bhd. All Rights Reserved
 Fundamental of Optimization

What is Optimization?(Cont.)

 Linear problem – solved by Simplex or Graphical methods.

 The solution of the linear problem lies on boundaries of the feasible
region.

Figure 3: Solution of linear problem Figure 4: Three dimensional solution of

non-linear problem
 Non-linear problem solution lies within and on the boundaries of the
feasible region.
© 2008 Solutions 4U Sdn Bhd. All Rights Reserved
 Fundamental of Optimization

Fundamentals of Non-Linear Optimization

 Single Objective function f(x) Maximize X1 + 1.5 X2

• Maximization Subject to:
X1 + X2 ≤ 150
• Minimization 0.25 X1 + 0.5 X2 ≤ 50
 Design Variables, xi , i=0,1,2,3….. X1 ≥ 50
X2 ≥ 25
 Constraints X1 ≥0, X2 ≥0
• Inequality Figure 5: Example of design variables and
• Equality constraints used in non-linear optimization.
 Optimal points
• Local minima/maxima points: A point or Solution x* is at local point
if there is no other x in its Neighborhood less than x*
• Global minima/maxima points: A point or Solution x** is at global
point if there is no other x in entire search space less than x**

© 2008 Solutions 4U Sdn Bhd. All Rights Reserved

 Fundamental of Optimization

Fundamentals of Non-Linear Optimization (Cont.)

Figure 6: Global versus local optimization. Figure 7: Local point is equal to global point if
the function is convex.

© 2008 Solutions 4U Sdn Bhd. All Rights Reserved

 Fundamental of Optimization

Fundamentals of Non-Linear Optimization (Cont.)

 Function f is convex if f(Xa) is less than value of the corresponding
point joining f(X1) and f(X2).
 Convexity condition – Hessian 2nd order derivative) matrix of
function f must be positive semi definite ( eigen values +ve or zero).

Figure 8: Convex and nonconvex set Figure 9: Convex function

© 2008 Solutions 4U Sdn Bhd. All Rights Reserved

 Fundamental of Optimization

Mathematical Background
 Slop or gradient of the objective function f – represent the
direction in which the function will decrease/increase most rapidly
df f ( x  x)  f ( x) f
 lim  lim
dx x0 x x 0 x

 Taylor series expansion

df 1 d2 f
f ( x p  x)  (x)  2
(x) 2  .......
dx xp 2! dx xp

 Jacobian – matrix of gradient of f with respect to several variables

 f f f 
 x y z 
J  
 g g g 
 x y z 

© 2008 Solutions 4U Sdn Bhd. All Rights Reserved

 Fundamental of Optimization

Mathematical Background (Cont.)

 Slope -First order Condition (FOC) – Provides function’s slope information
f ( X *)  0
 Hessian – Second derivative of function of several variables, Sign
indicates max.(+ve) or min.(-ve)
 2 f 2 f 
 2 
x yx 
H  2
 f 2 f 
 xy y 2 

 Second order condition (SOC)
• Eigen values of H(X*) are all positive
• Determinants of all lower order of H(X*) are +ve

© 2008 Solutions 4U Sdn Bhd. All Rights Reserved

 Fundamental of Optimization

Optimization Algorithm

 Deterministic - specific rules to move from one iteration to next ,

gradient, Hessian

 Stochastic – probalistic rules are used for subsequent iteration

 Optimal Design – Engineering Design based on

optimization algorithm

 Lagrangian method – sum of objective function and linear

combination of the constraints.

© 2008 Solutions 4U Sdn Bhd. All Rights Reserved

 Fundamental of Optimization

Optimization Methods
 Deterministic
• Direct Search – Use Objective function values to locate minimum
• Gradient Based – first or second order of objective function.
• Minimization objective function f(x) is used with –ve sign –
f(x) for maximization problem.
 Single Variable
• Newton – Raphson is Gradient based technique (FOC)
• Golden Search – step size reducing iterative method
 Multivariable Techniques ( Make use of Single variable Techniques
specially Golden Section)
• Unconstrained Optimization
a.) Powell Method – Quadratic (degree 2) objective function polynomial is
non-gradient based.
b.) Gradient Based – Steepest Descent (FOC) or Least Square minimum
(LMS)
c.) Hessian Based -Conjugate Gradient (FOC) and BFGS (SOC)
© 2008 Solutions 4U Sdn Bhd. All Rights Reserved
 Fundamental of Optimization

Optimization Methods - Constrained

• Constrained Optimization
a.) Indirect approach – by transforming into unconstrained
problem.
b.) Exterior Penalty Function (EPF) and Augmented Lagrange
Multiplier
c.) Direct Method Sequential Linear Programming (SLP), SQP and
Steepest Generalized Reduced Gradient Method (GRG)

Figure 10: Descent Gradient or LMS

© 2008 Solutions 4U Sdn Bhd. All Rights Reserved
 Fundamental of Optimization

Optimization Methods (Cont.)

 Global Optimization – Stochastic techniques

• Simulated Annealing (SA) method – minimum

energy principle of cooling metal crystalline structure
• Genetic Algorithm (GA) – Survival of the fittest
principle based upon evolutionary theory

© 2008 Solutions 4U Sdn Bhd. All Rights Reserved

 Fundamental of Optimization

Optimization Methods (Examples)

Multivariable Gradient based optimization

J is the cost function to be minimized in two
dimension

The contours of the J paraboloid shrinks as it is

decrease
Figure 11: Multivariable Gradient based optimization
function retval = Example6_1(x)
% example 6.1
retval = 3 + (x(1) - 1.5*x(2))^2 + (x(2) - 2)^2;

>> SteepestDescent('Example6_1', [0.5 0.5], 20, 0.0001, 0,

1, 20)
Where
[0.5 0.5] -initial guess value
20 -No. of iteration
0.001 -Golden search tol.
0 -initial step size
1 -step interval
20 -scanning step
>> ans
2.7585 1.8960
Figure 12: Steepest Descent
© 2008 Solutions 4U Sdn Bhd. All Rights Reserved
 Fundamental of Optimization

Numerical Optimization
Newton –Raphson Method
1. Root Solver – System of nonlinear equations
2. (MATLAB – Optimization – fsolve)
f ( xi )
xi 1  xi 
f ' ( xi 1 )
2. One dimensional Solver –
(MATLAB - OPTIMIZATION Method )
f ' ( xi )
xi 1  xi 
f ' ' ( xi 1 )
Steepest Gradient Ascent/Descent
Methods
xi 1  f ( xi   .d i )
 f ( xi )   .f ( xi ).d i
f ( xi ).d i Is the magnitude of descent direction.
d 2  f ( xi ) achieve the steepest descent
d 2  f ( xi ) achieve steepest ascent

© 2008 Solutions 4U Sdn Bhd. All Rights Reserved

 Fundamental of Optimization

Steepest Gradient
Solve the following for two step steepest ascent 1.2
1 0
f ( x, y)  2.25xy  1.75 y  1.5x 2  2 y 1
2
The partial derivatives can be evaluated at the initial guesses, x = 1 and 0.8 max
y = 1, f
 3x  2.25 y  3(1)  2.25(1)  0.75
x 0.6

f
 2.25 x  4 y  1.75  2.25(1)  4(1)  1.75  0 0.4
y
0.2
Therefore, the search direction is –0.75i.
0
f (1  0.75h, 1)  0.5  0.5625 h  0.84375 h 2 0 0.2 0.4 0.6 0.8 1 1.2

This can be differentiated and set equal to zero and solved for h* = 0.33333. Therefore, the result for the
first iteration is x = 1 – 0.75(0.3333) = 0.75 and y = 1 + 0(0.3333) = 1. For the second iteration, the
partial derivatives can be evaluated as,
f
 3(0.75)  2.25(1)  0
x
f
 2.25(0.75)  4(1)  1.75  0.5625
y
Therefore, the search direction is –0.5625j.
f (0.75, 1  0.5625 h)  0.59375  0.316406 h  0.63281h 2

This can be differentiated and set equal to zero and solved for h* = 0.25.
Therefore, the result for the second iteration is x = 0.75 + 0(0.25) = 0.75 and y = 1 + (–0.5625)0.25 = 0.859375.

© 2008 Solutions 4U Sdn Bhd. All Rights Reserved

Fundamental of Optimization

%chapra14.5 … Contd
clear
clc
Clf x2
ww1=0:0.01:1.2;
ww2=ww1;
[w1,w2]=meshgrid(ww1,ww2);
J=-1.5*w1.^2+2.25*w2.*w1-2*w2.^2+1.75*w2;
cs=contour(w1,w2,J,70);
%clabel(cs);
hold
grid
w1=1; w2=1; h=0;
for i=1:10
syms h
dfw1=-3*w1(i)+2.25*w2(i);
dfw2=2.25*w1(i)-4*w2(i)+1.75;
fw1=-1.5*(w1(i)+dfw1*h).^2 + 2.25*(w2(i)+dfw2*h).*(w1(i)+…
dfw1*h)-2*(w2(i)+dfw2*h).^2+1.75*(w2(i)+dfw2*h);
J=-1.5*w1(i)^2+2.25*w2(i)*w1(i)-2*w2(i)^2+1.75*w2(i)
g=solve(fw1);
h=sum(g)/2;
w1(i+1)=w1(i)+dfw1*h;
w2(i+1)=w2(i)+dfw2*h;
plot(w1,w2) xi
pause(0.05)
End
MATLAB OPTIMIZATION TOOLBOX
%startchapra.m
w1(i), w2(i) function J=chaprafun(x) clc
w1=x(1); clear
w2=x(2) x0=[1 1];
J=-(-1.5*w1^2+2.25*w2*w1-2*w2^2+1.75*w2); options=optimset('LargeScale','off','Display','iter','Maxiter',…
20,'MaxFunEvals',100,'TolX',1e-3,'TolFun',1e-3);
[x,fval]=fminunc(@chaprafun,x0,options)

© 2008 Solutions 4U Sdn Bhd. All Rights Reserved

 Fundamental of Optimization

Newton –Raphson – Four Bar Mechanism

Sine and Cosine angle components - All angles referenced from global x - axis

In above equations θ1 = 0 as it is along the x-axis and other three angles are time varying.
1st angular velocity derivative

© 2008 Solutions 4U Sdn Bhd. All Rights Reserved

 Fundamental of Optimization

• If input is applied to link2 - DC motor then ω2 would be the input to the syste

In Matrix for

2. Numerical Solution for Non Algebraic Equations

_
f ( xi )
xi 1  x i 
f ' ( xi 1 )

© 2008 Solutions 4U Sdn Bhd. All Rights Reserved

Fundamental of Optimization

_
f (q)
qi  
f ' (qi )

© 2008 Solutions 4U Sdn Bhd. All Rights Reserved

 Fundamental of Optimization
_
 f ( i )
i   i  _ Newton-Raphson
f ' ( i )

%fouropt.m
function f=fouropt(x)
the = 0;
r1=12; r2=4; r3=10; r4=7;
f=-[r2*cos(the)+r3*cos(x(1))-r1*cos(0)-r4*cos(x(2));
r2*sin(the)+r3*sin(x(1))-r1*sin(0)-r4*sin(x(2))];

%startfouropt.m
clc
clear
x0=[0.1 0.1];
options=optimset('LargeScale','off','Display','iter','Maxiter',…
200,'MaxFunEvals',100,'TolX',1e-8,'TolFun',1e-8);
[x,fval]=fsolve(@fouropt,x0,options);
theta3=x(1)*57.3 Foursimmechm.m, foursimmech.mdl and possol4.m
theta4=x(2)*57.3

© 2008 Solutions 4U Sdn Bhd. All Rights Reserved

Fundamental of Optimization

References:

1) Steven C. Chapra , Raymond P. Canale, Numerical Methods for Engineers, McGraw Hill,
Singapore, 2006
2) Kalyanmoy Deb, Optimization for Engineering Design, Prentice Hall, New Dehli, 1996
3) Optimization Toolbox for use with MATLAB, User guide Ver. 3, MathWorks, Natick, MA, USA,
2006

© 2008 Solutions 4U Sdn Bhd. All Rights Reserved