Optimization Techniques

Optimal Problem Formulation

Optimization
The idea behind optimization is to find the best
solution from a domain of possible solutions.
Optimization methods provide mathematical tools
that allow the search for this best solution to
carried out in a rational and efficient way.
Before these tools can be applied the design
problem needs to be recast in an appropriate
form.

Conventional Optimization
Translate a descriptive statement of the design
problem into a mathematical statement for
optimization using a five-step process
Identify and define the problems design
variables
Identify a function for the problem that needs to
be optimized
Identify and define the problems constraints

Optimization in Design
Need Identified
Choose Design Variables
Formulate Constraints
Formulate Objective Function
Set up Variable Bounds
Choose an Optimization Algo
Obtain solutions

Optimization
x2

Set of all workable

or functional designs

H 2 : x2 > c2

H1 : X1> c1

Set of all acceptable

designs.

Optimal Design
U(x1,x2) = Umax
x1

Steps for optimization

Step 1: Project/problem statement.
Step 2: Data and information collection.
Step 3: Identification/definition of design
variables.
Step 4: Identification of a criterion to be
optimized.
Step 5: Identification of constraints.

Step 1: Project/Problem Statement

The formulation process begins by developing a descriptive

statement for the
project/problem, which is usually done by the projects
owner/sponsor. The statement
describes the overall objectives of the project and the
requirements to be met.

Step 2: Data and Information

Collection
Is all the information available to solve the
problem?
To develop a mathematical formulation of the
problem, we need to gather material
properties, performance requirements,
resource limits, cost of raw materials, and other
relevant information. In addition, most
problems require the capability to analyze trial
designs. Therefore, analysis procedures and
analysis tools must be identified at this stage.

Step 3: Identification/Definition of
Design Variables
Identify a set of variables that describe the system,
called design variables.
They are referred to as optimization variables and are
regarded as free because we should be able to assign
any value to them. Different values for the variables
produce different designs.
The design variables should be independent of each
other as far as possible. If they are dependent, then their
values cannot be specified independently.
The number of independent design variables specifies
the design degrees of freedom for the problem.
For some problems, different sets of variables can be
identified to describe the same system. The problem
formulation will depend on the selected set. Once the
design variables are given numerical values, we have a
design of the system.

Step 3: Identification/Definition of
Design Variables
Design variables should be independent of each other as
far as possible. If they are not, then there must be some
equality constraints between them. Conversely, if there
are equality constraints in the problem, then the design
variables are dependent.
A minimum number of design variables required to
formulate a design optimization problem properly exists.
As many independent parameters as possible should be
designated as design variables at the problem
formulation phase. Later on, some of the variables can
be assigned fixed values.
A numerical value should be given to each variable once
design variables have been defined to determine if a trial
design of the system is specified.

Step 3: Identification/Definition of
Design Variables
Identify a set of variables that describe the system,
called design variables.
They are referred to as optimization variables and are
regarded as free because we should be able to assign
any value to them. Different values for the variables
produce different designs.
The design variables should be independent of each
other as far as possible. If they are dependent, then their
values cannot be specified independently.
The number of independent design variables specifies
the design degrees of freedom for the problem.
For some problems, different sets of variables can be
identified to describe the same system. The problem
formulation will depend on the selected set. Once the
design variables are given numerical values, we have a
design of the system.

Step 4: Identification of a Criterion to

Be Optimized
How do I know that my design is the best?
There can be many feasible designs for a system, and some are
better than others. To compare different designs, we must
have a criterion. The criterion must be a scalar function whose
numerical value can be obtained once a design is specified,
i.e., it must be a function of the design variable vector x.
Such a criterion is usually called an objective function for the
optimum design problem, which needs to be maximized or
minimized depending on problem requirements.
A criterion that is to be minimized is usually called the cost
function.
It is emphasized that a valid objective function must be influenced
directly or indirectly by the variables of the design problem;
otherwise, it is not a meaningful objective function. Note that an
optimized design has the best value for the objective function.

Step 5: Identification of Constraints

All restrictions placed on a design are collectively called
constraints. The final step in the formulation process is to
identify all constraints and develop expressions for them. Most
realistic systems must be designed and fabricated within given
resources and performance requirements.
Linear and Nonlinear Constraints
Feasible Design
A design meeting all requirements is called a feasible design
(acceptable or workable). An infeasible design (unacceptable)
does not meet one or more of the requirements.
Equality and Inequality Constraints
Implicit Constraints
When there are implicit functions in the problem formulation, it is
not possible to formulate the problem functions explicitly in
terms of design variables alone. Instead, we must use some
intermediate variables in the problem formulation.

Important definitions
Objective Function:

It represents the quantity (U) which is to be optimized (the
objective) as a function of one or more independent variables
(x1, x2, x3)

The best form to put the objective function in depends on the
optimization technique to be employed.
U = U( x1 , x2 , x3) Uopt

Important definitions
Design Variables:
The independent variables (x1, x2, x3) that the objective
function depends on.

It is generally best to minimize the number of design variables
the more variables the tougher the optimization will be.

Important definitions
Constraints:

Relations which limit the possible (physical limitations) or the
permissible (external constraints) solutions to the objective
function.

Constraints come in two mathematical flavors.
Equality Constraints: Often come from fundamental physics
considerations (e.g. cons. of mass)
Inequality Constraints: Often from safety, cost, space, material
strength limits etc.

Generally Equality Constraints are easier to deal with than

inequality constraints.

it is desirable to reduce the number of constraints.

Distinction between Equality and In

equality Constraints

Mathematical Formulation
Objective Function of n independent design variables:
For U( x1, x2, x3xn);

Find Uopt

Equality Constraints:
Gi( x1, x2, x3)=0

i=1,2,,m

Inequality Constraints:
Hj(x1, x2, x3) < or > Cj ;

j=1, 2 ,, l.

If n > m An Optimization problem results;

If n = m A unique solution existsjust solve all equations
simultaneously;

If n < m The problem is overconstrained no solution
which satisfies all of the constraints is possible.

Classification of Optimization Techniques

Calculus based Techniques

Lagrange Multipliers

Search Methods

Elimination Methods
Exhaustive
Fibonacci
golden section search

Hill Climbing techniques

Lattice Search
Steepest ascent

Programming methods

Linear Programming

Geometric Programming

Search- Exhaustive
x2
H2 : X2> c2

H1 : X1> c1

Optimal Design
U(x1,x2) = Umax

Note: None of the search

points exactly hits the
optimum. The space
between search points is
known as the interval of
uncertainty

x1

Search Lattice
1
3

*
2
4

Programming Methods
These methods have nothing to do with Programming in the sense that
you usually think of it!
Linear Programming: Very powerful, but very limited!

Applies only when the objective function and all constraints can be
expressed as Linear Functions Not in thermal/fluid systems

Quadratic Programming (QP) Problems

If the objective function is a quadratic function and all constraint functions are linear
functions ofoptimization variables/ the problem is called a quadratic programming
problem.

Dynamic Programming: Related to optimizing a process

Lets you find the best order to do steps in

useful in Project Management and Assembly optimization

Geometric Programming: Similar to Linear Programming, but now all

functions must be polynomials.

Reduces the restrictions on linear programming quite a bit

Very useful where empirical correlations for system behavior are known

More difficult and computationally intensive than Linear Prog.

Functions
Optimization deals with functions. A function is simply a mapping from one
space to another. (that is, a set of instructions describing how to get from
one location to another)

f : A B

This is the range

This is the domain

This is a function
For any

x A

and

yB

Note: A function maps each value of x to one and only one value for y

y = f (x)

Examples of Single variable Function

For

0 x 5
Domain

y = 3x + 5
f (x)

A = [0 ,5 ]

x A

5 y 20
Range

B = [ 5, 20 ]
f (3) = 14
14

Examples of Single variable Function

y

20

y = 3x + 5

Range

Y =14

A straight line is a function.

For

0 x5

x
0

Domain

5
X =3

Optimization of a function
It involves finding the maximum value for y over an allowable range.

max {3 x + 5}

20

Range

0 x 5

Here, the
optimum occurs
at x = 5 (y = 20)

x
0

Domain

What is the solution to this optimization problem?

1
max

5 x
0 x 10

x
5

10
There is no optimum
because f(x) is
discontinuous at x = 5

What is the solution to this optimization problem?

max{2 x}

12

0 x<6

There is no optimum
because the domain is open
(that is, the maximum
occurs at x = 6, but x = 6 is
NOT in the domain!)

x
0

What is the solution to this optimization problem?

y
12

max

{ x}

x0

There is no optimum
because the domain is
unbounded (x is allowed to
become arbitrarily large)

x
0

One-Variable case
The Weierstrass Theorem provides sufficient conditions for an optimum
to exist, the conditions are shown in next slides

Here suppose f(x) is continuous over the domain of

x.

Sufficient conditions guarantee a

solution, but are not required

max{ f ( x)}

Necessary conditions are required

for a solution to exist

The domain for f(x) is closed

and bounded.

Derivatives
Formally, the derivative of a function is defined as

df
f ( x + x) f ( x)
f ' (x ) =
= lim

0
dx
x

All you need to remember is the derivative represents a slope.

y
f ( x + x )
f ( x + x ) f ( x )

f (x)

f ( x + x) f ( x)

Slope =

x + x

Example:

f ( x) = x 2

f ( x + x) f ( x)

( x + x)2 x 2
=

x
x

2x

x 2 + 2 xx + x 2 x 2 2 xx + x 2
=
=
= 2 x + x
x
x

lim 2 x + x = 2 x

x 0

Unconstrained maximization
Strictly speaking, no problem is truly unconstrained. However,
sometimes the constraints dont influence the maximum.
First Order Necessary Conditions
If x* is a solution to the optimization problem:
Min{ f(x) } or Max{ f(x) } then

f ' ( x*) = 0

How can we be sure we are at a minimum?

Secondary Order Necessary Conditions
*
x
If
is a solution to the maximization problem:

max{ f ( x)}
x

then

f ' ' ( x*) < 0

*
x
If
is a solution to the minimization problem:

min{ f ( x)}
x

then

f ' ' ( x*) > 0

The second derivative is the rate of change of the first derivative

x
x*

x
x*

f ' ( x*) = 0

f ' ( x*) = 0

f ' ' ( x*) < 0

f " ( x*) > 0

Slope is decreasing

Slope is increasing

Example1
The purpose of this project is to design a can to hold at
least 400 ml of liquid, as well as to meet other design
requirements (1 ml = 1cm3). The cans will be produced
in the billions so it is desirable to minimize manufacturing
costs. Since cost can be directly related to the surface
area of the sheet metal, it is reasonable to minimize the
amount of sheet metal required to fabricate the can.
Fabrication, handling, aesthetics, and shipping
considerations impose the following restrictions on the
size of the can:
The diameter should be no more than 8 cm and no less
than 3.5 cm, whereas the height should be no more than
18 cm and no less than 8 cm.

Example 2
The goal of this project is to choose insulation
thickness t to minimize the life-cycle cooling
cost for a spherical tank. The cooling costs
include the cost of installing and running the
refrigeration equipment, and the cost of
installing the insulation. Assume a 10-year life,
10 percent annual interest rate, and no salvage
value. The tank has already been designed
having r (m) as its radius.

Quiz
Suppose that a company owns a corporate jet. The annual
expenses are as follows:
 You pay your flight crew (pilot, co-pilot, and navigator a
combined annual salary of $500,000.
 Annual insurance costs on the jet are $250,000
 Fuel/Supplies cost $1,500 per flight hour
 Per hour maintenance costs on the jet are proportional to the
number of hours flown per year.
 Maintenance costs (per flight hour) = 1.5(Annual Flight Hours)
If you like to minimize the hourly cost of your jet, how many hours
should company use it per year?

Quiz Sol
Let x = Number of Flight Hours

Hourly Cost =

$750 ,000
+ $1500 + $1.5 x
x

$750 ,000

min
+ $1500 + $1.5 x
x >0
x

First Order Necessary Conditions

$750,000
750,000
+
$
1
.
5
=
0

x
=
= 707hrs
2
x
1.5

Hourly Cost ($)

Quiz Sol

Annual Flight Hours

Quiz Sol
Let x = Number of Flight Hours
$ 750 ,000

+ $ 1500 + $ 1 .5 x
min
x
x

First Order Necessary Conditions

$750,000
750,000
+ $1.5 = 0 x =
= 707hrs
2
x
1.5
Second Order Necessary Conditions

$1,500,000
>0
3
x

For X>0