2 Nonlinear Programming Models

Topics Convex sets and convex programming First-order optimality conditions Examples Problem classes

General NLP
Minimize f(x) s.t. gi(x) (, , =) bi, i = 1,,m x = (x1,,xn)T is the n-dimensional vector of decision variables f (x) is the objective function gi(x) are the constraint functions bi are fixed known constants

Convex Sets
Definition: A set S n is convex if every point on the line segment connecting any two points x1, x2 S is also in S. Mathematically, this is equivalent to x0 = x1 + (1)x2 S for all such 0 1. x1

x1

x2

x1

x2 x2

(Nonconvex) Feasible Region

S = {(x1, x2) : (0.5x1 0.6)x2 1 2(x1)2 + 3(x2)2 27; x1, x2 0}

x2

x1

Convex Sets and Optimization

Let S = { x n : gi(x) bi, i = 1,,m } Fact: If gi(x) is a convex function for each i = 1,,m then S is a convex set. Convex Programming Theorem: Let x n and let f (x) be a convex function defined over a convex constraint set S. If a finite solution exists to the problem Minimize { f (x) : x S } then all local optima are global optima. If f (x) is strictly convex, the optimum is unique.

Convex Programming
Min f (x1,,xn) s.t. gi(x1,,xn) bi i = 1,,m x1 0,,xn 0 is a convex program if f is convex and each gi is convex. Max f (x1,,xn) s.t. gi(x1,,xn) bi i = 1,,m x1 0,,xn 0 is a convex program if f is concave and each gi is convex.

Linearly Constrained Convex Function with Unique Global Maximum

Maximize f (x) = (x1 2)2 + (x2 2)2 subject to 3x1 2x2 6 x1 + x2 3 x1 + x2 7 2x1 3x2 4
x2

5 4 3 2 1

x1

(Nonconvex) Optimization Problem

First-Order Optimality Conditions

Minimize { f (x) : gi(x) bi, i = 1,,m } Lagrangian:L(x,) = f (x) + i (gi (x) bi )
m i=1

Optimality conditions Stationarity: L(x,) = f (x) + igi (x) = 0 Complementarity: igi(x) = 0, i = 1,,m Feasibility: gi(x) bi, i = 1,,m Nonnegativity: i 0, i = 1,,m
i=1 m

Importance of Convex Programs

Commercial optimization software cannot guarantee that a solution is globally optimal to a nonconvex program. NLP algorithms try to find a point where the gradient of the Lagrangian function is zero a stationary point and complementary slackness holds. Given we want L(x,) = f(x) + (g(x) b) L(x,) = f(x) + g(x) = 0 (g(x) b) = 0 g(x) b 0, 0
For a convex program, all local solutions are global optima.

Example: Cylinder Design

We want to build a cylinder (with a top and a bottom) of maximum volume such that its surface area is no more than s units. Max V(r,h) = r2h s.t. 2r2 + 2rh = s r 0, h 0
There are a number of ways to approach this problem. One way is to solve the surface area constraint for h and substitute the result into the objective function.
r

Solution by Substitution
s 2r 2 h= 2r s 2r 2 rs Volume = V = r2 [ r 3 ] = 2 2r

s 1/2 s 1/2 dV s = 0 r=( ) h = 2r r = 2( ) dr 6 6 V = r 2h s 3/2 = 2 ( 6) s 1/2 r = ( ) 6 s 1/2 ) h = 2( 6

Is this a global optimal solution?

Test for Convexity

rs r 3 V(r ) = 2 s dV(r) 3r 2 = 2 dr d2V(r ) dr 2 = 6r

d2V 0 for all r 0 dr 2 Thus V(r ) is concave on r 0 so the solution is a global maximum.

Advertising (with Diminishing Returns)

A company wants to advertise in two regions. The marketing department says that if $x1 is spent in region 1, sales volume will be 6(x1)1/2. If $x2 is spent in region 2, sales volume will be 4(x2)1/2. The advertising budget is $100.

Model: Max f (x) = 6(x1)1/2 + 4(x2)1/2 s.t.

x1 + x2 100, x1 0, x2 0

Solution: x1* = 69.2, x2* = 30.8, f (x*) = 72.1 Is this a global optimum?

Excel Add-in Solution

A B C D 1 Nonlinear Model 2 72.111 Change 3 2 4 TRUE Solve 5 TRUE 100 Variables 6 Change Relation Name: 7 Values: 8 Lower Bounds: 9 10 11 Linear Obj. Coef.: 12 Nonlinear Obj. Terms: 13 Nonlinear Obj. Coef.: 14 Constraints 15 Num. Name Value Rel. RHS 16 1 Con1 100 <= 100 17 2 Con2 0 <= 10000 18 19 E F G H I J K L Name: Adv100 Objective Terms Solver: Excel Solver Type: NLP1 Linear: 0 Type: Nonlinear Goal: Max NonLinear 1: 72.111 Sens.: Yes 0 Objective: 72.111NonLinear 2: 1 2 X1 X2 69.231 30.769 0 0 M N O

Comp. Time 00:00 Status Optimal

0 8.3205 6

0 5.547 4

Linear Constraint Coefficients 1 1 0 0

Portfolio Selection with Risky Assets (Markowitz)

Suppose that we may invest in (up to) n stocks. Investors worry about (1) expected gain (2) risk. Let rj = random variable associated with return on stock j

j = expected return on stock j jj = variance of return for stock j

We are also concerned with the covariance terms:

ij = cov(ri, rj)
If ij > 0 then returns on i and j are positively correlated. If ij < 0 returns are negatively correlated.

Decision Variables: xj = # of shares of stock j purchased

Expected return of the portfolio: R(x) = jxj
j =1 n

Variance (measure of risk): V(x) =

11 12 1 1 = 21 22 1 1

i =1 j =1

ijxixj

 11 12 1 1 = If , then buying stock 2 is just like 21 22 1 1

buying additional shares of stock 1.

Nonlinear optimization models

Let pj = price of stock j b = our total budget

= risk-aversion factor (when = 0 risk is not a factor)

Consider 3 different models: 1) Max f (x) = R(x) V(x) s.t.
j =1 n

pj xj b, xj 0, j = 1,,n

where 0 is determined by the decision maker

2) Max f (x) = R(x) s.t. V(x) , pjxj b, xj 0, j = 1,,n

j =1 n

where 0 is determined by the investor. Smaller values of represent greater risk aversion. 3) Min f (x) = V(x) s.t. R(x) , pj xj b, xj 0, j = 1,,n
j =1 n

where 0 is the desired rate of return (minimum expectation) is selected by the investor.

Hanging Chain with Rigid Links

10ft

1 ft

x
y

each link

What is equilibrium shape of chain? Decision variables: Let (xj, yj), j = 1,,n, be the incremental horizontal and vertical displacement of each link, where n 10. Constraints: xj2 + yj2 = 1, j = 1,,n, each link has length 1 x1 + x2 + y1 + y2 +

+ xn = 10, net horizontal displacement + yn = 0, net vertical displacement

Objective: Minimize chains potential energy Assuming that the center of the mass of each link is at the center of the link. This is equivalent to minimizing
1 y + (y + 1 y ) + (y + y + 1 y ) + 1 1 1 2 2 3 2 2 2

+ (y1 + y2 +

1 + yn-1 + 2 yn)

1 1 = (n 1 + 2 ]y1 + (n 2 + 2 )y2 1 + (n 3 + 2 )y3 +

3 1 + 2 yn-1 + 2 yn

Summary
Min (n j + )yj
j =1 n

s.t. xj2 + yj2 = 1, j = 1,,n x1 + x2 + y1 + y2 +

+ xn = 10 + yn = 0

Is a local optimum guaranteed to be a global optimum?

No!
Constraints xj2 + yj2 = 1 for all j yield a nonconvex feasible region so there may be several local optima. Consider a chain with 4 links:

These solutions are both local minima.

Direct Current Network

10 20

I2

I4
I3

100v

I1
I6

10
I7

I5

20

Problem: Determine the current flows I1, I2,,I7 so that the total content is minimized Content: G(I) =

v(i)di for I 0 and G(I) = I v(i)di for I < 0

Solution Approach
Electrical Engineering: Use Kirchoffs laws to find currents when power source is given. Operations Research: Optimize performance measure in network taking flow balance into account. Linear resistor: Voltage, v(I ) = IR Content function, G(I ) = I 2R/2 Battery: Voltage, v(I ) = E Content function, G(I ) = EI

Network Flow Model

Network diagram:
2
5 I2
2

10 I 4

-100 I1

5 I3

10 I5

0 I6

0 I7

Minimize Z = 100I1 + 5I2 + 5I3 + 10I4 + 10I5 subject to I1 I2 = 0, I2 I3 I4 = 0, I5 I6 = 0, I5 + I7 = 0, I3 + I6 I7 = 0, I1 I6 = 0 Solution: I1 = I2 = 50/9, I3 = 40/9, I4 = I5 = 10/9, I6 = 50/9, I7 = 10/9

NLP Problem Classes

Constrained vs. unconstrained Convex programming problem Quadratic programming problem
f (x) = a + cTx + xTQx, Q 0

Separable programming problem

f (x) = j=1,n fj(xj)

Geometric programming problem

g(x) = t=1,T ctPt(x), Pt(x) = (x1 t1) . . . (xn tn), xj >0
a a

Equality constrained problems

What You Should Know About Nonlinear Programming

How to identify a convex program. How to write out the first-order optimality conditions. The difference between a local and global solution. How to classify problems.