MIE 1620: Linear Programming and Network Flows

Final Review Notes

Marlee Vandewouw
November 30, 2014

Contents
1 Introduction to Linear Programming
1.1 General and Standard Form LPs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1.2 Approximating Convex Optimization Problems using LPs . . . . . . . . . . . . . . . . . .

3
3
4

2 Geometry of Linear Programming

2.1 Polyhedra and Convex Sets . . . . . . . . . . . .
2.2 Definitions of a Polyhedrons Corner Point . . . .
2.3 Polyhedra and Basic Solutions in Standard Form
2.4 Degeneracy . . . . . . . . . . . . . . . . . . . . .
2.5 Existence and Optimality of Extreme Points . . .

.
.
.
.
.

4
4
4
5
6
6

Simplex Method
Theory Behind the Simplex Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Implementation of the Simplex Method . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Finding an Initial BFS for the Simplex Method . . . . . . . . . . . . . . . . . . . . . . . .

6
6
8
8

3 The
3.1
3.2
3.3

4 Duality
4.1 Fundamental Duality Theorems
4.2 Dual Simplex Method . . . . .
4.3 Duality, Degeneracy and Farkas
4.4 Cones and Extreme Rays . . .
4.5 Representation of Polyhedra . .

. . . . .
. . . . .
Lemma
. . . . .
. . . . .

5 Sensitivity Analysis
5.1 Changes in b . . . . . . . . . . .
5.2 Changes in c . . . . . . . . . . .
5.3 Changes in A . . . . . . . . . . .
5.4 Add a New Variable/Column . .
5.5 Add a New Inequality Constraint
5.6 Add a New Equality Constraint .

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

.
.
.
.
.

9
9
10
10
11
11

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

11
11
12
12
13
13
13

6 Network Flow Problems

6.1 Directed and Undirected Graphs . . . . . .
6.2 Graph Theorems . . . . . . . . . . . . . . .
6.3 Network Flow Formulation . . . . . . . . .
6.4 Special Cases of the Network Flow Problem
6.5 The Network Simplex Algorithm . . . . . .
6.6 The Maximum Flow Problem . . . . . . . .
6.7 Duality in Network Flow Problems . . . . .
6.8 Network Flow Sensitivity Analysis . . . . .
6.9 Network Flow Complementary Slackness . .
6.10 The Assignment Problem . . . . . . . . . .
6.11 The Shortest Path Problem . . . . . . . . .

.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.

13
13
14
14
15
15
17
17
18
18
18
19

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

.
.
.
.
.
.

7 Interior Point Methods

7.1 The Primal Path Following Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

20
20

8 Large Scale Optimization

8.1 Delayed Column Generation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
8.2 Cutting Plane Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
8.3 Dantzig-Wolfe Decomposition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

21
21
21
21

9 Robust Optimization
9.1 General Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
9.2 Inverse Optimization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

23
23
23

Chapter 1

Introduction to Linear Programming

1.1

General and Standard Form LPs

Definition 1.1.1 (LP Problem in General Form). An LP problem in general form is:
minimize c0 x
subject to a0i x bi
a0i x
a0i x

i M1

bi

i M2

= bi

i M3

xj 0

j N1

xj 0

j N2

Or, more compactly:

minimize c0 x
subject to Ax b

Definition 1.1.2 (LP Problem in Standard Form). An LP problem in standard form is:
minimize c0 x
subject to Ax = b
x0
To transform a general form to a standard form LP, follow the following algorithm.
Algorithm 1 Transform General Form to Standard Form
1: Eliminate the free variables:

xj = x+
j xj

x+
j 0
x
j 0
2:

Eliminate inequality constraints:

Ax b

Ax s = b,

New A matrix:

[A

I]

New x vector:

[x

s]0

s0

1.2

Approximating Convex Optimization Problems using LPs

Definition 1.2.1 (Convex Function). A function f : Rn R is convex if x, y Rn , and [0, 1] we

have:
f (x + (1 )y) f (x) + (1 )f (y)
Example 1.2.1 (Residual Problem). Since linear functions are convex, the maximum of a set of linear
functions is also convex. Let c0i x + di , i I be a set of linear functions. Let f (x) = maxiI {c0i x + di }.
Then:
min f (x)

min max{c0i x + di }

min

s.t.

z c0i x + di ,

iI

x,z

iI

Chapter 2

Geometry of Linear Programming

2.1

Polyhedra and Convex Sets

Definition 2.1.1 (Hyperplanes and Halfspaces). Let a be a non-zero vector in Rn . Let b be a scalar.
Then:
(a) {x Rn |a0 x = b} is a hyperplane
(b) {x Rn |a0 x b} is a halfspace (like a constraint)
Definition 2.1.2 (Polyhedron). A polyhedron is a set of the form {x Rn |Ax b} where A is an m n
matrix and b is a vector in Rm . In other words, a polyhedron is the intersection of a finite number of
halfspaces of the form {x|a0i x bi , i I}.
Definition 2.1.3 (Convex Set). A set S is convex if x, y S, and [0, 1], we have:
x + (1 )y S
Theorem 2.1.4 (Polyhedra and Convexity). A polyhedron P = {x Rn |Ax b} is convex.
Definition
2.1.5 (Convex Combination and Hull). Let x1 , . . . , xk Rn and 1 , . . . , k 0 with
Pk
i=1 i = 1. Then:
P
(a) y = ki=1 i xi is a convex combination of the vectors x1 , . . . , xk .
(b) The convex hull of x1 , . . . , xk is the set of all possible convex combinations of x1 , . . . , xk .

2.2

Definitions of a Polyhedrons Corner Point

Definition 2.2.1 (Extreme Point). x P is an extreme point of a polyhedron P if there do not exist
y 6= x, z 6= x, y, z P and [0, 1] such that x = y + (1 )z.
4

Definition 2.2.2 (Vertex). x P is a vertex of a polyhedron P if c such that c0 x < c0 y, y P , y 6= x.

Definition 2.2.3 (Active Constraint). Any constraint that holds with equality at a particular x is an
active or binding constraint at x.
Definition 2.2.4 (Basic Solution). Consider a polyhedron P Rn defined by linear equality and inequality constraints. Then:
(a) x is a basic solution if:
(i) all equality constraints are active
(ii) out of all constraints active at x, n of them have linearly independent ai
(b) x is a basic feasible solution (bfs) if x is a basic solution and it is feasible
Theorem 2.2.5 (Equivalence of Corner Point Definitions). Let P be a non-empty polyhedron and let
x P . Then, the following are equivalent:
(a) x is a vertex
(b) x is an extreme point
(c) x is a bfs

2.3

Polyhedra and Basic Solutions in Standard Form

Definition 2.3.1 (Polyhedra in Standard Form). In standard form, consider polyhedra of the form
P = {x|Ax = b, x 0} where x Rn , b Rm , and A Rmn . Assume the rows of A are linearly
independent to avoid redundancy, and m n
Definition 2.3.2 (Basic Solutions in Standard Form). In standard form, the constraint Ax = b contributes m active constraints, and thus n m of the constraints x 0 must be active. Overall, we need
n linearly independent active constraints.
Theorem 2.3.3 (Conditions for Basic Solutions). A vector x Rn is a basic solution (in a standard
form polyhedron) if and only if Ax = b and there exist the set of indices B(1), . . . , B(m) such that:
(a) The columns AB(1) , . . . , AB(m) are linearly independent
(b) If i
/ {B(1), . . . , B(m)}, then xi = 0
Definition 2.3.4 (Basic Matrix, Columns, and Variables). Let B be a matrix constructed with the
column vectors AB(i) , i {B(1), . . . , B(m)}, and xB the column vector consisting of entries xB(i) , i
{B(1), . . . , B(m)}. Then, B is called a basic matrix, AB(1) , . . . , AB(m) are basic columns which form a
basis of Rn , and xB is a vector of basic variables (other xj s are nonbasic).
To construct a basic solution, follow the following algorithm.
Algorithm 2 Construct a Basic Solution
1: Pick m indices B(1), . . . , B(m) such that AB(1) , . . . , AB(m) are linearly independent.
2: Set xj = 0 for j 6= B(1), . . . , B(m).
3: Solve Ax = b for unknown xB(1) , . . . , xB(m) , i.e. BxB = b.

2.4

Degeneracy

Definition 2.4.1 (Degenerate Basic Solution). A basic solution x Rn is degenerate if more than n
constraints are active at x.
Definition 2.4.2 (Degenerate Basic Solution in Standard Form Polyhedra). Let P = {x Rn |Ax =
b, x 0}, A Rmn . A basic solution x Rn is degenerate if more than n m of the components of x
are zero, i.e. if xB(i) = 0 form some i.

2.5

Existence and Optimality of Extreme Points

Definition 2.5.1 (Contains a Line). A polyhedron P Rn contains a line if there exists a vector x P
and a non-zero vector d Rn such that x + d P for all scalars .
Theorem 2.5.2 (Existence of Extreme Points). Suppose P = {x|Ax b}, where A is an m n matrix
with m n and P is non-empty. Then, the following are equivalent.
(a) P has at least one extreme point
(b) P does not contain a line
(c) There exists n rows of A that are linearly independent
Additionally, the following hold:
(a) Every nonempty bounded polyhedron has at least one bfs
(b) Every nonempty polyhedron in standard form has at least one bfs
Theorem 2.5.3 (Optimality of Extreme Points). Consider:
min c0 x
xP

If P has at least one extreme point, then either the optimal cost is , or there exists an extreme point
that is optimal.

Chapter 3

The Simplex Method

For the following chapter, only standard form problems will be considered where rank(A) = m n.

3.1

Theory Behind the Simplex Method

The objective behind the simplex method is:

(a) If c0 d 0, d, then terminate.
(b) If d such that c0 d < 0, move along the direction d.

Definition 3.1.1 (Feasible Direction). Let x P . A vector d Rn is a feasible direction at x is there

exists a positive scalar for which x + d P .
Definition 3.1.2 (Reduced Costs). For each variable xj :
cj = cj c0B B 1 Aj
is the reduced cost. The reduced cost of basic variables is always zero. The reduced cost of nonbasic
variables can be positive, zero, or negative.
Theorem 3.1.3 (Determining Optimality from Reduced Costs). Let x be a bfs with basic matrix B and
corresponding reduced costs c. Then,
(a) If c 0, then x is optimal.
(b) If c < 0, with j a nonbasic index, and x is non-degenerate, then x is not-optimal.
An optimal non-degenerate solution satisfies:
(a) B 1 b 0 (feasibility - non-negativity x)
(b) c0 = c0 c0B B 1 A 0 (non-negativity of reduced cost)
How far along the direction d can we go? Start from the point x and travel to x + d. If d improves the
cost, move as far as possible along the line while remaining feasible. Thus:
= max{ > 0|x + d P }
We only leave the polyhedron P is some xi becomes negative. There are two cases:
(a) d 0, then x + d 0, > 0, thus =
(b) di < 0 for some i, then:
xi + di 0 = i

xi
di

= = min
{i|di <0}

= =

xi
di

min

{i=1,...,m|dB(i) <0}

xB(i)
dB(i)

This leads to the first version of the Simplex Algorithm.

Algorithm 3 Simplex Method 1.0
1: Start with a bfs x and its corresponding basic columns AB(1) , . . . , AB(m)
2: Compute c
j = cj c0B B 1 Aj for all nonbasic indices j. If cj 0 for all j, the current bfs is optimal,
then STOP. Else, choose j such that cj < 0 as the entering variable.
3: Compute u = B 1 Aj (i.e. u = d). If ui 0 i, the optimal cost is . STOP.
xB(i)
x
= B(l)
4: = min{i=1,...,m|dB(i) <0} d
ul for some l, the exiting variable.
B(i)

5: xj , xB(i) xB(i) ui , xB(l) 0

6: New basis: Aj replaces AB(l) . Return to Step 2.

3.2

Implementation of the Simplex Method

The following algorithm is revised Simplex Algorithm.

Algorithm 4 Revised Simplex Method
1: Proceed the same as the Simplex Method 1.0, but also keep the matrix B 1 .
Do elementary row operations
2: Maintain the matrix [B 1 |u], where u is the non-identity row of B 1 B.
th
so u becomes a matrix with 0s except for the i element, which becomes 1.
Definition 3.2.1 (Full Tableau Implementation). Maintain and update the following tableau:
0

cB xB
|
c1
. . . cj
...
cn

xB(1)
|
|
. . . u1 . . .
|

..

..
.
|
|
...
.
...
|

xB(l)
|
B 1 A1 . . . ul
. . . B 1 An

..

..
.
|
|
...
.
...
|
xB(m)
|
|
. . . um . . .
|
where c0B xB is the negative of the objective function, cj is the entering variable, and the corresponding
column is the pivot column, the top row is the zeroth row (the reduced costs), xB(l) is the exiting variable,
x
and the corresponding row is the pivot row, and ul is the pivot element. Compute B(i)
ui for all ui > 0.
The smallest ratio corresponds to the exiting variable. Perform row operations to make ul = 1, and the
rest of the ui s to equal 0. Perform the same operations to the zeroth row, which will produce the correct
results.

3.3

Finding an Initial BFS for the Simplex Method

Definition 3.3.1 (Auxiliary Problem). Starting with the constraints Ax = b and x 0, introduce
artificial variables and form the auxiliary problem:
minimize y1 + . . . + ym
subject to Axy = b
x, y 0
Use the simplex method to solve the auxiliary problem using the initial bfs x = 0 and y = b. If the
optimal cost is 0, then y = 0, so there exists an x such that Ax = b, x 0. Otherwise, the original
problem is infeasible. If the auxiliary problem is feasible, if the basic variables are only xs, then drop
the columns corresponding to the ys. Otherwise, drive ys out of the basis until only xs remain.
This leads to the two-phase simplex method:

Algorithm 5 Two Phase Simplex Method

1: Phase I:
1: Formulate and solve the auxiliary problem.
2: If the optimal cost is greater than zero, the original problem is infeasible. Else, we have a feasible
solution to the original problem.
3: Drive out any artificial variables and eliminate redundant constraints (if any), and terminate with
the basic columns of A.
4: Phase II:
1: Start with the bfs x, a basis from the last step of Phase I.
2: Compute reduced costs.
3: Run the Simplex Method.

Chapter 4

Duality
Definition 4.0.2 (Primal and Dual). For every variable in the primal/dual, there is a constraint in the
dual/primal, so the number of constraints in the primal equals the number of variables in the dual, and
vice versa. To convert between the two, use the following table:
Primal
min c0 x
s.t. xj ()0
a0i x = bi
xj free
a0i x ()bi

Dual
max p0 b
s.t. p0 Aj ()cj
pi free
p0 Aj = cj
pi ()0

The dual of the dual is the primal, and equivalent problems have equivalent duals.

4.1

Fundamental Duality Theorems

Theorem 4.1.1 (Weak Duality). If x is the primal feasible solution, and p is a dual feasible solution,
then p0 b c0 x. Corollary:
(a) If the optimal cost of the primal is , then the dual is infeasible.
(b) If the optimal cost of the dual is +, then the primal is infeasible. item Let x be primal feasible, p
be dual feasible, and c0 x = p0 b. Then, x and p are the primal/dual optimal, respectively.
Theorem 4.1.2 (Strong Duality). If a linear program has an optimal solution x , then so does its dual
(p ) and, moreover, the optimal costs are equal, i.e. c0 x = (p )0 b.
The following table can be used to determine the existence of solutions for the primal and dual.
PP

PP Primal
PP
Dual
PP
P
Finite Optimum

Infeasible

Finite Optimum

Infeasible

3
7
7

7
7
3

7
3
3

Theorem 4.1.3 (Complementary Slackness). Let x be primal feasible and p be dual feasible. Then,
(x, p) is optimal if and only if:
(a) pi (a0i x bi ) = 0,
(b) (cj p0j Aj )xJ = 0,

4.2

i
j

Dual Simplex Method

Definition 4.2.1 (Dual Feasibility). Consider a standard form primal, with a basis B and a primal bfs
xB = B 1 b 0. B gives rise to a dual basic solution p0 = cB B 1 . Dual feasibility is p0 A c0 which is
equivalent to having non-negative reduced costs, i.e. c0 0.
For the dual simplex method, start with a dual feasible solution (and primal infeasible) and iterate until
it is primal feasible while maintaining dual feasibility. It is used when a dual bfs is readily available.
Algorithm 6 Dual Simplex Method
1: Select xB(l) < 0. If non exists, the we have an optimal bfs and the algorithm terminates.
2: Examine the lth row (xB(l) , v1 , . . . , vn ) where vi is the lth component of B 1 Ai . For each vi < 0,
c
compute |vcii | . Let j be such that |vjj | = mini:vi <0 |vcii | . If vi > 0 for all i then the optimal dual cost
is unbounded, and the algorithm terminates.
3: Pivot on vj : Aj enters the basis and AB(l) leaves the basis. Same mechanics (elementary row operations) as before for the primal simplex method.

4.3

Duality, Degeneracy and Farkas Lemma

Definition 4.3.1 (Dual Degeneracy). Dual degeneracy occurs when more than m reduced costs are equal
to 0, which is equivalent to at least one non-basic reduced cost equaling 0.
Theorem 4.3.2 (Corollary of Complementary Slackness). (a) If the primal has a non-degenerate optimal solution, the dual has a unique solution.
(b) If the dual has a non-degenerate optimal solution, the primal has a unique solution.
Theorem 4.3.3 (Farkas Lemma). Let A be an m n matrix, and b Rm . Exactly one of the two
alternatives hold:
(a) There exists some x 0 such that Ax = b.
(b) There exists some vector p such that p0 A 0 and p0 b < 0.
Theorem 4.3.4 (Farkas Lemma Corollary). Suppose the system Ax b has at least one solution
and let d be some scalar. Then, the following are equivalent:
(a) Every feasible solution to Ax b satisfies c0 x d.
(b) There exists some vector p > 0 such that p0 A = c0 and p0 b d.

10

4.4

Cones and Extreme Rays

Definition 4.4.1 (Cones). A set C Rn is a cone if 0, and x C, we have that x C.

(a) If C is nonempty, then ~0 C always.
(b) P = {x Rn |Ax 0} is a polyhedral cone.
(c) The only possible extreme point of a cone is the zero vector.
(d) A cone with an extreme point is pointed.
Theorem 4.4.2 (Cone Theorem). Let C = {x|Ax 0}. Then, the following are equivalent:
(a) The zero vector is an extreme point of C.
(b) C does not contain a line.
(c) n vectors out of a1 , . . . , am (rows of A) that are linearly independent.
Definition 4.4.3 (Recession Cones). A recession cone at y is the set of all directions d such that
y + d P , P . If P = {x|Ax b}, then the recession cone at y is {d|A(y + d) b, 0}, which
is equivalent to {d|Ad 0}, which is independent of y (similar for equality case).
Definition 4.4.4 (Rays). Nonzero elements of the recession cone are called rays.
(a) A non-zero element x of a polyhedral cone is an extreme ray if there are (n 1) linearly independent
active constraints at x.
(b) An extreme ray of a recession cone of P is an extreme ray of P .
Theorem 4.4.5 (Extreme Points and Extreme Rays). Consider minxP c0 x, where P = {x|Ax b} and
assume P has at least one extreme point. Then, the following are equivalent:
(a) The optimal cost is .
(b) There exists an extreme ray d such that c0 d < 0

4.5

Representation of Polyhedra

Theorem 4.5.1 (Resolution Theorem). Let P = {x|Ax b} be a nonempty polyhedron with at least
one extreme point. Let x1 , . . . , xk be the extreme points and w1 , . . . , wr be the complete set of extreme
rays of P . Let:
k
r
k
X
X
X
Q=
i xi +
j wj | 0,
i = 1, j 0
i=1

j=1

i=1

Then, Q = P . The converse holds as well. Also, a non-empty bounded polyhedron is the convex hull of
its extreme points.

Chapter 5

Sensitivity Analysis
5.1

Changes in b

Suppose bi is changed to bi + :
11

(a) Reduced costs are not a function of b, and thus not affected.
(b) Feasibility requires that:
B 1 (b + ei ) 0 xB + g,

where g = B 1 ei

If satisfies xB(j) + ji 0, j = 1, . . . , m, the current basis is optimal, where g = [1i , . . . , mi ]0 ,

so:
xB(j)
xB(j)
max
min
ji
j|ji >0 ji
j|ji <0
(c) For cost change, the optimal cost is:
c0B B 1 (b + ei ) = p0 b + pi
where p0 c0B B 1 . Thus, the change in cost is proportional to pi provided stays in the right range.

5.2

Changes in c

Suppose cj is changed to cj + :
(a) There are two cases for reduced cost:
(i) j is nonbasic. Then, cB doesnt change, so check:
cj + c0B B 1 Aj 0
Thus, if cj , then the current solution remains optimal. Otherwise, the reduced cost fo xj
becomes negative, so run the primal simplex method.
(ii) j is basic, corresponding to the lth basic variable. Then, cb becomes cB + el . Then, for i 6= j:
ci (cB + el )0 B 1 Ai ci qli
where qli is the lth entry of B 1 Ai . For j:
cj + (cB + el )0 B 1 Aj qlj 1
which is always satisfied.
(b) Feasibility is not a function of c, and thus not affected.

5.3

Changes in A

Suppose aij is changed to aij + :

(a) There are two cases for reduced cost:
(i) j is nonbasic. Only need to look at reduced cost of j th column affected:
cj = c0B B 1 (Aj + ei ) = cj pi 0
which gives a bound on .
(ii) j is basic, then B is affected and both feasibility and reduced costs are affected.
(b) There are two cases for feasibility:
(i) j is nonbasic. B doesnt change so feasibility is not affected.
(ii) j is basic, then B is affected and both feasibility and reduced costs are affected.
12

5.4

Add a New Variable/Column

Suppose a new variable/column is added, making the problem:

minimize c0 x + cn+1 xn+1
subject to Ax + An+1 xn+1 = b
x 0, xn+1 0
(a) Need to check whether the reduced costs satisfy:
cn+1 = cn+1 c0B B 1 An+1 0
If this holds, the current solution is optimal, otherwise, run the primal simplex method.
(b) (x, xn+1 ) = (x , 0) is a basic feasible solution to the new problem.

5.5

Add a New Inequality Constraint

Suppose a new inequality constraint is added a0m+1 x bm+1 . If the old solution x satisfies this constraint,
it is still optimal. Otherwise, need to move to a new optimal solution by running the dual simplex method.

5.6

Add a New Equality Constraint

Suppose a new inequality constraint is added a0m+1 x = bm+1 . Assume the old x violates this constraint.
Without loss of generality, assume a0m+1 x > bm+1 . Construct the auxiliary problem:
minimize c0 x + M xn+1
subject to Ax = b
a0m+1 x xn+1 = bm+1
x 0, xn+1 0
where M is a big number. Then (x , a0m+1 x b) is a primal bfs, so run the primal simplex number. If
the optimal solution to the auxiliary problem has xn+1 = 0, then we have an optimal solution to the new
problem.

Chapter 6

Network Flow Problems

6.1

Directed and Undirected Graphs

Definition 6.1.1 (Directed Graph). A directed graph G = (N, A) consists of the set of nodes N and set
of directed arcs A. An arc is an ordered pair (i, j) where i is the start node and j is the end node.
Definition 6.1.2 (Undirected Graph). An undirected graph G = (N, E) consists of the set of nodes N
and set of edges E joining two nodes.
Definition 6.1.3 (Connected Graph). A graph is connected if for every two distinct nodes i, j N ,
there is a path from i to j.
13

6.2

Graph Theorems

Definition 6.2.1 (Tree). An undirected graph G = (N, E) is a tree if it is directed and has no cycles.
A node in a tree is a leaf if it is only connected to one node (i.e. has degree 1).
Theorem 6.2.2 (Tree Theorem). (a) Every tree with more than one node has a leaf.
(b) An undirected graph is a tree if and only if it is connected and has |N | 1 edges.
(c) For any two distinct nodes i, j in a tree, there is a unique path between the two.
(d) Adding an edge to a tree will create exactly one cycle.
Definition 6.2.3 (Spanning Tree). Given a connected undirected graph G = (N, E), let E1 E such
that T = (N, E1 ) is a tree. Then, T is a spanning tree.
Theorem 6.2.4 (Spanning Tree Theorem). Let G = (N, E) be connected and undirected. Let E0 E
be a set of edges which do not form a cycle. E0 can be augmented to (E1 E0 E1 ) so that (N, E1 ) is a
spanning tree.

6.3

Network Flow Formulation

Definition 6.3.1 (Network Flow Problem). A network flow problem (NFP) is:
X
minimize
cij fij
(i,j)A

subject to bi +

fji =

j:(j,i)A

fij ,

i N

j:(i,j)A

0 fij uij ,

(i, j) A

Definition 6.3.2 (Flow Conservation). Corresponding to the first constraint, the total amount of flow
into node i equals the total flow out of i:
X
bi = 0
iN

Definition 6.3.3 (Capacity). Corresponding to the second constraint, the flow through can arc cannot
exceed its capacity. If uij = , it is an incapacitated problem.
Definition 6.3.4 (Matrix Notation for the Network Flow Problem). If there are n nodes and m arcs, A
is an n m vector, called the node-arc incidence matrix. The rows equal the number of nodes and the
columns the number of arcs.

if i is the start node of the k th arc

1
aik = 1 if i is the end node of the k th arc

0
otherwise
The matrix notation for flow conservation is Af = b. The sum of all rows of A is the zero vector, and thus
the rows of A are linearly dependent. Form A by throwing away the last row of A, same for b, forming
= b.
Ax

14

Definition 6.3.5 (Circulation). A flow vector that satisfies Af = 0 is a circulation, meaning that there
is no external supply and demand. Consider a cycle C with F being the set of forward arcs in a cycle,
and B being the backwards ones. Let the flow vector be:

if (i, j) F
1
c
hij = 1 if (i, j) B

0
otherwise
which satisfies Ahc = 0. The cost of cycle C is:
X
X
cij
cij
c0 hC =
(i,j)B

(i,j)F

Given a flow vector f , cycle C and a scalar , the flow vector f + hC is obtained by pushing units of
flow around C. The resulting cost change is c0 hC .

6.4

Special Cases of the Network Flow Problem

Definition 6.4.1 (Shortest Path Problem). Find the shortest path between two nodes. This problem
has no capacities, one unit of flow, and distance ij = cij .
Definition 6.4.2 (Maximum Flow Problem). Find the largest amount of flow that can be sent from a
given source to a given sink node. This problem has capacities, and bs = bt , and want to maximize bs .
Definition 6.4.3 (Transportation Problem).
Send
P flow from the sources to the sinks to meet demand
P
at minimal cost. This problem requires i si = j dj , and cij are the shipping costs.
Definition 6.4.4 (Assignment Problem). Assigns each source node to a sink node. In this problem, the
number of sources equals the number of sinks, and it has unit supply and demand.
Definition 6.4.5 (Equivalent Variants of the Network Flow Problem). (a) Every network flow problem
can be reduced to a one source/one sink problem.
(b) Every network flow problem can be reduced to one with no sources or sinks.

6.5

The Network Simplex Algorithm

Definition 6.5.1 (Standard Form Network Flow Problem). Consider the network flow problem:
minimize c0 f
subject to Af = b
f 0
Then, the standard form network flow problem is:
minimize c0 f
= b
subject to Af
f 0
Definition 6.5.2 (Tree Solution). A flow vector f is a tree solution if it is constructed as follows:
(a) Pick a set T A of n 1 arcs that form a tree (ignore directions)
15

(b) Set fij = 0, (i, j)

/T
= b
(c) Determine fij for (i, j) T using Af
If a tree solution satisfies f 0 it is a feasible tree solution. To carry out step (c), make n the root, start
at the leaves and work towards the root.
Theorem 6.5.3 (Unique Tree Solution). Once a tree T is fixed, a corresponding tree solution is uniquely
= b, fij = 0, (i, j)
determined. In other words, Af
/ T has a unique solution.
Theorem 6.5.4 (Tree and Basic Solution). A flow vector is a tree solution if and only if it is a basic
solution
Definition 6.5.5 (Change of Basis). Picking a nonbasic variable to enter is equivalent to choosing an
arc (i, j)
/ T , forming a cycle. Push flow around the cycle until a constraint is about to be violated, i.e.
choose to push flow equal to:
= min fkl
(k,l)B

The new flow in the cycle is given as:

(
fkl +
fkl =
fkl

if (k, l) F
if (k, l) B

If B is empty, = (extreme ray).

Definition 6.5.6 (Change in Cost). The change in cost is equal to:

X
X

ckl
ckl = cij
(k,l)F

(k,l)B

where (i, j) is the non-basic entering variable. Defining pn = 0, we get:

cij = cij (pi pj ),

(i, j) A

Since the reduced costs are zero for basic variables, the pi s are easily calculated.
An overview of the network simplex method is provided in the following algorithm.
Algorithm 7 Overview of the Network Simplex Method
1: Start with a tree T and a bfs f . To initialize, assume a network with one source and sink, and add
an auxiliary arc from the source to the sink with a high cost M . Let the auxiliary arc be in the tree
and all flow goes through it. This is a bfs. Run the NSM using this as the initial bfs.
2: Compute p : pn = 0, pi pj = cij , (i, j T ).
3: Compute cij = cij (pi pj ), (i, j)
/ T . If cij , (i, j), then the current bfs is optimal. Else, choose
(i, j) with cij < 0 to enter the basis.
4: (i, j) forms a cycle. If all arcs are oriented in the same direction as (i, j), B = , and the optimal
cost is .
5: Let = min(k,l)B fkl . Compute the new flow vector, and remove from the basis an arc that now
has zero flow.
Definition 6.5.7 (Network Simplex for Capacitated Problems). Now, consider constraints of the form
dij fij uij , (i, j) A. For arcs not in the tree solution, either fij = dij or fij = uij . If cij < 0 and
fij = dij , push flow around the cycle in the direction of (i, j). If cij > 0 and fij = uij , push flow around
the cycle in the opposite direction of (i, j). The rest of the algorithm is essentially the same.
16

6.6

The Maximum Flow Problem

Definition 6.6.1 (The Maximum Flow Problem). Given capacities uij , no costs, we want to maximize
the flow from the source s to the sink t:
maximize bs
subject to Af = b
bt = bs
i 6= s, t

bi = 0,

0 fij uij ,

(i, j) A

By setting the costs all toPzero, introducing a new infinite capacity arc (t, s) with cost cts = 1, this is
equivalent to minimizing (i,j)A cij fij .
Definition 6.6.2 (Augmented Path). Let f be a feasible flow vector. An augmented path is a path from
s to t such that fij < uij for all forward arcs, and fij > 0 for all backward arcs in the path.
Definition 6.6.3 (Labeling). A labeled node i means that there exists an unsaturated path from s t
i. A scanned node i means that all neighbours of i have been looked at and have been attempted to
be labeled. If a node i is labeled, and (i, j) is an arc, and fij < uij , then j should be labeled as well.
Similarly, if i is labeled and (j, i) is an arc and fji > 0, j should be labeled. This finds an augmented
path.
Definition 6.6.4 (Maximum Flow that can be Pushed Through a Path). The maximum amount of flow
that can be pushed along a path P is:
(P ) = min{ min (uij fij ), min fij }
(i,j)F

(i,j)B

This gives rise to the Ford-Fulkerson algorithm for solving maximum flow problems.
Algorithm 8 Ford-Fulkerson Algorithm
1: Start with a feasible flow f (i.e. f = 0).
2: Search for an augmenting path. If non, stop (optimal).
3: Else, found P and push (P ) along the path and return to Step 2. If (P ) = , then algorithm
terminated with optimal cost of .
Definition 6.6.5 (Cuts). An s-t cut is a subset S N such that s S and t
/ S. A capacity of a cut
is given by:
X
C(S) =
uij
{(i,j)A|iS,j S}
/

Theorem 6.6.6 (Max-Flow Min-Cut Theorem). (a) If the Ford-Fulkerson algorithm terminates because
no augmenting path can be found, then the current flow is optimal.
(b) The value of the max-flow equals the value of the minimum capacity cut.

6.7

Duality in Network Flow Problems

Definition 6.7.1 (Dual Network Flow Problem). Consider

the standard form network flow problem, and
P
assume the network is connected, uncapacitated and iN bi = 0. The corresponding dual is:
minimize p0 b
subject to p0 A c0
p free
17

which is equivalent to:

minimize p0b
subject to p0 A c0
p free
Because of the structure of A, dual constraints are of the form:
pi pj cij ,

6.8

, (i, j) A

Network Flow Sensitivity Analysis

Suppose bi becomes bi +  and bj becomes bj  to maintain the flow balance. Assume non-degeneracy
and the same basis remains optimal. Then, the change in cost is:
(pi pj )

6.9

Network Flow Complementary Slackness

An optimal solution (f, p) satisfies the complementary slackness conditions:

(a) The first condition is:
pi (a0i f bi ) = 0
If pi 6= 0, then a0i f = bi for all feasible flows.
(b) The secon condition is:
fij (pi pj cij ) = 0
If fij > 0, then pi pj = cij . If pi pj < cij , then there exists a way of sending flow from i to j at
less cost than using (i, j). Hence, (i, j) should not carry flow, so fij = 0.

6.10

The Assignment Problem

Definition 6.10.1 (The Assignment Problem). Given capacities uij , no costs, we want to maximize the
flow from the source s to the sink t:
minimize

subject to

n X
n
X

cij fij

i=1 j=1
n
X
i=1
n
X

fij = 1,

j = 1, . . . , n

fij = 1,

i = 1, . . . , n

j=1

fij 0,

(i, j) A

There are n people and n projects. The first constraint can be interpreted as for any project j, at most
one person works on it. The second constraint can be interpreted as for any person i, at most one works
on one job. If person i cannot do j, then cij = . The flow fij is equal to one if i is assigned to j, and
zero otherwise.
18

Definition 6.10.2 (The Bipartite Matching Problem). For this special case of the assignment problem,
cij = 0 if and only if person i is compatible with project j. We want to find as many person-project
matchings as possible. If the optimal cost is zero, there exists a perfect matching.
minimize

subject to

n X
n
X

cij fij

i=1 j=1
n
X

fij = 1,

j = 1, . . . , n

fij = 1,

i = 1, . . . , n

i=1
n
X
j=1

which is equivalent to:

maximize

n
X

ri +

i=1

n
X

pj

j=1

subject to ri + pj cij ,

i, j

Once the pj s are determined, ri can be determined as ri = minj=1,...,m {cij pj } and can be rewritten
accordingly.
Definition 6.10.3 (Complementary Slackness). The complementary slackness condition can be written
as:
(a) Flow must be conserved.
(b) If fij > 0, then ri + pj = cij pj cij = maxk {pk cik }
where the second condition can be interpreted as each project k has a reward pk and if peron i is assigned
to it, there is a cost cik . The difference between the two is the profit to person i derived from carrying
out project k. Thus, it says that each person should be assigned to a most profitable project.

6.11

The Shortest Path Problem

Definition 6.11.1 (The Shortest Path Problem Formulation). Consider a directed graph with cost cij .
We want to find the shortest directed path from the origin to the destination. Solve the network flow as
a linear program, and if the optimal cost is , there exists a negative cost cycle, otherwise, then an
optimal tree solution exists, which is the tree of shortest paths.
Definition 6.11.2 (Dual of the Shortest Path Problem). A degenerate primal optimal solution implies
that the dual has a unique optimal solution. Let pi be the cost of sending flow from i to n, which is also
the length of the shortest path from i to n. If (i, j) T , then pi = pj + cij , otherwise, pi pj + cij .
Definition 6.11.3 (Bellmans Equation). The dual optimal solution p satisfies the following equation:
pi = min{pj + cij },

i = 1, . . . , n 1

pn = 0
Definition 6.11.4 (Bellman Ford Algorithm). Let pi (t) be the shortest path from i to n that uses less
than t arcs. pi (t) = if you cannot do it in less than t arcs. Then, let pi (t + 1) = min{cij + pj (t)}.
Keep track of the indices which attain min and then find nodes in the shortest path. The termination
conditions are:
19

(a) If there is no negative cost cycle, then there exists a shortest walk that is also the shortest path with
less than n 1 arcs, i.e. pi (n 1) = pi and pi (n) = pi (n 1).
(b) If there exists a negative cost cycle, then pi (t) is for some i.
Definition 6.11.5 (Dijkstras Algorithm). The algorithm can only be used if cij > 0 and if we assume
that cij is defined for all i, j with i 6= n but may be for some parts. The idea is to work backwards
from destination nodes.
(a) Find a node l 6= n such that cln cin , i 6= n. Set pl = cln .
(b) For all nodes i 6= l, n, set cin = min{cin , cil + cln }.
(c) Remove node l from the graph and repeat with remaining graph.
Alternatively,
(a) Move back from destination node and label its neighbour.
(b) Pick the smallest one and finalize (move into set of shortest path lengths).
(c) Label neighbours of finalized node and pick the smallest one of the remaining nodes, etc.

Chapter 7

Interior Point Methods

7.1

The Primal Path Following Algorithm

Set up a barrier problem to prevent xj from reaching the boundary xj . Consider the standard form
problem:
minimize c0 x
subject to Ax = b
x0
and its corresponding dual:
maximize p0 b
subject to p0 A + s0 = c0
s0 0
Define the barrier function as:
0

B (x) = c x

n
X

log xj ,

j=1

Then, we want to:

minimize B (x)
subject to Ax = b

20

Solve the barrier problem for successively smaller values of . In particular, as the limit approaches zero,
x () x .

Chapter 8

Large Scale Optimization

8.1

Delayed Column Generation

Suppose in the standard form problem, the matrix A has a very large number of columns. All the simplex
method really needs to do is find a j such that cj < 0 when one exists. Thus, the idea behind delayed
column generation is to solve an auxiliary problem: minimize cj . If the optimal value is negative, then
Aj enters the basis, otherwise, it is at the optimal solution. The ability to solve this auxiliary problem
efficiently depends on the problem structure.
The idea is to solve the restricted master problem, which uses the constraints i I {1, . . . , n}. This
generates a solution set (x, p). Then solve the subproblem of minimizing the reduced cost. If the optimal
condition is met, than an optimal solution is found; otherwise, add the appropriate constraint into the
restricted master problem.

8.2

Cutting Plane Methods

The cutting plane method recognizes that a primal with many columns is a dual with many rows. It then
solves the relaxed dual problem, for only constraints i I {1, . . . , n}. Suppose p is a solution to the
relaxes problem. Then, there are two cases:
(a) p is feasible for the original problem, and thus it is optimal for the original problem.
(b) p is infeasible for the original problem, and thus it must violate one of the original constraints, which
is added to the relaxed problem.
To determine whether an optimal solution to the relaxed problem is feasible for the original problem,
minimize ci (p )0 Ai over all i. If the optimal value is nonnegative, p is feasible and thus optimal to the
original problem. Otherwise, the i which causes the value to be negative should be added to the relaxed
problem.

8.3

Dantzig-Wolfe Decomposition

Consider an LP of the form:

minimize c01 x1 + c2 x2
subject to D1 x1 + D2 x2 = b0 ,

dimension m0

F1 x1 = b1 ,

dimension m1

F2 x2 = b2 ,

dimension m2

x1, x2 0,

dimension n1 , n2 , respectively

Let:
Pi = {xi 0|Fi xi = bi },
21

i = 1, 2

Then, an equivalent problem is:

minimize c01 x1 + c2 x2
subject to D1 x1 + D2 x2 = b0
xi Pi ,

i = 1, 2

Let xji , j Ji be the extreme points of Pi and wik , k Ki be a complete set of extreme rays of Pi . By the
resolution theorem, any xi Pi can be represented as the sum:
X j j
X
xi =
i xi +
ik wik
jJi

where ji , ik 0,

jJi

kKi

ji = 1, for i = 1, 2. Substituting this in, the constraints can be written as:

b0
D1 w1k
D2 w2k
D1 xj1
D2 xj2
X
X
X
j1 1 +
1k 0 +
j2 0 +
1k 0 = 1
jJ1
jJ2
kK1
kK2
1
0
0
0
1
X

Then, give a basis B, p0 = c0B B 1 , where p0 = [q, r1 , r2 ], apply the column generation idea: i.e. examine
the reduced costs for the first subproblem:
For variable j1 : c01 xj1 [q, r1 , r2 ][D1 xj1 , 1, 0]0 = (c01 q 0 D1 )xj1 r1
For variable 1k : c01 w1k [q, r1 , r2 ][D1 w1k , 0, 0]0 = (c01 q 0 D1 )w1k
And solve:
minimize (c01 q 0 D1 )x1
subject to x1 P1
Do the same for x2 . Then, there are three possibilities:
(a) The optimal cost is , i.e. the simplex method finds some extreme ray w1k such that (c01 q 0 D1 )w1k <
0. Then, generate the column [D1 w1k , 0, 0]0
(b) The optimal cost is finite and less than r1 , i.e. the simplex method finds an extreme point xj1 such
that (c01 q 0 D1 )xj1 r1 < 0. Then, generate the column [D1 x1 , 1, 0]0
(c) The optimal cost is finite and greater or equal to r1 . Thus, all reduced costs are nonnegative so an
optimal solution is obtained.
This gives rise to the Dantzig-Wolfe algorithm.
Algorithm 9 Dantzig-Wolfe Algorithm
1: Typical iteration starts with m0 + 2 extreme points and extreme rays of P1 , P2 bfs B 1
p0 = c0B B 1
2: Form and solve the two subproblems. If the optimal cost of subproblem i ri , i, all reduced costs
in the master problem are nonnegative, so at an optimal solution.
3: If the optimal cost in the ith subproblem is , there exists an extreme ray wik associated with the
variable ik with negative reduced cost. This variable enters the master problem.
j
4: If the optimal cost in the ith subproblem is < ri , there exists an extreme point xi associated with the
variable ji with negative reduced cost. This variable enters the master problem.
5: Carry out an iteration of the revised simplex method for the master problem and update B 1 , p.
22

Chapter 9

Robust Optimization
Robust optimization aims to ensure feasibility of the following problem at a good cost:
minimize c0 x
subject to Ax = b,

9.1

A A, b B

General Approach

Consider the problem:

minimize c0 x
subject to Ax b
y 0 x f,

y P

where P = {y Rn |M y g} where M is an l n matrix. Note that y 0 x f , y P is equivalent to

minimizing y 0 x f over all y P .
Theorem 9.1.1 (General Approach Theorem). A given x
satisfies y 0 x
f , y P if and only if there
l
exists a p R satisfying:
p0 M = x

p0
0

pgf
Thus, the problem can be rewritten as:
minimize c0 x
subject to Ax b
p 0 M = x0
p0 g f
p0
which has more variables but a finite number of constraints.

9.2

Inverse Optimization

The inverse optimization problem is given a feasible x

, find a cost vector c that makes x
optimal. It is
formulated as:
minimize 0
subject to p0 A c
p0 b = c0 x

p0
cf ree

23