Professional Documents
Culture Documents
Course
Outline
Review of Math
Basics of Optimization
Review of Matlab
Examples of Optimization in Computer, Telecom and Power applications
Graphical Optimization
Optimization Types Constraint and Unconstraint
Optimization Problem Types Linear NonLinear etc
Linear Optimization
Non Linear Optimization
Integer and Mixed integer programming
Complexity Analysis
2
Set Theory
Sets
Subset
A={1,3,5,7,9},
A B
A B x[ x A x B ]
CB
C is a subset of B.
A B x[ x A x B ]
x[ x A x B]
Set Theory
Conceptually, sets may be infinite (i.e., not finite, without end).
Symbols for some special infinite sets:
N = {0, 1, 2, }
[ a, b] {x R | a x b} closed interval
( a, b) {x R | a x b} open interval
[ a, b) {x R | a x b} half-open interval
( a, b] {x R | a x b} half-open interval
4
Set Theory
For real numbers a,b with a<b
[ a, b] {x R | a x b} closed interval
( a, b) {x R | a x b} open interval
[ a, b) {x R | a x b} half-open interval
( a, b] {x R | a x b} half-open interval
xS (x is in S) is the proposition that object x is an element or member
of set S.
e.g. 3N, a{x | x is a letter of the alphabet}
Can define set equality in terms of relation:
S,T: S=T (x: xS xT)
Two sets are equal iff they have all the same members.
xS : (xS)
x is not in S
5
Function
f (x)
x5
f ( x) x 2 5
Function
f (x)
x5
f ( x) x 2 5
Example:
Domain: All Real
Range: {y: y-5}
5x4x3
3x2x1
Answer:
10 committees
n
n!
C ( n, r )
r !(n r )!
r
Essentially unordered permutations
P ( n, r ) C ( n , r ) P ( r , r )
Note that C(n,r) = C(n, nr)
n
P (n, r ) n! /( n r )!
n!
C (n, r )
P(r , r )
r!
r!(n r )!
r
10
Linear Algebra
Algebra means, roughly, relationships.
Linear Algebra means, roughly, line-like relationships.
If 3 feet forward has a 1-foot rise, then going 10x as far should give a 10x
rise (30 feet forward is a 10-foot rise)
Think of a vector as a directed line segment y
v
x
a
v b
c
11
Linear Algebra
Vector in Rn is an ordered
1
6
3
4
vector:
as opposed to a row vector:
6 3 4
1 2 8
4 78 6
9 3 2
12
Linear Algebra
use the head-to-tail method
to combine vectors
Line equation: y = mx + c
Matrix equation: y = Mx + c
A
B
C
B
A
A B ( x1 , x2 ) ( y1 , y2 ) ( x1 y1 , x2 y2 )
Scalar Product: A+B
av a( x1 , x2 ) (ax1 , ax2 )
av
v
13
Linear Algebra
Vectors: Magnitude (Length) and Phase (direction)
v ( x , x , , x )T
1 2
n
n
v x2
i
i 1
If v 1, v is a unit vector
Alternate representations:
Polar coords: (||v||, )
Complex numbers: ||v||ej
phase
14
Linear Algebra
Vector norm: afunction that assigns a strictly positive length
orsizeto each vectorin avector space [wikipedia]
Forann-dimensionalvector x [ x1
thevectornorm: x p x
x2 ... xn ]
p
@ xi
i
1/ p
; p 1, 2,...
Specialcase : x @max xi
i
Example: v = (1, 2,
Name
Symbol value
3)
L1 norm
|v|1
L2 norm
|v|2
14 3.74
L3 norm
|v|3
361/33.3
L norm
|v|
1/2
Properties:
1. x 0whenx 0; x 0iff x 0
2. kx k x scalark
3. x y x y
15
Linear Algebra
5 0 1
5
0 5 1
5
0 1 1
1
1 0 1
1
0 1 1
0
1 0 0
1
1 0 1
1
0 0 1
0
1 c x
x cy
y
0 1 y
(stretching)
(rotation)
(reflection)
(projection)
(shearing)
16
Linear Algebra
Matrices as sets of constraints
x y z 1
Special matrices
2x y z 2
a 0 0
0
b
0
0 0 c
diagonal
a b
0 d
0 0
e
f
upper-triangular
a 0
b c
d e
0
f
lower-triangular
0
0
x
1 1 1
1
y
2 1 1
2
z
1 0 0
0
1
0
0 0 1
1
2
5
6
Symmetric Matrix: AT = A
Identity Matrix: A square matrix whose diagonal elements are all 1 and off-
Linear Algebra
Determinants
Example
3 5
A
1
2
3 5
det A
(3)(2) (5)(1) 6 5 1
1 2
Def: Minors
Let A =[aij] be an nxn matrix . The ijth minor of A ( or the minor of
aij) is the determinant Mij of the (n-1)x(n-1) submatrix after you
delete the ith row and the jth column of A.
Example
Find
M 23 , M 32 , M 33 ,
1 0 2
A 4 3 1
3 5 1
18
Linear Algebra
Def: Cofactors
Let A =[aij] be an nxn matrix . The ijth cofactor of A ( or the
cofactor of aij) is defined to be Aij ( 1) i j M ij
Example
1 0 2
A 4 3 1
3 5 1
signs
19
Linear Algebra
0 2 1
A 3 1 2
4 0 1
Find all the minors and cofactors of A, and then
find the determinant of A.
Sol:
1 2
3 2
3 1
M 11
1, M 12
4
4 1
4 0
C13 4
C11 1
C12 5
M 21 2, M 22 4, M 23 8, C21 2, C22 4, C23 8,
M 31 5, M 32 3, M 33 6. C31 5, C32 3, C33 6.
0
5,
M 13
Linear Algebra
Principal Minor
Principal minor of order k: A sub-matrix obtained by deleting any n-k
rows and their corresponding columns from an n x n matrix Q.
Consider
1 2 3
Q 4 5 6
7 8 9
1 2 1 3
5 6
4 5 , 7 9 and 8 9
21
Linear Algebra
Leading Principal Minor
The leading principal minor of order k of an n x n matrix is obtained
by deleting the last n-k rows and their corresponding columns.
1 2 3
Q 4 5 6
7 8 9
1 2
4 5
22
Linear Algebra
An nxn matrix M is said to be positive definite if zTMz is positive for all
non-zero column vectors z. Matrix M is symmetric.
Tests for positive definite matrices
All diagonal elements must be positive.
All the leading principal determinants must be positive (z>0).
semi-definiteness.
23
Linear Algebra
24
Linear Algebra
25
Linear Algebra
Eigenvalues
If the action of a matrix on a (nonzero) vector changes its magnitude but
not its direction, then the vector is called an eigenvector of that matrix.
Each eigenvector is, in effect, multiplied by a scalar, called the eigenvalue
corresponding to that eigenvector. The eigenspace corresponding to one
eigenvalue of a given matrix is the set of all eigenvectors of the matrix with
that eigenvalue
Given a linear transformation A, a non-zero vector x is defined to be an
eigenvector of the transformation if it satisfies the eigenvalue equation
AX X
for some scalar . In this situation, the scalar is called an eigenvalue of
A corresponding to the eigenvector x.
26
Linear Algebra
Computation of eigenvalues, and the characteristic equation
When a transformation is represented by a square matrix A, the
eigenvalue equation can be expressed as
AX IX 0
Solve
det(A I) = 0.
27
Linear Algebra
28
Linear Algebra
29
Linear Algebra
30
Linear Algebra
31
Differential Calculus
The derivative, or derived function of f(x) denoted f`(x)
is defined as
f ( x h) f ( x)
f `( x) lim
h 0
h
y
Q
f ( x h) f ( x )
P
h
x
mPQ
x+h
f ( x h) f ( x )
h
dy
y
Leibniz Notation: f `( x) lim
ho x
dx
32
Calculus
The Product Rule
If k ( x) f ( x).g ( x) , then:
k `( x) f `( x) g ( x) g `( x) f ( x)
OR
dy
f `g g ` f
dx
1. Differentiate y x 2 sin x
f ( x) x 2
g ( x) sin x
f `( x) 2 x
g `( x) cos x
dy
f `g g ` f
dx
2 x sin x x 2 cos x
33
Calculus
The Quotient Rule
dy f `g g ` f
dx
g2
d x3
1. Find
dx sin x
f ( x) x3
g ( x) sin x
f `( x) 3 x 2 g `( x) cos x
dy f `g g ` f
3x 2 sin x x3 cos x
2
dx
g
sin 2 x
34
Calculus
A secant line is a straight line joining two points on a function
A tangent line is a straight line that touches a function at only one point.
The tangent line represents the instantaneous rate of change of the
function at that one point. The slope of the tangent line at a point on the
function is equal to the derivative of the function at the same point.
35
35
Calculus
A stationary point is an input to a
function where the derivative is zero
Stationary points (red circles) and
inflection points (blue squares).
The stationary points in this graph
are all relative maxima or relative
minima.
Calculus
Consider the
function:
x2 2 x 1
f ( x) x
1 x 2
1
x 1 2 x 4
2
y
5
Domain [ 2, 4)
2x
f `( x) 1
1
2 x 1
1 x 2
2 x 4
37
Calculus
Local extrema
43215 21
321 321
Local extreme values occur either at the end points of the function, turning
points or critical points within the interval of the domain.
y
x2 2x
3 x 1
f ( x) x3
1 x 1
2 x 2 8 x 5 1 x 3
Domain [ 3,3)
If extrema occurs at end points then they are end point maximums or end
point minimums.
(i) Local maximum / minimum turning points
(ii) End point values
(iii) Critical points
39
Calculus
The Nature of Stationary Points.
If f `(a) = 0 then a table of values over a suitable interval centred at a
provides evidence of the nature of the stationary point that must exist at a.
A simpler test does exist.
It is the second derivative test.
40
Calculus
100
Orig
1st Derv.
2nd Derv.
80
60
y = x 4-4x 3+5
40
dy = 4 x. 3 - 12*x 2
d2y = 12 x 2 - 24 x
20
0
-20
-40
-60
x = -2:0.001:4;
y = x.^4 - 4*x.^3 + 5;
dy = 4 * x.^3 - 12*x.^2;
d2y = 12 * x.^2 - 24*x;
-80
-2
-1
1
x
41
Graph Theory
Network = graph
Informally a graph is a set of nodes joined
by a set of lines or arrows.
Nodes and edges
G(V, E)
Undirected graph
Directed graph
V:={1,2,3,4,5,6}
E:={{1,2},{1,5},{2,3},{2,5},{3,4},{4,5},{4,6}}
Many problems can be stated in terms of a graph
The properties of graphs are well-studied
Many algorithms exists to solve graph problems
Many problems are already known to be intractable
42
Graph Theory
Undirected graph
loop
Directed graph
loop
G=(V,E)
isolated vertex
multiple
edges
adjacent
43
Graph Theory
x
d
c
Graph Theory
simple graph: an undirected graph with no loops or multiple edges
Simple graph
b
K5
d
c
Disconnected graph
with two components
d
c
45
Graph Theory
tree: a connected, acyclic graph
directed acyclic graph (DAG): a digraph with no cycles
weighted graph: any graph with weights associated with the edges
b
5
8
-3
2
e
10
46
MathTerminologies
A Theorem is a major result
A Corollary is a theorem that follows on from another theorem
A Lemma is a small results (less important than a theorem)
Example: Here is a Theorem, a Corollary to it, and also a Lemma!
Theorem:
If m and n are any two whole numbers and
a = m2 n2
b = 2mn
c = m 2 + n2
then a2 + b2 = c2
Proof:
a2 + b2 = (m2 n2)2 + (2mn)2
= m4 2m2n2 + n4 + 4m2n2
= (m2 + n2)2
= c2
MathTerminologies
A Theorem is a major result
A Corollary is a theorem that follows on from another theorem
A Lemma is a small results (less important than a theorem)
Example: Here is a Theorem, a Corollary to it, and also a Lemma!
Theorem:
Corollary
If m and n are any two whole numbers and
a, b and c, as defined above, are
2
2
a=m n
a Pythagorean Triple
b = 2mn
Proof:
c = m 2 + n2
From the Theorem a2 + b2 = c2, so
then a2 + b2 = c2
a, b and c are a Pythagorean Triple
Proof:
(That result "followed on" from the
previous Theorem.)
a2 + b2 = (m2 n2)2 + (2mn)2
= m4 2m2n2 + n4 + 4m2n2
= (m2 + n2)2
= c2
MathTerminologies
In a nutshell
A Theorem is a major result
A Corollary is a theorem that follows on from another theorem
A Lemma is a small results (less important than a theorem)
Example: Here is a Theorem, a Corollary to it, and also a Lemma!
Theorem:
Corollary
If m and n are any two whole numbers and
a, b and c, as defined above, are
2
2
a=m n
a Pythagorean Triple
b = 2mn
Proof:
c = m 2 + n2
From the Theorem a2 + b2 = c2, so
then a2 + b2 = c2
a, b and c are a Pythagorean Triple
Proof:
(That result "followed on" from the
previous Theorem.)
a2 + b2 = (m2 n2)2 + (2mn)2
= m4 2m2n2 + n4 + 4m2n2
= (m2 + n2)2
= c2
MathTerminologies
Proposition a proved and often interesting result, but generally less
important than a theorem. if x is odd then x2is odd.
Conjecture a statement that is unproved, but is believed to be true.
Every even number larger than 2 can be written as a sum of two
primes.
Axiom/Postulate a statement that is assumed to be true without proof.
These are the basic building blocks from which all theorems are proved.
One of the axioms of arithmetic is that a + b = b + a. You cant prove that,
but it is the basis of arithmetic and something we use rather often.
50
MATLAB
MATLAB is an interactive environment
Commands are interpreted one line at a time
Commands may be scripted to create your own functions or
procedures
Variables are created when they are used
Variables are typed, but variable names may be reused for different
types
Basic data structure is the matrix
Matrix dimensions are set dynamically
Operations on matrices are applied to all elements of a matrix at
once
Removes the need for looping over elements one by one!
Makes for fast & efficient programmes
51
MATLAB
Worksp
ace
Command
Window
Command
History
52
Variables
Names
MATLAB
equal symbol =
>> A = 32
To find out the value of a variable simply
type the name in
53
MATLAB
54
MATLAB
A MATLAB matrix is a rectangular array of numbers
Scalars and vectors are regarded as special cases of matrices
MATLAB allows you to work with a whole array at a time
MATLAB
The colon : is actually an operator, that generates a row vector
This row vector may be treated as a set of indices when accessing a
elements of a matrix
The more general form is
[start:stepsize:end]
>> [11:2:21]
11 13 15 17 19 21
>>
Stepsize does not have to be integer (or positive)
>> [22:-2.07:11]
56
MATLAB
Mathematical Operators:
Add: +
Subtract: Divide: ./
Multiply: .*
Power: .^ (e.g. .^2 means squared)
57
MATLAB
Combining this with methods from Accessing Matrix Elements
58
MATLAB
Logical Operators:
Greater Than: >
Less Than: <
Greater Than or Equal To: >=
Less Than or Equal To: <=
Is Equal: ==
Not Equal To: ~=
For example, you can find data that is above a certain limit:
>> r = results(:,1)
>> ind = r > 0.2
Boolean Operators:
AND: &
OR: |
NOT: ~
59
MATLAB
There are a number of special functions that provide useful
constants
pi = 3.14159265.
i or j
= square root of -1
Inf = infinity
NaN
= not a number
Passing a vector to a function like sum, mean, std will calculate the property within
the vector
>> sum([1,2,3,4,5])
= 15
>> mean([1,2,3,4,5])
=3
>> max([1,2,3,4,5])
=5
60
MATLAB
The plot function can be used in different ways:
>> plot(data)
>> plot(x, y)
>> plot(data, r.-)
Colour: r, b, g, c, k, y etc.
Point style: . + * x o > etc.
Line style: - -- : .-
1
0.8
A basic plot
>> x =
[0:0.1:2*pi]
>> y = sin(x)
>> plot(x, y,
r.-)
0.6
0.4
0.2
0
-0.2
-0.4
-0.6
-0.8
-1
61
MATLAB
>> x = [0:0.1:2*pi];
>> y = sin(x);
>> plot(x, y, 'b*-')
0.5
sin(x)
2*sin(x)
1.5
>> hold on
Sin Plots
0
-0.5
-1
>> xlabel(x);
-1.5
>> ylabel(y);
>> hold of
-2
3
x
62
MATLAB
For command
Use a for loop to repeat one or more statements
The end keyword tells Matlab where the loop finishes
You control the number of times a loop is repeated by defining the
This uses the colon operator again, so index values do not need
to be integer
For example
>> for i = 1:4
a(i) = i * 2
end
63
MATLAB
The counter can be used to index different rows or columns
E.g.
end
..although you could do this in one step
m = mean(results);
64
MATLAB
The if command is used with logical operators
Again, the end command is used to tell Matlab where the statement
ends.
For example, the following code loops through a matrix performing
calculations on each column
>> for i = 1:size(results, 2)
m = results(:, i)
if m > 1
do something
else
do something different
end
end
65