Computer Algorithms PDF

Introduction to Algorithms
6.046J/18.401J/SMA5503
Lecture 1
Prof. Charles E. Leiserson
Welcome to Introduction to
Algorithms, Fall 2001
Handouts
1.
2.
3.
4.
5.
6.
Day 1
Course Information
Calendar
Registration (MIT students only)
References
Objectives and Outcomes
Diagnostic Survey
L1.2
Course information
1.
2.
3.
4.
5.
6.
7.
Staff
Distance learning
Prerequisites
Lectures
Recitations
Handouts
Textbook (CLRS)
8. Website
9. Extra help
10.Registration (MIT only)
11.Problem sets
12.Describing algorithms
13.Grading policy
14.Collaboration policy
Course information handout

Day 1
L1.3
Analysis of algorithms
The theoretical study of computer-program
performance and resource usage.
Whats more important than performance?
modularity
user-friendliness
correctness
programmer time
maintainability
simplicity
functionality
extensibility
robustness
reliability
Day 1
L1.4
Why study algorithms and

performance?
Algorithms help us to understand scalability.
Performance often draws the line between what
is feasible and what is impossible.
Algorithmic mathematics provides a language
for talking about program behavior.
The lessons of program performance generalize
to other computing resources.
Speed is fun!
Day 1
L1.5
The problem of sorting

Input: sequence a1, a2, , an of numbers.
Output: permutation a'1, a'2, , a'n such
that a'1 a'2 a'n .
Example:
Input: 8 2 4 9 3 6
Output: 2 3 4 6 8 9
Day 1
L1.6
Insertion sort
pseudocode
INSERTION-SORT (A, n)
A[1 . . n]
for j 2 to n
do key A[ j]
ij1
while i > 0 and A[i] > key
do A[i+1] A[i]
ii1
A[i+1] = key
j
A:
sorted
Day 1
key
L1.7
Example of insertion sort

8
Day 1
L1.8

8
Day 1
L1.9
Day 1
L1.10
Day 1
L1.11
Day 1
L1.12
Day 1
L1.13
Day 1
L1.14
Day 1
L1.15
Day 1
L1.16
Day 1
L1.17
Day 1
9 done
L1.18
Running time
The running time depends on the input: an
already sorted sequence is easier to sort.
Parameterize the running time by the size of
the input, since short sequences are easier to
sort than long ones.
Generally, we seek upper bounds on the
running time, because everybody likes a
guarantee.
Day 1
L1.19
Kinds of analyses
Worst-case: (usually)
T(n) = maximum time of algorithm
on any input of size n.
Average-case: (sometimes)
T(n) = expected time of algorithm
over all inputs of size n.
Need assumption of statistical
distribution of inputs.
Best-case: (bogus)
Cheat with a slow algorithm that
works fast on some input.
Day 1
L1.20
Machine-independent time
What is insertion sorts worst-case time?
It depends on the speed of our computer:
relative speed (on the same machine),
absolute speed (on different machines).
BIG IDEA:
Ignore machine-dependent constants.
Look at growth of T(n) as n .
Asymptotic Analysis
Day 1
L1.21
-notation
Math:
(g(n)) = { f (n) : there exist positive constants c1, c2, and

n0 such that 0 c1 g(n) f (n) c2 g(n)
for all n n0 }
Engineering:
Drop low-order terms; ignore leading constants.
Example: 3n3 + 90n2 5n + 6046 = (n3)
Day 1
L1.22
Asymptotic performance
When n gets large enough, a (n2) algorithm
always beats a (n3) algorithm.
T(n)
n
Day 1
n0
We shouldnt ignore
asymptotically slower
algorithms, however.
Real-world design
situations often call for a
careful balancing of
engineering objectives.
Asymptotic analysis is a
useful tool to help to
structure our thinking.
L1.23
Insertion sort analysis

Worst case: Input reverse sorted.
T ( n) =
2)
(
(
j
)
=
[arithmetic series]
j =2
Average case: All permutations equally likely.
T ( n) =
( j / 2) = (n 2 )
j =2
Is insertion sort a fast sorting algorithm?

Moderately so, for small n.
Not at all, for large n.
Day 1
L1.24
Merge sort
MERGE-SORT A[1 . . n]
1. If n = 1, done.
2. Recursively sort A[ 1 . . n/2 ]
and A[ n/2+1 . . n ] .
3. Merge the 2 sorted lists.
Key subroutine: MERGE
Day 1
L1.25
Merging two sorted arrays

20 12
13 11
7
Day 1
L1.26

20 12
13 11
7
1
1
Day 1
L1.27

20 12
20 12
13 11
13 11
Day 1
L1.28

20 12
20 12
13 11
13 11
Day 1
L1.29

20 12
20 12
20 12
13 11
13 11
13 11
Day 1
L1.30

20 12
20 12
20 12
13 11
13 11
13 11
Day 1
L1.31

20 12
20 12
20 12
20 12
13 11
13 11
13 11
13 11
Day 1
L1.32

20 12
20 12
20 12
20 12
13 11
13 11
13 11
13 11
Day 1
L1.33

20 12
20 12
20 12
20 12
20 12
13 11
13 11
13 11
13 11
13 11
Day 1
L1.34

20 12
20 12
20 12
20 12
20 12
13 11
13 11
13 11
13 11
13 11
Day 1
11
L1.35

20 12
20 12
20 12
20 12
20 12
20 12
13 11
13 11
13 11
13 11
13 11
13
Day 1
11
L1.36

20 12
20 12
20 12
20 12
20 12
20 12
13 11
13 11
13 11
13 11
13 11
13
Day 1
11
12
L1.37

20 12
20 12
20 12
20 12
20 12
20 12
13 11
13 11
13 11
13 11
13 11
13
11
12
Time = (n) to merge a total

of n elements (linear time).
Day 1
L1.38
Analyzing merge sort

T(n)
MERGE-SORT A[1 . . n]
(1)
1. If n = 1, done.
2T(n/2) 2. Recursively sort A[ 1 . . n/2 ]
Abuse
and A[ n/2+1 . . n ] .
(n)
3. Merge the 2 sorted lists
Sloppiness: Should be T( n/2 ) + T( n/2 ) ,
but it turns out not to matter asymptotically.
Day 1
L1.39
Recurrence for merge sort

T(n) =
(1) if n = 1;
2T(n/2) + (n) if n > 1.
We shall usually omit stating the base

case when T(n) = (1) for sufficiently
small n, but only when it has no effect on
the asymptotic solution to the recurrence.
CLRS and Lecture 2 provide several ways
to find a good upper bound on T(n).
Day 1
L1.40
Recursion tree
Solve T(n) = 2T(n/2) + cn, where c > 0 is constant.
Day 1
L1.41
Recursion tree
T(n)
Day 1
L1.42
Recursion tree
cn
T(n/2)
T(n/2)
Day 1
L1.43
Recursion tree
cn
cn/2
cn/2
T(n/4)
Day 1
T(n/4)
T(n/4)
T(n/4)
L1.44
Recursion tree
cn
cn/2
cn/2
cn/4
cn/4
cn/4
cn/4
(1)
Day 1
L1.45
Recursion tree
cn
cn/2
cn/2
cn/4
cn/4
cn/4
h = lg n cn/4
(1)
Day 1
L1.46
Recursion tree
cn
cn
cn/2
cn/2
cn/4
cn/4
cn/4
h = lg n cn/4
(1)
Day 1
L1.47
Recursion tree
cn
cn
cn/2
cn/2
cn/4
cn/4
cn/4
h = lg n cn/4
cn
(1)
Day 1
L1.48
Recursion tree
cn
cn
cn/2
cn/2
cn/4
cn/4
cn/4
cn
h = lg n cn/4
cn
(1)
Day 1
L1.49
Recursion tree
cn
cn
cn/2
cn/2
cn/4
cn/4
(1)
Day 1
cn/4
cn
h = lg n cn/4
cn
#leaves = n
(n)
L1.50
Recursion tree
cn
cn
cn/2
cn/2
cn/4
cn/4
(1)
cn/4
cn
h = lg n cn/4
cn
#leaves = n
(n)
Total = (n lg n)
Day 1
L1.51
Conclusions
(n lg n) grows more slowly than (n2).
Therefore, merge sort asymptotically
beats insertion sort in the worst case.
In practice, merge sort beats insertion
sort for n > 30 or so.
Go test it out for yourself!
Day 1
L1.52
6.046J/18.401J/SMA5503
Lecture 2
Prof. Erik Demaine
Solving recurrences
The analysis of merge sort from
Lecture 1 required us to solve a
recurrence.
Recurrences are like solving integrals,
differential equations, etc.
o Learn a few tricks.
Lecture 3: Applications of recurrences.
Day 3
L2.2
Substitution method
The most general method:
1. Guess the form of the solution.
2. Verify by induction.
3. Solve for constants.
Example: T(n) = 4T(n/2) + n
[Assume that T(1) = (1).]
Guess O(n3) . (Prove O and separately.)
Assume that T(k) ck3 for k < n .
Prove T(n) cn3 by induction.
Day 3
L2.3
Example of substitution
T (n) = 4T (n / 2) + n
4c ( n / 2 ) 3 + n
= ( c / 2) n 3 + n
desired residual
= cn3 ((c / 2)n3 n)
cn3 desired
whenever (c/2)n3 n 0, for example,
if c 2 and n 1.
residual
Day 3
L2.4
Example (continued)
We must also handle the initial conditions,
that is, ground the induction with base
cases.
Base: T(n) = (1) for all n < n0, where n0
is a suitable constant.
For 1 n < n0, we have (1) cn3, if we
pick c big enough.
This bound is not tight!
Day 3
L2.5
A tighter upper bound?

We shall prove that T(n) = O(n2).
Assume that T(k) ck2 for k < n:
T (n) = 4T (n / 2) + n
4cn 2 + n
= O(n) Wrong! We must prove the I.H.
= cn 2 ( n) [ desired residual ]
cn 2
for no choice of c > 0. Lose!
Day 3
L2.6
A tighter upper bound!

IDEA: Strengthen the inductive hypothesis.
Subtract a low-order term.
Inductive hypothesis: T(k) c1k2 c2k for k < n.
T (n) = 4T (n / 2) + n
4(c1 (n / 2) 2 c2 (n / 2) + n
= c1n 2 2c2 n + n
= c1n 2 c2 n (c2 n n)
c1n 2 c2 n if c2 > 1.
Pick c1 big enough to handle the initial conditions.

Day 3
L2.7
Recursion-tree method
A recursion tree models the costs (time) of a
recursive execution of an algorithm.
The recursion tree method is good for
generating guesses for the substitution method.
The recursion-tree method can be unreliable,
just like any method that uses ellipses ().
The recursion-tree method promotes intuition,
however.
Day 3
L2.8
Example of recursion tree

Solve T(n) = T(n/4) + T(n/2) + n2:
Day 3
L2.9

Solve T(n) = T(n/4) + T(n/2) + n2:
T(n)
Day 3
L2.10

Solve T(n) = T(n/4) + T(n/2) + n2:
n2
T(n/4)
Day 3
T(n/2)
L2.11

Solve T(n) = T(n/4) + T(n/2) + n2:
n2
(n/4)2
T(n/16)
Day 3
T(n/8)
(n/2)2
T(n/8)
T(n/4)
L2.12

Solve T(n) = T(n/4) + T(n/2) + n2:
n2
(n/4)2
(n/8)2
(n/8)2
(n/4)2
(n/16)2
(n/2)2
(1)
Day 3
L2.13

Solve T(n) = T(n/4) + T(n/2) + n2:
n2
(n/4)2
(n/8)2
(n/2)2
(n/8)2
(n/4)2
(n/16)2
n2
(1)
Day 3
L2.14

Solve T(n) = T(n/4) + T(n/2) + n2:
n2
(n/4)2
(n/8)2
(n/8)2
5 n2
16
(n/4)2
(n/16)2
(n/2)2
n2
(1)
Day 3
L2.15

Solve T(n) = T(n/4) + T(n/2) + n2:
n2
(n/8)2
(n/8)2
(n/16)2
(n/2)2
(n/4)2
5 n2
16
25 n 2
256
(n/4)2
n2
(1)
Day 3
L2.16

Solve T(n) = T(n/4) + T(n/2) + n2:
n2
(n/2)2
(n/8)2
(n/8)2
(1)
Day 3
(n/4)2
( ) +( )
2
5
5
1 + 16 + 16
Total = n
= (n2)
5 n2
16
25 n 2
256
(n/4)2
(n/16)2
n2
5 3
16
+L
geometric series
L2.17
The master method

The master method applies to recurrences of
the form
T(n) = a T(n/b) + f (n) ,
where a 1, b > 1, and f is asymptotically
positive.
Day 3
L2.18
Three common cases

Compare f (n) with nlogba:
1. f (n) = O(nlogba ) for some constant > 0.
f (n) grows polynomially slower than nlogba
(by an n factor).
Solution: T(n) = (nlogba) .
2. f (n) = (nlogba lgkn) for some constant k 0.
f (n) and nlogba grow at similar rates.
Solution: T(n) = (nlogba lgk+1n) .
Day 3
L2.19
Three common cases (cont.)

Compare f (n) with nlogba:
3. f (n) = (nlogba + ) for some constant > 0.
f (n) grows polynomially faster than nlogba (by
an n factor),
and f (n) satisfies the regularity condition that
a f (n/b) c f (n) for some constant c < 1.
Solution: T(n) = ( f (n)) .
Day 3
L2.20
Examples
Ex. T(n) = 4T(n/2) + n
a = 4, b = 2 nlogba = n2; f (n) = n.
CASE 1: f (n) = O(n2 ) for = 1.
T(n) = (n2).
Ex. T(n) = 4T(n/2) + n2
a = 4, b = 2 nlogba = n2; f (n) = n2.
CASE 2: f (n) = (n2lg0n), that is, k = 0.
T(n) = (n2lg n).
Day 3
L2.21
Examples
Ex. T(n) = 4T(n/2) + n3
a = 4, b = 2 nlogba = n2; f (n) = n3.
CASE 3: f (n) = (n2 + ) for = 1
and 4(cn/2)3 cn3 (reg. cond.) for c = 1/2.
T(n) = (n3).
Ex. T(n) = 4T(n/2) + n2/lg n
a = 4, b = 2 nlogba = n2; f (n) = n2/lg n.
Master method does not apply. In particular,
for every constant > 0, we have n = (lg n).
Day 3
L2.22
General method (Akra-Bazzi)

k
T (n) = aiT (n / bi ) + f (n)

i =1
Let p be the unique solution to

p
(
a
/b
i i ) = 1.
k
i =1
Then, the answers are the same as for the

master method, but with np instead of nlogba.
(Akra and Bazzi also prove an even more
general result.)
Day 3
L2.23
Idea of master theorem

f (n)
a
f (n/b) f (n/b) f (n/b)
a
h = logbn
f (n/b2) f (n/b2) f (n/b2)
(1)
Day 3
#leaves = ah
= alogbn
= nlogba
f (n)
a f (n/b)
a2 f (n/b2)
Recursion tree:
nlogba (1)
L2.24

f (n)
a
f (n/b) f (n/b) f (n/b)
a
h = logbn
f (n/b2) f (n/b2) f (n/b2)
C
CASE
ASE 1:
1: The
The weight
weight increases
increases
geometrically
geometrically from
from the
the root
root to
to the
the
(1) leaves.
leaves. The
The leaves
leaves hold
hold aa constant
constant
fraction
fraction of
of the
the total
total weight.
weight.
Day 3
f (n)
a f (n/b)
a2 f (n/b2)
Recursion tree:
nlogba (1)
(nlogba)
L2.25

Recursion tree:
f (n)
(1)
Day 3
a f (n/b)
a2 f (n/b2)
a
f (n/b) f (n/b) f (n/b)
a
h = logbn
f (n/b2) f (n/b2) f (n/b2)
f (n)
C
CASE
ASE 2:
2: (k
(k == 0)
0) The
The weight
weight
isis approximately
approximately the
the same
same on
on
each
each of
of the
the log
logbbnn levels.
levels.
nlogba (1)
(nlogbalg n)
L2.26

f (n)
a
f (n/b) f (n/b) f (n/b)
a
h = logbn
f (n/b2) f (n/b2) f (n/b2)
C
CASE
ASE 3:
3: The
The weight
weight decreases
decreases
geometrically
geometrically from
from the
the root
root to
to the
the
(1) leaves.
leaves. The
The root
root holds
holds aa constant
constant
fraction
fraction of
of the
the total
total weight.
weight.
Day 3
f (n)
a f (n/b)
a2 f (n/b2)
Recursion tree:
nlogba (1)
( f (n))
L2.27
Conclusion
Next time: applying the master method.

For proof of master theorem, see CLRS.
Day 3
L2.28
Appendix: geometric series

n +1
x
1
for x 1
1 + x + x2 + L + xn =
1 x
1
1+ x + x +L =
for |x| < 1
1 x
2
Return to last
slide viewed.
Day 3
L2.29
6.046J/18.401J/SMA5503
Lecture 3
Prof. Erik Demaine
The divide-and-conquer
design paradigm
1. Divide the problem (instance)
into subproblems.
2. Conquer the subproblems by
solving them recursively.
3. Combine subproblem solutions.
Day 4
L3.2
Example: merge sort

1. Divide: Trivial.
2. Conquer: Recursively sort 2 subarrays.
3. Combine: Linear-time merge.
T(n) = 2 T(n/2) + O(n)
# subproblems
subproblem size
Day 4
work dividing
and combining
L3.3
Master theorem (reprise)

T(n) = a T(n/b) + f (n)
CASE 1: f (n) = O(nlogba )
T(n) = (nlogba) .
CASE 2: f (n) = (nlogba lgkn)
T(n) = (nlogba lgk+1n) .
CASE 3: f (n) = (nlogba + ) and a f (n/b) c f (n)
T(n) = ( f (n)) .
Merge sort: a = 2, b = 2 nlogba = n
CASE 2 (k = 0) T(n) = (n lg n) .
Day 4
L3.4
Binary search
Find an element in a sorted array:
1. Divide: Check middle element.
2. Conquer: Recursively search 1 subarray.
3. Combine: Trivial.
Example: Find 9
3
Day 4
12
15
L3.5
Binary search
Example: Find 9
3
Day 4
12
15
L3.6
Binary search
Example: Find 9
3
Day 4
12
15
L3.7
Binary search
Example: Find 9
3
Day 4
12
15
L3.8
Binary search
Example: Find 9
3
Day 4
12
15
L3.9
Binary search
Example: Find 9
3
Day 4
12
15
L3.10
Recurrence for binary search

T(n) = 1 T(n/2) + (1)
# subproblems
subproblem size
work dividing
and combining
nlogba = nlog21 = n0 = 1 CASE 2 (k = 0)

T(n) = (lg n) .
Day 4
L3.11
Powering a number
Problem: Compute a n, where n N.
Naive algorithm: (n).
Divide-and-conquer algorithm:
an
a n/2 a n/2
a (n1)/2 a (n1)/2 a
if n is even;
if n is odd.
T(n) = T(n/2) + (1) T(n) = (lg n) .

Day 4
L3.12
Fibonacci numbers
Recursive definition:
0
if n = 0;
if n = 1;
Fn = 1
Fn1 + Fn2 if n 2.
0
8 13 21 34 L
Naive recursive algorithm: ( n)

(exponential time), where = (1 + 5) / 2
is the golden ratio.
Day 4
L3.13
Computing Fibonacci
numbers
Naive recursive squaring:
Fn = n/ 5 rounded to the nearest integer.
Recursive squaring: (lg n) time.
This method is unreliable, since floating-point
arithmetic is prone to round-off errors.
Bottom-up:
Compute F0, F1, F2, , Fn in order, forming
each number by summing the two previous.
Running time: (n).
Day 4
L3.14
Recursive squaring
Fn +1
Theorem:
Fn
Fn 1 1
.
=
Fn 1 1 0
Algorithm: Recursive squaring.

Time = (lg n) .
Proof of theorem. (Induction on n.)

1
F
F
2
1 1 1
Base (n = 1):
.
=
F1 F0 1 0
Day 4
L3.15
Recursive squaring
Inductive step (n 2):
Fn +1
F
n
Day 4
.
Fn Fn
Fn 1 1 1
Fn 1 Fn 1 Fn 2 1 0
n1
1 1
1 1
=
1 0
1 0
n
1 1
=
1
0
L3.16
Matrix multiplication
Input: A = [aij], B = [bij].
Output: C = [cij] = A B.
c11 c12
c c
21 22
M M
c c
n1 n 2
L c1n a11 a12

L c2 n a21 a22
=
M
O M M
L cnn an1 an 2
i, j = 1, 2, , n.
L a1n b11 b12

L a2 n b21 b22
O M M M
L ann bn1 bn 2
L b1n
L b2 n
O M
L bnn
cij = aik bkj

k =1
Day 4
L3.17
Standard algorithm
for i 1 to n
do for j 1 to n
do cij 0
for k 1 to n
do cij cij + aik bkj
Running time = (n3)
Day 4
L3.18
Divide-and-conquer algorithm
IDEA:
nn matrix = 22 matrix of (n/2)(n/2) submatrices:
r s a b e f
t u = c d g h
C
r
s
t
u
= ae + bg
= af + bh
= ce + dh
= cf + dg
Day 4
8 mults of (n/2)(n/2) submatrices

4 adds of (n/2)(n/2) submatrices
L3.19
Analysis of D&C algorithm

T(n) = 8 T(n/2) + (n2)
# submatrices
submatrix size
work adding
submatrices
nlogba = nlog28 = n3 CASE 1 T(n) = (n3).

No better than the ordinary algorithm.
Day 4
L3.20
Strassens idea
Multiply 22 matrices with only 7 recursive mults.
P1 = a ( f h)
P2 = (a + b) h
P3 = (c + d) e
P4 = d (g e)
P5 = (a + d) (e + h)
P6 = (b d) (g + h)
P7 = (a c) (e + f )
Day 4
r
s
t
u
= P5 + P4 P2 + P6
= P1 + P2
= P3 + P4
= P5 + P1 P3 P7
77 mults,
mults, 18
18 adds/subs.
adds/subs.
Note:
Note: No
No reliance
reliance on
on
commutativity
commutativity of
of mult!
mult!
L3.21
Strassens idea
Multiply 22 matrices with only 7 recursive mults.
P1 = a ( f h)
P2 = (a + b) h
P3 = (c + d) e
P4 = d (g e)
P5 = (a + d) (e + h)
P6 = (b d) (g + h)
P7 = (a c) (e + f )
Day 4
r = P5 + P4 P2 + P6
= (a + d) (e + h)
+ d (g e) (a + b) h
+ (b d) (g + h)
= ae + ah + de + dh
+ dg de ah bh
+ bg + bh dg dh
= ae + bg
L3.22
Strassens algorithm
1. Divide: Partition A and B into
(n/2)(n/2) submatrices. Form terms
to be multiplied using + and .
2. Conquer: Perform 7 multiplications of
(n/2)(n/2) submatrices recursively.
3. Combine: Form C using + and on
(n/2)(n/2) submatrices.
T(n) = 7 T(n/2) + (n2)

Day 4
L3.23
Analysis of Strassen
T(n) = 7 T(n/2) + (n2)
nlogba = nlog27 n2.81 CASE 1 T(n) = (nlg 7).
The number 2.81 may not seem much smaller than
3, but because the difference is in the exponent, the
impact on running time is significant. In fact,
Strassens algorithm beats the ordinary algorithm
on todays machines for n 30 or so.
Best to date (of theoretical interest only): (n2.376L).
Day 4
L3.24
VLSI layout
Problem: Embed a complete binary tree
with n leaves in a grid using minimal area.
W(n)
H(n)
H(n) = H(n/2) + (1) W(n) = 2 W(n/2) + (1)
= (lg n)
= (n)
Area = (n lg n)
Day 4
L3.25
H-tree embedding
L(n)
L(n) = 2 L(n/4) + (1)
= ( n )
L(n)
Area = (n)
L(n/4) (1) L(n/4)
Day 4
L3.26
Conclusion
Divide and conquer is just one of several
powerful techniques for algorithm design.
Divide-and-conquer algorithms can be
analyzed using recurrences and the master
method (so practice this math).
Can lead to more efficient algorithms
Day 4
L3.27
6.046J/18.401J/SMA5503
Lecture 4
Quicksort
Proposed by C.A.R. Hoare in 1962.
Divide-and-conquer algorithm.
Sorts in place (like insertion sort, but not
like merge sort).
Very practical (with tuning).
Day 6
L4.2
Divide and conquer

Quicksort an n-element array:
1. Divide: Partition the array into two subarrays
around a pivot x such that elements in lower
subarray x elements in upper subarray.
xx
xx
xx
2. Conquer: Recursively sort the two subarrays.
Key: Linear-time partitioning subroutine.
Day 6
L4.3
Partitioning subroutine
PARTITION(A, p, q) A[ p . . q]
x A[ p]
pivot = A[ p]
Running
Running time
time
ip
== O(n)
O(n) for
for nn
for j p + 1 to q
elements.
elements.
do if A[ j] x
then i i + 1
exchange A[i] A[ j]
exchange A[ p] A[i]
return i
Invariant: xx
p
Day 6
xx
xx
i
??
j
q
L4.4
Example of partitioning
66 10
10 13
13 55
i
j
Day 6
88
33
22 11
11
L4.5
66 10
10 13
13 55
i
j
Day 6
88
33
22 11
11
L4.6
66 10
10 13
13 55
i
j
Day 6
88
33
22 11
11
L4.7
66 10
10 13
13 55
66
Day 6
88
33
22 11
11
55 13
13 10
10 88
i
j
33
22 11
11
L4.8
66 10
10 13
13 55
66
Day 6
88
33
22 11
11
55 13
13 10
10 88
i
j
33
22 11
11
L4.9
66 10
10 13
13 55
66
Day 6
88
33
22 11
11
55 13
13 10
10 88
i
33
j
22 11
11
L4.10
66 10
10 13
13 55
Day 6
88
33
22 11
11
66
55 13
13 10
10 88
33
22 11
11
66
55
33 10
10 88 13
13 22 11
11
i
j
L4.11
66 10
10 13
13 55
Day 6
88
33
22 11
11
66
55 13
13 10
10 88
33
22 11
11
66
55
33 10
10 88 13
13 22 11
11
i
j
L4.12
66 10
10 13
13 55
Day 6
88
33
22 11
11
66
55 13
13 10
10 88
33
22 11
11
66
55
33 10
10 88 13
13 22 11
11
66
55
33
22
i
88 13
13 10
10 11
11
j
L4.13
66 10
10 13
13 55
Day 6
88
33
22 11
11
66
55 13
13 10
10 88
33
22 11
11
66
55
33 10
10 88 13
13 22 11
11
66
55
33
22
i
88 13
13 10
10 11
11
j
L4.14
66 10
10 13
13 55
Day 6
88
33
22 11
11
66
55 13
13 10
10 88
33
22 11
11
66
55
33 10
10 88 13
13 22 11
11
66
55
33
22
i
88 13
13 10
10 11
11
L4.15
66 10
10 13
13 55
Day 6
88
33
22 11
11
66
55 13
13 10
10 88
33
22 11
11
66
55
33 10
10 88 13
13 22 11
11
66
55
33
22
88 13
13 10
10 11
11
22
55
33
66
i
88 13
13 10
10 11
11
L4.16
Pseudocode for quicksort

QUICKSORT(A, p, r)
if p < r
then q PARTITION(A, p, r)
QUICKSORT(A, p, q1)
QUICKSORT(A, q+1, r)
Initial call: QUICKSORT(A, 1, n)
Day 6
L4.17
Analysis of quicksort
Assume all input elements are distinct.
In practice, there are better partitioning
algorithms for when duplicate input
elements may exist.
Let T(n) = worst-case running time on
an array of n elements.
Day 6
L4.18
Worst-case of quicksort
Input sorted or reverse sorted.
Partition around min or max element.
One side of partition always has no elements.
T (n) = T (0) + T (n 1) + (n)

= (1) + T (n 1) + (n)
= T (n 1) + (n)
= ( n 2 )
Day 6
(arithmetic series)
L4.19
Worst-case recursion tree

T(n) = T(0) + T(n1) + cn
Day 6
L4.20

T(n) = T(0) + T(n1) + cn
T(n)
Day 6
L4.21

T(n) = T(0) + T(n1) + cn
cn
T(0) T(n1)
Day 6
L4.22

T(n) = T(0) + T(n1) + cn
cn
T(0) c(n1)
T(0) T(n2)
Day 6
L4.23

T(n) = T(0) + T(n1) + cn
cn
T(0) c(n1)
T(0) c(n2)
T(0)
O
(1)
Day 6
L4.24

T(n) = T(0) + T(n1) + cn
cn
T(0) c(n1)
T(0) c(n2)
T(0)
n
k = (n 2 )
k =1
O
(1)
Day 6
L4.25

T(n) = T(0) + T(n1) + cn
cn
(1) c(n1)
h=n
(1) c(n2)
(1)
n
k = (n 2 )
k =1
T(n) = (n) + (n2)
= (n2)
(1)
Day 6
L4.26
Best-case analysis
(For intuition only!)
If were lucky, PARTITION splits the array evenly:

T(n) = 2T(n/2) + (n)
(same as merge sort)
= (n lg n)
What if the split is
1 9
always 10 : 10 ?
T (n) = T (101 n ) + T (109 n ) + (n)

What is the solution to this recurrence?
Day 6
L4.27
Analysis of almost-best case

T (n)
Day 6
L4.28

cn
T (101 n )
Day 6
T (109 n )
L4.29

cn
1
10
cn
9
10
cn
9
9
81
1
T (100
n ) T (100
n ) T (100
n )T (100
n)
Day 6
L4.30

1
10
cn
9
100
9
10
cn
cn
log10/9n
9
81
cn
cn
100
100
(1)
O(n)
O(n) leaves
leaves
1
100
cn
cn
cn
cn
cn
(1)
Day 6
L4.31

1
10
9
100
9
10
cn
cn
log10/9n
9
81
cn
cn
100
100
(1)
(n lg n)
Lucky!
Day 6
O(n)
O(n) leaves
leaves
log10n
1
cn
100
cn
cn
cn
cn
cn
(1)
cn log10n T(n) cn log10/9n + (n)
L4.32
More intuition
Suppose we alternate lucky, unlucky,
lucky, unlucky, lucky, .
L(n) = 2U(n/2) + (n) lucky
U(n) = L(n 1) + (n) unlucky
Solving:
L(n) = 2(L(n/2 1) + (n/2)) + (n)
= 2L(n/2 1) + (n)
= (n lg n) Lucky!
How can we make sure we are usually lucky?
Day 6
L4.33
Randomized quicksort
IDEA: Partition around a random element.
Running time is independent of the input
order.
No assumptions need to be made about
the input distribution.
No specific input elicits the worst-case
behavior.
The worst case is determined only by the
output of a random-number generator.
Day 6
L4.34
Randomized quicksort
analysis
Let T(n) = the random variable for the running
time of randomized quicksort on an input of size
n, assuming random numbers are independent.
For k = 0, 1, , n1, define the indicator
random variable
Xk =
1 if PARTITION generates a k : nk1 split,

0 otherwise.
E[Xk] = Pr{Xk = 1} = 1/n, since all splits are

equally likely, assuming elements are distinct.
Day 6
L4.35
Analysis (continued)
T(n) =
T(0) + T(n1) + (n) if 0 : n1 split,

T(1) + T(n2) + (n) if 1 : n2 split,
M
T(n1) + T(0) + (n) if n1 : 0 split,
n 1
X k (T (k ) + T (n k 1) + (n)) .
k =0
Day 6
L4.36
Calculating expectation
n 1
E[T (n)] = E X k (T (k ) + T (n k 1) + (n) )

k =0
Take expectations of both sides.
Day 6
L4.37
n 1
E[T (n)] = E X k (T (k ) + T (n k 1) + (n) )

k =0
n 1
E[ X k (T (k ) + T (n k 1) + (n) )]
k =0
Linearity of expectation.
Day 6
L4.38
n 1
E[T (n)] = E X k (T (k ) + T (n k 1) + (n) )

k =0
=
=
n 1
E[ X k (T (k ) + T (n k 1) + (n) )]
k =0
n 1
E[ X k ] E[T (k ) + T (n k 1) + (n)]
k =0
Independence of Xk from other random

choices.
Day 6
L4.39
n 1
E[T (n)] = E X k (T (k ) + T (n k 1) + (n) )

k =0
=
=
n 1
E[ X k (T (k ) + T (n k 1) + (n) )]
k =0
n 1
E[ X k ] E[T (k ) + T (n k 1) + (n)]
k =0
n 1
n 1
n 1
= 1 E [T (k )] + 1 E [T (n k 1)] + 1 (n)
n k =0
n k =0
n k =0
Linearity of expectation; E[Xk] = 1/n .

Day 6
L4.40
n 1
E[T (n)] = E X k (T (k ) + T ( n k 1) + (n) )
k =0
=
=
n 1
E[ X k (T (k ) + T (n k 1) + (n) )]
k =0
n 1
E[ X k ] E[T (k ) + T (n k 1) + (n)]
k =0
n 1
n 1
n 1
= 1 E [T (k )] + 1 E [T (n k 1)] + 1 (n)
n k =0
n k =0
n k =0
n 1
= 2 E [T (k )] + (n)
n k =1
Day 6
Summations have
identical terms.
L4.41
Hairy recurrence
n 1
E[T (n)] = 2 E [T (k )] + (n)

n k =2
(The k = 0, 1 terms can be absorbed in the (n).)
Prove: E[T(n)] an lg n for constant a > 0 .
Choose a large enough so that an lg n
dominates E[T(n)] for sufficiently small n 2.
n 1
Use fact:
1 n 2 lg n 1n 2
k
lg
k
(exercise).
2
8
k =2
Day 6
L4.42
Substitution method
n 1
E [T (n)] 2 ak lg k + (n)
n k =2
Substitute inductive hypothesis.
Day 6
L4.43
Substitution method
n 1
E [T (n)] 2 ak lg k + (n)
n k =2
2a 1 n 2 lg n 1 n 2 + (n)
n 2
8
Use fact.
Day 6
L4.44
Substitution method
n 1
E [T (n)] 2 ak lg k + (n)
n k =2
2a 1 n 2 lg n 1 n 2 + (n)
n 2
8
= an lg n an (n)
4
Express as desired residual.
Day 6
L4.45
Substitution method
n 1
E [T (n)] 2 ak lg k + (n)
n k =2
= 2a 1 n 2 lg n 1 n 2 + (n)
n 2
8
= an lg n an (n)
4
an lg n ,
if a is chosen large enough so that
an/4 dominates the (n).
Day 6
L4.46
Quicksort in practice
Quicksort is a great general-purpose
sorting algorithm.
Quicksort is typically over twice as fast
as merge sort.
Quicksort can benefit substantially from
code tuning.
Quicksort behaves well even with
caching and virtual memory.
Day 6
L4.47
6.046J/18.401J/SMA5503
Lecture 5
Prof. Erik Demaine
How fast can we sort?

All the sorting algorithms we have seen so far
are comparison sorts: only use comparisons to
determine the relative order of elements.
E.g., insertion sort, merge sort, quicksort,
heapsort.
The best worst-case running time that weve
seen for comparison sorting is O(n lg n) .
Is O(n lg n) the best we can do?
Decision trees can help us answer this question.
2001 by Charles E. Leiserson
Day 8
L5.2
Decision-tree example
Sort a1, a2, , an
1:2
1:2
2:3
2:3
123
123
1:3
1:3
213
213
1:3
1:3
132
132
312
312
2:3
2:3
231
231
321
321
Each internal node is labeled i:j for i, j {1, 2,, n}.

The left subtree shows subsequent comparisons if ai aj.
The right subtree shows subsequent comparisons if ai aj.
Day 8
L5.3
Sort a1, a2, a3
= 9, 4, 6 :
1:2
1:2
94
2:3
2:3
123
123
1:3
1:3
213
213
1:3
1:3
132
132
312
312
2:3
2:3
231
231
321
321

Day 8
L5.4
Sort a1, a2, a3
= 9, 4, 6 :
1:2
1:2
2:3
2:3
123
123
1:3
1:3
213
213
1:3
1:3
132
132
312
312
96
2:3
2:3
231
231
321
321

Day 8
L5.5
Sort a1, a2, a3
= 9, 4, 6 :
1:2
1:2
2:3
2:3
123
123
1:3
1:3
213
213 4 6 2:3
2:3
1:3
1:3
132
132
312
312
231
231
321
321

Day 8
L5.6
Sort a1, a2, a3
= 9, 4, 6 :
1:2
1:2
2:3
2:3
123
123
1:3
1:3
213
213
1:3
1:3
132
132
312
312
2:3
2:3
231
231
321
321
469
Each leaf contains a permutation (1), (2),, (n) to
indicate that the ordering a(1) a(2) L a(n) has been
established.
Day 8
L5.7
Decision-tree model
A decision tree can model the execution of
any comparison sort:
One tree for each input size n.
View the algorithm as splitting whenever
it compares two elements.
The tree contains the comparisons along
all possible instruction traces.
The running time of the algorithm = the
length of the path taken.
Worst-case running time = height of tree.
Day 8
L5.8
Lower bound for decisiontree sorting

Theorem. Any decision tree that can sort n
elements must have height (n lg n) .
Proof. The tree must contain n! leaves, since
there are n! possible permutations. A height-h
binary tree has 2h leaves. Thus, n! 2h .
h lg(n!)
(lg is mono. increasing)
(Stirlings formula)
lg ((n/e)n)
= n lg n n lg e
= (n lg n) .
Day 8
L5.9
Lower bound for comparison

sorting
Corollary. Heapsort and merge sort are
asymptotically optimal comparison sorting
algorithms.
Day 8
L5.10
Sorting in linear time

Counting sort: No comparisons between elements.
Input: A[1 . . n], where A[ j]{1, 2, , k} .
Output: B[1 . . n], sorted.
Auxiliary storage: C[1 . . k] .
Day 8
L5.11
Counting sort
for i 1 to k
do C[i] 0
for j 1 to n
do C[A[ j]] C[A[ j]] + 1 C[i] = |{key = i}|
for i 2 to k
do C[i] C[i] + C[i1]
C[i] = |{key i}|
for j n downto 1
do B[C[A[ j]]] A[ j]
C[A[ j]] C[A[ j]] 1
Day 8
L5.12
Counting-sort example
A:
44
11
33
44
33
C:
B:
Day 8
L5.13
Loop 1
A:
44
11
33
44
33
C:
00
00
00
00
B:
for i 1 to k
do C[i] 0
Day 8
L5.14
Loop 2
A:
44
11
33
44
33
C:
00
00
00
11
B:
for j 1 to n
do C[A[ j]] C[A[ j]] + 1 C[i] = |{key = i}|
Day 8
L5.15
Loop 2
A:
44
11
33
44
33
C:
11
00
00
11
B:
for j 1 to n
do C[A[ j]] C[A[ j]] + 1 C[i] = |{key = i}|
Day 8
L5.16
Loop 2
A:
44
11
33
44
33
C:
11
00
11
11
B:
for j 1 to n
do C[A[ j]] C[A[ j]] + 1 C[i] = |{key = i}|
Day 8
L5.17
Loop 2
A:
44
11
33
44
33
C:
11
00
11
22
B:
for j 1 to n
do C[A[ j]] C[A[ j]] + 1 C[i] = |{key = i}|
Day 8
L5.18
Loop 2
A:
44
11
33
44
33
C:
11
00
22
22
B:
for j 1 to n
do C[A[ j]] C[A[ j]] + 1 C[i] = |{key = i}|
Day 8
L5.19
Loop 3
A:
44
11
33
44
33
B:
C:
11
00
22
22
C':
11
11
22
22
for i 2 to k
C[i] = |{key i}|

Day 8
L5.20
Loop 3
A:
44
11
33
44
33
B:
C:
11
00
22
22
C':
11
11
33
22
for i 2 to k
C[i] = |{key i}|

Day 8
L5.21
Loop 3
A:
44
11
33
44
33
B:
C:
11
00
22
22
C':
11
11
33
55
for i 2 to k
C[i] = |{key i}|

Day 8
L5.22
Loop 4
A:
B:
44
11
33
44
33
33
C:
11
11
33
55
C':
11
11
22
55
for j n downto 1
C[A[ j]] C[A[ j]] 1
Day 8
L5.23
Loop 4
A:
B:
44
11
33
44
33
44
33
C:
11
11
22
55
C':
11
11
22
44
for j n downto 1
C[A[ j]] C[A[ j]] 1
Day 8
L5.24
Loop 4
A:
B:
44
11
33
44
33
33
33
44
C:
11
11
22
44
C':
11
11
11
44
for j n downto 1
C[A[ j]] C[A[ j]] 1
Day 8
L5.25
Loop 4
1
A:
44
11
33
44
33
B:
11
33
33
44
C:
11
11
11
44
C':
00
11
11
44
for j n downto 1
C[A[ j]] C[A[ j]] 1
Day 8
L5.26
Loop 4
1
A:
44
11
33
44
33
B:
11
33
33
44
44
C:
00
11
11
44
C':
00
11
11
33
for j n downto 1
C[A[ j]] C[A[ j]] 1
Day 8
L5.27
Analysis
(k)
(n)
(k)
(n)
for i 1 to k
do C[i] 0
for j 1 to n
do C[A[ j]] C[A[ j]] + 1
for i 2 to k
for j n downto 1
C[A[ j]] C[A[ j]] 1
(n + k)
Day 8
L5.28
Running time
If k = O(n), then counting sort takes (n) time.
But, sorting takes (n lg n) time!
Wheres the fallacy?
Answer:
Comparison sorting takes (n lg n) time.
Counting sort is not a comparison sort.
In fact, not a single comparison between
elements occurs!
Day 8
L5.29
Stable sorting
Counting sort is a stable sort: it preserves
the input order among equal elements.
A:
44
11
33
44
33
B:
11
33
33
44
44
Exercise: What other sorts have this property?

Day 8
L5.30
Radix sort
Origin: Herman Holleriths card-sorting
machine for the 1890 U.S. Census. (See
Appendix .)
Digit-by-digit sort.
Holleriths original (bad) idea: sort on
most-significant digit first.
Good idea: Sort on least-significant digit
first with auxiliary stable sort.
Day 8
L5.31
Operation of radix sort

329
457
657
839
436
720
355
720
355
436
457
657
329
839
720
329
436
839
355
457
657
329
355
436
457
657
720
839
Day 8
L5.32
Correctness of radix sort

Induction on digit position
Assume that the numbers
are sorted by their low-order
t 1 digits.
Sort on digit t
720
329
436
839
355
457
657
329
355
436
457
657
720
839
Day 8
L5.33

t 1 digits.
Sort on digit t
Two numbers that differ in
digit t are correctly sorted.
720
329
436
839
355
457
657
329
355
436
457
657
720
839
Day 8
L5.34

t 1 digits.
Sort on digit t
Two numbers that differ in
digit t are correctly sorted.
Two numbers equal in digit t
are put in the same order as
the input correct order.
720
329
436
839
355
457
657
329
355
436
457
657
720
839
Day 8
L5.35
Analysis of radix sort

Assume counting sort is the auxiliary stable sort.
Sort n computer words of b bits each.
Each word can be viewed as having b/r base-2r
digits.
8
8
8
8
Example: 32-bit word
r = 8 b/r = 4 passes of counting sort on
base-28 digits; or r = 16 b/r = 2 passes of
counting sort on base-216 digits.
How many passes should we make?
Day 8
L5.36
Recall: Counting sort takes (n + k) time to
sort n numbers in the range from 0 to k 1.
If each b-bit word is broken into r-bit pieces,
each pass of counting sort takes (n + 2r) time.
Since there are b/r passes, we have
T (n, b) = b (n + 2 r ) .
r
Choose r to minimize T(n, b):
Increasing r means fewer passes, but as
r >> lg n, the time grows exponentially.
Day 8
L5.37
Choosing r
T (n, b) = b (n + 2 r )
r
Minimize T(n, b) by differentiating and setting to 0.
Or, just observe that we dont want 2r >> n, and
theres no harm asymptotically in choosing r as
large as possible subject to this constraint.
Choosing r = lg n implies T(n, b) = (bn/lg n) .
For numbers in the range from 0 to n d 1, we
have b = d lg n radix sort runs in (d n) time.
Day 8
L5.38
Conclusions
In practice, radix sort is fast for large inputs, as
well as simple to code and maintain.
Example (32-bit numbers):
At most 3 passes when sorting 2000 numbers.
Merge sort and quicksort do at least lg 2000 =
11 passes.
Downside: Unlike quicksort, radix sort displays
little locality of reference, and thus a well-tuned
quicksort fares better on modern processors,
which feature steep memory hierarchies.
Day 8
L5.39
Appendix: Punched-card
technology
Herman Hollerith (1860-1929)
Punched cards
Holleriths tabulating system
Operation of the sorter
Origin of radix sort
Modern IBM card
Web resources on punchedcard technology
Day 8
L5.40
Herman Hollerith
(1860-1929)
The 1880 U.S. Census took almost
10 years to process.
While a lecturer at MIT, Hollerith
prototyped punched-card technology.
His machines, including a card sorter, allowed
the 1890 census total to be reported in 6 weeks.
He founded the Tabulating Machine Company in
1911, which merged with other companies in 1924
to form International Business Machines.
Day 8
L5.41
Punched cards
Punched card = data record.
Hole = value.
Algorithm = machine + human operator.
Replica of punch
card from the
1900 U.S. census:
[Howells 2000]
Day 8
L5.42
Holleriths
tabulating
system
See figure from

[Howells 2000] .
Pantograph card
punch
Hand-press reader
Dial counters
Sorting box
Day 8
L5.43
Operation of the sorter

An operator inserts a card into
the press.
Pins on the press reach through
the punched holes to make
electrical contact with mercuryfilled cups beneath the card.
Whenever a particular digit
value is punched, the lid of the
corresponding sorting bin lifts.
The operator deposits the card
into the bin and closes the lid.
When all cards have been processed, the front panel is opened, and
the cards are collected in order, yielding one pass of a stable sort.
Day 8
L5.44
Origin of radix sort

Holleriths original 1889 patent alludes to a mostsignificant-digit-first radix sort:
The most complicated combinations can readily be
counted with comparatively few counters or relays by first
assorting the cards according to the first items entering
into the combinations, then reassorting each group
according to the second item entering into the combination,
and so on, and finally counting on a few counters the last
item of the combination for each group of cards.
Least-significant-digit-first radix sort seems to be

a folk invention originated by machine operators.
Day 8
L5.45
Modern IBM card

One character per column.
See examples on the WWW Virtual Punch-Card Server.

.
So, thats why text windows have 80 columns!

Day 8
L5.46
Web resources on punchedcard technology

Doug Joness punched card index
Biography of Herman Hollerith
The 1890 U.S. Census
Early history of IBM
Pictures of Holleriths inventions
Holleriths patent application (borrowed
from Gordon Bells CyberMuseum)
Impact of punched cards on U.S. history
Day 8
L5.47
6.046J/18.401J/SMA5503
Lecture 6
Prof. Erik Demaine
Order statistics
Select the ith smallest of n elements (the
element with rank i).
i = 1: minimum;
i = n: maximum;
i = (n+1)/2 or (n+1)/2: median.
Naive algorithm: Sort and index ith element.
Worst-case running time = (n lg n) + (1)
= (n lg n),
using merge sort or heapsort (not quicksort).
Day 9
L6.2
Randomized divide-andconquer algorithm

RAND-SELECT(A, p, q, i) ith smallest of A[ p . . q]
if p = q then return A[ p]
r RAND-PARTITION(A, p, q)
krp+1
k = rank(A[r])
if i = k then return A[ r]
if i < k
then return RAND-SELECT(A, p, r 1, i )
else return RAND-SELECT(A, r + 1, q, i k )
k
A[r]
A[r]
A[r]
A[r]
p
r
q
Day 9
L6.3
Example
Select the i = 7th smallest:
66 10
10 13
13 55
pivot
Partition:
22 55
33
66
88
33
22 11
11
i=7
88 13
13 10
10 11
11
k=4
Select the 7 4 = 3rd smallest recursively.

Day 9
L6.4
Intuition for analysis

(All our analyses today assume that all elements
are distinct.)
Lucky:
n log10 / 9 1 = n 0 = 1
T(n) = T(9n/10) + (n)
CASE 3
= (n)
Unlucky:
arithmetic series
T(n) = T(n 1) + (n)
= (n2)
Worse than sorting!
Day 9
L6.5
Analysis of expected time

The analysis follows that of randomized
quicksort, but its a little different.
Let T(n) = the random variable for the running
time of RAND-SELECT on an input of size n,
assuming random numbers are independent.
For k = 0, 1, , n1, define the indicator
random variable
1 if PARTITION generates a k : nk1 split,
Xk =
0 otherwise.
Day 9
L6.6
To obtain an upper bound, assume that the ith
element always falls in the larger side of the
partition:
T(max{0, n1}) + (n) if 0 : n1 split,
T(max{1, n2}) + (n) if 1 : n2 split,
T(n) =
M
T(max{n1, 0}) + (n) if n1 : 0 split,
=
n 1
X k (T (max{k , n k 1}) + (n)) .
k =0
Day 9
L6.7
n 1
E[T (n)] = E X k (T (max{k , n k 1}) + (n) )

k =0
Take expectations of both sides.
Day 9
L6.8
n 1
E[T (n)] = E X k (T (max{k , n k 1}) + (n) )

k =0
n 1
E[ X k (T (max{k , n k 1}) + (n) )]
k =0
Day 9
L6.9
n 1
E[T (n)] = E X k (T (max{k , n k 1}) + (n) )

k =0
=
=
n 1
E[ X k (T (max{k , n k 1}) + (n) )]
k =0
n 1
E[ X k ] E[T (max{k , n k 1}) + (n)]
k =0
Independence of Xk from other random

choices.
Day 9
L6.10
n 1
E[T (n)] = E X k (T (max{k , n k 1}) + (n) )

k =0
=
=
n 1
E[ X k (T (max{k , n k 1}) + (n) )]
k =0
n 1
E[ X k ] E[T (max{k , n k 1}) + (n)]
k =0
n 1
n 1
= 1 E [T (max{k , n k 1})] + 1 (n)

n k =0
n k =0
Linearity of expectation; E[Xk] = 1/n .

Day 9
L6.11
n 1
E[T (n)] = E X k (T (max{k , n k 1}) + (n) )
k =0
=
=
n 1
E[ X k (T (max{k , n k 1}) + (n) )]
k =0
n 1
E[ X k ] E[T (max{k , n k 1}) + (n)]
k =0
n 1
n 1
= 1 E [T (max{k , n k 1})] + 1 (n)

n k =0
n k =0
n 1
2 E [T (k )] + (n)
n k = n / 2
Upper terms
appear twice.
Day 9
L6.12
Hairy recurrence
(But not quite as hairy as the quicksort one.)
n 1
E[T (n)] = 2 E [T (k )] + (n)

n k= n/2

Prove: E[T(n)] cn for constant c > 0 .
The constant c can be chosen large enough
so that E[T(n)] cn for the base cases.
n 1
3n 2
k
8 (exercise).
Use fact:
k = n / 2
Day 9
L6.13
Substitution method
n 1
E [T (n)] 2 ck + (n)
n k= n/2

Substitute inductive hypothesis.
Day 9
L6.14
Substitution method
n 1
E [T (n)] 2 ck + (n)
n k= n/2

2c 3 n 2 + (n)
n 8
Use fact.
Day 9
L6.15
Substitution method
n 1
E [T (n)] 2 ck + (n)
n k= n/2

2c 3 n 2 + (n)
n 8
= cn cn (n)
4
Express as desired residual.
Day 9
L6.16
Substitution method
n 1
E [T (n)] 2 ck + (n)
n k= n/2

2c 3 n 2 + (n)
n 8
= cn cn (n)
4
cn ,
if c is chosen large enough so
that cn/4 dominates the (n).
Day 9
L6.17
Summary of randomized
order-statistic selection
Works fast: linear expected time.
Excellent algorithm in practice.
But, the worst case is very bad: (n2).
Q. Is there an algorithm that runs in linear
time in the worst case?
A. Yes, due to Blum, Floyd, Pratt, Rivest,
and Tarjan [1973].
IDEA: Generate a good pivot recursively.
Day 9
L6.18
Worst-case linear-time order

statistics
SELECT(i, n)
1. Divide the n elements into groups of 5. Find
the median of each 5-element group by rote.
2. Recursively SELECT the median x of the n/5
group medians to be the pivot.
3. Partition around the pivot x. Let k = rank(x).
4. if i = k then return x
elseif i < k
then recursively SELECT the ith
smallest element in the lower part
else recursively SELECT the (ik)th
smallest element in the upper part
Same as
RANDSELECT
Day 9
L6.19
Choosing the pivot
Day 9
L6.20
Choosing the pivot
1. Divide the n elements into groups of 5.
Day 9
L6.21
Choosing the pivot
1. Divide the n elements into groups of 5. Find lesser

greater
Day 9
L6.22
Choosing the pivot
1. Divide the n elements into groups of 5. Find lesser

greater
Day 9
L6.23
Analysis
At least half the group medians are x, which

is at least n/5 /2 = n/10 group medians.
lesser
greater
Day 9
L6.24
Analysis
(Assume all elements are distinct.)

Therefore, at least 3 n/10 elements are x.
lesser
greater
Day 9
L6.25
Analysis
(Assume all elements are distinct.)

Therefore, at least 3 n/10 elements are x.
Similarly, at least 3 n/10 elements are x.
lesser
greater
Day 9
L6.26
Minor simplification
For n 50, we have 3 n/10 n/4.
Therefore, for n 50 the recursive call to
SELECT in Step 4 is executed recursively
on 3n/4 elements.
Thus, the recurrence for running time
can assume that Step 4 takes time
T(3n/4) in the worst case.
For n < 50, we know that the worst-case
time is T(n) = (1).
Day 9
L6.27
Developing the recurrence

T(n)
(n)
T(n/5)
(n)
T(3n/4)
SELECT(i, n)
1. Divide the n elements into groups of 5. Find
3. Partition around the pivot x. Let k = rank(x).
4. if i = k then return x
elseif i < k
then recursively SELECT the ith
smallest element in the lower part
else recursively SELECT the (ik)th
smallest element in the upper part
Day 9
L6.28
Solving the recurrence

T (n) = T 1 n + T 3 n + (n)
4
5
T (n) 1 cn + 3 cn + (n)
4
5
= 19 cn + (n)
20
= cn 1 cn (n)
20
cn ,
if c is chosen large enough to handle both the
(n) and the initial conditions.
Substitution:
T(n) cn
Day 9
L6.29
Conclusions
Since the work at each level of recursion
is a constant fraction (19/20) smaller, the
work per level is a geometric series
dominated by the linear work at the root.
In practice, this algorithm runs slowly,
because the constant in front of n is large.
The randomized algorithm is far more
practical.
Exercise: Why not divide into groups of 3?
Day 9
L6.30
6.046J/18.401J/SMA5503
Lecture 7
Symbol-table problem
Symbol table T holding n records:
x
record
key[x]
key[x]
Other fields
containing
satellite data
Operations on T:
INSERT(T, x)
DELETE(T, x)
SEARCH(T, k)
How should the data structure T be organized?

Day 11
L7.2
Direct-access table
IDEA: Suppose that the set of keys is K {0,
1, , m1}, and keys are distinct. Set up an
array T[0 . . m1]:
x
if k K and key[x] = k,
T[k] =
NIL otherwise.
Then, operations take (1) time.
Problem: The range of keys can be large:
64-bit numbers (which represent
18,446,744,073,709,551,616 different keys),
character strings (even larger!).
Day 11
L7.3
Hash functions
Solution: Use a hash function h to map the
universe U of all keys into
T
{0, 1, , m1}:
0
k1
K
k2
k4
h(k1)
h(k4)
k5
h(k2) = h(k5)
k3
h(k3)
m1
When a record to be inserted maps to an already

As
each key
h maps
it to a slot of T.
occupied
slotisininserted,
T, a collision
occurs.
Day 11
L7.4
Resolving collisions by
chaining
Records in the same slot are linked into a list.
T
49
49
86
86
52
52
h(49) = h(86) = h(52) = i
Day 11
L7.5
Analysis of chaining
We make the assumption of simple uniform
hashing:
Each key k K of keys is equally likely to
be hashed to any slot of table T, independent
of where other keys are hashed.
Let n be the number of keys in the table, and
let m be the number of slots.
Define the load factor of T to be
= n/m
= average number of keys per slot.
Day 11
L7.6
Search cost
Expected time to search for a record with
a given key = (1 + ).
apply hash
function and
access slot
search
the list
Expected search time = (1) if = O(1),

or equivalently, if n = O(m).
Day 11
L7.7
Choosing a hash function

The assumption of simple uniform hashing
is hard to guarantee, but several common
techniques tend to work well in practice as
long as their deficiencies can be avoided.
Desirata:
A good hash function should distribute the
keys uniformly into the slots of the table.
Regularity in the key distribution should
not affect this uniformity.
Day 11
L7.8
Division method
Assume all keys are integers, and define
h(k) = k mod m.
Deficiency: Dont pick an m that has a small
divisor d. A preponderance of keys that are
congruent modulo d can adversely affect
uniformity.
Extreme deficiency: If m = 2r, then the hash
doesnt even depend on all the bits of k:
If k = 10110001110110102 and r = 6, then
h(k) = 0110102 .
h(k)
Day 11
L7.9
Division method (continued)

h(k) = k mod m.
Pick m to be a prime not too close to a power
of 2 or 10 and not otherwise used prominently
in the computing environment.
Annoyance:
Sometimes, making the table size a prime is
inconvenient.
But, this method is popular, although the next
method well see is usually superior.
Day 11
L7.10
Multiplication method
Assume that all keys are integers, m = 2r, and our
computer has w-bit words. Define
h(k) = (Ak mod 2w) rsh (w r),
where rsh is the bit-wise right-shift operator
and A is an odd integer in the range 2w1 < A < 2w.
Dont pick A too close to 2w.
Multiplication modulo 2w is fast.
The rsh operator is fast.
Day 11
L7.11
Multiplication method
example
h(k) = (Ak mod 2w) rsh (w r)
Suppose that m = 8 = 23 and that our computer
has w = 7-bit words:
1011001 =A
1101011 =k
10010100110011
A
h(k)
0
7 1
5 4 3
.
2A
Modular wheel
3A
Day 11
L7.12
Dot-product method
Randomized strategy:
Let m be prime. Decompose key k into r + 1
digits, each with value in the set {0, 1, , m1}.
That is, let k = k0, k1, , km1, where 0 ki < m.
Pick a = a0, a1, , am1 where each ai is chosen
randomly from {0, 1, , m1}.
Define ha (k ) =
ai ki mod m.
i =0
Excellent in practice, but expensive to compute.

Day 11
L7.13
Resolving collisions by open

addressing
No storage is used outside of the hash table itself.
Insertion systematically probes the table until an
empty slot is found.
The hash function depends on both the key and
probe number:
h : U {0, 1, , m1} {0, 1, , m1}.
The probe sequence h(k,0), h(k,1), , h(k,m1)
should be a permutation of {0, 1, , m1}.
The table may fill up, and deletion is difficult (but
not impossible).
Day 11
L7.14
Example of open addressing

Insert key k = 496:
0. Probe h(496,0)
586
133
204
collision
481
m1
Day 11
L7.15

Insert key k = 496:
0. Probe h(496,0)
1. Probe h(496,1)
586
133
collision
204
481
m1
Day 11
L7.16

Insert key k = 496:
0. Probe h(496,0)
1. Probe h(496,1)
2. Probe h(496,2)
586
133
204
496
481
insertion
m1
Day 11
L7.17

Search for key k = 496:
0. Probe h(496,0)
1. Probe h(496,1)
2. Probe h(496,2)
586
133
204
496
481
Search uses the same probe

sequence, terminating sucm1
cessfully if it finds the key
and unsuccessfully if it encounters an empty slot.
Day 11
L7.18
Probing strategies
Linear probing:
Given an ordinary hash function h(k), linear
probing uses the hash function
h(k,i) = (h(k) + i) mod m.
This method, though simple, suffers from primary
clustering, where long runs of occupied slots build
up, increasing the average search time. Moreover,
the long runs of occupied slots tend to get longer.
Day 11
L7.19
Probing strategies
Double hashing
Given two ordinary hash functions h1(k) and h2(k),
double hashing uses the hash function
h(k,i) = (h1(k) + ih2(k)) mod m.
This method generally produces excellent results,
but h2(k) must be relatively prime to m. One way
is to make m a power of 2 and design h2(k) to
produce only odd numbers.
Day 11
L7.20
Analysis of open addressing

We make the assumption of uniform hashing:
Each key is equally likely to have any one of
the m! permutations as its probe sequence.
Theorem. Given an open-addressed hash
table with load factor = n/m < 1, the
expected number of probes in an unsuccessful
search is at most 1/(1).
Day 11
L7.21
Proof of the theorem

Proof.
At least one probe is always necessary.
With probability n/m, the first probe hits an
occupied slot, and a second probe is necessary.
With probability (n1)/(m1), the second probe
hits an occupied slot, and a third probe is
necessary.
With probability (n2)/(m2), the third probe
hits an occupied slot, etc.
n
i
n
<
= for i = 1, 2, , n.
Observe that
mi m
Day 11
L7.22
Proof (continued)
Therefore, the expected number of probes is
2
n
n
1
n
1
1 + 1 +
L
L 1 +
1 +
m m 1 m 2 m n + 1
1 + (1 + (1 + (L (1 + )L)))
1+ + 2 +3 +L
=
i =0
= 1 .
1
The textbook has a

more rigorous proof.
Day 11
L7.23
Implications of the theorem

If is constant, then accessing an openaddressed hash table takes constant time.
If the table is half full, then the expected
number of probes is 1/(10.5) = 2.
If the table is 90% full, then the expected
number of probes is 1/(10.9) = 10.
Day 11
L7.24
6.046J/18.401J/SMA5503
Lecture 8
A weakness of hashing
Problem: For any hash function h, a set
of keys exists that can cause the average
access time of a hash table to skyrocket.
An adversary can pick all keys from
{k U : h(k) = i} for some slot i.
IDEA: Choose the hash function at random,
independently of the keys.
Even if an adversary can see your code,
he or she cannot find a bad set of keys,
since he or she doesnt know exactly
which hash function will be chosen.
Day 12
L8.2
Universal hashing
Definition. Let U be a universe of keys, and
let H be a finite collection of hash functions,
each mapping U to {0, 1, , m1}. We say
H is universal if for all x, y U, where x y,
we have |{h H : h(x) = h(y)}| = |H|/m.
That is, the chance
of a collision
between x and y is
1/m if we choose h
randomly from H.
{h : h(x) = h(y)}
|H|
m
Day 12
L8.3
Universality is good
Theorem. Let h be a hash function chosen
(uniformly) at random from a universal set H
of hash functions. Suppose h is used to hash

n arbitrary keys into the m slots of a table T.
Then, for a given key x, we have
E[#collisions with x] < n/m.
Day 12
L8.4
Proof of theorem
Proof. Let Cx be the random variable denoting
the total number of collisions of keys in T with
x, and let
1 if h(x) = h(y),
cxy =
0 otherwise.
Note: E[cxy] = 1/m and C x =
cxy .
yT {x}
Day 12
L8.5
Proof (continued)
E[C x ] = E c xy
yT { x}
Take expectation
of both sides.
Day 12
L8.6
Proof (continued)
E[C x ] = E c xy
yT { x}
=
E[cxy ]
yT { x}
Take expectation
of both sides.
Linearity of
expectation.
Day 12
L8.7
Proof (continued)
E[C x ] = E c xy
yT { x}
=
E[cxy ]
Linearity of
expectation.
1/ m
E[cxy] = 1/m.
yT { x}
Take expectation
of both sides.
yT { x}
Day 12
L8.8
Proof (continued)
E[C x ] = E c xy
yT { x}
=
E[cxy ]
Linearity of
expectation.
1/ m
E[cxy] = 1/m.
yT { x}
Take expectation
of both sides.
yT { x}
= n 1 .
m
Algebra.
Day 12
L8.9
Constructing a set of
universal hash functions
Let m be prime. Decompose key k into r + 1
digits, each with value in the set {0, 1, , m1}.
That is, let k = k0, k1, , kr, where 0 ki < m.
Randomized strategy:
Pick a = a0, a1, , ar where each ai is chosen
randomly from {0, 1, , m1}.
r
Dot
product,
Define ha (k ) = ai ki mod m .
modulo
m
i =0
How big is H = {ha}? |H| = mr + 1. REMEMBER
THIS!
Day 12
L8.10
Universality of dot-product
hash functions
Theorem. The set H = {ha} is universal.
Proof. Suppose that x = x0, x1, , xr and y =

y0, y1, , yr be distinct keys. Thus, they differ
in at least one digit position, wlog position 0.
For how many ha Hdo x and y collide?
We must have ha(x) = ha(y), which implies that
r
i =0
i =0
ai xi ai yi
(mod m) .
Day 12
L8.11
Proof (continued)
Equivalently, we have
r
ai ( xi yi ) 0
(mod m)
i =0
or
r
a0 ( x0 y0 ) + ai ( xi yi ) 0
(mod m) ,
i =1
which implies that
a0 ( x0 y0 ) ai ( xi yi )
(mod m) .
i =1
Day 12
L8.12
Fact from number theory

Theorem. Let m be prime. For any z Zm
such that z 0, there exists a unique z1 Zm
such that
z z1 1 (mod m).
Example: m = 7.
1 2 3 4 5 6
z1
1 4 5 2 3 6
Day 12
L8.13
Back to the proof

We have
a0 ( x0 y0 ) ai ( xi yi )
(mod m) ,
i =1
and since x0 y0 , an inverse (x0 y0 )1 must exist,

which implies that
a0 ai ( xi yi ) ( x0 y0 ) 1
i =1
(mod m) .
Thus, for any choices of a1, a2, , ar, exactly

one choice of a0 causes x and y to collide.
Day 12
L8.14
Proof (completed)
Q. How many has cause x and y to collide?
A. There are m choices for each of a1, a2, , ar ,
but once these are chosen, exactly one choice
for a0 causes x and y to collide, namely
1
a0 = ai ( xi yi ) ( x0 y0 ) mod m .
i =1
Thus, the number

of hras that cause x and y
r
to collide is m 1 = m = |H|/m.
Day 12
L8.15
Perfect hashing
Given a set of n keys, construct a static hash
table of size m = O(n) such that SEARCH takes
(1) time in the worst case.
IDEA: Twolevel scheme
with universal
hashing at
both levels.
No collisions
at level 2!
T
0
1 44 31
31
2
3
4 11 00
00
5
6 99 86
86
m a
S1
14
27
1427
S4
26
26
h31(14) = h31(27) = 1
S6
40
22
40 37
37
22
0 1 2 3 4 5 6 7 8
Day 12
L8.16
Collisions at level 2
Theorem. Let H be a class of universal hash
functions for a table of size m = n2. Then, if we
use a random h H to hash n keys into the table,
the expected number of collisions is at most 1/2.
Proof. By the definition of universality, the
probability that 2 given keys in the table collide
n
2
under h is 1/m = 1/n . Since there are (2 ) pairs
of keys that can possibly collide, the expected
number of collisions is
n 1
n(n 1) 1
2 < 1.
2 =
2
2
n
2 n
Day 12
L8.17
No collisions at level 2
Corollary. The probability of no collisions
is at least 1/2.
Proof. Markovs inequality says that for any

nonnegative random variable X, we have
Pr{X t} E[X]/t.
Applying this inequality with t = 1, we find
that the probability of 1 or more collisions is
at most 1/2.
Thus, just by testing random hash functions
in H, well quickly find one that works.
Day 12
L8.18
Analysis of storage
For the level-1 hash table T, choose m = n, and
let ni be random variable for the number of keys
that hash to slot i in T. By using ni2 slots for the
level-2 hash table Si, the expected total storage
required for the two-level scheme is therefore
m1
2
E (ni ) = (n) ,
i =0
since the analysis is identical to the analysis from

recitation of the expected running time of bucket
sort. (For a probability bound, apply Markov.)
Day 12
L8.19
6.046J/18.401J/SMA5503
Lecture 9
Binary-search-tree sort
T
Create an empty BST
for i = 1 to n
do TREE-INSERT(T, A[i])
Perform an inorder tree walk of T.
Example:
A = [3 1 8 2 6 7 5]
33
11
Tree-walk time = O(n),

but how long does it
take to build the BST?
88
22
66
55
77
Day 17
L9.2
Analysis of BST sort

BST sort performs the same comparisons as
quicksort, but in a different order!
3 1 8 2 6 7 5
1 2
8 6 7 5
2
675
5
The expected time to build the tree is asymptotically the same as the running time of quicksort.
Day 17
L9.3
Node depth
The depth of a node = the number of comparisons
made during TREE-INSERT. Assuming all input
permutations are equally likely, we have
Average node depth
n
1
= E (# comparisons to insert node i )
n i =1
= 1 O(n lg n)
n
= O(lg n) .
(quicksort analysis)
Day 17
L9.4
Expected tree height

But, average node depth of a randomly built
BST = O(lg n) does not necessarily mean that its
expected height is also O(lg n) (although it is).
Example.
lg n
Ave. depth 1 n lg n + n n
n
2
= O(lg n)
h= n
Day 17
L9.5
Height of a randomly built

binary search tree
Outline of the analysis:
Prove Jensens inequality, which says that
f(E[X]) E[f(X)] for any convex function f and
random variable X.
Analyze the exponential height of a randomly
built BST on n nodes, which is the random
variable Yn = 2Xn, where Xn is the random
variable denoting the height of the BST.
Prove that 2E[Xn] E[2Xn ] = E[Yn] = O(n3),
and hence that E[Xn] = O(lg n).
Day 17
L9.6
Convex functions
A function f : R R is convex if for all
, 0 such that + = 1, we have
f(x + y) f(x) + f(y)
for all x,y R.
f
f(y)
f(x) + f(y)
f(x)
f(x + y)
x
x + y
y
Day 17
L9.7
Convexity lemma
Lemma. Let f : R R be a convex function,
and let {1, 2 , , n} be a set of nonnegative
constants such that k k = 1. Then, for any set
{x1, x2, , xn} of real numbers, we have
n
n
f k xk k f ( xk ) .
k =1
k =1
Proof. By induction on n. For n = 1, we have
1 = 1, and hence f(1x1) 1f(x1) trivially.
Day 17
L9.8
Proof (continued)
Inductive step:
n
f k xk =
k =1
n 1
k
f n xn + (1 n )
xk
k =11 n
Algebra.
Day 17
L9.9
Proof (continued)
Inductive step:
n 1
k
f n xn + (1 n )
xk
k =11 n
n1 k
n f ( xn ) + (1 n ) f
xk
k =11 n
f k xk =
k =1
Convexity.
Day 17
L9.10
Proof (continued)
Inductive step:
n 1
k
f n xn + (1 n )
xk
k =11 n
n1 k
n f ( xn ) + (1 n ) f
xk
k =11 n
n
f k xk =
k =1
n 1
k
n f ( xn ) + (1 n )
f ( xk )
k =11 n
Induction.
Day 17
L9.11
Proof (continued)
Inductive step:
n 1
k
f n xn + (1 n )
xk
k =11 n
n1 k
n f ( xn ) + (1 n ) f
xk
k =11 n
n
f k xk =
k =1
n 1
k
n f ( xn ) + (1 n )
f ( xk )
k =11 n
n
= k f ( xk ) .
k =1
Algebra.
Day 17
L9.12
Jensens inequality
Lemma. Let f be a convex function, and let X
be a random variable. Then, f (E[X]) E[ f (X)].
Proof.

f ( E[ X ]) = f k Pr{ X = k}
k =
Definition of expectation.
Day 17
L9.13
Jensens inequality
Proof.

f ( E[ X ]) = f k Pr{ X = k}
k =
f (k ) Pr{X = k}
k =
Convexity lemma (generalized).

Day 17
L9.14
Jensens inequality
Proof.

f ( E[ X ]) = f k Pr{ X = k}
k =
f (k ) Pr{X = k}
k =
= E[ f ( X )] .
Tricky step, but truethink about it.

Day 17
L9.15
Analysis of BST height

Let Xn be the random variable denoting
the height of a randomly built binary
search tree on n nodes, and let Yn = 2Xn
be its exponential height.
If the root of the tree has rank k, then
Xn = 1 + max{Xk1, Xnk} ,
since each of the left and right subtrees
of the root are randomly built. Hence,
we have
Yn = 2 max{Yk1, Ynk} .
Day 17
L9.16
Define the indicator random variable Znk as
1 if the root has rank k,
Znk =
0 otherwise.
Thus, Pr{Znk = 1} = E[Znk] = 1/n, and
n
Yn = Z nk (2 max{Yk 1 , Ynk }) .
k =1
Day 17
L9.17
Exponential height recurrence

n
E [Yn ] = E Z nk (2 max{Yk 1 , Ynk })

k =1
Take expectation of both sides.
Day 17
L9.18
n
k =1
n
= E [Z nk (2 max{Yk 1 , Ynk })]

k =1
Day 17
L9.19

n
k =1
n
= E [Z nk (2 max{Yk 1 , Ynk })]

k =1
n
= 2 E[ Z nk ] E[max{Yk 1 , Ynk }]
k =1
Independence of the rank of the root

from the ranks of subtree roots.
Day 17
L9.20

n
k =1
n
= E [Z nk (2 max{Yk 1 , Ynk })]

k =1
n
= 2 E[ Z nk ] E[max{Yk 1 , Ynk }]
k =1
n
2 E[Yk 1 + Ynk ]
n k =1
The max of two nonnegative numbers

is at most their sum, and E[Znk] = 1/n.
Day 17
L9.21

n
E[Yn ] = E Z nk (2 max{Yk 1 , Ynk })
k =1
n
= E[Z nk (2 max{Yk 1 , Ynk })]

k =1
n
= 2 E[ Z nk ] E[max{Yk 1 , Ynk }]
k =1
n
2 E[Yk 1 + Ynk ]
n k =1
n 1
= 4 E[Yk ]
n k =0
Each term appears

twice, and reindex.
Day 17
L9.22

Use substitution to
show that E[Yn] cn3
for some positive
constant c, which we
can pick sufficiently
large to handle the
initial conditions.
n 1
E [Yn ] = 4 E[Yk ]
n k =0
Day 17
L9.23

Use substitution to
show that E[Yn] cn3
for some positive
large to handle the
initial conditions.
n 1
E [Yn ] = 4 E[Yk ]
n k =0
n 1
4 ck 3
n k =0
Substitution.
Day 17
L9.24

Use substitution to
show that E[Yn] cn3
for some positive
large to handle the
initial conditions.
n 1
E [Yn ] = 4 E[Yk ]
n k =0
n 1
4 ck 3
n k =0
n 3
4
c
x dx
n 0
Integral method.
Day 17
L9.25

Use substitution to
show that E[Yn] cn3
for some positive
large to handle the
initial conditions.
n 1
E [Yn ] = 4 E[Yk ]
n k =0
n 1
4 ck 3
n k =0
n 3
4
c
x dx
n 0
4
4
c
n
=
n 4
Solve the integral.

Day 17
L9.26

Use substitution to
show that E[Yn] cn3
for some positive
large to handle the
initial conditions.
n 1
E [Yn ] = 4 E[Yk ]
n k =0
n 1
4 ck 3
n k =0
n 3
c
4
x dx
n 0
4
c
n
4
=
n 4
= cn3. Algebra.
Day 17
L9.27
The grand finale

Putting it all together, we have
2E[Xn] E[2Xn ]
Jensens inequality, since
f(x) = 2x is convex.
Day 17
L9.28
The grand finale

2E[Xn] E[2Xn ]
= E[Yn]
Definition.
Day 17
L9.29
The grand finale

2E[Xn] E[2Xn ]
= E[Yn]
cn3 .
What we just showed.
Day 17
L9.30
The grand finale

2E[Xn] E[2Xn ]
= E[Yn]
cn3 .
Taking the lg of both sides yields
E[Xn] 3 lg n +O(1) .
Day 17
L9.31
Post mortem
Q. Does the analysis have to be this hard?
Q. Why bother with analyzing exponential
height?
Q. Why not just develop the recurrence on
Xn = 1 + max{Xk1, Xnk}
directly?
Day 17
L9.32
Post mortem (continued)

A. The inequality
max{a, b} a + b .
provides a poor upper bound, since the RHS
approaches the LHS slowly as |a b| increases.
The bound
max{2a, 2b} 2a + 2b
allows the RHS to approach the LHS far more
quickly as |a b| increases. By using the
convexity of f(x) = 2x via Jensens inequality,
we can manipulate the sum of exponentials,
resulting in a tight analysis.
Day 17
L9.33
Thought exercises
See what happens when you try to do the
analysis on Xn directly.
Try to understand better why the proof
uses an exponential. Will a quadratic do?
See if you can find a simpler argument.
(This argument is a little simpler than the
one in the bookI hope its correct!)
Day 17
L9.34
6.046J/18.401J/SMA5503
Lecture 10
Prof. Erik Demaine
Balanced search trees

Balanced search tree: A search-tree data
structure for which a height of O(lg n) is
guaranteed when implementing a dynamic
set of n items.
Examples:
AVL trees
2-3 trees
2-3-4 trees
B-trees
Red-black trees
Day 18
L10.2
Red-black trees
This data structure requires an extra onebit color field in each node.
Red-black properties:
1. Every node is either red or black.
2. The root and leaves (NILs) are black.
3. If a node is red, then its parent is black.
4. All simple paths from any node x to a
descendant leaf have the same number
of black nodes = black-height(x).
Day 18
L10.3
Example of a red-black tree

77
33
NIL
18
18
NIL
10
10
88
11
11
NIL NIL NIL NIL
h=4
22
22
NIL
26
26
NIL
NIL
Day 18
L10.4

77
33
NIL
18
18
NIL
10
10
88
22
22
11
11
NIL
NIL NIL NIL NIL
26
26
NIL
NIL
1. Every node is either red or black.

Day 18
L10.5

77
33
NIL
18
18
NIL
10
10
88
22
22
11
11
NIL NIL NIL NIL
NIL
26
26
NIL
NIL
2. The root and leaves (NILs) are black.

Day 18
L10.6

77
33
NIL
18
18
NIL
10
10
88
22
22
11
11
NIL NIL NIL NIL
NIL
26
26
NIL
NIL
3. If a node is red, then its parent is black.

Day 18
L10.7

77 bh = 2
33
NIL
18
18 bh = 2
NIL
bh = 1 10
10
bh = 1
88
22
22
11
11
bh = 0 NIL NIL NIL NIL
NIL
26
26
NIL
NIL
4. All simple paths from any node x to a

descendant leaf have the same number of
black nodes = black-height(x).
Day 18
L10.8
Height of a red-black tree

Theorem. A red-black tree with n keys has height
h 2 lg(n + 1).
Proof. (The book uses induction. Read carefully.)
INTUITION:
Merge red nodes
into their black
parents.
Day 18
L10.9

h 2 lg(n + 1).
INTUITION:
Merge red nodes
into their black
parents.
Day 18
L10.10

h 2 lg(n + 1).
INTUITION:
Merge red nodes
into their black
parents.
Day 18
L10.11

h 2 lg(n + 1).
INTUITION:
Merge red nodes
into their black
parents.
Day 18
L10.12

h 2 lg(n + 1).
INTUITION:
Merge red nodes
into their black
parents.
Day 18
L10.13

h 2 lg(n + 1).
INTUITION:
Merge red nodes
h
into their black
parents.
This process produces a tree in which each node
has 2, 3, or 4 children.
The 2-3-4 tree has uniform depth h of leaves.
Day 18
L10.14
Proof (continued)
We have
h h/2, since
at most half
the leaves on any path
are red.
The number of leaves

in each tree is n + 1
n + 1 2h'
lg(n + 1) h' h/2
h 2 lg(n + 1).
Day 18
L10.15
Query operations
Corollary. The queries SEARCH, MIN,
MAX, SUCCESSOR, and PREDECESSOR
all run in O(lg n) time on a red-black
tree with n nodes.
Day 18
L10.16
Modifying operations
The operations INSERT and DELETE cause
modifications to the red-black tree:
the operation itself,
color changes,
restructuring the links of the tree via
rotations.
Day 18
L10.17
Rotations
RIGHT-ROTATE(B)
BB
LEFT-ROTATE(A)
AA
AA
BB
Rotations maintain the inorder ordering of keys:

a , b , c a A b B c.
A rotation can be performed in O(1) time.
Day 18
L10.18
Insertion into a red-black tree

IDEA: Insert x in tree. Color x red. Only redblack property 3 might be violated. Move the
violation up the tree by recoloring until it can
be fixed with rotations and recoloring.
Example:
77
33
18
18
10
10
88
22
22
11
11
26
26
Day 18
L10.19

Example:
33
Insert x =15.
Recolor, moving the
violation up the tree.
77
18
18
10
10
88
22
22
11
11
26
26
15
15
Day 18
L10.20

Example:
33
Insert x =15.
Recolor, moving the
RIGHT-ROTATE(18).
77
18
18
10
10
88
22
22
11
11
26
26
15
15
Day 18
L10.21

77
Example:
33
Insert x =15.
88
Recolor, moving the
RIGHT-ROTATE(18).
LEFT-ROTATE(7) and recolor.
10
10
18
18
11
11
22
22
15
15
26
26
Day 18
L10.22

Example:
77
Insert x =15.
Recolor, moving the 33
88
RIGHT-ROTATE(18).
LEFT-ROTATE(7) and recolor.
10
10
18
18
11
11
22
22
15
15
26
26
Day 18
L10.23
Pseudocode
RB-INSERT(T, x)
TREE-INSERT(T, x)
color[x] RED only RB property 3 can be violated
while x root[T] and color[p[x]] = RED
do if p[x] = left[p[p[x]]
then y right[p[p[x]]
y = aunt/uncle of x
if color[y] = RED
then Case 1
else if x = right[p[x]]
then Case 2 Case 2 falls into Case 3
Case 3
else then clause with left and right swapped
color[root[T]] BLACK
Day 18
L10.24
Graphical notation
Let
All
denote a subtree with a black root.

s have the same black-height.
Day 18
L10.25
Case 1
Recolor
CC
DD
AA
new x
DD
AA
BB
(Or, children of
A are swapped.)
CC
BB
Push Cs black onto

A and D, and recurse,
since Cs parent may
be red.
Day 18
L10.26
Case 2
CC
AA
LEFT-ROTATE(A)
CC
BB
BB
AA
Transform to Case 3.
Day 18
L10.27
Case 3
CC
BB
RIGHT-ROTATE(C)
y
AA
AA
BB
CC
Done! No more
violations of RB
property 3 are
possible.
Day 18
L10.28
Analysis
Go up the tree performing Case 1, which only
recolors nodes.
If Case 2 or Case 3 occurs, perform 1 or 2
rotations, and terminate.
Running time: O(lg n) with O(1) rotations.
RB-DELETE same asymptotic running time
and number of rotations as RB-INSERT (see
textbook).
Day 18
L10.29
6.046J/18.401J/SMA5503
Lecture 11
Prof. Erik Demaine
Dynamic order statistics

OS-SELECT(i, S): returns the i th smallest element
in the dynamic set S.
OS-RANK(x, S): returns the rank of x S in the
sorted order of Ss elements.
IDEA: Use a red-black tree for the set S, but keep
subtree sizes in the nodes.
Notation for nodes:
key
key
size
size
Day 20
L11.2
Example of an OS-tree
M
M
99
CC
55
PP
33
AA
11
FF
33
DD
11
NN
11
QQ
11
HH
11
size[x] = size[left[x]] + size[right[x]] + 1

Day 20
L11.3
Selection
Implementation trick: Use a sentinel
(dummy record) for NIL such that size[NIL] = 0.
OS-SELECT(x, i) ith smallest element in the
subtree rooted at x
k size[left[x]] + 1 k = rank(x)
if i = k then return x
if i < k
then return OS-SELECT(left[x], i )
else return OS-SELECT(right[x], i k )
(OS-RANK is in the textbook.)
Day 20
L11.4
Example
OS-SELECT(root, 5)
i=5
k=6
i=5
k=2
M
M
99
CC
55
PP
33
AA
11
FF
33
DD
11
i=3
k=2
HH
11
NN
11
QQ
11
i=1
k=1
Running time = O(h) = O(lg n) for red-black trees.

Day 20
L11.5
Data structure maintenance

Q. Why not keep the ranks themselves
in the nodes instead of subtree sizes?
A. They are hard to maintain when the
red-black tree is modified.
Modifying operations: INSERT and DELETE.
Strategy: Update subtree sizes when
inserting or deleting.
Day 20
L11.6
Example of insertion
INSERT(K)
M
M
10
910
9
CC
6565
PP
33
AA
11
FF
4343
DD
11
NN
11
QQ
11
HH
2121
KK
11
Day 20
L11.7
Handling rebalancing
Dont forget that RB-INSERT and RB-DELETE may
also need to modify the red-black tree in order to
maintain balance.
Recolorings: no effect on subtree sizes.
Rotations: fix up subtree sizes in O(1) time.
Example:
EE
16
16
CC
11
11
7
CC
16
16
4
EE
88
7
3
RB-INSERT and RB-DELETE still run in O(lg n) time.

Day 20
L11.8
Data-structure augmentation
Methodology: (e.g., order-statistics trees)
1. Choose an underlying data structure (redblack trees).
2. Determine additional information to be
stored in the data structure (subtree sizes).
3. Verify that this information can be
maintained for modifying operations (RBINSERT, RB-DELETE dont forget rotations).
4. Develop new dynamic-set operations that use
the information (OS-SELECT and OS-RANK).
These steps are guidelines, not rigid rules.
Day 20
L11.9
Interval trees
Goal: To maintain a dynamic set of intervals,
such as time intervals.
i = [7, 10]
low[i] = 7
5
4
10 = high[i]
11
17
15
19
18 22
23
Query: For a given query interval i, find an

interval in the set that overlaps i.
Day 20
L11.10
Following the methodology

1. Choose an underlying data structure.
Red-black tree keyed on low (left) endpoint.
2. Determine additional information to be
stored in the data structure.
Store in each node x the largest value m[x]
in the subtree rooted at x, as well as the
interval int[x] corresponding to the key.
int
int
m
m
Day 20
L11.11
Example interval tree

17,19
17,19
23
23
5,11
5,11
18
18
4,8
4,8
88
22,23
22,23
23
23
15,18
15,18
18
18
7,10
7,10
10
10
m[x] = max
high[int[x]]
m[left[x]]
m[right[x]]
Day 20
L11.12
Modifying operations
3. Verify that this information can be maintained
for modifying operations.
INSERT: Fix ms on the way down.
Rotations Fixup = O(1) time per rotation:
11,15
11,15
30
30
6,20
6,20
30
30
30
30
6,20
6,20
30
30
19
19
11,15
11,15
19
19
30
30
14
14
14
14
19
19
Total INSERT time = O(lg n); DELETE similar.

Day 20
L11.13
New operations
4. Develop new dynamic-set operations that use
the information.
INTERVAL-SEARCH(i)
x root
while x NIL and (low[i] > high[int[x]]
or low[int[x]] > high[i])
do i and int[x] dont overlap
if left[x] NIL and low[i] m[left[x]]
then x left[x]
else x right[x]
return x
Day 20
L11.14
Example 1: INTERVAL-SEARCH([14,16])
x
17,19
17,19
23
23
5,11
5,11
18
18
4,8
4,8
88
22,23
22,23
23
23
15,18
15,18
18
18
7,10
7,10
10
10
x root
[14,16] and [17,19] dont overlap
14 18 x left[x]
Day 20
L11.15
17,19
17,19
23
23
5,11
5,11
18
18
4,8
4,8
88
22,23
22,23
23
23
15,18
15,18
18
18
7,10
7,10
10
10

14 > 8 x right[x]
Day 20
L11.16
17,19
17,19
23
23
5,11
5,11
18
18
4,8
4,8
88
22,23
22,23
23
23
x
7,10
7,10
10
10
15,18
15,18
18
18
[14,16] and [15,18] overlap

return [15,18]
Day 20
L11.17
x
17,19
17,19
23
23
5,11
5,11
18
18
4,8
4,8
88
22,23
22,23
23
23
15,18
15,18
18
18
7,10
7,10
10
10
x root
12 18 x left[x]
Day 20
L11.18
17,19
17,19
23
23
5,11
5,11
18
18
4,8
4,8
88
22,23
22,23
23
23
15,18
15,18
18
18
7,10
7,10
10
10

12 > 8 x right[x]
Day 20
L11.19
17,19
17,19
23
23
5,11
5,11
18
18
4,8
4,8
88
22,23
22,23
23
23
x
7,10
7,10
10
10
15,18
15,18
18
18

12 > 10 x right[x]
Day 20
L11.20
17,19
17,19
23
23
5,11
5,11
18
18
4,8
4,8
88
22,23
22,23
23
23
15,18
15,18
18
18
7,10
7,10
10
10
x
x = NIL no interval that
overlaps [12,14] exists
Day 20
L11.21
Analysis
Time = O(h) = O(lg n), since INTERVAL-SEARCH
does constant work at each level as it follows a
simple path down the tree.
List all overlapping intervals:
Search, list, delete, repeat.
Insert them all again at the end.
Time = O(k lg n), where k is the total number of
overlapping intervals.
This is an output-sensitive bound.
Best algorithm to date: O(k + lg n).
Day 20
L11.22
Correctness
Theorem. Let L be the set of intervals in the
left subtree of node x, and let R be the set of
intervals in xs right subtree.
If the search goes right, then
{ i L : i overlaps i } = .
If the search goes left, then
{i L : i overlaps i } =
{i R : i overlaps i } = .
In other words, its always safe to take only 1
of the 2 children: well either find something,
or nothing was to be found.
Day 20
L11.23
Correctness proof
Proof. Suppose first that the search goes right.
If left[x] = NIL, then were done, since L = .
Otherwise, the code dictates that we must have
low[i] > m[left[x]]. The value m[left[x]]
corresponds to the right endpoint of some
interval j L, and no other interval in L can
have a larger right endpoint than high( j).
i
L
high( j) = m[left[x]]
low(i)
Therefore, {i L : i overlaps i } = .
Day 20
L11.24
Proof (continued)
Suppose that the search goes left, and assume that
{i L : i overlaps i } = .
Then, the code dictates that low[i] m[left[x]] =
high[ j] for some j L.
Since j L, it does not overlap i, and hence
high[i] < low[ j].
But, the binary-search-tree property implies that
for all i R, we have low[ j] low[i].
But then {i R : i overlaps i } = .
i
j
i
Day 20
L11.25
6.046J/18.401J/SMA5503
Lecture 12
Prof. Erik Demaine
Computational geometry
Algorithms for solving geometric problems
in 2D and higher.
Fundamental objects:
point
line segment
line
Basic structures:
point set
2001 by Erik D. Demaine
polygon
Day 21
L12.2
Computational geometry
Algorithms for solving geometric problems
in 2D and higher.
Fundamental objects:
point
line segment
line
Basic structures:
triangulation
convex hull
Day 21
L12.3
Orthogonal range searching

Input: n points in d dimensions
E.g., representing a database of n records
each with d numeric fields
Query: Axis-aligned box (in 2D, a rectangle)
Report on the points inside the box:
Are there any points?
How many are there?
List the points.
Day 21
L12.4
Orthogonal range searching

Input: n points in d dimensions
Query: Axis-aligned box (in 2D, a rectangle)
Report on the points inside the box
Goal: Preprocess points into a data structure
to support fast queries
Primary goal: Static data structure
In 1D, we will also obtain a
dynamic data structure
supporting insert and delete
Day 21
L12.5
1D range searching
In 1D, the query is an interval:
First solution using ideas we know:
Interval trees
Represent each point x by the interval [x, x].
Obtain a dynamic structure that can list
k answers in a query in O(k lg n) time.
Day 21
L12.6
1D range searching
Second solution using ideas we know:
Sort the points and store them in an array
Solve query by binary search on endpoints.
Obtain a static structure that can list
k answers in a query in O(k + lg n) time.
Goal: Obtain a dynamic structure that can list
k answers in a query in O(k + lg n) time.
Day 21
L12.7
1D range searching
New solution that extends to higher dimensions:
Balanced binary search tree
New organization principle:
Store points in the leaves of the tree.
Internal nodes store copies of the leaves
to satisfy binary search property:
Node x stores in key[x] the maximum
key of any leaf in the left subtree of x.
Day 21
L12.8
Example of a 1D range tree
17
17
11
66
88 12
12 14
14
43
43
26
35 41
41 42
42
26 35
59
61
59 61
Day 21
L12.9
Example of a 1D range tree

xx
17
17
88
42
42
11
14
14
66
11
66
12
12
88 12
12 14
14
35
35
17
17
26
26
43
43
41
41
26
35 41
41 42
42
26 35
>x
43
43
59
59
59
61
59 61
Day 21
L12.10
Example of a 1D range query

xx
17
17
88
42
42
11
14
14
66
11
66
12
12
88 12
12 14
14
35
35
17
17
26
26
>x
43
43
41
41
26
35 41
41 42
42
26 35
43
43
59
59
59
61
59 61
RANGE-QUERY([7, 41])
Day 21
L12.11
General 1D range query

root
split node
Day 21
L12.12
Pseudocode, part 1:
Find the split node
1D-RANGE-QUERY(T, [x1, x2])
w root[T]
while w is not a leaf and (x2 key[w] or key[w] < x1)
do if x2 key[w]
then w left[w]
else w right[w]
w is now the split node
[traverse left and right from w and report relevant subtrees]
Day 21
L12.13
Pseudocode, part 2: Traverse

left and right from split node
1D-RANGE-QUERY(T, [x1, x2])
[find the split node]
w is now the split node
if w is a leaf
then output the leaf w if x1 key[w] x2
else v left[w]
Left traversal
while v is not a leaf
do if x1 key[v]
then output the subtree rooted at right[v]
v left[v]
else v right[v]
output the leaf v if x1 key[v] x2
[symmetrically for right traversal]
Day 21
L12.14
Analysis of 1D-RANGE-QUERY
Query time: Answer to range query represented
by O(lg n) subtrees found in O(lg n) time.
Thus:
Can test for points in interval in O(lg n) time.
Can count points in interval in O(lg n) time
if we augment the tree with subtree sizes.
Can report the first k points in
interval in O(k + lg n) time.
Space: O(n)
Preprocessing time: O(n lg n)
Day 21
L12.15
2D range trees
Store a primary 1D range tree for all the points
based on x-coordinate.
Thus in O(lg n) time we can find O(lg n) subtrees
representing the points with proper x-coordinate.
How to restrict to points with proper y-coordinate?
Day 21
L12.16
2D range trees
Idea: In primary 1D range tree of x-coordinate,
every node stores a secondary 1D range tree
based on y-coordinate for all points in the subtree
of the node. Recursively search within each.
Day 21
L12.17
Analysis of 2D range trees

Query time: In O(lg2 n) = O((lg n)2) time, we can
represent answer to range query by O(lg2 n) subtrees.
Total cost for reporting k points: O(k + (lg n)2).
Space: The secondary trees at each level of the
primary tree together store a copy of the points.
Also, each point is present in each secondary
tree along the path from the leaf to the root.
Either way, we obtain that the space is O(n lg n).
Preprocessing time: O(n lg n)
Day 21
L12.18
d-dimensional range trees

Each node of the secondary y-structure stores
a tertiary z-structure representing the points
in the subtree rooted at the node, etc.
Query time: O(k + lgd n) to report k points.
Space: O(n lgd 1 n)
Preprocessing time: O(n lgd 1 n)
Best data structure to date:
Query time: O(k + lgd 1 n) to report k points.
Space: O(n (lg n / lg lg n)d 1)
Preprocessing time: O(n lgd 1 n)
Day 21
L12.19
Primitive operations:
Crossproduct
Given two vectors v1 = (x1, y1) and v2 = (x2, y2),
is their counterclockwise angle
convex (< 180),
v2
v1
v
reflex (> 180), or
v2
1
borderline (0 or 180)? convex

reflex
Crossproduct v1 v2 = x1 y2 y1 x2
= |v1| |v2| sin .
Thus, sign(v1 v2) = sign(sin ) > 0 if convex,
< 0 if reflex,
= 0 if borderline.
Day 21
L12.20
Orientation test
Given three points p1, p2, p3 are they
in clockwise (cw) order,
in counterclockwise (ccw) order, or
collinear?
(p2 p1) (p3 p1)
> 0 if ccw
< 0 if cw
p2
= 0 if collinear
p1
cw
p1
p3
p2
p1
collinear
p3
ccw
p3
p2
Day 21
L12.21
Sidedness test
Given three points p1, p2, p3 are they
in clockwise (cw) order,
in counterclockwise (ccw) order, or
collinear?
Let L be the oriented line from p1 to p2.
Equivalently, is the point p3
right of L,
p2
left of L, or
p1
cw
p1
on L?
p
p3
p2
p1
collinear
p3
ccw
p2
Day 21
L12.22
Line-segment intersection
Given n line segments, does any pair intersect?
Obvious algorithm: O(n2).
e
d
a
c
f
b
Day 21
L12.23
Sweep-line algorithm
Sweep a vertical line from left to right
(conceptually replacing x-coordinate with time).
Maintain dynamic set S of segments
that intersect the sweep line, ordered
(tentatively) by y-coordinate of intersection.
Order changes when
new segment is encountered, segment
existing segment finishes, or endpoints
two segments cross
Key event points are therefore segment endpoints.
Day 21
L12.24
d
a a
a c c
a b b b
e
d
c
b
d
c
b
e
e d
d b
b f
e
b
d
f
b
e e
d d
f f
e
d
a
c
f
b
Day 21
L12.25
Sweep-line algorithm
Process event points in order by sorting segment
endpoints by x-coordinate and looping through:
For a left endpoint of segment s:
Add segment s to dynamic set S.
Check for intersection between s
and its neighbors in S.
For a right endpoint of segment s:
Remove segment s from dynamic set S.
Check for intersection between
the neighbors of s in S.
Day 21
L12.26
Analysis
Use red-black tree to store dynamic set S.
Total running time: O(n lg n).
Day 21
L12.27
Correctness
Theorem: If there is an intersection,
the algorithm finds it.
Proof: Let X be the leftmost intersection point.
Assume for simplicity that
only two segments s1, s2 pass through X, and
no two points have the same x-coordinate.
At some point before we reach X,
s1 and s2 become consecutive in the order of S.
Either initially consecutive when s1 or s2 inserted,
or became consecutive when another deleted.
Day 21
L12.28
6.046J/18.401J/SMA5503
Lecture 13
Prof. Erik Demaine
Fixed-universe
successor problem
Goal: Maintain a dynamic subset S of size n
of the universe U = {0, 1, , u 1} of size u
subject to these operations:
INSERT(x U \ S): Add x to S.
DELETE(x S): Remove x from S.
SUCCESSOR(x U): Find the next element in S
larger than any element x of the universe U.
PREDECESSOR(x U): Find the previous
element in S smaller than x.
Day 23
L12.2
Solutions to fixed-universe
successor problem
Goal: Maintain a dynamic subset S of size n
of the universe U = {0, 1, , u 1} of size u
subject to INSERT, DELETE, SUCCESSOR, PREDECESSOR.
Balanced search trees can implement operations in
O(lg n) time, without fixed-universe assumption.
In 1975, Peter van Emde Boas solved this problem
in O(lg lg u) time per operation.
If u is only polynomial in n, that is, u = O(nc),
then O(lg lg n) time per operation-exponential speedup!
Day 23
L12.3
O(lg lg u)?!
Where could a bound of O(lg lg u) arise?
Binary search over O(lg u) things
T(u) = T( u ) + O(1)
T(lg u) = T((lg u)/2) + O(1)
= O(lg lg u)
Day 23
L12.4
(1) Starting point: Bit vector

Bit vector v stores, for each x U,
1 if x S
vx = 0 if x S
Example: u = 16; n = 4; S = {1, 9, 10, 15}.
0 1 0 0 0 0 0 0 0 1 1 0 0 0 0 1
0 1 2
3 4 5 6 7
8 9 10 11 12 13 14 15
Insert/Delete run in O(1) time.

Successor/Predecessor run in O(u) worst-case time.
Day 23
L12.5
(2) Split universe into widgets

Carve universe of size u into u widgets
W0, W1, , W u 1 each of size u .
Example: u = 16, u = 4 .
W0
W1
W2
W3
0 1 0 0
0 0 0 0 0 1 1 0
0 0 0 1
0 1 2
4 5 6 7
12 13 14 15
8 9 10 11
Day 23
L12.6

Carve universe of size u into u widgets
W0, W1, , W u 1 each of size u .
W0 represents 0, 1, , u 1 U;
W1 represents u , u + 1, , 2 u 1 U;
:
Wi represents i u , i u + 1, , (i + 1) u 1 U;
:
W u 1 represents u u , u u + 1 , , u 1 U.
Day 23
L12.7

x=9
Define high(x) 0 and low(x) 0

1 0 0 1
so that x = high(x) u + low(x).
That is, if we write x U in binary, high(x) low(x)
=2
=1
high(x) is the high-order half of the bits,
and low(x) is the low-order half of the bits.
For x U, high(x) is index of widget containing x
and low(x) is the index of x within that widget.
W0
W1
W2
W3
0 1 0 0
0 0 0 0 0 1 1 0
0 0 0 1
0 1 2
4 5 6 7
12 13 14 15
8 9 10 11
Day 23
L12.8

INSERT(x)
insert x into widget Whigh(x) at position low(x).
mark Whigh(x) as nonempty.
Running time T(n) = O(1).
Day 23
L12.9

SUCCESSOR(x)
look for successor of x within widget Whigh(x)
starting after position low(x).
if successor found
then return it
else find smallest i > high(x)
for which Wi is nonempty.
return smallest element in Wi
O( u )
O( u )
O( u )
Running time T(u) = O( u ).

Day 23
L12.10
Revelation
SUCCESSOR(x)
look for successor of x within widget Whigh(x)
starting after position low(x).
if successor found
then return it
else find smallest i > high(x)
for which Wi is nonempty.
return smallest element in Wi
recursive
successor
recursive
successor
recursive
successor
Day 23
L12.11
(3) Recursion
Represent universe by widget of size u.
Recursively split each widget W of size |W|
into W subwidgets sub[W][0], sub[W][1], ,
sub[W][ W 1 ] each of size W .
Store a summary widget summary[W] of size W
representing which subwidgets are nonempty.
W
summary[W] sub[W][0] sub[W][1]
W
W
sub[W][ W 1 ]
W
Day 23
L12.12
(3) Recursion
Define high(x) 0 and low(x) 0
so that x = high(x) W + low(x).
INSERT(x, W)
if sub[W][high(x)] is empty
then INSERT(high(x), summary[W])
INSERT(low(x), sub[W][high(x)])
Running time T(u) = 2 T( u ) + O(1)

T(lg u) = 2 T((lg u) / 2) + O(1)
= O(lg u) .
Day 23
L12.13
(3) Recursion
SUCCESSOR(x, W)
j SUCCESSOR(low(x), sub[W][high(x)])
if j <
then return high(x) W + j
else i SUCCESSOR(high(x), summary[W])
j SUCCESSOR( , sub[W][i])
T( u )
T( u )
T( u )
return i W + j

T(lg u) = 3 T((lg u) / 2) + O(1)
= O((lg u) lg 3) .
Day 23
L12.14
Improvements
Need to reduce INSERT and SUCCESSOR
down to 1 recursive call each.
1 call: T(u) = 1 T( u ) + O(1)
= O(lg lg n)
2 calls: T(u) = 2 T( u ) + O(1)
= O(lg n)
3 calls: T(u) = 3 T( u ) + O(1)
= O((lg u) lg 3)
Were closer to this goal than it may seem!
Day 23
L12.15
Recursive calls in successor

If x has a successor within sub[W][high(x)],
then there is only 1 recursive call to SUCCESSOR.
Otherwise, there are 3 recursive calls:
SUCCESSOR(low(x), sub[W][high(x)])
discovers that sub[W][high(x)] hasnt successor.
SUCCESSOR(high(x), summary[W])
finds next nonempty subwidget sub[W][i].
SUCCESSOR( , sub[W][i])
finds smallest element in subwidget sub[W][i].
Day 23
L12.16
Reducing recursive calls

in successor
If x has no successor within sub[W][high(x)],
there are 3 recursive calls:
SUCCESSOR(low(x), sub[W][high(x)])
discovers that sub[W][high(x)] hasnt successor.
Could be determined using the maximum
value in the subwidget sub[W][high(x)].
SUCCESSOR(high(x), summary[W])
finds next nonempty subwidget sub[W][i].
SUCCESSOR( , sub[W][i])
finds minimum element in subwidget sub[W][i].
Day 23
L12.17
(4) Improved successor

INSERT(x, W)
if sub[W][high(x)] is empty
then INSERT(high(x), summary[W])
if x < min[W] then min[W] x
new (augmentation)
if x > max[W] then max[W] x

T(lg u) = 2 T((lg u) / 2) + O(1)
= O(lg u) .
Day 23
L12.18
(4) Improved successor

SUCCESSOR(x, W)
if low(x) < max[sub[W][high(x)]]
then j SUCCESSOR(low(x), sub[W][high(x)])
return high(x) W + j
j min[sub[W][i]]
T( u )
T( u )
return i W + j

= O(lg lg u) .
Day 23
L12.19
Recursive calls in insert

If sub[W][high(x)] is already in summary[W],
then there is only 1 recursive call to INSERT.
Otherwise, there are 2 recursive calls:
INSERT(high(x), summary[W])
Idea:We know that sub[W][high(x)]) is empty.
Avoid second recursive call by specially
storing a widget containing just 1 element.
Specifically, do not store min recursively.
Day 23
L12.20
(5) Improved insert

INSERT(x, W)
if x < min[W] then exchange x min[W]
if sub[W][high(x)] is nonempty, that is,
min[sub[W][high(x)] NIL
then INSERT(low(x), sub[W][high(x)])
else min[sub[W][high(x)]] low(x)
INSERT(high(x), summary[W])
if x > max[W] then max[W] x

= O(lg lg u) .
Day 23
L12.21
(5) Improved insert

SUCCESSOR(x, W)
if x < min[W] then return min[W] new
if low(x) < max[sub[W][high(x)]]
then j SUCCESSOR(low(x), sub[W][high(x)])
return high(x) W + j
j min[sub[W][i]]
T( u )
T( u )
return i W + j

= O(lg lg u) .
Day 23
L12.22
Deletion
DELETE(x, W)
if min[W] = NIL or x < min[W] then return
if x = min[W]
then i min[summary[W]]
x i W + min[sub[W][i]]
min[W] x
DELETE(low(x), sub[W][high(x)])
if sub[W][high(x)] is now empty, that is,
min[sub[W][high(x)] = NIL
then DELETE(high(x), summary[W])
(in this case, the first recursive call was cheap)
Day 23
L12.23
6.046J/18.401J/SMA5503
Lecture 14
How large should a hash

table be?
Goal: Make the table as small as possible, but
large enough so that it wont overflow (or
otherwise become inefficient).
Problem: What if we dont know the proper size
in advance?
Solution: Dynamic tables.
IDEA: Whenever the table overflows, grow it
by allocating (via malloc or new) a new, larger
table. Move all items from the old table into the
new one, and free the storage for the old table.
Day 24
L14.2
Example of a dynamic table

1. INSERT
2. INSERT
overflow
Day 24
L14.3

1. INSERT
2. INSERT
overflow
11
Day 24
L14.4

1. INSERT
2. INSERT
11
2
Day 24
L14.5

1. INSERT
2. INSERT
3. INSERT
11
22
overflow
Day 24
L14.6

1. INSERT
2. INSERT
3. INSERT
1
2
overflow
Day 24
L14.7

1. INSERT
2. INSERT
3. INSERT
1
2
Day 24
L14.8

1.
2.
3.
4.
INSERT
INSERT
INSERT
INSERT
1
2
3
4
Day 24
L14.9

1.
2.
3.
4.
5.
INSERT
INSERT
INSERT
INSERT
INSERT
1
2
3
4
overflow
Day 24
L14.10

1.
2.
3.
4.
5.
INSERT
INSERT
INSERT
INSERT
INSERT
1
2
3
4
overflow
Day 24
L14.11

1.
2.
3.
4.
5.
INSERT
INSERT
INSERT
INSERT
INSERT
1
2
3
4
Day 24
L14.12

1.
2.
3.
4.
5.
6.
7.
INSERT
INSERT
INSERT
INSERT
INSERT
INSERT
INSERT
1
2
3
4
5
6
7
Day 24
L14.13
Worst-case analysis
Consider a sequence of n insertions. The
worst-case time to execute one insertion is
(n). Therefore, the worst-case time for n
insertions is n (n) = (n2).
WRONG! In fact, the worst-case cost for
n insertions is only (n) (n2).
Lets see why.
Day 24
L14.14
Tighter analysis
Let ci = the cost of the i th insertion
i if i 1 is an exact power of 2,
=
1 otherwise.
i
sizei
16 16
ci
Day 24
10
1
L14.15
Tighter analysis
Let ci = the cost of the i th insertion
i if i 1 is an exact power of 2,
=
1 otherwise.
i
sizei
16 16
1
1
1
2
1
4
1
8
ci
Day 24
10
1
L14.16
Tighter analysis (continued)

n
Cost of n insertions = ci
i =1
n+
lg( n 1)
2j
j =0
3n
= ( n ) .
Thus, the average cost of each dynamic-table
operation is (n)/n = (1).
Day 24
L14.17
Amortized analysis
An amortized analysis is any strategy for
analyzing a sequence of operations to
show that the average cost per operation is
small, even though a single operation
within the sequence might be expensive.
Even though were taking averages, however,
probability is not involved!
An amortized analysis guarantees the
average performance of each operation in
the worst case.
Day 24
L14.18
Types of amortized analyses

Three common amortization arguments:
the aggregate method,
the accounting method,
the potential method.
Weve just seen an aggregate analysis.
The aggregate method, though simple, lacks the
precision of the other two methods. In particular,
the accounting and potential methods allow a
specific amortized cost to be allocated to each
operation.
Day 24
L14.19
Accounting method
Charge i th operation a fictitious amortized cost
i, where $1 pays for 1 unit of work (i.e., time).
This fee is consumed to perform the operation.
Any amount not immediately consumed is stored
in the bank for use by subsequent operations.
The bank balance must not go negative! We
must ensure that n
n
ci ci
i =1
i =1
for all n.
Thus, the total amortized costs provide an upper
bound on the total true costs.
Day 24
L14.20
Accounting analysis of
dynamic tables
Charge an amortized cost of i = $3 for the i th
insertion.
$1 pays for the immediate insertion.
$2 is stored for later table doubling.
When the table doubles, $1 pays to move a
recent item, and $1 pays to move an old item.
Example:
$0
$0 $0
$0 $0
$0 $2
$2 $2
$2 $2 $2 overflow
$0 $0
Day 24
L14.21
dynamic tables
insertion.
Example:
overflow
$0
$0 $0
$0 $0
$0 $0
$0 $0
$0 $0
$0 $0
$0
$0 $0
Day 24
L14.22
dynamic tables
insertion.
Example:
$0
$0 $0
$0 $0
$0 $0
$0 $0
$0 $0
$0 $0
$0 $2 $2 $2
$0 $0
Day 24
L14.23
Accounting analysis
(continued)
Key invariant: Bank balance never drops below 0.
Thus, the sum of the amortized costs provides an
upper bound on the sum of the true costs.
i
sizei
16 16
ci
2* 3
banki
10
*Okay, so I lied. The first operation costs only $2, not $3.
Day 24
L14.24
Potential method
IDEA: View the bank account as the potential
energy ( la physics) of the dynamic set.
Framework:
Start with an initial data structure D0.
Operation i transforms Di1 to Di.
The cost of operation i is ci.
Define a potential function : {Di} R,
such that (D0 ) = 0 and (Di ) 0 for all i.
The amortized cost i with respect to is
defined to be i = ci + (Di) (Di1).
Day 24
L14.25
Understanding potentials
i = ci + (Di) (Di1)
potential difference i
If i > 0, then i > ci. Operation i stores

work in the data structure for later use.
If i < 0, then i < ci. The data structure
delivers up stored work to help pay for
operation i.
Day 24
L14.26
The amortized costs bound

the true costs
The total amortized cost of n operations is
n
i =1
i =1
ci = (ci + ( Di ) ( Di1 ))
Summing both sides.
Day 24
L14.27

the true costs
n
i =1
i =1
n
ci = (ci + ( Di ) ( Di1 ))
= ci + ( Dn ) ( D0 )
i =1
The series telescopes.
Day 24
L14.28

the true costs
n
i =1
i =1
n
ci = (ci + ( Di ) ( Di1 ))
= ci + ( Dn ) ( D0 )
i =1
n
ci
i =1
since (Dn) 0 and

(D0 ) = 0.
Day 24
L14.29
Potential analysis of table

doubling
Define the potential of the table after the ith
insertion by (Di) = 2i 2lg i. (Assume that
2lg 0 = 0.)
Note:
(D0 ) = 0,
(Di) 0 for all i.
Example:
= 26 23 = 4
$0
$0 $0
$0 $0
$0 $2
$2 $2
$2
$0 $0
accounting method)
Day 24
L14.30
Calculation of amortized costs

The amortized cost of the i th insertion is
i = ci + (Di) (Di1)
=
i + (2i 2lg i) (2(i 1) 2lg (i1))

if i 1 is an exact power of 2,
1 + (2i 2lg i) (2(i 1) 2lg (i1))
otherwise.
Day 24
L14.31
Calculation (Case 1)
Case 1: i 1 is an exact power of 2.
i = i + (2i 2lg i) (2(i 1) 2lg (i1))

= i + 2 (2lg i 2lg (i1))
= i + 2 (2(i 1) (i 1))
= i + 2 2i + 2 + i 1
=3
Day 24
L14.32
Calculation (Case 2)
Case 2: i 1 is not an exact power of 2.
i = 1 + (2i 2lg i) (2(i 1) 2lg (i1))

= 1 + 2 (2lg i 2lg (i1))
=3
Therefore, n insertions cost (n) in the worst case.

Exercise: Fix the bug in this analysis to show that
the amortized cost of the first insertion is only 2.
Day 24
L14.33
Conclusions
Amortized costs can provide a clean abstraction
of data-structure performance.
Any of the analysis methods can be used when
an amortized analysis is called for, but each
method has some situations where it is arguably
the simplest.
Different schemes may work for assigning
amortized costs in the accounting method, or
potentials in the potential method, sometimes
yielding radically different bounds.
Day 24
L14.34
6.046J/18.401J/SMA5503
Lecture 15
Dynamic programming
Design technique, like divide-and-conquer.
Example: Longest Common Subsequence (LCS)
Given two sequences x[1 . . m] and y[1 . . n], find
a longest subsequence common to them both.
a not the
x: A B
C
B
D A B
BCBA =
LCS(x, y)
y: B
D C
A B
A
functional notation,
but not a function
Day 26
L15.2
Brute-force LCS algorithm

Check every subsequence of x[1 . . m] to see
if it is also a subsequence of y[1 . . n].
Analysis
Checking = O(n) time per subsequence.
2m subsequences of x (each bit-vector of
length m determines a distinct subsequence
of x).
Worst-case running time = O(n2m)
= exponential time.
Day 26
L15.3
Towards a better algorithm

Simplification:
1. Look at the length of a longest-common
subsequence.
2. Extend the algorithm to find the LCS itself.
Notation: Denote the length of a sequence s
by | s |.
Strategy: Consider prefixes of x and y.
Define c[i, j] = | LCS(x[1 . . i], y[1 . . j]) |.
Then, c[m, n] = | LCS(x, y) |.
Day 26
L15.4
Recursive formulation
Theorem.
c[i1, j1] + 1
if x[i] = y[j],
c[i, j] = max{c[i1, j], c[i, j1]} otherwise.
Proof. Case x[i] = y[ j]:
x:
1
1
2
2
y:
L
j
Let z[1 . . k] = LCS(x[1 . . i], y[1 . . j]), where c[i, j]

= k. Then, z[k] = x[i], or else z could be extended.
Thus, z[1 . . k1] is CS of x[1 . . i1] and y[1 . . j1].
Day 26
L15.5
Proof (continued)
Claim: z[1 . . k1] = LCS(x[1 . . i1], y[1 . . j1]).
Suppose w is a longer CS of x[1 . . i1] and
y[1 . . j1], that is, | w | > k1. Then, cut and
paste: w || z[k] (w concatenated with z[k]) is a
common subsequence of x[1 . . i] and y[1 . . j]
with | w || z[k] | > k. Contradiction, proving the
claim.
Thus, c[i1, j1] = k1, which implies that c[i, j]
= c[i1, j1] + 1.
Other cases are similar.
Day 26
L15.6
Dynamic-programming
hallmark #1
Optimal substructure
An optimal solution to a problem
(instance) contains optimal
solutions to subproblems.
If z = LCS(x, y), then any prefix of z is
an LCS of a prefix of x and a prefix of y.
Day 26
L15.7
Recursive algorithm for LCS

LCS(x, y, i, j)
if x[i] = y[ j]
then c[i, j] LCS(x, y, i1, j1) + 1
else c[i, j] max{ LCS(x, y, i1, j),
LCS(x, y, i, j1)}
Worst-case: x[i] y[ j], in which case the
algorithm evaluates two subproblems, each
with only one parameter decremented.
Day 26
L15.8
Recursion tree
m = 3, n = 4:
3,4
3,4
2,4
2,4
1,4
1,4
3,3
3,3
same
subproblem
2,3
2,3
1,3
1,3
3,2
3,2
2,3
2,3
2,2
2,2
1,3
1,3
m+n
2,2
2,2
Height = m + n work potentially exponential.,

but were solving subproblems already solved!
Day 26
L15.9
Dynamic-programming
hallmark #2
Overlapping subproblems
A recursive solution contains a
small number of distinct
subproblems repeated many times.
The number of distinct LCS subproblems for
two strings of lengths m and n is only mn.
Day 26
L15.10
Memoization algorithm
Memoization: After computing a solution to a
subproblem, store it in a table. Subsequent calls
check the table to avoid redoing work.
LCS(x, y, i, j)
if c[i, j] = NIL
then if x[i] = y[j]
then c[i, j] LCS(x, y, i1, j1) + 1
else c[i, j] max{ LCS(x, y, i1, j),
LCS(x, y, i, j1)}
same
as
before
Time = (mn) = constant work per table entry.

Space = (mn).
Day 26
L15.11
Dynamic-programming
algorithm
IDEA:
Compute the
table bottom-up.
Time = (mn).
A B C B D
00 00 00 00 00 00
B 00 00 11 11 11 11
D 00 00 11 11 11 22
C 00 00 11
A 00 11 11
B 00 11 22
A 00 11 22
A B
00 00
11 11
22 22
22 22 22 22 22
22 22 22 33 33
22 33 33 33 44
22 33 33 44 44
Day 26
L15.12
Dynamic-programming
algorithm
IDEA:
Compute the
table bottom-up.
Time = (mn).
Reconstruct
LCS by tracing
backwards.
Space = (mn).
Exercise:
O(min{m, n}).
A B C B D
00 00 00 00 00 00
B 00 00 11 11 11 11
D 00 00 11 11 11 22
C 00 00 11
A 00 11 11
B 00 11 22
A 00 11 22
A B
00 00
11 11
22 22
22 22 22 22 22
22 22 22 33 33
22 33 33 33 44
22 33 33 44 44
Day 26
L15.13
6.046J/18.401J/SMA5503
Lecture 16
Graphs (review)
Definition. A directed graph (digraph)
G = (V, E) is an ordered pair consisting of
a set V of vertices (singular: vertex),
a set E V V of edges.
In an undirected graph G = (V, E), the edge
set E consists of unordered pairs of vertices.
In either case, we have | E | = O(V 2). Moreover,
if G is connected, then | E | | V | 1, which
implies that lg | E | = (lg V).
(Review CLRS, Appendix B.)
Day 27
L16.2
Adjacency-matrix
representation
The adjacency matrix of a graph G = (V, E), where
V = {1, 2, , n}, is the matrix A[1 . . n, 1 . . n]
given by
1 if (i, j) E,
A[i, j] =
0 if (i, j) E.
22
11
33
44
A 1 2 3 4
1 0 1 1 0
2 0 0 1 0
3 0 0 0 0
4 0 0 1 0
(V 2) storage
dense
representation.
Day 27
L16.3
Adjacency-list representation
An adjacency list of a vertex v V is the list Adj[v]
of vertices adjacent to v.
Adj[1] = {2, 3}
22
11
Adj[2] = {3}
Adj[3] = {}
Adj[4] = {3}
33
44
For undirected graphs, | Adj[v] | = degree(v).
For digraphs, | Adj[v] | = out-degree(v).
Handshaking Lemma: vV = 2 |E| for undirected
graphs adjacency lists use (V + E) storage
a sparse representation (for either type of graph).
Day 27
L16.4
Minimum spanning trees

Input: A connected, undirected graph G = (V, E)
with weight function w : E R.
For simplicity, assume that all edge weights are
distinct. (CLRS covers the general case.)
Output: A spanning tree T a tree that connects
all vertices of minimum weight:
w(T ) = w(u , v) .
(u ,v )T
Day 27
L16.5
Example of MST
6
12
9
5
14
8
3
15
10
Day 27
L16.6
MST T:
(Other edges of G
are not shown.)
u
T1
T2
v
Remove any edge (u, v) T. Then, T is partitioned

into two subtrees T1 and T2.
Theorem. The subtree T1 is an MST of G1 = (V1, E1),
the subgraph of G induced by the vertices of T1:
V1 = vertices of T1,
E1 = { (x, y) E : x, y V1 }.
Similarly for T2.
Day 27
L16.7
Proof of optimal substructure

Proof. Cut and paste:
w(T) = w(u, v) + w(T1) + w(T2).
If T1 were a lower-weight spanning tree than T1 for
G1, then T = {(u, v)} T1 T2 would be a
lower-weight spanning tree than T for G.
Do we also have overlapping subproblems?
Yes.
Great, then dynamic programming may work!
Yes, but MST exhibits another powerful property
which leads to an even more efficient algorithm.
Day 27
L16.8
Hallmark for greedy

algorithms
Greedy-choice property
A locally optimal choice
is globally optimal.
Theorem. Let T be the MST of G = (V, E),
and let A V. Suppose that (u, v) E is the
least-weight edge connecting A to V A.
Then, (u, v) T.
Day 27
L16.9
Proof of theorem
Proof. Suppose (u, v) T. Cut and paste.
T:
v
A
VA
u
(u, v) = least-weight edge
connecting A to V A
Day 27
L16.10
Proof of theorem
T:
v
A
VA
u
connecting A to V A
Consider the unique simple path from u to v in T.
Day 27
L16.11
Proof of theorem
T:
v
A
VA
u
connecting A to V A

Swap (u, v) with the first edge on this path that
connects a vertex in A to a vertex in V A.
Day 27
L16.12
Proof of theorem
T :
A
VA
v
u
connecting A to V A

Swap (u, v) with the first edge on this path that
connects a vertex in A to a vertex in V A.
A lighter-weight spanning tree than T results.
Day 27
L16.13
Prims algorithm
IDEA: Maintain V A as a priority queue Q. Key
each vertex in Q with the weight of the leastweight edge connecting it to a vertex in A.
QV
key[v] for all v V
key[s] 0 for some arbitrary s V
while Q
do u EXTRACT-MIN(Q)
for each v Adj[u]
do if v Q and w(u, v) < key[v]
then key[v] w(u, v)
DECREASE-KEY
[v] u
At the end, {(v, [v])} forms the MST.

Day 27
L16.14
Example of Prims algorithm

A
VA
14
8
3
12
9
7
15
00
10
Day 27
L16.15

A
VA
14
8
3
12
9
7
15
00
10
Day 27
L16.16

A
VA
14
77
7
8
3
12
00
10
10
9
15
15
15
10
Day 27
L16.17

A
VA
14
77
7
8
3
12
00
10
10
9
15
15
15
10
Day 27
L16.18

A
VA
12
12
55
14
77
7
8
3
12
00
10
10
9
15
99
15
15
10
Day 27
L16.19

A
VA
12
12
55
14
77
7
8
3
12
00
10
10
9
15
99
15
15
10
Day 27
L16.20

A
VA
66
55
14
14
14
77
7
8
3
12
00
88
9
15
99
15
15
10
Day 27
L16.21

A
VA
66
55
14
14
14
77
7
8
3
12
00
88
9
15
99
15
15
10
Day 27
L16.22

A
VA
66
55
14
14
14
77
7
8
3
12
00
88
9
15
99
15
15
10
Day 27
L16.23

A
VA
66
55
14
3
77
7
33
12
00
88
9
15
99
15
15
10
Day 27
L16.24

A
VA
66
55
14
3
77
7
33
12
00
88
9
15
99
15
15
10
Day 27
L16.25

A
VA
66
55
14
3
77
7
33
12
00
88
9
15
99
15
15
10
Day 27
L16.26

A
VA
66
55
14
3
77
7
33
12
00
88
9
15
99
15
15
10
Day 27
L16.27
Analysis of Prim
QV
(V)
key[v] for all v V
total
key[s] 0 for some arbitrary s V
while Q
do u EXTRACT-MIN(Q)
for each v Adj[u]
|V |
do if v Q and w(u, v) < key[v]
times degree(u)
times
then key[v] w(u, v)
[v] u
Handshaking Lemma (E) implicit DECREASE-KEYs.
Time = (V)TEXTRACT-MIN + (E)TDECREASE-KEY

Day 27
L16.28
Analysis of Prim (continued)

Q
TEXTRACT-MIN TDECREASE-KEY
Total
array
O(V)
O(1)
O(V2)
binary
heap
O(lg V)
O(lg V)
O(E lg V)
Fibonacci O(lg V)
heap
amortized
O(1)
O(E + V lg V)
amortized worst case
Day 27
L16.29
MST algorithms
Kruskals algorithm (see CLRS):
Uses the disjoint-set data structure (Lecture 20).
Running time = O(E lg V).
Best to date:
Karger, Klein, and Tarjan [1993].
Randomized algorithm.
O(V + E) expected time.
Day 27
L16.30
6.046J/18.401J/SMA5503
Lecture 17
Prof. Erik Demaine
Paths in graphs
Consider a digraph G = (V, E) with edge-weight
function w : E R. The weight of path p = v1
v2 L vk is defined to be
k 1
w( p ) = w(vi , vi +1 ) .
i =1
Example:
vv11
vv22
vv33
vv44
vv55
w(p) = 2
Day 29
L17.2
Shortest paths
A shortest path from u to v is a path of
minimum weight from u to v. The shortestpath weight from u to v is defined as
(u, v) = min{w(p) : p is a path from u to v}.
Note: (u, v) = if no path from u to v exists.
Day 29
L17.3
Theorem. A subpath of a shortest path is a
shortest path.
Proof. Cut and paste:
Day 29
L17.4
Triangle inequality
Theorem. For all u, v, x V, we have
(u, v) (u, x) + (x, v).
Proof.
(u, v)
uu
(u, x)
vv
(x, v)
xx
Day 29
L17.5
Well-definedness of shortest
paths
If a graph G contains a negative-weight cycle,
then some shortest paths may not exist.
Example:
<0
uu
vv
Day 29
L17.6
Single-source shortest paths

Problem. From a given source vertex s V, find
the shortest-path weights (s, v) for all v V.
If all edge weights w(u, v) are nonnegative, all
shortest-path weights must exist.
IDEA: Greedy.
1. Maintain a set S of vertices whose shortestpath distances from s are known.
2. At each step add to S the vertex v V S
whose distance estimate from s is minimal.
3. Update the distance estimates of vertices
adjacent to v.
Day 29
L17.7
Dijkstras algorithm
d[s] 0
for each v V {s}
do d[v]
S
QV
Q is a priority queue maintaining V S
while Q
do u EXTRACT-MIN(Q)
S S {u}
for each v Adj[u]
relaxation
do if d[v] > d[u] + w(u, v)
then d[v] d[u] + w(u, v)
step
Implicit DECREASE-KEY
Day 29
L17.8
Example of Dijkstras
algorithm
Graph with
nonnegative
edge weights:
10
AA
1 4
3
BB
CC
2
8
D
D
7 9
EE
Day 29
L17.9
algorithm
BB
Initialize:
10
0 AA
Q: A B C D E
0
1 4
3
CC
2
8
D
D
7 9
EE
S: {}
Day 29
L17.10
algorithm
A EXTRACT-MIN(Q):
10
0 AA
Q: A B C D E
0
BB
1 4
3
CC
2
8
D
D
7 9
EE
S: { A }
Day 29
L17.11
algorithm
Relax all edges leaving A:
10
0 AA
Q: A B C D E
0
10
10
BB
1 4
3
CC
3
2
8
D
D
7 9
EE
S: { A }
Day 29
L17.12
algorithm
C EXTRACT-MIN(Q):
10
0 AA
Q: A B C D E
0
10
10
BB
1 4
3
CC
3
2
8
D
D
7 9
EE
S: { A, C }
Day 29
L17.13
algorithm
Relax all edges leaving C:
10
0 AA
Q: A B C D E
0
10
7
11
7
BB
1 4
3
CC
3
2
8
11
D
D
7 9
EE
5
S: { A, C }
Day 29
L17.14
algorithm
E EXTRACT-MIN(Q):
10
0 AA
Q: A B C D E
0
10
7
11
7
BB
1 4
3
CC
3
2
8
11
D
D
7 9
EE
5
S: { A, C, E }
Day 29
L17.15
algorithm
Relax all edges leaving E:
10
0 AA
Q: A B C D E
0
10
7
7
11
11
7
BB
1 4
3
CC
3
2
8
11
D
D
7 9
EE
5
S: { A, C, E }
Day 29
L17.16
algorithm
B EXTRACT-MIN(Q):
10
0 AA
Q: A B C D E
0
10
7
7
11
11
7
BB
1 4
3
CC
3
2
8
11
D
D
7 9
EE
5
S: { A, C, E, B }
Day 29
L17.17
algorithm
Relax all edges leaving B:
10
0 AA
Q: A B C D E
0
10
7
7
11
11
9
7
BB
1 4
3
CC
3
9
D
D
2
8
7 9
EE
5
S: { A, C, E, B }
Day 29
L17.18
algorithm
D EXTRACT-MIN(Q):
10
0 AA
Q: A B C D E
0
10
7
7
11
11
9
7
BB
1 4
3
CC
3
2
8
9
D
D
7 9
EE
5
S: { A, C, E, B, D }
Day 29
L17.19
Correctness Part I
Lemma. Initializing d[s] 0 and d[v] for all
v V {s} establishes d[v] (s, v) for all v V,
and this invariant is maintained over any sequence
of relaxation steps.
Proof. Suppose not. Let v be the first vertex for
which d[v] < (s, v), and let u be the vertex that
caused d[v] to change: d[v] = d[u] + w(u, v). Then,
d[v] < (s, v)
supposition
(s, u) + (u, v) triangle inequality
(s,u) + w(u, v) sh. path specific path
d[u] + w(u, v)
v is first violation
Contradiction.
Day 29
L17.20
Correctness Part II
Theorem. Dijkstras algorithm terminates with
d[v] = (s, v) for all v V.
Proof. It suffices to show that d[v] = (s, v) for every
v V when v is added to S. Suppose u is the first
vertex added to S for which d[u] (s, u). Let y be the
first vertex in V S along a shortest path from s to u,
and let x be its predecessor:
uu
S, just before
adding u.
ss
xx
yy
Day 29
L17.21
Correctness Part II
(continued)
S
ss
uu
xx
yy
Since u is the first vertex violating the claimed invariant,

we have d[x] = (s, x). Since subpaths of shortest paths
are shortest paths, it follows that d[y] was set to (s, x) +
w(x, y) = (s, y) when (x, y) was relaxed just after x was
added to S. Consequently, we have d[y] = (s, y) (s, u)
d[u]. But, d[u] d[y] by our choice of u, and hence d[y]
= (s, y) = (s, u) = d[u]. Contradiction.
Day 29
L17.22
Analysis of Dijkstra
|V |
times
while Q
do u EXTRACT-MIN(Q)
S S {u}
for each v Adj[u]
degree(u)
times
Handshaking Lemma (E) implicit DECREASE-KEYs.

Note: Same formula as in the analysis of Prims
minimum spanning tree algorithm.
Day 29
L17.23
Analysis of Dijkstra
(continued)
Q
TEXTRACT-MIN TDECREASE-KEY
Total
array
O(V)
O(1)
O(V2)
binary
heap
O(lg V)
O(lg V)
O(E lg V)
Fibonacci O(lg V)
heap
amortized
O(1)
O(E + V lg V)
amortized worst case
Day 29
L17.24
Unweighted graphs
Suppose w(u, v) = 1 for all (u, v) E. Can the
code for Dijkstra be improved?
Use a simple FIFO queue instead of a priority
queue.
Breadth-first search
while Q
do u DEQUEUE(Q)
for each v Adj[u]
do if d[v] =
then d[v] d[u] + 1
ENQUEUE(Q, v)
Analysis: Time = O(V + E).

Day 29
L17.25
Example of breadth-first
search
aa
ff
hh
dd
bb
gg
ee
ii
cc
Q:
Day 29
L17.26
search
0
aa
ff
hh
dd
bb
gg
ee
ii
cc
0
Q: a
Day 29
L17.27
search
0
aa
ff
hh
dd
1
bb
gg
ee
ii
cc
1 1
Q: a b d
Day 29
L17.28
search
0
aa
ff
hh
dd
1
bb
gg
ee
cc
ii
2
1 2 2
Q: a b d c e
Day 29
L17.29
search
0
aa
ff
hh
dd
1
bb
gg
ee
cc
ii
2
2 2
Q: a b d c e
Day 29
L17.30
search
0
aa
ff
hh
dd
1
bb
gg
ee
cc
ii
2
2
Q: a b d c e
Day 29
L17.31
search
0
aa
dd
1
bb
cc
ff
1
3
hh
gg
ee
ii
3
3 3
Q: a b d c e g i
Day 29
L17.32
search
4
0
aa
dd
1
bb
cc
ff
1
3
hh
gg
ee
ii
3
3 4
Q: a b d c e g i f
Day 29
L17.33
search
0
aa
dd
1
bb
cc
ff
hh
gg
ee
ii
3
4 4
Q: a b d c e g i f h
Day 29
L17.34
search
0
aa
dd
1
bb
cc
ff
hh
gg
ee
ii
3
4
Day 29
L17.35
search
0
aa
dd
1
bb
cc
ff
hh
gg
ee
ii
Day 29
L17.36
search
0
aa
dd
1
bb
cc
ff
hh
gg
ee
ii
Day 29
L17.37
Correctness of BFS
while Q
do u DEQUEUE(Q)
for each v Adj[u]
do if d[v] =
then d[v] d[u] + 1
ENQUEUE(Q, v)
Key idea:
The FIFO Q in breadth-first search mimics
the priority queue Q in Dijkstra.
Invariant: v comes after u in Q implies that
d[v] = d[u] or d[v] = d[u] + 1.
Day 29
L17.38
6.046J/18.401J/SMA5503
Lecture 18
Prof. Erik Demaine
Negative-weight cycles
Recall: If a graph G = (V, E) contains a negativeweight cycle, then some shortest paths may not exist.
Example:
<0
uu
vv
Bellman-Ford algorithm: Finds all shortest-path

lengths from a source s V to all v V or
determines that a negative-weight cycle exists.
Day 31
L18.2
Bellman-Ford algorithm
d[s] 0
for each v V {s}
do d[v]
initialization
for i 1 to | V | 1
do for each edge (u, v) E
relaxation
step
for each edge (u, v) E

then report that a negative-weight cycle exists
At the end, d[v] = (s, v). Time = O(VE).

Day 31
L18.3
Example of Bellman-Ford
BB
AA
4
CC
1
5
EE
D
D
A B C D E
0
Day 31
L18.4
1
BB
AA
4
CC
1
5
EE
D
D
A B C D E
0
0 1
Day 31
L18.5
1
BB
AA
4
CC
4
1
5
EE
D
D
A
0
0
0
1
1
Day 31
L18.6
1
BB
AA
4
CC
4
2
1
5
EE
D
D
A
0
0
0
0
1
1
1
4
2
Day 31
L18.7
1
BB
AA
4
CC
2
1
5
EE
D
D
A
0
0
0
0
1
1
1
4
2
Day 31
L18.8
1
BB
AA
4
CC
2
1
5
EE
D
D
A
0
0
0
0
0
1
1
1
1
4
2
2
Day 31
L18.9
1
BB
AA
4
CC
2
1
5
EE
D
D
1
A
0
0
0
0
0
0
1
1
1
1
1
4
2
2
2
Day 31
1
1
L18.10
1
BB
AA
4
CC
2
1
5
EE
D
D
2
1
A
0
0
0
0
0
0
0
1
1
1
1
1
1
4
2
2
2
2
1
2
Day 31
1
1
1
L18.11
1
BB
AA
4
EE
3
CC 5 D
D
2
1
2
Note: Values decrease
monotonically.
A
0
0
0
0
0
0
0
1
1
1
1
1
1
4
2
2
2
2
1
2
Day 31
1
1
1
L18.12
Correctness
Theorem. If G = (V, E) contains no negativeweight cycles, then after the Bellman-Ford
algorithm executes, d[v] = (s, v) for all v V.
Proof. Let v V be any vertex, and consider a shortest
path p from s to v with the minimum number of edges.
s
p: vv0
0
vv11
vv22
vv33
vvkk
Since p is a shortest path, we have

(s, vi) = (s, vi1) + w(vi1, vi) .
Day 31
L18.13
Correctness (continued)
s
p: vv0
0
vv11
vv22
vv33
v
vvkk
Initially, d[v0] = 0 = (s, v0), and d[s] is unchanged by

subsequent relaxations (because of the lemma from
Lecture 17 that d[v] (s, v)).
After 1 pass through E, we have d[v1] = (s, v1).
After 2 passes through E, we have d[v2] = (s, v2).
M
After k passes through E, we have d[vk] = (s, vk).
Since G contains no negative-weight cycles, p is simple.
Longest simple path has |V| 1 edges.
Day 31
L18.14
Detection of negative-weight
cycles
Corollary. If a value d[v] fails to converge after
|V| 1 passes, there exists a negative-weight
cycle in G reachable from s.
Day 31
L18.15
DAG shortest paths

If the graph is a directed acyclic graph (DAG), we first
topologically sort the vertices.
Determine f : V {1, 2, , | V |} such that (u, v) E
f (u) < f (v).
O(V + E) time using depth-first search.
11
s 22
44
33
55
77
66
88
99
Walk through the vertices u V in this order, relaxing

the edges in Adj[u], thereby obtaining the shortest paths
from s in a total of O(V + E) time.
Day 31
L18.16
Linear programming
Let A be an mn matrix, b be an m-vector, and c
be an n-vector. Find an n-vector x that maximizes
cTx subject to Ax b, or determine that no such
solution exists.
n
m
.
A
maximizing
x b
cT
x
Day 31
L18.17
Linear-programming
algorithms
Algorithms for the general problem
Simplex methods practical, but worst-case
exponential time.
Ellipsoid algorithm polynomial time, but
slow in practice.
Interior-point methods polynomial time and
competes with simplex.
Feasibility problem: No optimization criterion.
Just find x such that Ax b.
In general, just as hard as ordinary LP.
Day 31
L18.18
Solving a system of difference

constraints
Linear programming where each row of A contains
exactly one 1, one 1, and the rest 0s.
Example:
Solution:
x1 = 3
x1 x2 3
x2 = 0
xj xi wij
x2 x3 2
x3 = 2
x1 x3 2
Constraint graph:
xj xi wij
vvii
wij
vvjj
(The A
matrix has
dimensions
|E | |V |.)
Day 31
L18.19
Unsatisfiable constraints
Theorem. If the constraint graph contains
a negative-weight cycle, then the system of
differences is unsatisfiable.
Proof. Suppose that the negative-weight cycle is
v1 v2 L vk v1. Then, we have
x2 x1
x3 x2
w12
w23
M
xk xk1 wk1, k
x1 xk wk1
0
weight of cycle
<0
Therefore, no
values for the xi
can satisfy the
constraints.
Day 31
L18.20
Satisfying the constraints

Theorem. Suppose no negative-weight cycle
exists in the constraint graph. Then, the
constraints are satisfiable.
Proof. Add a new vertex s to V with a 0-weight edge

to each vertex vi V.
vv11
vv44
vv77
vv99
vv33
Note:
No negative-weight
cycles introduced
shortest paths exist.
Day 31
L18.21
Proof (continued)
Claim: The assignment xi = (s, vi) solves the constraints.
Consider any constraint xj xi wij, and consider the
shortest paths from s to vj and vi:
ss
(s, vi)
(s, vj)
vvii
wij
vvjj
The triangle inequality gives us (s,vj) (s, vi) + wij.

Since xi = (s, vi) and xj = (s, vj), the constraint xj xi
wij is satisfied.
Day 31
L18.22
Bellman-Ford and linear

programming
Corollary. The Bellman-Ford algorithm can
solve a system of m difference constraints on n
variables in O(m n) time.
Single-source shortest paths is a simple LP
problem.
In fact, Bellman-Ford maximizes x1 + x2 + L + xn
subject to the constraints xj xi wij and xi 0
(exercise).
Bellman-Ford also minimizes maxi{xi} mini{xi}
(exercise).
Day 31
L18.23
Application to VLSI layout

compaction
Integrated
-circuit
features:
minimum separation
Problem: Compact (in one dimension) the
space between the features of a VLSI layout
without bringing any features too close together.
Day 31
L18.24
VLSI layout compaction

d1
11
x1
x2
x2 x1 d 1 +
Bellman-Ford minimizes maxi{xi} mini{xi},
which compacts the layout in the x-dimension.
Constraint:
Day 31
L18.25
6.046J/18.401J/SMA5503
Lecture 19
Prof. Erik Demaine
Shortest paths
Single-source shortest paths
Nonnegative edge weights
Dijkstras algorithm: O(E + V lg V)
General
Bellman-Ford: O(VE)
DAG
One pass of Bellman-Ford: O(V + E)
All-pairs shortest paths

Nonnegative edge weights
Dijkstras algorithm |V| times: O(VE + V 2 lg V)
General
Three algorithms today.
Day 32
L19.2
All-pairs shortest paths

Input: Digraph G = (V, E), where |V | = n, with
edge-weight function w : E R.
Output: n n matrix of shortest-path lengths
(i, j) for all i, j V.
IDEA #1:
Run Bellman-Ford once from each vertex.
Time = O(V 2E).
Dense graph O(V 4) time.
Good first try!
Day 32
L19.3
Dynamic programming
Consider the n n adjacency matrix A = (aij)
of the digraph, and define
dij(m) = weight of a shortest path from
i to j that uses at most m edges.
Claim: We have
0 if i = j,
(0)
dij =
if i j;
and for m = 1, 2, , n 1,
dij(m) = mink{dik(m1) + akj }.
Day 32
L19.4
Proof of claim
ks
dij(m) = mink{dik(m1) + akj }
es
g
d
1e
ii
Relaxation!
s
e
g
d
e
1
m
m
1
edg
es
for k 1 to n
do if dij > dik + akj
then dij dik + akj
jj
M
m 1 edges
Note: No negative-weight cycles implies

(i, j) = dij (n1) = dij (n) = dij (n+1) = L
Day 32
L19.5
Compute C = A B, where C, A, and B are n n
matrices:
n
cij = aik bkj .
k =1
Time = (n3) using the standard algorithm.

What if we map + min and +?
cij = mink {aik + bkj}.
Thus, D(m) = D(m1) A.
Identity matrix = I =
0
0
0
0
= D0 = (dij(0)).
Day 32
L19.6
(continued)
The (min, +) multiplication is associative, and
with the real numbers, it forms an algebraic
structure called a closed semiring.
Consequently, we can compute
D(1) = D(0) A = A1
D(2) = D(1) A = A2
M
M
D(n1) = D(n2) A = An1 ,
yielding D(n1) = ((i, j)).
Time = (nn3) = (n4). No better than n B-F.
Day 32
L19.7
Improved matrix
multiplication algorithm
Repeated squaring: A2k = Ak Ak.
lg(n1)
2
4
2
.
Compute A , A , , A
O(lg n) squarings
Note: An1 = An = An+1 = L.
Time = (n3 lg n).
To detect negative-weight cycles, check the
diagonal for negative values in O(n) additional
time.
Day 32
L19.8
Floyd-Warshall algorithm
Also dynamic programming, but faster!
Define cij(k) = weight of a shortest path from i
to j with intermediate vertices
belonging to the set {1, 2, , k}.
ii
kk
kk
kk
kk
jj
Thus, (i, j) = cij(n). Also, cij(0) = aij .

Day 32
L19.9
Floyd-Warshall recurrence
cij(k) = mink {cij(k1), cik(k1) + ckj(k1)}
cik
ii
(k1)
cij(k1)
ckj(k1)
jj
intermediate vertices in {1, 2, , k}
Day 32
L19.10
Pseudocode for FloydWarshall

for k 1 to n
do for i 1 to n
do for j 1 to n
do if cij > cik + ckj
then cij cik + ckj
relaxation
Notes:
Okay to omit superscripts, since extra relaxations
cant hurt.
Runs in (n3) time.
Simple to code.
Efficient in practice.
Day 32
L19.11
Transitive closure of a
directed graph
Compute tij =
1 if there exists a path from i to j,

0 otherwise.
IDEA: Use Floyd-Warshall, but with (, ) instead

of (min, +):
tij(k) = tij(k1) (tik(k1) tkj(k1)).

Time = (n3).
Day 32
L19.12
Graph reweighting
Theorem. Given a label h(v) for each v V, reweight
each edge (u, v) E by
(u, v) = w(u, v) + h(u) h(v).
Then, all paths between the same two vertices are
reweighted by the same amount.
Proof. Let p = v1 v2 L vk be a path in the graph.
k 1
Then, we have w ( p ) = w ( vi ,vi +1 )

=
=
i =1
k 1
( w(vi ,vi +1 )+ h (vi )h (vi +1 ) )
i =1
k 1
w(vi ,vi +1 ) + h (v1 ) h (vk )
i =1
= w( p ) + h ( v1 ) h ( v k ) .
Day 32
L19.13
Johnsons algorithm
1. Find a vertex labeling h such that (u, v) 0 for all
(u, v) E by using Bellman-Ford to solve the
difference constraints
h(v) h(u) w(u, v),
or determine that a negative-weight cycle exists.
Time = O(V E).
2. Run Dijkstras algorithm from each vertex using .
Time = O(V E + V 2 lg V).
3. Reweight each shortest-path length (p) to produce
the shortest-path lengths w(p) of the original graph.
Time = O(V 2).
Total time = O(V E + V 2 lg V).

Day 32
L19.14
6.046J/18.401J/SMA5503
Lecture 20
Prof. Erik Demaine
Disjoint-set data structure

(Union-Find)
Problem: Maintain a dynamic collection of
pairwise-disjoint sets S = {S1, S2, , Sr}.
Each set Si has one element distinguished as the
representative element, rep[Si].
Must support 3 operations:
MAKE-SET(x): adds new set {x} to S
with rep[{x}] = x (for any x Si for all i).
UNION(x, y): replaces sets Sx, Sy with Sx Sy
in S for any x, y in distinct sets Sx, Sy .
FIND-SET(x): returns representative rep[Sx]
of set Sx containing element x.
Day 33
L20.2
Simple linked-list solution

Store each set Si = {x1, x2, , xk} as an (unordered)
doubly linked list. Define representative element
rep[Si] to be the front of the list, x1.
Si :
x1
rep[Si]
x2
xk
MAKE-SET(x) initializes x as a lone node. (1)

FIND-SET(x) walks left in the list containing x
(n)
until it reaches the front of the list.
UNION(x, y) concatenates the lists containing
x and y, leaving rep. as FIND-SET[x].
(n)
Day 33
L20.3
Simple balanced-tree solution

Store each set Si = {x1, x2, , xk} as a balanced tree
(ignoring keys). Define representative element
rep[Si] to be the root of the tree.
MAKE-SET(x) initializes x
(1)
as a lone node.
FIND-SET(x) walks up the
tree containing x until it
reaches the root.
(lg n)
UNION(x, y) concatenates
the trees containing x and y,
(lg n)
changing rep.
Si = {x1, x2, x3, x4, x5}

rep[Si] x1
x4
x2
x3
x5
Day 33
L20.4
Plan of attack
We will build a simple disjoint-union data structure
that, in an amortized sense, performs significantly
better than (lg n) per op., even better than
(lg lg n), (lg lg lg n), etc., but not quite (1).
To reach this goal, we will introduce two key tricks.
Each trick converts a trivial (n) solution into a
simple (lg n) amortized solution. Together, the
two tricks yield a much better solution.
First trick arises in an augmented linked list.
Second trick arises in a tree structure.
Day 33
L20.5
Augmented linked-list solution

Store set Si = {x1, x2, , xk} as unordered doubly
linked list. Define rep[Si] to be front of list, x1.
Each element xj also stores pointer rep[xj] to rep[Si].
rep
Si :
x1
rep[Si]
x2
xk
FIND-SET(x) returns rep[x].

(1)
UNION(x, y) concatenates the lists containing
x and y, and updates the rep pointers for
(n)
all elements in the list containing y.
Day 33
L20.6
Example of
augmented linked-list solution
Each element xj stores pointer rep[xj] to rep[Si].
UNION(x, y)
concatenates the lists containing x and y, and
updates the rep pointers for all elements in the
list containing y.
rep
Sx :
x1
rep[Sx]
x2
rep
Sy :
y1
rep[Sy]
y2
y3
Day 33
L20.7
Example of
UNION(x, y)
list containing y.
rep
Sx Sy :
x1
rep[Sx]
x2
rep
y1
rep[Sy]
y2
y3
Day 33
L20.8
Example of
UNION(x, y)
list containing y.
rep
Sx Sy :
x1
rep[Sx Sy]
x2
y1
y2
y3
Day 33
L20.9
Alternative concatenation
UNION(x, y) could instead
concatenate the lists containing y and x, and
update the rep pointers for all elements in the
list containing x.
rep
rep
Sy :
y1
rep[Sy]
y2
Sx :
x1
rep[Sx]
x2
y3
Day 33
L20.10
list containing x.
rep
Sx Sy :
y1
rep[Sy]
x1
rep[Sx]
rep
y2
x2
y3
Day 33
L20.11
list containing x.
rep
Sx Sy :
y1
rep[Sx Sy]
x1
rep
y2
x2
y3
Day 33
L20.12
Trick 1: Smaller into larger

To save work, concatenate smaller list onto the end
of the larger list. Cost = (length of smaller list).
Augment list to store its weight (# elements).
Let n denote the overall number of elements
(equivalently, the number of MAKE-SET operations).
Let m denote the total number of operations.
Let f denote the number of FIND-SET operations.
Theorem: Cost of all UNIONs is O(n lg n).
Corollary: Total cost is O(m + n lg n).
Day 33
L20.13
Analysis of Trick 1
To save work, concatenate smaller list onto the end
of the larger list. Cost = (1 + length of smaller list).
Theorem: Total cost of UNIONs is O(n lg n).
Proof. Monitor an element x and set Sx containing it.
After initial MAKE-SET(x), weight[Sx] = 1. Each
time Sx is united with set Sy, weight[Sy] weight[Sx],
pay 1 to update rep[x], and weight[Sx] at least
doubles (increasing by weight[Sy]). Each time Sy is
united with smaller set Sy, pay nothing, and
weight[Sx] only increases. Thus pay lg n for x.
Day 33
L20.14
Representing sets as trees

Store each set Si = {x1, x2, , xk} as an unordered,
potentially unbalanced, not necessarily binary tree,
storing only parent pointers. rep[Si] is the tree root.
MAKE-SET(x) initializes x
Si = {x1, x2, x3, x4, x5 , x6}
(1)
as a lone node.
rep[Si] x1
FIND-SET(x) walks up the
tree containing x until it
x4
x3
reaches the root. (depth[x])
UNION(x, y) concatenates
the trees containing x and y
x2 x5 x6
Day 33
L20.15
Trick 1 adapted to trees

UNION(x, y) can use a simple concatenation strategy:
Make root FIND-SET(y) a child of root FIND-SET(x).
FIND-SET(y) = FIND-SET(x).
x1
We can adapt Trick 1
to this context also:
x
x
y1
4
3
Merge tree with smaller
weight into tree with
x2 x5 x6
y4 y3
larger weight.
Height of tree increases only when its size y
2
doubles, so height is logarithmic in weight.
Thus total cost is O(m + f lg n).
Day 33
y5
L20.16
Trick 2: Path compression

When we execute a FIND-SET operation and walk
up a path p to the root, we know the representative
for all the nodes on path p.
x1
Path compression makes
x4
x3 y1
all of those nodes direct
children of the root.
x2 x5 x6
y4 y3
Cost of FIND-SET(x)
is still (depth[x]).
FIND-SET(y2) y2 y5
Day 33
L20.17

x1
x4
x3 y1
x2 x5 x6
y4 y3
Cost of FIND-SET(x)
FIND-SET(y2) y2 y5
Day 33
L20.18

x1
x4
x3 y1 y2 y3
x2 x5 x6 y4
y5
Cost of FIND-SET(x)
FIND-SET(y2)
Day 33
L20.19
Analysis of Trick 2 alone

Theorem: Total cost of FIND-SETs is O(m lg n).
Proof: Amortization by potential function.
The weight of a node x is # nodes in its subtree.
Define (x1, , xn) = i lg weight[xi].
UNION(xi, xj) increases potential of root FIND-SET(xi)
by at most lg weight[root FIND-SET(xj)] lg n.
Each step down p c made by FIND-SET(xi),
except the first, moves cs subtree out of ps subtree.
Thus if weight[c] weight[p], decreases by 1,
paying for the step down. There can be at most lg n
steps p c for which weight[c] < weight[p].
Day 33
L20.20
Analysis of Trick 2 alone

Theorem: If all UNION operations occur before
all FIND-SET operations, then total cost is O(m).
Proof: If a FIND-SET operation traverses a path
with k nodes, costing O(k) time, then k 2 nodes
are made new children of the root. This change
can happen only once for each of the n elements,
so the total cost of FIND-SET is O(f + n).
Day 33
L20.21
Ackermanns function A
j + 1 if k = 0,
Define Ak ( j ) = ( j +1)
Ak 1 ( j ) if k 1. iterate j+1 times
A0(1) = 2
A0(j) = j + 1
A1(1) = 3
A1(j) ~ 2 j
A2(1) = 7
A2(j) ~ 2j 2j > 2j
j
A3(1) = 2047
2
.
2
..
.2
2
A3(j) > 2
2
A4(j) is a lot bigger. A4(1) > 2
..
2047
2048
Define (n) = min {k : Ak(1) n} 4 for practical n.

Day 33
L20.22
Analysis of Tricks 1 + 2
Theorem: In general, total cost is O(m (n)).
(long, tricky proof see Section 21.4 of CLRS)
Day 33
L20.23
Application:
Dynamic connectivity
Suppose a graph is given to us incrementally by
ADD-VERTEX(v)
ADD-EDGE(u, v)
and we want to support connectivity queries:
CONNECTED(u, v):
Are u and v in the same connected component?
For example, we want to maintain a spanning forest,
so we check whether each new edge connects a
previously disconnected pair of vertices.
Day 33
L20.24
Application:
Dynamic connectivity
Sets of vertices represent connected components.
Suppose a graph is given to us incrementally by
ADD-VERTEX(v) MAKE-SET(v)
ADD-EDGE(u, v) if not CONNECTED(u, v)
then UNION(v, w)
and we want to support connectivity queries:
CONNECTED(u, v): FIND-SET(u) = FIND-SET(v)
Are u and v in the same connected component?
For example, we want to maintain a spanning forest,
so we check whether each new edge connects a
previously disconnected pair of vertices.
Day 33
L20.25
6.046J/18.401J/SMA5503
Lecture 21
Take-home quiz
No notes (except this one).
Day 35
L21.2
6.046J/18.401J/SMA5503
Lecture 22
Flow networks
Definition. A flow network is a directed graph
G = (V, E) with two distinguished vertices: a
source s and a sink t. Each edge (u, v) E has
a nonnegative capacity c(u, v). If (u, v) E,
then c(u, v) = 0.
Example:
2
3
3
1
ss
2
3
tt
2
2
Day 38
L22.2
Flow networks
Definition. A positive flow on G is a function
p : V V R satisfying the following:
Capacity constraint: For all u, v V,
0 p(u, v) c(u, v).
Flow conservation: For all u V {s, t},
p(u, v) p(v, u ) = 0 .
vV
vV
The value of a flow is the net flow out of the

source:
p ( s , v ) p (v, s ) .
vV
vV
Day 38
L22.3
A flow on a network
positive
flow
capacity
2:2
2:3
1:3
0:1
ss
2:2
1:3 1:1 2:3
2:3
tt
1:2
1:2
Flow conservation (like Kirchoffs current law):

Flow into u is 2 + 1 = 3.
Flow out of u is 0 + 1 + 2 = 3.
The value of this flow is 1 0 + 2 = 3.
Day 38
L22.4
The maximum-flow problem

Maximum-flow problem: Given a flow network
G, find a flow of maximum value on G.
2:2
2:3
2:3
0:1
ss
2:2
0:3 1:1 2:3

3:3
tt
1:2
2:2
The value of the maximum flow is 4.

Day 38
L22.5
Flow cancellation
Without loss of generality, positive flow goes
either from u to v, or from v to u, but not both.
vv
2:3
vv
1:2
uu
1:3
0:2
uu
Net flow from

u to v in both
cases is 1.
The capacity constraint and flow conservation

are preserved by this transformation.
INTUITION: View flow as a rate, not a quantity.
Day 38
L22.6
A notational simplification
IDEA: Work with the net flow between two
vertices, rather than with the positive flow.
Definition. A (net) flow on G is a function
f : V V R satisfying the following:
Capacity constraint: For all u, v V,
f (u, v) c(u, v).
Flow conservation: For all u V {s, t},
summation
f (u, v) = 0. One
instead of two.
vV
Skew symmetry: For all u, v V,
f (u, v) = f (v, u).
Day 38
L22.7
Equivalence of definitions
Theorem. The two definitions are equivalent.
Proof. () Let f (u, v) = p(u, v) p(v, u).
Capacity constraint: Since p(u, v) c(u, v) and
p(v, u) 0, we have f (u, v) c(u, v).
Flow conservation:
( p(u, v) p(v, u ) )
f (u , v) =
vV
= p (u , v) p (v, u )
vV
vV
vV
Skew symmetry:
f (u, v) = p(u, v) p(v, u)
= (p(v, u) p(u, v))
= f (v, u).
Day 38
L22.8
Proof (continued)
() Let
p(u, v) =
f (u, v) if f(u, v) > 0,

0
if f(u, v) 0.
Capacity constraint: By definition, p(u, v) 0. Since f

(u, v) c(u, v), it follows that p(u, v) c(u, v).
Flow conservation: If f (u, v) > 0, then p(u, v) p(v, u)
= f (u, v). If f (u, v) 0, then p(u, v) p(v, u) = f (v, u)
= f (u, v) by skew symmetry. Therefore,
p(u, v) p(v, u) = f (u, v) .

vV
vV
vV
Day 38
L22.9
Notation
Definition. The value of a flow f, denoted by | f |,
is given by
f = f ( s, v )
vV
= f ( s, V ) .
Implicit summation notation: A set used in
an arithmetic formula represents a sum over
the elements of the set.
Example flow conservation:
f (u, V) = 0 for all u V {s, t}.
Day 38
L22.10
Simple properties of flow

Lemma.
f (X, X) = 0,
f (X, Y) = f (Y, X),
f (XY, Z) = f (X, Z) + f (Y, Z) if XY = .
Theorem. | f | = f (V, t).
Proof.
|f| =
=
=
=
=
f (s, V)
f (V, V) f (Vs, V)
Omit braces.
f (V, Vs)
f (V, t) + f (V, Vst)
f (V, t).
Day 38
L22.11
Flow into the sink

2:2
2:3
2:3
0:1
ss
0:3 1:1 1:3
2:2
3:3
| f | = f (s, V) = 4
tt
0:2
2:2
f (V, t) = 4
Day 38
L22.12
Cuts
Definition. A cut (S, T) of a flow network G =
(V, E) is a partition of V such that s S and t T.
If f is a flow on G, then the flow across the cut is
f (S, T).
2:2
2:3
2:3
0:1
ss
2:2
0:3 1:1 1:3

3:3
tt
0:2
2:2
f (S, T) = (2 + 2) + ( 2 + 1 1 + 2)
=4
S
T
Day 38
L22.13
Another characterization of
flow value
Lemma. For any flow f and any cut (S, T), we
have | f | = f (S, T).
Proof.
f (S, T) = f (S, V) f (S, S)

= f (S, V)
= f (s, V) + f (Ss, V)
= f (s, V)
= | f |.
Day 38
L22.14
Capacity of a cut
Definition. The capacity of a cut (S, T) is c(S, T).
2:2
2:3
2:3
0:1
ss
2:2
0:3 1:1 1:3
tt
0:2
3:3
S
T
2:2
c(S, T) = (3 + 2) + (1 + 2 + 3)
= 11
Day 38
L22.15
Upper bound on the maximum

flow value
Theorem. The value of any flow is bounded
above by the capacity of any cut.
Proof.
f = f (S ,T )
= f (u , v)
uS vT
c(u , v)
uS vT
= c( S , T ) .
Day 38
L22.16
Residual network
Definition. Let f be a flow on G = (V, E). The
residual network Gf (V, Ef ) is the graph with
strictly positive residual capacities
cf (u, v) = c(u, v) f (u, v) > 0.
Edges in Ef admit more flow.
Example:
G:
0:1
uu
vv
Gf :
3:5
uu
vv
2
Lemma. |Ef | 2|E|.

Day 38
L22.17
Augmenting paths
Definition. Any path from s to t in Gf is an augmenting path in G with respect to f. The flow
value can be increased along an augmenting
path p by c f ( p ) = min {c f (u , v)}.
(u ,v ) p
Ex.:
3:5
G:
2:5
tt
2
5:5
7
2:3
2
ss
tt
3
0:2
ss
cf (p) = 2
Gf :
2:6
2
2
Day 38
L22.18
Max-flow, min-cut theorem

Theorem. The following are equivalent:
1. f is a maximum flow.
2. f admits no augmenting paths.
3. | f | = c(S, T) for some cut (S, T).
Proof (and algorithms). Next time.
Day 38
L22.19
6.046J/18.401J/SMA5503
Lecture 23
Recall from Lecture 22

Flow value: | f | = f (s, V).
Cut: Any partition (S, T) of V such that s S
and t T.
Lemma. | f | = f (S, T) for any cut (S, T).
Corollary. | f | c(S, T) for any cut (S, T).
Residual graph: The graph Gf = (V, Ef ) with
strictly positive residual capacities cf (u, v) =
c(u, v) f (u, v) > 0.
Augmenting path: Any path from s to t in Gf .
Residual capacity of an augmenting path:
c f ( p ) = min {c f (u , v)} .
(u ,v ) p
Day 40
L23.2
Max-flow, min-cut theorem

Theorem. The following are equivalent:
1. | f | = c(S, T) for some cut (S, T).
2. f is a maximum flow.
3. f admits no augmenting paths.
Proof.
(1) (2): Since | f | c(S, T) for any cut (S, T) (by
the corollary from Lecture 22), the assumption that
| f | = c(S, T) implies that f is a maximum flow.
(2) (3): If there were an augmenting path, the
flow value could be increased, contradicting the
maximality of f.
Day 40
L23.3
Proof (continued)
(3) (1): Suppose that f admits no augmenting paths.
Define S = {v V : there exists a path in Gf from s to v},
and let T = V S. Observe that s S and t T, and thus
(S, T) is a cut. Consider any vertices u S and v T.
ss
path in Gf
uu
vv
S
We must have cf (u, v) = 0, since if cf (u, v) > 0, then v S,

not v T as assumed. Thus, f (u, v) = c(u, v), since cf (u, v)
= c(u, v) f (u, v). Summing over all u S and v T
yields f (S, T) = c(S, T), and since | f | = f (S, T), the theorem
follows.
Day 40
L23.4
Ford-Fulkerson max-flow
algorithm
Algorithm:
f [u, v] 0 for all u, v V

while an augmenting path p in G wrt f exists
do augment f by cf (p)
Can be slow:
109
G:
ss
109
109
tt
109
Day 40
L23.5
algorithm
Algorithm:

Can be slow:
0:109
G:
0:1
ss
0:109
0:109
tt
0:109
Day 40
L23.6
algorithm
Algorithm:

Can be slow:
0:109
G:
0:1
ss
0:109
0:109
tt
0:109
Day 40
L23.7
algorithm
Algorithm:

Can be slow:
1:109
G:
1:1
ss
0:109
0:109
tt
1:109
Day 40
L23.8
algorithm
Algorithm:

Can be slow:
1:109
G:
1:1
ss
0:109
0:109
tt
1:109
Day 40
L23.9
algorithm
Algorithm:

Can be slow:
1:109
G:
0:1
ss
1:109
1:109
tt
1:109
Day 40
L23.10
algorithm
Algorithm:

Can be slow:
1:109
G:
0:1
ss
1:109
1:109
tt
1:109
Day 40
L23.11
algorithm
Algorithm:

Can be slow:
2:109
G:
1:109
1:1
ss
1:109
tt
2:109
2 billion iterations on a graph with 4 vertices!

Day 40
L23.12
Edmonds-Karp algorithm
Edmonds and Karp noticed that many peoples
implementations of Ford-Fulkerson augment along
a breadth-first augmenting path: a shortest path in
Gf from s to t where each edge has weight 1. These
implementations would always run relatively fast.
Since a breadth-first augmenting path can be found
in O(E) time, their analysis, which provided the first
polynomial-time bound on maximum flow, focuses
on bounding the number of flow augmentations.
(In independent work, Dinic also gave polynomialtime bounds.)
Day 40
L23.13
Monotonicity lemma
Lemma. Let (v) = f (s, v) be the breadth-first
distance from s to v in Gf . During the EdmondsKarp algorithm, (v) increases monotonically.
Proof. Suppose that f is a flow on G, and augmentation
produces a new flow f . Let (v) = f (s, v). Well
show that (v) (v) by induction on (v). For the base
case, (s) = (s) = 0.
For the inductive case, consider a breadth-first path s
L u v in Gf . We must have (v) = (u) + 1, since
subpaths of shortest paths are shortest paths. Certainly,
(u, v) Ef , and now consider two cases depending on
whether (u, v) Ef .
Day 40
L23.14
Case 1
Case: (u, v) Ef .
We have
(v) (u) + 1
(u) + 1
= (v)
(triangle inequality)
(induction)
(breadth-first path),
and thus monotonicity of (v) is established.
Day 40
L23.15
Case 2
Case: (u, v) Ef .
Since (u, v) Ef , the augmenting path p that produced
f from f must have included (v, u). Moreover, p is a
breadth-first path in Gf :
p=sLvuLt.
Thus, we have
(v) = (u) 1
(breadth-first path)
(u) 1
(induction)
(v) 2
< (v) ,
thereby establishing monotonicity for this case, too.
Day 40
L23.16
Counting flow augmentations

Theorem. The number of flow augmentations
in the Edmonds-Karp algorithm (Ford-Fulkerson
with breadth-first augmenting paths) is O(V E).
Proof. Let p be an augmenting path, and suppose that
we have cf (u, v) = cf (p) for edge (u, v) p. Then, we
say that (u, v) is critical, and it disappears from the
residual graph after flow augmentation.
cf (p) = 2
Example:
2
Gf :
ss
tt
3
1
2
Day 40
L23.17

Theorem. The number of flow augmentations
in the Edmonds-Karp algorithm (Ford-Fulkerson
with breadth-first augmenting paths) is O(V E).
Proof. Let p be an augmenting path, and suppose that
the residual capacity of edge (u, v) p is cf (u, v) = cf (p).
Then, we say (u, v) is critical, and it disappears from the
residual graph after flow augmentation.
Example:
Gf :
ss
tt
5
4
Day 40
L23.18

(continued)
The first time an edge (u, v) is critical, we have (v) =

(u) + 1, since p is a breadth-first path. We must wait
until (v, u) is on an augmenting path before (u, v) can
be critical again. Let be the distance function when
(v, u) is on an augmenting path. Then, we have
(u) = (v) + 1
(v) + 1
(monotonicity)
= (u) + 2
(breadth-first path).
Example:
uu
ss
tt
vv
Day 40
L23.19

(continued)

(u) = (v) + 1
(v) + 1
(monotonicity)
= (u) + 2
Example:
(u) = 5
uu
ss
tt
vv
(v) = 6
Day 40
L23.20

(continued)

(u) = (v) + 1
(v) + 1
(monotonicity)
= (u) + 2
Example:
(u) = 5
uu
ss
tt
vv
(v) = 6
Day 40
L23.21

(continued)

(u) = (v) + 1
(v) + 1
(monotonicity)
= (u) + 2
Example:
(u) 7
uu
ss
tt
vv
(v) 6
Day 40
L23.22

(continued)

(u) = (v) + 1
(v) + 1
(monotonicity)
= (u) + 2
Example:
(u) 7
uu
ss
tt
vv
(v) 6
Day 40
L23.23

(continued)

(u) = (v) + 1
(v) + 1
(monotonicity)
= (u) + 2
Example:
(u) 7
uu
ss
tt
vv
(v) 8
Day 40
L23.24
Running time of EdmondsKarp

Distances start out nonnegative, never decrease, and are
at most |V| 1 until the vertex becomes unreachable.
Thus, (u, v) occurs as a critical edge O(V) times, because
(v) increases by at least 2 between occurrences. Since
the residual graph contains O(E) edges, the number of
flow augmentations is O(V E).
Corollary. The Edmonds-Karp maximum-flow

algorithm runs in O(V E 2) time.
Proof. Breadth-first search runs in O(E) time, and all
other bookkeeping is O(V) per augmentation.
Day 40
L23.25
Best to date
The asymptotically fastest algorithm to date for
maximum flow, due to King, Rao, and Tarjan,
runs in O(V E logE/(V lg V)V) time.
If we allow running times as a function of edge
weights, the fastest algorithm for maximum
flow, due to Goldberg and Rao, runs in time
O(min{V 2/3, E 1/2} E lg (V 2/E + 2) lg C),
where C is the maximum capacity of any edge
in the graph.
Day 40
L23.26
Lectures 24 & 25
Higher Layer Protocols:
TCP/IP and ATM
Eytan Modiano
Massachusetts Institute of Technology
Laboratory for Information and Decision Systems
Eytan Modiano
Slide 1
Outline
Network Layer and Internetworking
The TCP/IP protocol suit
ATM
MPLS
Eytan Modiano
Slide 2
Higher Layers
Application
Application
Virtual network service
Presentation
Presentation
Virtual session
Session
Session
Virtual link for end to end messages
Transport
Transport
TCP, UDP
Virtual link for end to end packets

Network
Network
Network
Network
Virtual link for

reliable packets
Data link
Control
physical
interface
DLC
DLC
DLC
DLC
Virtual bit pipe

phys. int. phys. int.
phys. int. phys. int.
Data link
Control
physical
interface
Physical link
External
Site
Eytan Modiano
Slide 3
subnet
node
subnet
node
External
site
IP, ATM
Packet Switching
Datagram packet switching
Route chosen on packet-by-packet basis

Different packets may follow different routes
Packets may arrive out of order at the destination
E.g., IP (The Internet Protocol)
Virtual Circuit packet switching
All packets associated with a session follow the same path

Route is chosen at start of session
Packets are labeled with a VC# designating the route
The VC number must be unique on a given link but can change from
link to link
Imagine having to set up connections between 1000 nodes in a mesh
Unique VC numbers imply 1 Million VC numbers that must be represented
and stored at each node
Eytan Modiano
Slide 4
E.g., ATM (Asynchronous transfer mode)
Virtual Circuits Packet Switching
For datagrams, addressing information must uniquely distinguish

each network node and session
Need unique source and destination addresses
For virtual circuits, only the virtual circuits on a link need be

distinguished by addressing
Global address needed to set-up virtual circuit

Once established, local virtual circuit numbers can then be used to
represent the virtual circuits on a given link: VC number changes from
link to link
VC7
Merits of virtual circuits

Save on route computation
Need only be done once
at start of session
Eytan Modiano
Slide 5
Save on header size

More complex
Less flexible
VC4
VC13
5
VC3
8
VC3
VC7
2
Node 5 table
(3,5) VC13 -> (5,8) VC3
(3,5) VC7 -> (5,8) VC4
(6,5) VC3 -> (5,8) VC7
The TCP/IP Protocol Suite
Transmission Control Protocol / Internet Protocol
Developed by DARPA to connect Universities and Research Labs

Four Layer model
Applications
Transport
Network
Link
Telnet, FTP, email, etc.
TCP, UDP
IP, ICMP, IGMP
Device drivers, interface cards
TCP - Transmission Control Protocol
UDP - User Datagram Protocol
IP - Internet Protocol
Eytan Modiano
Slide 6
Internetworking with TCP/IP
FTP
client
FTP Protocol
TCP
TCP Protocol
IP
FTP
server
TCP
ROUTER
IP Protocol
IP Protocol
IP
Ethernet
driver
Ethernet
Protocol
Ethernet
Eytan Modiano
Slide 7
Ethernet
driver
token ring
driver
token ring
Protocol
token ring
IP
token ring
driver
Encapsulation
user data
Appl
header
Application
user data
TCP
TCP
header
application data
IP
TCP segment
IP
header
TCP
header
application data
Ethernet
driver
IP datagram
Ethernet
header
14
IP
header
20
TCP
header
20
Ethernet frame
46 to 1500 bytes
Eytan Modiano
Slide 8
application data
Ethernet
trailer
4
Ethernet
Bridges, Routers and Gateways
A Bridge is used to connect multiple LAN segments

Layer 2 routing (Ethernet)
Does not know IP address
Varying levels of sophistication
Simple bridges just forward packets
smart bridges start looking like routers
A Router is used to route connect between different networks

using network layer address
A Gateway connects between networks using different protocols
Eytan Modiano
Slide 9
Within or between Autonomous Systems

Using same protocol (e.g., IP, ATM)
Protocol conversion
Address resolution
These definitions are often mixed and seem to evolve!
Bridges, routers and gateways
Small company
Ethernet A
Bridge
Another providers
Frame Relay
Backbone
IP
Router
Ethernet B
Gateway
Gateway
ATM switches
(routers)
Eytan Modiano
Slide 10
Service
providers
ATM
backbone
IP addresses
32 bit address written as four decimal numbers
One per byte of address (e.g., 155.34.60.112)
Hierarchical address structure
IP Address classes:
Network ID/ Host ID/ Port ID

Complete address called a socket
Network and host ID carried in IP Header
Port ID (sending process) carried in TCP header
32
0 Net ID
Class A Nets
Host ID
16
10
Net ID
32
Class B Nets
Host ID
32
24
110
Net ID
Host ID
Class D is for multicast traffic
Eytan Modiano
Slide 11
Class C Nets
Host Names
Each machine also has a unique name
Domain name System: A distributed database that provides a

mapping between IP addresses and Host names
E.g., 155.34.50.112 => plymouth.ll.mit.edu
Eytan Modiano
Slide 12
Internet Standards
Internet Engineering Task Force (IETF)
Request for Comments (RFCs)
Eytan Modiano
Slide 13
Development on near term internet standards

Open body
Meets 3 times a year
Official internet standards

Available from IETF web page: http://www.ietf.org
The Internet Protocol (IP)
Routing of packet across the network

Unreliable service
Best effort delivery

Recovery from lost packets must be done at higher layers
Connectionless
Packets are delivered (routed) independently

Can be delivered out of order
Re-sequencing must be done at higher layers
Current version V4
Future V6
Eytan Modiano
Slide 14
Add more addresses (40 byte header!)

Ability to provide QoS
Header Fields in IP
8
Header
length
Ver
16
type of service
16 - bit identification
TTL
32
Total length (bytes)
Flags
Protocol
13 - bit fragment offset
Header Checksum
Source IP Address
Destination IP Address
Options (if any)
Data
Note that the minimum size header is 20 bytes; TCP

also has 20 byte header
Eytan Modiano
Slide 15
IP HEADER FIELDS
Vers:
Version # of IP (current version is 4)
HL:
Header Length in 32-bit words
Service:
Mostly Ignored
SA & DA:
Network Addresses
Options:
Record Route,Source Route,TimeStamp
Eytan Modiano
Slide 16
Total length Length of IP datagram

ID
Unique datagram ID
Flags:
NoFrag, More
FragOffset:
Fragment offset in units of 8 Octets
TTL:
Time to Live in "seconds or Hops
Protocol:
Higher Layer Protocol ID #
HDR Cksum: 16 bit 1's complement checksum (on header only!)
FRAGMENTATION
X.25
ethernet
mtu=1500
G
MTU = 512
G
ethernet
mtu=1500
A gateway fragments a datagram if length is too great for next

network (fragmentation required because of unknown paths).
Each fragment needs a unique identifier for datagram plus

identifier for position within datagram
In IP, the datagram ID is a 16 bit field counting datagram from

given host
Eytan Modiano
Slide 17
POSITION OF FRAGMENT
Fragment offset field gives starting position of fragment within

datagram in 8 byte increments (13 bit field)
Length field in header gives the total length in bytes (16 bit field)
Maximum size of IP packet 64K bytes
A flag bit denotes last fragment in datagram
IP reassembles fragments at destination and throws them away if

one or more is too late in arriving
Eytan Modiano
Slide 18
IP Routing
Routing table at each node contains for each destination the next
hop router to which the packet should be sent
Not all destination addresses are in the routing table
Look for net ID of the destination Prefix match
Use default router
Eytan Modiano
Slide 19
Routers do not compute the complete route to the destination but

only the next hop router
IP uses distributed routing algorithms: RIP, OSPF
In a LAN, the host computer sends the packet to the default
router which provides a gateway to the outside world
Subnet addressing
Class A and B addresses allocate too many hosts to a given net

Subnet addressing allows us to divide the host ID space into
smaller sub networks
Simplify routing within an organization
Smaller routing tables
Potentially allows the allocation of the same class B address to more
than one organization
32 bit Subnet Mask is used to divide the host ID field into

subnets
1 denotes a network address field

0 denotes a host ID field
16 bit net ID
Class B
Address
Mask
Eytan Modiano
Slide 20
16 bit host ID
140.252
Subnet ID
Host ID
111111 111 1111111
11111111
00000000
Classless inter-domain routing (CIDR)
Class A and B addresses allocate too many hosts to an

organization while class C addresses dont allocate enough
This leads to inefficient assignment of address space
Classless routing allows the allocation of addresses outside of

class boundaries (within the class C pool of addresses)
Allocate a block of contiguous addresses

E.g., 192.4.16.1 - 192.4.32.155
Bundles 16 class C addresses
The first 20 bits of the address field are the same and are essentially the
network ID
Network numbers must now be described using their length and

value (I.e., length of network prefix)
Routing table lookup using longest prefix match
Eytan Modiano
Slide 21
Notice similarity to subnetting - supernetting
Dynamic Host Configuration (DHCP)
Automated method for assigning network numbers
Computers contact DHCP server at Boot-up time

Server assigns IP address
Allows sharing of address space
More efficient use of address space

Adds scalability
Addresses are least for some time
Eytan Modiano
Slide 22
IP addresses, default routers
Not permanently assigned
Address Resolution Protocol
IP addresses only make sense within IP suite

Local area networks, such as Ethernet, have their own addressing
scheme
To talk to a node on LAN one must have its physical address
(physical interface cards dont recognize their IP addresses)
ARP provides a mapping between IP addresses and LAN

addresses
RARP provides mapping from LAN addresses to IP addresses
This is accomplished by sending a broadcast packet requesting
the owner of the IP address to respond with their physical address
All nodes on the LAN recognize the broadcast message

The owner of the IP address responds with its physical address
An ARP cache is maintained at each node with recent mappings

IP
ARP
RARP
Ethernet
Eytan Modiano
Slide 23
Routing in the Internet
The internet is divided into sub-networks, each under the control

of a single authority known as an Autonomous System (AS)
Routing algorithms are divided into two categories:
Interior protocols (within an AS)

Exterior protocols (between ASs)
Interior Protocols use shortest path algorithms (more later)

Distance vector protocols based on Bellman-ford algorithm
Nodes exchange routing tables with each other
E.g., Routing Information Protocol (RIP)
Link state protocols based on Dijkstras algorithm

Nodes monitor the state of their links (e.g., delay)
Nodes broadcast this information to all of the network
E.g., Open Shortest Path First (OSPF)
Exterior protocols route packets across ASs

Issues: no single cost metric, policy routing, etc..
Routes often are pre-computed
Example protocols: Exterior Gateway protocol (EGP) and Border
Gateway protocol (BGP)
Eytan Modiano
Slide 24
IPv6
Effort started in 1991 as IPng

Motivation
Need to increase IP address space

Support for real time application - QoS
Security, Mobility, Auto-configuration
Major changes
31
ver class Flow label

length nexthd Hop limit
Source address
Increased address space (6 bytes)

1500 IP addresses per sq. ft. of earth!
Address partition similar to CIDR
Most of the reasons for IPv6 have been

taken care of in IPv4
Eytan Modiano
Slide 25
Support for QoS via Flow Label field

Simplified header
Is IPv6 really needed?

Complex transition from V4 to V6
Destination address
Resource Reservation (RSVP)
Service classes (defined by IETF)

Best effort
Guaranteed service
Max packet delay
Controlled load
emulate lightly loaded network via priority queueing mechanism
Need to reserve resources at routers along the path

RSVP mechanism
Packet classification
Associate packets with sessions (use flow field in IPv6)
Receiver initiated reservations to support multicast

soft state - temporary reservation that expires after 30 seconds
Simplify the management of connections
Requires refresh messages
Packet scheduling to guarantee service

Proprietary mechanisms (e.g., Weighted fair queueing)
Scalability Issues
Each router needs to keep track of large number of flows that grows
with the size (capacity) of the router
Eytan Modiano
Slide 26
Differentiated Services (Diffserv)
Unlike RSVP Diffserv does not need to keep track of individual

flows
Allocate resources to a small number of classes of traffic
Queue packets of the same class together
E.g., two classes of traffic - premium and regular

Use one bit to differential between premium and regular packets
Issues
Who sets the premium bit?
How is premium service different from regular?
IETF propose to use TOS field in IP header to identify

traffic classes
Eytan Modiano
Slide 27
Potentially more than just two classes
User Datagram Protocol (UDP)
Transport layer protocol
Delivery of messages across network
Datagram oriented
Unreliable
No error control mechanism
Connectionless
Not a stream protocol
Max packet length 65K bytes

UDP checksum
Covers header and data
Optional
Can be used by applications
Eytan Modiano
Slide 28
UDP allows applications to interface directly to IP with minimal

additional processing or protocol overhead
UDP header format
IP Datagram
IP header
data
UDP header
16 bit source port number
16 bit destination port number

16 bit checksum
16 bit UDP length
Data
The port numbers identifie the sending and receiving processes
UDP length = length of packet in bytes
Minimum of 8 and maximum of 2^16 - 1 = 65,535 bytes
Checksum covers header and data
Eytan Modiano
Slide 29
I.e., FTP, email, etc..

Allow UDP to multiplex the data onto a single stream
Optional, UDP does not do anything with the checksum
Transmission Control Protocol (TCP)
Transport layer protocol
Connection oriented
Stream traffic
Must re-sequence out of order IP packets
Reliable
Reliable transmission of messages
ARQ mechanism
Notice that packets have a sequence number and an ack number
Notice that packet header has a window size (for Go Back N)
Flow control mechanism

Slow start
Limits the size of the window in response to congestion
Eytan Modiano
Slide 30
Basic TCP operation
At sender
At receiver
Eytan Modiano
Slide 31
Application data is broken into TCP segments

TCP uses a timer while waiting for an ACK of every packet
Un-ACKd packets are retransmitted
Errors are detected using a checksum

Correctly received data is acknowledged
Segments are reassembled into their proper order
Duplicate segments are discarded
Window based retransmission and flow control
TCP header fields
16
Source port
32
Destination port
Sequence number
Request number
Data
Offset
Reserved
Control
Check sum
Urgent pointer
Options (if any)
Data
Eytan Modiano
Slide 32
Window
TCP header fields
Ports number are the same as for UDP

32 bit SN uniquely identify the application data contained in the
TCP segment
32 bit RN is used for piggybacking ACKs
SN is in bytes!
It identify the first byte of data
RN indicates the next byte that the received is expecting
Implicit ACK for all of the bytes up to that point
Data offset is a header length in 32 bit words (minimum 20 bytes)

Window size
Used for error recovery (ARQ) and as a flow control mechanism

Sender cannot have more than a window of packets in the network
simultaneously
Specified in bytes
Window scaling used to increase the window size in high speed networks
Eytan Modiano
Slide 33
Checksum covers the header and data
TCP error recovery
Error recovery is done at multiple layers
Link, transport, application
Transport layer error recovery is needed because

Packet losses can occur at network layer
E.g., buffer overflow
SN and RN are used for error recovery in a similar way to Go Back

N at the link layer
Large SN needed for re-sequencing out of order packets
TCP uses a timeout mechanism for packet retransmission
Eytan Modiano
Slide 34
Some link layers may not be reliable
Timeout calculation
Fast retransmission
TCP timeout calculation
Based on round trip time measurement (RTT)
Weighted average
RTT_AVE = a*(RTT_measured) + (1-a)*RTT_AVE
Timeout is a multiple of RTT_AVE (usually two)
Short Timeout would lead to too many retransmissions

Long Timeout would lead to large delays and inefficiency
In order to make Timeout be more tolerant of delay variations it

has been proposed (Jacobson) to set the timeout value based on
the standard deviation of RTT
Timeout = RTT_AVE + 4*RTT_SD

Eytan Modiano
Slide 35
In many TCP implementations the minimum value of Timeout is

500 ms due to the clock granularity
Fast Retransmit
When TCP receives a packet with a SN that is greater than the

expected SN, it sends an ACK packet with a request number of the
expected packet SN
If a packet is lost then duplicate RNs will be sent by TCP until the
packet it correctly received
After 3 duplicate RNs are received the packet is retransmitted

After retransmission, continue to send new data
Fast retransmit allows TCP retransmission to behave more like

Selective repeat ARQ
Eytan Modiano
Slide 36
But the packet will not be retransmitted until a Timeout occurs

This leads to added delay and inefficiency
Fast retransmit assumes that if 3 duplicate RNs are received by

the sending module that the packet was lost
This could be due to out-of-order delivery or packet loss
Future option for selective ACKs (SACK)
TCP congestion control
TCP uses its window size to perform end-to-end congestion

control
More on window flow control later
Basic idea
With window based ARQ the number of packets in the network
cannot exceed the window size (CW)
Last_byte_sent (SN) - last_byte_ACKd (RN) <= CW
Transmission rate when using window flow control is equal to one

window of packets every round trip time
R = CW/RTT
Eytan Modiano
Slide 37
By controlling the window size TCP effectively controls the rate
Effect Of Window Size
The window size is the number of bytes that are allowed to be in

transport simultaneously
WASTED BW
WINDOW
WINDOW
Too small a window prevents continuous transmission
To allow continuous transmission window size must exceed round-trip

delay time
Eytan Modiano
Slide 38
Length of a bit (traveling at 2/3C)
At 300 bps
1 bit = 415 miles
3000 miles = 7 bits
At 3.3 kbps
1 bit = 38 miles
3000 miles = 79 bits
At 56 kbps
1 bit = 2 miles
3000 miles = 1.5 kbits
At 1.5 Mbps
1 bit = 438 ft.
3000 miles = 36 kbits
At 150 Mbps
1 bit = 4.4 ft.
3000 miles = 3.6 Mbits
At 1 Gbps
1 bit = 8 inches
3000 miles = 240 Mbits
Eytan Modiano
Slide 39
Dynamic adjustment of window size
TCP starts with CW = 1 packet and increases the window size

slowly as ACKs are received
Slow start phase

Congestion avoidance phase
Slow start phase

During slow start TCP increases the window by one packet for every
ACK that is received
When CW = Threshold TCP goes to Congestion avoidance phase
Notice: during slow start CW doubles every round trip time
Exponential increase!
Congestion avoidance phase

During congestion avoidance TCP increases the window by one
packet for every window of ACKs that it receives
Notice that during congestion avoidance CW increases by 1 every
round trip time - Linear increase!
Eytan Modiano
Slide 40
TCP continues to increase CW until congestion occurs
Reaction to congestion
Many variations: Tahoe, Reno, Vegas

Basic idea: when congestion occurs decrease the window size
There are two congestion indication mechanisms
Duplicate ACKs - could be due to temporary congestion

Timeout - more likely due to significant congstion
TCP Reno - most common implementation
If Timeout occurs, CW = 1 and go back to slow start phase
If duplicate ACKs occur CW = CW/2 stay in congestion avoidance

phase
Eytan Modiano
Slide 41
Understanding TCP dynamics
Slow start phase is actually fast

TCP spends most of its time in Congestion avoidance phase
While in Congestion avoidance
CW increases by 1 every RTT

CW decreases by a factor of two with every loss
Additive Increase / Multiplicative decrease
Saw-tooth Behavior
CW
Time
Eytan Modiano
Slide 42
Random Early Detection (RED)
Instead of dropping packet on queue overflow, drop them probabilistically earlier
Motivation

Dropped packets are used as a mechanism to force the source to slow down
If we wait for buffer overflow it is in fact too late and we may have to drop many packets
Leads to TCP synchronization problem where all sources slow down simultaneously
RED provides an early indication of congestion

Randomization reduces the TCP synchronization problem
Mechanism

Use weighted average queue size

If AVE_Q > Tmin drop with prob. P
If AVE_Q > Tmax drop with prob. 1
RED can be used with explicit congestion

notification rather than packet dropping
RED has a fairness property
Large flows more likely to be dropped

Eytan Modiano
Slide 43
Threshold and drop probability values

are an area of active research
1
P
max
Tmin
Ave queue length
Tmax
TCP Error Control
EFFICIENCY VS. BER

1
SRP
1 SEC R/T DELAY

T-1 RATE
1000 BIT PACKETS
0.9
0.8
GO BACK N
0.7
WITH TCP
WINDOW CONSTRAINT
0.6
0.5
0.4
0.3
0.2
0.1
0
1E-07
1E-06
1E-05
1E-04
1E-03
1E-02
CHANNEL BER

Eytan Modiano
Slide 44
Original TCP designed for low BER, low delay links

Future versions (RFC 1323) will allow for larger windows and selective
retransmissions
Impact of transmission errors on
TCP congestion control
EFFICIENCY VS BER FOR TCP'S

CONGESTION CONTROL
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
2.4 KBPS
1,544 KBPS
1.00E-07
1.00E-06
16 KBPS
64 KBPS
1.00E-05
1.00E-04
1.00E-03
BER
TCP assumes dropped packets are due to congestion and responds

by reducing the transmission rate

Over a high BER link dropped packets are more likely to be due to
errors than to congestion
TCP extensions (RFC 1323)
Fast retransmit mechanism, fast recovery, window scaling
Eytan Modiano
Slide 45
TCP releases
TCP standards are published as RFCs

TCP implementations sometimes differ from one another
May not implement the latest extensions, bugs, etc.
The de facto standard implementation is BSD

Computer system Research group at UC-Berkeley
Most implementations of TCP are based on the BSD implementations
SUN, MS, etc.
BSD releases
4.2BSD - 1983
First widely available release
4.3BSD Tahoe - 1988

Slow start and congestion avoidance
4.3BSD Reno - 1990

Header compression
4.4BSD - 1993
Multicast support, RFC 1323 for high performance
Eytan Modiano
Slide 46
The TCP/IP Suite
ping
Telnet&
Rlogin
FTP
SMTP
Trace
route
TFTP
UDP
TPC
ICMP
IP
ARP
Data Link
media
Eytan Modiano
Slide 47
DNS
IGMP
RARP
BOOTP
SNMP
RPC
NFS
Asynchronous Transfer Mode (ATM)
1980s effort by the phone companies to develop an integrated

network standard (BISDN) that can support voice, data, video, etc.
ATM uses small (53 Bytes) fixed size packets called cells
Why cells?
Cell switching has properties of both packet and circuit switching
Easier to implement high speed switches
Why 53 bytes?
Small cells are good for voice traffic (limit sampling delays)
For 64Kbps voice it takes 6 ms to fill a cell with data
ATM networks are connection oriented
Eytan Modiano
Slide 48
Virtual circuits
ATM Reference Architecture
Upper layers
Applications
TCP/IP
ATM adaptation layer
Similar to transport layer

Provides interface between
upper layers and ATM
Break messages into cells and
reassemble
ATM layer
ATM designed for SONET

Synchronous optical network
TDMA transmission scheme with
125 s frames
Eytan Modiano
Slide 49
A T M A daptation
L ayer (A A L )
AT M
Cell switching
Congestion control
Physical layer
Upper Layers
Physical
ATM Cell format
5 Bytes
ATM Cell
Header
48 Bytes
Data
ATM Header (NNI)
Virtual circuit numbers
(notice relatively small address

space!)
PTI - payload type

CLP - cell loss priority (1 bit!)
Virtual channel ID
Virtual path ID
4
5
Eytan Modiano
Slide 50
VPI
VCI
VCI
Mark cells that can be dropped
HEC - CRC on header
VPI
VCI
PTI
HEC
C
L
P
VPI/VCI
VPI identifies a physical path between the source and destination

VCI identifies a logical connection (session) within that path
Approach allows for smaller routing tables
and simplifies route computation
Private
network
ATM Backbone
Private
network
Private
network
Use VPI for switching
in backbone
Eytan Modiano
Slide 51
Use VCI to ID connection

Within private network
ATM HEADER CRC
ATM uses an 8 bit CRC that is able to correct 1 error

It checks only on the header of the cell, and alternates between
two modes
In detection mode it does not correct any errors but is able to detect
more errors
In correction mode it can correct up to one error reliably but is less
able to detect errors
When the channel is relatively good it makes sense to be in

correction mode, however when the channel is bad you want to be
in detection mode to maximize the detection capability
No detected
errors
Correct
errors
Eytan Modiano
Slide 52
Detect double error

Correct single
error
No detected errors
Detect
errors
Detected
errors
ATM Service Categories
Constant Bit Rate (CBR) - e.g. uncompressed voice
Variable Bit Rate (rt-VBR) - e.g. compressed video
For bursty traffic with limited BW guarantees and congestion control
Unspecified Bit Rate (UBR) - e.g. Internet
Eytan Modiano
Slide 53
Real-time and non-real-time
Available Bit Rate (ABR) - e.g. LAN interconnect
Circuit emulation
ABR without BW guarantees and congestion control
ATM service parameters
(examples)
Peak cell rate (PCR)

Sustained cell rate (SCR)
Maximum Burst Size (MBS)
Minimum cell rate (MCR)
Cell loss rate (CLR)
Cell transmission delay (CTD)
Cell delay variation (CDV)
Not all parameters apply to all service categories
Network guarantees QoS provided that the user conforms to his

contract as specified by above parameters
Eytan Modiano
Slide 54
E.g., CBR specifies PCR and CDV

VBR specifies MBR and SCR
When users exceed their rate network can drop those packets
Cell rate can be controlled using rate control scheme (leaky bucket)
Flow control in ATM networks (ABR)
ATM uses resource management cells to control rate parameters
RM cells contain
Forward resource management (FRM)

Backward resource management (BRM)
Congestion indicator (CI)
No increase Indicator (NI)
Explicit cell rate (ER)
Current cell rate (CCR)
Min cell rate (MCR)
Source generates RM cells regularly

As RM cells pass through the networked they can be marked with
CI=1 to indicate congestion
RM cells are returned back to the source where
CI = 1 => decrease rate by some fraction
CI = 1 => Increase rate by some fraction
Eytan Modiano
Slide 55
ER can be used to set explicit rate
End-to-End RM-Cell Flow
ABR
Source
ABR
Switch
BRM
FRM
ABR
Destin
ation
ABR
Switch
BRM
= data cell
FRM = forward RM cell

BRM = backward RM cell
At the destination the RM cell is turned around

and sent back to the source
Eytan Modiano
Slide 56
FRM
ATM Adaptation Layers
Interface between ATM layer and higher layer packets

Four adaptation layers that closely correspond
to ATMs service classes
AAL-1 to support CBR traffic

AAL-2 to support VBR traffic
AAL-3/4 to support bursty data traffic
AAL-5 to support IP with minimal overhead
The functions and format of the adaptation layer depend on the

class of service.
For example, stream type traffic requires sequence numbers to
identify which cells have been dropped.
Each class of service has

A different header format
(in addition to the 5 byte
ATM header)
USER PDU (DLC or NL)
ATM CELL
ATM CELL
Eytan Modiano
Slide 57
Example: AAL 3/4
ATM CELL PAYLOAD (48 Bytes)

ST SEQ
MID
10
LEN
CRC
10
44 Byte User Payload
ST: Segment Type (1st, Middle, Last)

SEQ:4-bit sequence number (detect lost cells)
MID: Message ID (reassembly of multiple msgs)
44 Byte user payload (~84% efficient)
LEN: Length of data in this segment
CRC: 10 bit segment CRC
AAL 3/4 allows multiplexing, reliability, & error detection but is

fairly complex to process and adds much overhead
AAL 5 was introduced to support IP traffic
Eytan Modiano
Slide 58
Fewer functions but much less overhead and complexity
ATM cell switches
Input 1
Cell
Processing
Output 1
Input 2
Cell
Processing
Output
Switch
Fabric
Input m
Cell
Processing
Output m
Input
Q's
Output
Q's
Control
S/W
Design issues
Eytan Modiano
Slide 59
Input vs. output queueing

Head of line blocking
Fabric speed
ATM summary
ATM is mostly used as a core network technology
ATM Advantages
Ability to provide QoS

Ability to do traffic management
Fast cell switching using relatively short VC numbers
ATM disadvantages
It not IP - most everything was design for TCP/IP
Its not naturally an end-to-end protocol
Does not work well in heterogeneous environment
Was not design to inter-operate with other protocols
Not a good match for certain physical media (e.g., wireless)
Many of the benefits of ATM can be borrowed by IP

Cell switching core routers
Label switching mechanisms
Eytan Modiano
Slide 60
Multi-Protocol Label Switching (MPLS)
As more services with fixed throughput and

delay requirements become more common, IP will
need virtual circuits (although it will probably call
them something else)
RG, April 28, 1994
Eytan Modiano
Slide 61
Label Switching
Router makers realize that in order to increase the speed and

capacity they need to adopt a mechanism similar to ATM
Switch based on a simple tag not requiring complex routing table
look-ups
Use virtual circuits to manage the traffic (QoS)
Use cell switching at the core of the router
First attempt: IP-switching
Routers attempt to identify flows

Define a flow based on observing a number of packets between a given
source and destination (e.g., 5 packets within a second)
Map IP source-destination pairs to ATM VCs

Distributed algorithm where each router makes its own decision
Multi-protocol label switching (MPLS)

Also known as Tag switching
Does not depend on ATM
Add a tag to each packet to serve as a VC number
Tags can be assigned permanently to certain paths
Eytan Modiano
Slide 62
Label switching can be used to create a virtual
mesh with the core network
Routers at the edge of the core

Core network
network can be connected to

each other using labels
Packets arriving at an edge router
can be tagged with the label to
the destination edge router
Tunneling
Significantly simplifies routing

in the core
Interior routers need not

remember all IP prefixes of
outside world
Allows for traffic engineering

Assign capacity to labels based
on demand
Eytan Modiano
Slide 63
D
Label switched routes
References
TCP/IP Illustrated (Vols. 1&2), Stevens
Computer Networks, Peterson and Davie
High performance communication networks, Walrand and Varaiya
Eytan Modiano
Slide 64
Class A
7 bits
24 bits
netid
hostid
14 bits
Class B
16 bits
netid
hostid
8 bits
21 bits
Class C
netid
hostid
28 bits
Class D
netid
hostid
27 bits
Class E
Eytan Modiano
Slide 65
(reserved for future use)

Computer Algorithms PDF

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Computer Algorithms PDF

Uploaded by

Copyright:

Available Formats

Introduction to Algorithms

Course information handout

Why study algorithms and

The problem of sorting

Example of insertion sort

Example of insertion sort

Example of insertion sort

Example of insertion sort

Example of insertion sort

Example of insertion sort

Example of insertion sort

Example of insertion sort

Example of insertion sort

Example of insertion sort

Example of insertion sort

(g(n)) = { f (n) : there exist positive constants c1, c2, and

Insertion sort analysis

Average case: All permutations equally likely.

Is insertion sort a fast sorting algorithm?

Merging two sorted arrays

Merging two sorted arrays

Merging two sorted arrays

Merging two sorted arrays

Merging two sorted arrays

Merging two sorted arrays

Merging two sorted arrays

Merging two sorted arrays

Merging two sorted arrays

Merging two sorted arrays

Merging two sorted arrays

Merging two sorted arrays

Merging two sorted arrays

Time = (n) to merge a total

Analyzing merge sort

Recurrence for merge sort

We shall usually omit stating the base

A tighter upper bound?

A tighter upper bound!

Pick c1 big enough to handle the initial conditions.

Example of recursion tree

Example of recursion tree

Example of recursion tree

Example of recursion tree

Example of recursion tree

Example of recursion tree

Example of recursion tree

Example of recursion tree

Example of recursion tree

The master method

Three common cases

Three common cases (cont.)

General method (Akra-Bazzi)

T (n) = aiT (n / bi ) + f (n)

Let p be the unique solution to

Then, the answers are the same as for the

Idea of master theorem

Idea of master theorem

Idea of master theorem

Idea of master theorem

Next time: applying the master method.

Appendix: geometric series

Example: merge sort