X1 X 2

Algorithms
Professor John Reif

ALG 1.1

Read-only
input tape

Xn

r0

location
counter

Models of Computation:

...

Program

Accumulator

r1
r2

..
.

(a) Random Access Machines (RAMs)

(b) Straight Line Programs and Circuits
(c) Decision Trees
(d) Machines That Make Random Choices

Main Reading Selection:

CLR, Chapter 1
1
Auxillary Reading Selections:
AHU-Design, Chapter 1
BB, Sections 1.1-1.5, 1.8
AHU-Data, Chapter 1

Y1

Memory
Registers

...
write-only output tape

RANDOM ACCESS MACHINE

RAM assumptions
(1)

each register holds an integer

(2)

program can't modify itself

(3)

memory instructions involve simple arithmetic

input X
size n = |X|

fi

Algorithm A

fi

output

Complexity Measures of Algorithms

a) addition, subtraction

Time A (X) = time cost of Algorithm A, input X

b) multiplication, division

Space

and control statements (goto, if-then, etc.)

(X) = space "

worst case
time complexity

Written in "Pidgin Algol"

r constant

r3 r1 op r2

op {+, -, , }

examples:
goto label

if r = 0 then goto L

read ( r )

write ( r )

average case
complexity
for random inputs
worst case
space complexity
average case
complexity
for random inputs

"

TA (n) =max (Time

{x:|x|=n}

E(T A (n)) =

"

(X))

S Time A(X)Prob(X)

{x:|x|=n}

SA (n) =max(Space

(x))

{x:|x|=n}

E(S A (n)) =

S Space A (x)Prob(x)
{x:|x|=n}

N o t e : "time" and "space" depend on machine

3

UNIFORM COST CRITERIA

{
{

t i me =# RA M i n s t r u c t i on s

Varieties of
Computing Machine
Models

s p ac e =# RA M me mor y r e gi s t e r s

LOGORITHMIC COST CRITERIA

time = L(i) units per RAM instruction
on integer size i
space = L(i) units per RAM register
size i

wh e r e L( i ) =

example

Z2
for k = 1

uniform

logarithmic

l og |i | i 0
1

i =0

to n do

output Z = 2

RAMs
straight line programs

circuits
bit vectors

lisp machines
..
.
Turing Machines

2 n

Z Z Z

time cost = n
time cost > 2 n
5

Example

Given polynomial
n

p ( x ) = an x + an - 1 x

Straight Line Programs

n- 1

+ . . . + a1 x + a0

with constant coefficients a

idea

fix n = input size

Horner's Rule
for
Polynomial
Evaluation
RA M
p r ogr am
i n 2 n s t e ps

,a 1 ,...,a

unroll each iteration loop

until result is
Pn
loop-free program
note : this is only possible if
we can eliminate all branching
and all indirect addressing

n=1

P1 :

Y a1 X

fi for e ach n > 0 ,

Y Y + a0

ge t a dis tinct program P

i np ut X
Y an
f or i = n - 1 b y - 1 t o 0 d o
Y ( Y X) + a i
ou t p u t Y

n=2

Y a2 X

Y Y + a1

Y Y X
Y Y + a0

n=3
Y
Y
Y
Y
Y
Y

a3 X
Y + a2
Y X
Y + a1
Y X
Y + a

fi fi

Straight Line
Programs

1-1
correspondence

(for VLSI design)

(DAG) graph model for straight line programs

input

Y a3

Y Y

+ a
2

Y Y

Y Y

a1

Y Y

Y Y

a0

inputs
a2

a3

Boolean Circuits

Circuits

a1

x1

x2

a0

output

restrictions:

(1) all memory registers have

(2) use only
logical operations

output

value 0 or 1

, , ,

"and" "or" "parity" "not"

9

10

Decision Trees

Vector Machines
Example

( modeling CONVEX computer)

applied to

vector

memory locations hold

0

Sorting

, , ,

logical operations

a<b?
N

elements

Y
b<c?

boolean vectors
Y

...

input a,b,c

0
output
a,b,c

may also allow vector shift operations

1
1

2
1

3
0

graph G

0
1

output
a,c,b

output
b,a,c

b<c?

output
c,a,b

output
b,c,a

output
c,b,a

To sort n keys
any decision tree must have
n! output leaves

1
0

a<c?
Y

Example

a<c?

(n! = # permutations of n keys)

hence height of tree is

>log (n!)
2
11

>c n log n
12

Lisp Machines

"The VW of Machines"

Complexity theory for AI

...

"The Ultimate Program Language of Theory"

(TM)
Turing Machine

xn

input tape
registers
r
r

program

...

Y 1 Y2

...

X1 X2

input tape
0

Finite State
Control

1
2

Yn

Y2

read/write
memory tapes

write only
output tape

output tape

Invented by Turing
operations
CDR (r)

...

Heap

read only

...

x1 x

r i CAR ( r )

CAR

Built by British for WWII crytography!

CDR

r i CDR ( r j )

CAR ( r i )

CELL

CDR (r i)

Some problems are

faster on LISP machine

(a Cambridge logician)

13

T(n) =

time cost

= max steps of TM

S(n) =

space cost

= max cells written by TM

on memory tapes

14

Reductions
between
TM and RAM
models
(1) Given

TM time cost T(n)

$ equivalent

time
cost

then

Ram (obvious)

Extensions of RAMS:
Reasonable

c T(n)
if uniform cost
c T(n) (log n)
if logarithmic cost

(0) Modifiable Program

(1) Random Choices

T(n) with logarithmic cost

(2) Given RAM time cost
then $ equivalent TM with
time cost c' T(n)

r1

Not Reasonable
(3) Non-deterministic Choices

idea

proof

registers
r0

(2) Non-uniformity

r2

rk

read
write
memory

do arithmetic by Grammer School Method

15

16

Randomized Machines
Extend RAM
instructions to
include

RASP Machine
same as RAM but allow

r RANDOM(k)
gives a random k
bit number

Let A R(x) denote randomized algorithm

with input x,
random choices R

program to change itself

Expected Time

Proof idea

Time (X)

Let RAM

use memory registers

to store
modifiable program of
(due to Von
RASP

input X

"R

Time (X)

Prob (R)

Expected Time Complexity

Neumann)

T(n) = max Time (x)

{x|n = |x|}

17

18

A Randomized Computation
P

T(n)
R

1/2

1/2

O
B

N
1/2
Y

1/2

B
1/2

1/2

1/2

1/2

1/2

1/2

...
...

...
...

...

...

T
Y

output
0

T I M E T(n)

19

output
1

output
0

20

Non-uniformity

(1) If machine
outputs value v with prob >
then v is considered its output.
(2) machine
(3) has

accepts input X

1-sided error

for each input size n,

allow the program a distinct, finite,
"advice tape " to read

1
2

if outputs 1 with prob>

1
2

if when not accepting 1

outputs only 0.
note
if advice tape length 2 n
can solve any boolean output
problem with n boolean inputs
(obvious).

21

22

Surprising Result

Nondeterministic
Machines

[Adelman]

allow "nondeterministic choice" b r a n c h e s

Given any polynomial time

randomized
algorithm with 1 side error,

nondeterministic
choice

a non-uniform
deterministic
algorithm with polynomial time
and polynomial advice!
( tricky proof )
Gives way of derandomizing

a randomized algorithm.

accept x

reject x

if any
sequence of choices succeed
to accept x, then computation a c c e p t s .
23

24

NP = languages accepted by
polynomial
time nondeterministic
TM machines.
- includes many

Another Surprising Result

Levin

hard problems:
If P=NP but we don't know the proof
(i.e., the polynomial time algorithm
for NP
find an optimal algorithm to
find the solution of any
solvable NP search problem,
in polynomial time!

(1) Traveling Salesman Problem

(2) Propositional Satisfiability
(3) Integer Programming

= languages accepted by
polynomial time deterministic

proof depends on assumption

that there is a
finite length
program
for NP search problems, running in poly time)

TM machines

not known
probably
no

P = NP
?
25

26

Conclusion

many possible
(1) There are
machine models
(2) Most (but
not Nondeterministic)
are "Constructable" - so might
be used if we have efficient
algorithms to execute on
machines.
(3) New machine models can help
us invent new algorithms , and
vice versa!

27