A probability measure

is a mapping from
a set of events
to the reals such that

Algorithms
Professor John Reif

ALG 2.0
Probability Theory

(Prob)

(a) Random Variables: Binomial

and Geometric
(b) Useful Probabilistic Bounds
and Inequalities

(1) For any event A

0 Prob(A) 1
(2) Prob ( all possible events) = 1

Main Reading Selections:

CLR, Chapter 6
Auxillary Reading Selections:
BB, Chapter 8
Handout: "Probability Theory
Refresher"

(3) If A,B are mutually exclusive

events, then
Prob (A B) = Prob (A) + Prob (B)
1

Bayes' Theorem
If A

,..., A

are

mutually exclusive and contain all events

Conditional Probability
then

Pi

Prob (A i | B ) =

Pj

j =1

where P

define Prob(A|B) =

= Prob(B|A j)

Prob (A B)
Prob (B)

Prob(A j)

B
A1

for Prob(B) > 0

A2

...

Ai

...

An

Random Variable

(over real numbers)

prob Distribution Function

x

F A(x) = Prob (A x) =

Density Function

f A(x) dx

f (x)=Prob(A=x)
A

f (x)

FA

(x)

FA (x)
0
x
0

x
6

Expectation of Random Variable

A

If for Random Variables A,B

"x

F A (x) F B (x)

then
"A upper bounds B"
and
"B lower bounds A"

E(A) = A =

x f A(x) dx

A is als o called "av erage of A"

and "mean of A" = mA

FB

(X)

FA

(X)

F A (x) = Prob (A x)
F B (x) = Prob (B x)

Variance of Random Variable A

2

s A = (A-A) = A

- (A)

A2 =

where 2nd moment

n'th Moments of Random Variable A

f A(x) dx

moment generating function

MA(s ) =
f (x)
A

small

x 2 fA (x) dx

small variance

example

sx

f A(x) dx

( s A)

= E e
x

note

large variance

example

s is a formal parameter
n

large

A =

f (x)
A

d MA (s)
ds

n
s= 0

x
9

10

Discrete

Random Variable A

Discrete Random Variable A

over nonnegative integers

density function
f (x)

expectation E(A) = A =

= prob (A=x)

n' th moment A =

x f A(x)

f A(x)

x=0

x=0
1

probability generating function

. . .

G A(z) =

distribution function
x

F (x) =
A

i =0

x=0

f A(x) = E

( zA)

fA ( i ) = prob (A x)

1st derivative
2nd derivative

'

GA (1) = A
GA" (1) = A2 - A

variance s A2 = GA" (1) + GA(1) - (GA(1))2

1

...

11

12

A,B independent if
Prob(A B) = Prob(A)

Combinatorics

Prob(B)

equivalent definition of independence

f A B (x) = f

MA B (s) = M
GA B (z) = G
If A1 ,...,A n

independent

(x) f B (x)

= number of permutations of n objects

(s) M B(s)

(z) GB(z)

Stirling's formula
n! = f(n) (1+o(1))
n

for i=1 , . . . , n

where f(n) = n e

Then if B = A1 A2 . . . An

(M

A1

note
tighter bound

(
)
(s ) , G (z) = G
)
(
f A (x)

f B (x) =
MB (s ) =

2 1

with same distribution

f A (x) = f A (x)
1

n! = n (n-1)

f(n) e

A1

(z)

1
(12n+1)

-n

pn
2p

< n! < f(n) e

1
12n

n!
= number of permutations of n
(n-k)!
object s taken k at a time
13

14

( )
n
k

n!
k! (n-k)!

Bernoulli Variable
and 0 with prob 1-P
A i is 1 with prob P

= number of (unordered)
Binomial Variable

combinations of n objects

B is sum of n independent
Bernoulli variables A i
each with some probability p

taken k at a time
Bounds (due to Erdos & Spencer, p. 18)
2

( )
n
k

n e

procedure

k
k
2n 6n2

begin

(1-o(1))

k!

for k = o

end

BINOMIAL with parameters n,p

B0
for i=1
output

to n do
with probability P
B

do B B+1

()
n

3
4

15

16

B is Binomial Variable with parameters n,p

B 0
i 0

i i + 1
m
i>n

yes

output B

1-P

mean

random
choice

m = n p

variance

s = np (1-p )

d ensity fn =

B B+1

Prob(B=x) =

()
( nk) p
n
x

x) =
distribution fn = Prob(B
17

(1-p)
k

n-x

(1-p)

n-k

k=0

18

GENERATING FUNCTION

Poisson Trial
n

G(z) = (1-p+pz) =

zk

k=0

k
n -k
n
( k ) p (1-p)

Ai is 1 with prob Pi
and 0 with prob 1-Pi

interesting fact

( )

m) = W
Prob(B=m

1
n

Suppose B' is the

sum of n independent Poisson trials
A i with probability Pi for i > 1,...,n

19

20

Geometric Variable
Hoeffding's Theorem
B' is upper bound

V parameter p

"x 0

by a Binomial Variable
B

p(1-p)

procedure
GEOMETRIC parameter p
n

parameters n, p w here p =

Prob(V=x)

V 0
loop: with probability p
goto exit

Pi
i=1

begin

V
F

V+1

goto loop

B'

exit: output

1-p

m ean m =

21

1-p
p

output V
V

V+1

22

Probabilistic Inequalities
for Random Variable A

GENERATING FUNCTION

G(Z) =

k=0

(p(1-p) ) =

p
1-(1-p)Z

mean m = A
2

v ariance s = A - (A)

lower tail

D D

23

upper tail

24

Markov Inequality

(uses only

Prob (A x)
Chebychev Inequality

(uses

m
x

Gaussian Density function

mean )

mean and variance )

m| D ) s 2
Prob (|A-m
D

Y(x) =

1
P
2P

x
2

example
If B is a Binomial

"Bell Shaped" Curve

with parameters n,p

np
x) x
Then Prob (B
Prob (|B-np| D)

np (1-p)

m=0

s=1
25

26

Normal Distribution

F (X ) =

Let S n be the
Y (Y) dY

sum of n independently
distributed variables
A 1 ,...,A n

Bounds

"x > 0

(Feller, p. 175)

Y (x)

1
1
- 3
x x

2
m
s
variance
each w ith mean n and
n

Y(x)
1 - F(x)
x

"x e [0, 1]
1
x
= x Y(1) F (x) - x Y(0) =
2
Pe
2P

S o Sn has mean m and variance s

x
P
2P

27

28

Strong Law of Large Numbers

Chernoff Bound
of Random Variable A
(uses all moments)

The probability dens ity function of

Tn =

(S n - m)
s

limits as n

Prob (A x)

to normal dis tribution F (x)

e
e

Henc e Prob
m| sx) F (x) as n
(|S n-m

s o Prob
(|S n - m| sx) 2(1- F (x))
2 Y (x)/x

-s x

MA (s )

g ( s )-s x

for s 0

w here g (s ) = ln (MA(s ))

g (s ) - s g ' (s )

by s etting

x = g' (s )

1s t derivative minimizes bounds

need moment generating function

F (x) Y (x)/x)
(s ince 1-F
29

30

Chernoff Bounds

Chernoff Bound

for

of

Discrete

Binomial B

Random Variable A

with parameters n,p

-x

Prob (A x) z

choose z=z o to minimize above bound

Prob (B x)

need
Probability Generating

G A(z) =

x 0

( )

z f A(x) = E z

m
n-m
n-x

m
x-m

e
31

n-x

( ) ( )

function
x

x m

Abov e mean

G A ( z ) for z 1

m
x

( )

m
-x -m

m
x

s ince

for x m e

1 -

1
x

<

e-1

32

Anguin-Valiant's Bounds
for

Binomial B

Below Mean x m

with parameters n,p

Prob (B x)

Just above mean

n-x

( ) ()
m
n-m
n-x

m
x

m = np for 0<

e <1

Prob ( B (1 + e ) m ) e

-e 2

m
2

Just below mean

m for 0< e <1
m)
Prob (B ( 1 - e )m

33

-e 2m / 3

34

Binomial Variable B
with Parameters p, N
and expectation m = pN

-ee2 m / 2
e

(1- e)m

-ee2 m /3

By Chernoff
Bound for p < 1/2

(1+e)m

N
Prob ( B ) < 2 N p 2
2

fi tails are bounded by N o r m a l

distributions

Raghavan-Spencer

bound

For any

>0

Prob ( B (1 + ) m )

(1 + )(1+ )

in FOCS'86.
35

36