Probability

1 Sample Space and Events

Definition 1 A sample space, denoted S, is the collection of all outcomes when an
experiment is performed.

Example 1 Toss a coin two times, the sample space is S = {HH, HT, T H, T T } 4

Example 2 Toss a coin until a head appears, the sample space is

S = {H, T H, T T H, T T T H, . . .} 4

Definition 2 An event is a set outcomes form the sample space. The events are
usually denoted A, B, C, . . .. If we regard S as a set, then an event is a subset of S.

Union and intersection of events

- The union of two events A and B, denoted A B, is the set of all outcomes that are
either in A or in B

- The intersection of two events A and B, denoted A B, is the set of all outcomes
that are both in A and in B

- The complement of an event A, denoted A, is the set of all outcomes that are not in A

Most of elementary operations on sets can be expressed in terms of union, intersection

and complement.

Example 3 A \ B is the set of outcomes that are in A but not in B; it is left as an

exercise to show that A \ B = A B 4

Mutually exclusive and exhaustive events

Definition 3 A set of events A1 , A2 , A3 , . . . is mutually exclusive if and only if

Ai Aj = , i 6= j (1)

Definition 4 A set of events A1 , A2 , A3 , . . . is exhaustive if

A1 A2 A3 . . . = S (2)

Definition 5 A set of events A1 , A2 , A3 , . . . is mutually exclusive and exhaustive (called

also partition) if (1) and (2) are both satisfied.

1
2 Probability Basics
Axiom 1 A probability, denoted P is an application, P : S [0, 1], satisfying:

a. P () = 0, P (S) = 1, and

b. If A1 , A2 , A3 , . . . is a set of mutually exclusive events, then

P
P (A1 A2 A3 . . .) = P (Ai ) = P (A1 ) + P (A2 ) + P (A3 ) + . . .

Property 1 (basic rules)

a. (union rule) P (A B) = P (A) + P (B) P (A B)

b. (complement rule) P (A) = 1 P (A)

The proof of the above property is left as an exercise. Note that other basic rules of
probability can be expressed in terms of the the union and the complement rules.

Example 4 P (A \ B) = P (A) P (A B) 4

Proof: A = (A \ B) (A B) (mutually exclusive events)

P (A) = P (A \ B) + P (A B), hence P (A \ B) = P (A) P (A B).

Exercise 1 Let A, B and C be three events, show that

P (AB C) = P (A)+P (B)+P (C)P (AB)P (AC)P (B C)+P (AB C)
5

Inclusion-Exclusion formula:
More generally, if A1 , A2 , . . . , An are n events, then
X n Xn n
X
P (A1 A2 . . . An ) = P (Ai ) P (Ai Aj ) + P (Ai Aj Ak ) . . . +
i=1 i,j=1 i,j,k=1
n1
(1) P (A1 A2 . . . An )

Proposition 1 (not required for STAT230)

Let[An , n = 1, 2.3, . . . be a increasing set of events (A1 A2 A3 . . .), then
P ( An ) = lim P (An ). 2
n+
n

Proof: Let B1 = A1 , B2 = A2 A1 , B3 = AS 3 A2 , . . ., S
Bn = An An1 . The events
Bn , n = 1, 2.3, . . . are mutually exclusive, and n=1 Bn = +
+
n=1 An . By axiom 1,b),
S+ P+ Pk S k
P ( n=1 An ) = n=1 P (Bn ) = lim n=1 P (Bn ) = lim P ( n=1 Bn ) = lim P (Ak ).
k+ k+ k+

2
3 Counting
Definition 6 A selection of k objects out of n objects without replacement and with-
n!
out order is called combination, denoted Cnk , and defined by: Cnk = k!(nk)!

Definition 7 A selection of k objects out of n objects without replacement and with

n!
order is called permutation, denoted k Pn , and defined by: k Pn = (nk)!

Example 5 a) In how many different ways can a group of 2 persons be formed if 5

persons are available ? b) In how many different ways can a this be done if the group
must contain a president and a vice-president ? 4

Property 2 The number of permutations of n distinct objects is n!. 2

Property 3 (Basic counting rule) If an experiment has n1 outcomes, and if another

experiment has n2 outcomes, then the total number of outcomes is n1 n2 . 2

Exercise 2 In how many different ways can a team of 2 men and 3 women be formed
if 5 men and 7 women are available ? In how many ways can this be done if one man
and one woman refuse to be together in the team ? 5

Exercise 3 In how many different ways can a group of 10 persons be seated in a row
if 3 of them insist to be next to each other ? 5

Exercise 4 Find the probability of a full-house in a poker hand. 5

Permutations of indistinguishable objects

Property 4 The number of permutations of n objects, from which n1 are the same,
n2 are the same, ..., and nk are the same (n1 + n2 + . . . + nk = n) is
n!
2
n1 ! n2 ! . . . nk !
n2
Proof: There are Cnn1 choices for the first n1 objects, and Cnn 1
choices for the
nk
n2 objects, ..., and Cnn1 n2 ...nk1 choices for the nk objects. By the basic counting
rule, the total number of choices is
n2 nk n!
Cnn1 Cnn . . . Cnn 1 n2 ...nk1
=
1
n1 ! n2 ! . . . nk !

Example 6 How many different words can be formed by using the letters of the word
MISSISSIPPI ? 4

Exercise 5 In how many different ways can 15 students be seated in 3 classes of size
5 each ? In how many ways can this be done if two of the students, Asma and Maya,
want to be in the same class room ? 5

3
4 Conditional Probability
Definition 8 Let A and B be two events with P (B) > 0. The conditional probability
of A given B, denoted by P (A|B), is defined by
P (A B)
P (A|B) =
P (B)

Example 7 5 cards are drawn at random and without replacement from a deck of 52
playing cards. Find the probability that all 5 cards are spades if at least 3 cards are
spades. 4

Solution: Let A={event all 5 cards are spades}, and B={at least 3 cards are
spades}
5
P (A B) P (A) C13
P (A|B) = = = 3 2 4 1 5
P (B) P (B) C13 C39 + C13 C39 + C52

Proposition 2 The conditional probability P (|B) is itself a probability. 2

Proof:
P ( B) P () P (S B) P (B)
a. P (|B) = = = 0 ; P (S|B) = = =1
P (B) P (B) P (B) P (B)
b. Let A1 , A2 , A3 , . . . be mutually exclusive events,
P ((A1 A2 A3 . . .) B)
P (A1 A2 A3 . . . |B) =
P (B)
P ((A1 B) (A2 B) (A3 B) . . .)
=
P (B)
P (A1 B) P (A2 B) P (A3 B)
= + + + ...
P (B) P (B) P (B)
= P (A1 |B) + P (A2 |B) + P (A3 |B) + . . .

The previous proposition says that all rules of probability still apply to conditional
probability; for example, P (A|C) = 1 P (A|C)

Property 5 (Multiplication rule)

a. P (A B) = P (A|B) P (B) = P (B|A) P (A), and more generally

b. P (A1 . . .An ) = P (A1 )P (A2 |A1 )P (A3 |A1 A2 ) . . .P (An |A1 A2 . . .An1 )
2

Example 8 Cards are drawn at random and without replacement from a deck of 52
cards. Find the probability that the sixth spade is drawn at the tenth draw. 4

4
Solution: Let A={event five spades in the first nine cards}, and B={sixth card is
spade}
5 4
C13 C39 8
P (A B) = P (A) P (B|A) = 9

C52 43

5 Independence
Definition 9 Two events A and B are independent if and only if P (A|B) = P (A) or
P (B|A) = P (B).
Proposition 3 Two events A and B are independent if and only if P (A B) =
P (A) P (B) 2
Property 6 If A and B are independent, then so are
a) A and B b) A and B c) A and B 2
Proof: a) P (B|A) = 1 P (B|A) = 1 P (B) = P (B)
parts b) and c) are left as an exercise.

Proposition 4 If n events A1 , A2 , . . . , An are mutually independent, then

P (A1 A2 . . . An ) = P (A1 ) P (A2 ) . . . P (An ) 2
Note that the events A1 , A2 , . . . , An can be ind two by two but not mutually indepen-
dent.
Example 9 A fair coin is tossed twice. Let
A1 ={first toss is head}, A2 ={second toss is head}, A3 ={the two tosses show up the
same face}

P (A1 ) = P (A2 ) = P (A3 ) = 12 , and

1 1
P (A1 A2 A3 ) = P (A1 A2 ) = 4
6= 8
= P (A1 )P (A2 )P (A3 ), hence A1 , A2 , A3 are
not mutually independent.

But P (A1 A2 ) = 14 = P (A1 )P (A2 ),

P (A2 A3 ) = P (A2 A1 ) = 14 = P (A2 )P (A3 ).
P (A1 A3 ) = P (A1 A2 ) = 14 = P (A1 )P (A3 )
hence A1 , A2 , A 3 are independent two by two. 4

6 Bayes Rule
Proposition 5 (Total probability formula) Let A1 , A2 , . . . , An be a set of mutually
exclusive and exhaustive events, with P (Ai ) > 0 for i = 1, 2, . . . , n. Then for any event
B,
X n
P (B) = P (Ai )P (B|Ai )
i=1
2

5
Proof: P (B) = P (B S) = P (B (A1 A2 . . . An ))

= P ((B A1 ) (B A2 ) . . . (B An ))

= P (B A1 ) + P (B A2 ) + . . . + P (B An )

= P (B|A1 )P (A1 ) + P (B|A2 )P (A2 ) + . . . + P (B|An )P (An )

A1 A2
B
A3

A4 ...

Proposition 6 (Bayes rule) Let A1 , A2 , . . . , An be a set of mutually exclusive and

exhaustive events, with P (Ai ) > 0 for i = 1, 2, . . . , n. Let B be any event with
P (B) > 0, then
P (B|Ai )P (Ai )
P (Ai |B) = n
X
P (Ai )P (B|Ai )
i=1
2
P (Ai B) P (B|Ai )P (Ai ) P (B|Ai )P (Ai )
Proof: P (Ai |B) = = = n
P (B) P (B) X
P (B|Ai )P (Ai )
i=1

Example 10 (Urn model)

Three balls are selected at random and without replacement from an urn U1 containing
5 white balls and 4 blue balls, and put in an empty urn U2 . A ball is then drawn from
U2 , its blue. Find the probability that all three balls transferred from U1 to U2 were
blue. 4

Solution: Let Bi ={event i blue balls selected from U1 } (i = 0, 1, 2, 3), and

B={event blue ball selected from U2 }

P (B|B3 )P (B3 )
P (B3 |B) =
P (B|B0 )P (B0 ) + P (B|B1 )P (B1 ) + P (B|B2 )P (B2 ) + P (B|B3 )P (B3 )
1 C43 3
= 3 1 1 2 2 2 1 3
=
0 C5 + 3 C4 C5 + 3 C4 C5 + 1 C4 28