Stochastic Systems Chapter 1-2

Chapter 1-2
Intro. to Probability Theory

ECEN/MAE 5513
School of Electrical and Computer Engineering

Oklahoma State University
Textbook
 Probability and Random Processes (2nd Edition)
 Authors: Scott L. Miller and Donald Childers
 Publisher: Academic Press 978-0-12-386981-4
Prof. Scott L. Miller Prof. Donald Childers

ECE@TAMU ECE@UFL
2
Two Kinds of Signals
 A deterministic signal is one which may be represented or
perfectly reconstructed by some parameter values.
Where does
it come from?
 A non-deterministic signal has an element of randomness

which prevents them from being perfectly reconstructed.
3
Speech Signals
4
Probability Theory: Applications
5
Probability Theory: Origination
“Concepts of probability have been

around for thousands of years, but
probability theory did not arise as a
branch of mathematics until the mid-
seventeenth century.”
http://www.teacherlink.org/content/math/interactive/probability/history/briefhistory/home.html
6
Game of Chance
 We often use a game of chance to illustrate the
ideas of probability where the outcome is
strongly influenced by some randomizing device.
Experiment An experiment (E) is a procedure we perform that produces some result.
For example, the experiment E5 consists of tossing a coin five times.
An outcome () is a possible result of an experiment.

Outcome For example, the outcome 1 of experiment E5 represent the sequence of
tosses Heads-Heads-Tails-Heads-Tails (HHTHT).
An event (denoted by C) is a certain set of outcomes of an experiment.

Event For example, the event C associated with experiment E5 might be
C={all outcomes consisting of an even number of heads}.
The sample space (S) is the collection of set of “all possible” distinct
Sample Space
outcomes of an experiment.
7
Examples
 Example 2.1
 Consider the example of flipping a fair coin,
where fair means that the coin is not biased
in weight to a particular side. There are two
possible outcomes, namely, a head or a tail.
Thus, the sample space, S, consists of two
outcomes, 1=H and 2=T.
 Example 2.2
 A cubical die with numbered faces is rolled
and the result observed. The sample space
consists of six possible outcomes, 1=1,
2=2, 3=3, 4=4, 5=5, 6=6, indicating the
possible faces of the cubical die.
8
Examples
 Example 2.3:
 Consider the experiment of rolling two dices and observing the results.
 The sample space consists of 36 outcomes, which may be labeled by the
ordered pairs 1  {1,1},  2  {1,2},  3  {1,3},...,  6  {1,6},....,  36  {6,6}
 Many events can be defined from this experiment:

 A={the sum of the outcomes of the two rolls =4}
{1,3}, {2,2}, {3,1}
 B={the outcomes of the two rolls are identical}
{1,1}, {2,2}, {3,3}, {4,4}, {5,5}, {6,6} Arithmetic Series
 C={the first roll was bigger than the second} an  a1  (n  1)d
{2,1}, {3,1 : 2}, {4,1 : 3}, {5,1 : 4}, {6,1 : 5} n(a1  an )
Sn  (sum)
1  2  3  4  5  15 2
9
Examples
 Example 2.4
 Let us flip a coin until a tails occurs. The
experiment is then terminated.
 The sample space consists of a collection of

sequences of coin tosses, and the possible
outcomes that may occur are
1  {T},  2  {H, T},  3  {H, H, T},... Geometric Series
an  a1  r n 1
 Questions
a1 (1  r n )
 How many possible outcomes in the sample space? Sn 
 What are the probabilities of each outcome to occur? 1 r
1 1 1  
1
 Pr( n )  
1
Pr(1 )  Pr( 2 )  Pr( 3 )  Pr( n )  n n 1
2 4 8 2 n 1 n 1 2
10
x  a  a  x  a
Examples x  a  x  a or x  a
 Example 2.5
 Consider a random number generator which selects a number in an
arbitrary manner from the semi-closed interval [0,1).
 The sample space consists of all real numbers, for which 0x<1.
 We can define events on a continuous space as well

A  {x  1 / 2} B  { x  1 / 2  1 / 4}
1
x  1/ 2  1/ 4
C  {x  1 / 2} 3/4
 1/ 4  x  1/ 2  1/ 4
C B
1/ 4  x  3 / 4 1/2
B  {1 / 4  x  3 / 4}
1/4 A
 Questions
0
 What is the probability of each event?
11
Definitions of Game of Chance
Experiment, Outcome, Event and Sample Space
 Definition 2.1:
Experiment
 An experiment (denoted by E) is a procedure we
perform that produce some result.
 Definition 2.2:
 An outcome (denoted by ) is a possible result of an
experiment. Outcome
 Definition 2.3:
 An event (denoted by C) is a certain set of outcomes of
an experiment. Event
 Definition 2.4:
 The sample space (denoted by S) is the collection of set
of “all possible” distinct outcomes of an experiment. Sample Space
12
Set Theory
 It is the branch of mathematics that deals with the formal
properties of sets as units and where three basic operations
are defined: intersection, union and complement.
Intersection Union
Complement Complement
http://www.edwardbosworth.com/CPSC2105/MyTextbook2105_HTM/MyText2105_Ch04_V06.htm
13
Set Theory: Associative Law
𝐴 ∪ 𝐵 ∪ 𝐶 = (𝐴 ∪ 𝐵) ∪ 𝐶 = 𝐴 ∪ (𝐵 ∪ 𝐶)
𝐴 ∩ 𝐵 ∩ 𝐶 = (𝐴 ∩ 𝐵) ∩ 𝐶 = 𝐴 ∩ (𝐵 ∩ 𝐶)
S S
B B
A A
C C
Union Intersection
14
Set Theory: Distributive Law
𝐴 ∩ (𝐵 ∪ 𝐶) = (𝐴 ∩ 𝐵) ∪ (𝐴 ∩ 𝐶)
𝐴 ∪ (𝐵 ∩ 𝐶) = (𝐴 ∪ 𝐵) ∩ (𝐴 ∪ 𝐶)
S S
B B
A A
C C
Left Right
15
Set Theory: De Morgan’s Law
 The complement of a union (intersection) of two sets equals
the intersection (union) of the individual complements.
𝐴 ∪ 𝐵 = ഥ𝐴 ∩ 𝐵ത = {Orange portion}
𝐴 ∩ 𝐵 = ഥ𝐴 ∪ 𝐵ത
S S
A B A B
16
Set Theory: Example
 To verify De Morgan’s Law: (𝐴 ∩ 𝐵) = ഥ𝐴 ∪ ഥ
𝐵
 Given S = {2 < s  24}, A = {2 < a 16}, B = {5 < b  22}
C C  A B C
A B
B B
A
C  A  B  {5  c  16} C  A  B {2  c  5, 16  c  24}
A  {16  a  24} B  {2  b  5, 22  b  24}
A  B  {2  b  5, 16  b  24}  C  ( A  B)
17
Two Key Elements in Probability
 Element 1: Rules of probabilistic computation
 Axioms, theorems, and corollaries of probability.
Theorems Theorems Theorems

Corollary
Axioms
 Element 2: Probability of atomic outcomes

 Finding atomic outcomes and assigning their probabilities.
Atomic Atomic Atomic Atomic

outcome outcome outcome outcome
18
Axioms
 Probability is a measure of the likelihood of various events, and it
is a function of an event that produces a numerical quantity that
measures the likelihood of that event.
 Axioms
 Axiom 2.1: For any event A, Pr 𝐴 ≥ 0.
 Axiom 2.2: If S is the same space for a given experiment, Pr(𝑆) = 1.
 Axiom 2.3a: If 𝐴 ∩ 𝐵 = ∅, then Pr 𝐴 ∪ 𝐵 = Pr 𝐴 + Pr(𝐵).
 Axiom 2.3b: If 𝐴𝑖 ∩ 𝐴𝑗 = ∅, for all 𝑖 ≠ 𝑗, then Pr ‫∞ڂ‬ 𝐴
𝑖=1 𝑖 = σ ∞
𝑖=1 Pr(𝐴𝑖 ).
S S S
𝐴2
𝐴3
A A B
𝐴1 𝐴4
19
Corollary and Theorems
 Corollary
 Corollary 2.1 𝐴𝑖 ∩ 𝐴𝑗 = ∅, for all 𝑖 ≠ 𝑗, then Pr ‫𝑀ڂ‬ 𝑀
𝑖=1 𝐴𝑖 = σ𝑖=1 Pr(𝐴𝑖 )
S
𝐴2
𝐴3
𝐴1 𝐴4
 Theorems
 Theorem 2.1 Pr 𝐴 ∪ 𝐵 = Pr 𝐴 + Pr 𝐵 − Pr 𝐴 ∩ 𝐵 .
 Theorem 2.2 S = A ∪ ഥ𝐴, Pr 𝐴ҧ = 1 − Pr(𝐴).
S S
Pr 𝐴 + Pr 𝐴ҧ = 1
A B A 𝐴ҧ S = A + ഥ𝐴 A ∩ ഥ𝐴 = ∅
Pr(𝑆) = 1
20 𝐴∩𝐵
Proof of Theorem 2.1
Pr 𝐴 ∪ 𝐵 = Pr 𝐴 + Pr 𝐵 − Pr 𝐴 ∩ 𝐵
Axiom 2.3a: If 𝐴 ∩ 𝐵 = ∅, then Pr 𝐴 ∪ 𝐵 = Pr 𝐴 + Pr(𝐵).

Corollary 2.1: 𝐴𝑖 ∩ 𝐴𝑗 = ∅, for all 𝑖 ≠ 𝑗, then Pr ‫𝑀ڂ‬
𝑖=1 𝐴 𝑖 = σ𝑀
𝑖=1 Pr(𝐴𝑖 )
Can we represent 𝐴 ∪ 𝐵 by the union of some mutually exclusive sets?

S 𝐴 ∪ 𝐵= 𝐴 ∩ 𝐴 ∩ 𝐵 ∪ 𝐴 ∩ 𝐵 ∪ 𝐵 ∩ 𝐴 ∩ 𝐵
A B 𝐴 = (𝐴 ∩ 𝐴 ∩ 𝐵) ∪ (𝐴 ∩ 𝐵)
Axiom 2.3a
Pr(𝐴) = Pr(𝐴 ∩ 𝐴 ∩ 𝐵) + Pr(𝐴 ∩ 𝐵)

𝐴∩𝐵 Pr(𝐴 ∩ 𝐴 ∩ 𝐵)= Pr 𝐴 − Pr(𝐴 ∩ 𝐵)
𝐵∩𝐴∩𝐵
𝐴∩𝐴∩𝐵 Pr(𝐵 ∩ 𝐴 ∩ 𝐵)= Pr 𝐵 − Pr(𝐴 ∩ 𝐵)
Pr(𝐴 ∪ 𝐵) = Pr 𝐴 ∩ 𝐴 ∩ 𝐵 + Pr 𝐴 ∩ 𝐵 + Pr 𝐵 ∩ 𝐴 ∩ 𝐵 = Pr 𝐴 + Pr 𝐵 − Pr 𝐴 ∩ 𝐵
21
Assigning Probability:
Atomic outcomes
 In many experiments, it is possible to specify all of the
outcomes of the experiment in terms of some fundamental
outcomes which we refer to as atomic outcomes.
 Some Examples:
 Atomic outcomes of tossing a cubical dice are six numbers.
 Atomic outcomes of tossing a coin are Heads and Tails.
 Atomic outcomes are the most basic events that cannot be simpler.
 We can build more complicated and more interesting events.
 Often we can justify assigning equal probabilities to all atomic outcomes.
 Given the M atomic outcomes of an experiment E 1 ,  2 ,...,  M
which are mutually exclusive, exhaustive, and equally probable,
then we assign Pr( i )  1 / M . Why?
Pr(S )  Pr(1   2  ...   M )  Pr(1 )  Pr( 2 )  ...  Pr( M )  1
22
Examples
 Example 2.6
 Consider the coin flipping experiment where there are two atomic
outcomes, i.e., 1  H ,  2  T, which are equally probable. Therefore,
these events should be assigned probabilities of Pr(1 )  Pr( 2 )  1 / 2.
 Example 2.7
 Consider a dice rolling experiment where there are six atomic
outcomes, i.e., Pr(1)  Pr(2)...  Pr(6)  1 / 6.
 Then we can determine the probability of more complicated events, such
Pr(even number is rolled)  Pr(2  4  6)
 Pr(2)  Pr(4)  Pr(6)
 1/ 6 1/ 6 1/ 6
 1/ 2
23
Examples
 Example 2.8
 A pair of dice were rolled.
 How many atomic outcomes in this experiment and what are their probability?
Pr(1,1)  Pr(1,2)...  Pr(6,6)  1 / 36.

 What is the probability of the event A={sum of two dice =5}
A  {(1,4), (2,3), (3,2), (4,1)}
Pr( A)  Pr(1,4)  (2,3)  (3,2)  (4,1)
 Pr 1,4   Pr 2,3  Pr 3,2   Pr 4,1
 1 / 36  1 / 36  1 / 36  1 / 36
 1/ 9
24
Examples
 Example 2.8 (Cont’d)
 A pair of dice were rolled.
 What is the probability of the event B={the 1st dice is greater than the 2nd)?
B  {(2,1), (3,1 : 2), (4,1 : 3), (5,1 : 4), (6,1 : 5)}
15 5
1  2  3  4  5  15 Pr( B )  
36 12
 What is the probability of the event C = 𝐴 ∩ 𝐵?
B  {(2,1), (3,1 : 2), (4,1 : 3), (5,1 : 4), (6,1 : 5)} C  {(3,2), (4,1)}
2 1
A  {(1,4), (2,3), (3,2), (4,1)} Pr( B)  
36 18
 What is the probability of the vent D = 𝐴 ∪ 𝐵?
Pr( D)  Pr( A  B)  Pr( A)  Pr( B)  Pr( A  B)
1 5 1 17
   
9 12 18 36
25
Prob. 2.18
 We flip a coin until the first occurrence of tails or until we flip
the coin four times, whichever comes first.
 List the possible outcomes of this experiment. How many are there?
Would you expect each outcome to be equally probable?
{T } {HT } {HHT } {HHHT } {HHHH }
 These outcomes are NOT equally probable.
 Can you develop a method to use the classical approach to assign
probabilities to this experiment (using atomic outcomes)?
 In the classical approach, each outcome is equally probable. Consider a new
experiment that involves four coin tosses. What are the possible outcomes?
{HHHH }, {HHHT }, {HHTH }, {HHTT }, {HTHH }, {HTHT }, {HTTH }, {HTTT } Pr(.)  1 / 16
{THHH }, {THHT }, {THTH }, {THTT }, {TTHH }, {TTHT }, {TTTH }, {TTTT }
 We combine those outcomes to correspond to the previous five outcomes.
Pr{T }  8 / 16  1 / 2 Pr{HHT }  2 / 16  1 / 8 Pr{HHHT }  Pr{HHHH }  1 / 16
Pr{HT }  4 / 16  1 / 4 Does this assignment satisfy the axioms of probability?
26
Goals
 To review the basics of probability theory.
 To define the joint probability of two events.
 To define the conditional probability of two events.
27
Axioms, Corollary and Theorems
 Probability is a measure of the likelihood of various events, and it
is a function of an event that produces a numerical quantity that
measures the likelihood of that event.
 Axioms
 Axiom 2.1: For any event A, Pr 𝐴 ≥ 0.
 Axiom 2.2: If S is the same space for a given experiment, Pr(𝑆) = 1.
 Axiom 2.3a: If 𝐴 ∩ 𝐵 = ∅, then Pr 𝐴 ∪ 𝐵 = Pr 𝐴 + Pr(𝐵).
 Axiom 2.3b: If 𝐴𝑖 ∩ 𝐴𝑗 = ∅, for all 𝑖 ≠ 𝑗, then Pr ‫∞ڂ‬ 𝐴
𝑖=1 𝑖 = σ ∞
𝑖=1 Pr(𝐴𝑖 ).
 Corollary and Theorems

 Corollary 2.1 𝐴𝑖 ∩ 𝐴𝑗 = ∅, for all 𝑖 ≠ 𝑗, then Pr ‫𝑀ڂ‬ 𝑀
𝑖=1 𝐴𝑖 = σ𝑖=1 Pr(𝐴𝑖 )
 Theorem 2.1 Pr 𝐴 ∪ 𝐵 = Pr 𝐴 + Pr 𝐵 − Pr 𝐴 ∩ 𝐵 .
 Theorem 2.2 S = A + ഥ𝐴, Pr 𝐴ҧ = 1 − Pr(𝐴).
28
Problem Solving Exercise (2.9)
 Prove Pr 𝐴 ∪ 𝐵 ∪ 𝐶 = Pr 𝐴 + Pr(𝐵) + Pr 𝐶
− Pr 𝐴 ∩ 𝐵 − Pr 𝐴 ∩ 𝐶 − Pr 𝐵 ∩ 𝐶 + Pr 𝐴 ∩ 𝐵 ∩ 𝐶
Pr 𝐴 ∪ 𝐵 = Pr 𝐴 + Pr 𝐵 − Pr 𝐴 ∩ 𝐵 (Theorem 2.1)
Pr 𝐴 ∪ 𝐵 ∪ 𝐶
= Pr 𝐴 ∪ 𝐵 ∪ 𝐶 𝐴 ∩ (𝐵 ∪ 𝐶) = (𝐴 ∩ 𝐵) ∪ 𝐴 ∩ 𝐶 (Distributive)
= Pr 𝐴 ∪ 𝐵) + Pr (𝐶 − Pr 𝐴 ∪ 𝐵 ∩ 𝐶
= Pr 𝐴 + Pr 𝐵 − Pr 𝐴 ∩ 𝐵 + Pr 𝐶 − Pr( 𝐴 ∩ 𝐶 ∪ 𝐵 ∩ 𝐶 )
Pr( 𝐴 ∩ 𝐶 ∪ 𝐵 ∩ 𝐶 )=Pr 𝐴 ∩ 𝐶 + Pr 𝐵 ∩ 𝐶 − Pr 𝐴 ∩ 𝐶 ∩ 𝐵 ∩ 𝐶
= Pr 𝐴 ∩ 𝐶 + Pr 𝐵 ∩ 𝐶 − Pr 𝐴 ∩ 𝐵 ∩ 𝐶
= Pr 𝐴 + Pr 𝐵 +Pr 𝐶 − Pr 𝐴 ∩ 𝐵 − Pr 𝐴 ∩ 𝐶 − Pr 𝐵 ∩ 𝐶 + Pr(𝐴 ∩ 𝐵 ∩ 𝐶)
29
Prob. 2.18
 We flip a coin until the first occurrence of tails or until we flip
the coin four times, whichever comes first.
 List the possible outcomes of this experiment. How many are there?
Would you expect each outcome to be equally probable?
{T } {HT } {HHT } {HHHT } {HHHH }
 These outcomes are NOT equally probable.
 Can you develop a method to use the classical approach to assign
probabilities to this experiment (using atomic outcomes)?
 In the classical approach, each outcome is equally probable. Consider a new
experiment that involves four coin tosses. What are the possible outcomes?
{HHHH }, {HHHT }, {HHTH }, {HHTT }, {HTHH }, {HTHT }, {HTTH }, {HTTT } Pr(.)  1 / 16
{THHH }, {THHT }, {THTH }, {THTT }, {TTHH }, {TTHT }, {TTTH }, {TTTT }
 We combine those outcomes to correspond to the previous five outcomes.
Pr{T }  8 / 16  1 / 2 Pr{HHT }  2 / 16  1 / 8 Pr{HHHT }  Pr{HHHH }  1 / 16
Pr{HT }  4 / 16  1 / 4 Does this assignment satisfy the axioms of probability?
30
Problem Solving Exercise (Prob. 2.16)
 We are given a number of darts. Suppose it is known that each
time we throw a dart at a target, we have a probability of 1/4
of hitting the target. An experiment consists of throwing three
darts at the target and observing the sequence of hits and
misses (e.g., one possible outcome might be (H, M, M)).
 (a) Find a probability assignment for the eight outcomes of this
experiment that leads to a probability of 1/4 of hitting the target on any
toss. Note, your assignment must satisfy the axioms of probability.
{HHH } {HHM } {HMH } {HMM }

 {MHH
(b) Is the}probability {MMH } {MMM
{MHM } assignment } unique? That is, are there
you chose
other possible probability assignments that lead to a probability of 1/4 of
hitting the target on any toss, or is yours the only valid assignment?
Carefully prove your answer.
31
Problem Solving Exercise (Prob. 2.16)
 Solution
 First, we group eight outcomes into four based on their probability.
Pr{HHH }  x
Pr{HHM }  Pr{MHH }  Pr{HMH }  y
Pr{HMM }  Pr{MHM }  Pr{MMH }  z
 Now, wePr{  wconstraints for those four probability values.
MMMa}few
develop
 According to Axiom 2.1, we have
  3 yhad
According to the condition that anyxtoss  3azprobability
 w  1 ¼ to hit the target
 First toss
x  2 y  z  1/ 4
 Second toss
How many constraints and
 Third toss x  2 y  z  1/ 4 how many variables?
 x  2 y  z  1/ 4
Any other constraint? Therefore, solutions are not unique.
0  x  y  z  1/ 4
32
Joint Probability
 For two events A and B, the probability Pr 𝐴, 𝐵 = Pr(𝐴 ∩ 𝐵)
is called joint probability
Pr( A  B)  Pr( A)  Pr( B)  Pr( A  B) Theorem 2.1
Pr( A, B)  Pr( A  B)  Pr( A)  Pr( B)  Pr( A  B)
A B   S A B   S
A B A B
33
Joint Probability: Example 2.11
Spade Heart
Diamond Club
 Given a deck of 52 playing cards that are sufficiently shuffled

and one card is drawn from the deck. Define the events
 A={red card selected}, B={number card selected}, C={heart selected)
Pr( A)  26 / 52  1 / 2 Pr( B )  40 / 52  10 / 13 Pr(C )  13 / 52  1 / 4
 Now let’s find the joint probabilities
Pr( A, B) 20 / 52  5 / 13 (20 outcomes are in common between A and B)
Pr( B, C )  10 / 52  5 / 26 (10 outcomes are in common between B and C)
Pr( A, C )  13 / 52  1 / 4 C  A  C  A  C  Pr(C  A)  Pr(C )

34
Conditional Probability
 Definition 2.5:
 For two events A and B, if the probability of A may depend on B, then
P(A│B) is called conditional probability, defined as S
Pr  A, B 
Pr  A B  
A B
with Pr(B) > 0 A B
Pr B 
 Or we can compute joint probabilities Pr  A B   Pr  A, B 
Pr  A, B   Pr  A  B   Pr  A B  Pr B 
 This idea can be extended to more than two events as
Pr  A, B, C   Pr C | A, B  Pr  A, B   Pr C A, B  Pr B | A Pr( A)
 Or in general for M events, Chain rule (the product rule) Pr  A1 , A2 , A3 
Pr  A1 , A2 ,..., AM   Pr  A1 , A2 
Pr  AM | A1 ,..., AM 1  Pr  AM 1 | A1 ,..., AM  2 .... Pr  A4 | A1 , A2 , A3  Pr  A3 | A1 , A2  Pr  A2 | A1  Pr  A1 

35
Conditional Probability:
Example 2.12
 We select two cards at random from the deck. When we select the
second card, we do not return the first card to the deck. We define
two events, A={1st card was a spade} and B={2nd card was a spade}.
 What is the joint probability Pr(𝐴, 𝐵)?
 How many possible outcomes of selecting two cards?
 There are 52*51 possible outcomes.
 How many possible outcomes that belong to the joint event 𝐴 ∩ 𝐵
 There are 13*12 possible outcomes.
 Then the joint probability Pr  A, B   (13 * 12) /(52 * 51)  1 / 17
 What is the conditional probability Pr(𝐵|𝐴)?
Pr B | A  Pr  A, B  / Pr  A  (1 / 17) /(1 / 4)  4 / 17
 Or we can find Pr(𝐵|𝐴) directly and then Pr(𝐴, 𝐵) as follows
12 4 4 1
Pr B | A   Pr  A, B   Pr B | A* Pr  A  *  1 / 17
51 17 17 4
36
Conditional Probability:
Example 2.13
 In a game of poker you are deal 5 cards from a 52-card deck
 What is the probability you are dealt a flush in spades?
 What is the probability you are dealt a flush in any suit?
 Let 𝐴𝑖 = {𝑖th card dealt to us is a spade}, 𝑖 = 1,2, … , 5
Pr  A1 , A2 , A3 , A4 , A5   Pr  A5 | A1 , A2 , A3 , A4  Pr  A1 , A2 , A3 , A4   (9 / 48) * (11 / 4165)  33 / 66640
Pr  A4 | A1 , A2 , A3  Pr  A1 , A2 , A3   (10 / 49) * (11 / 850)  11 / 4165
Pr  A3 | A1 , A2  Pr  A1 , A2   (11 / 50) * (1 / 17)  11 / 850
Pr  A2 | A1  Pr  A1   (12 / 51) * (1 / 4)  1 / 17
 To find the probability of being dealt a flush in any suite
Pr flush   Pr flush in spades   Pr flush in hearts   Pr flush in diamonds  Pr flush in clubs
Pr flush   4 * Pr flush in spades  
4 * 33
 0.001981
66640
37
Prob. 2.23
 A box of 30 diodes is known to contain five defective ones. If
two diodes are selected at random without replacement, what
is the probability that at least one of these diodes is defective?
 Let’s define A={first is defective}, B={first is NO defective}, C={second is
defective}, D={second is NOT defective}, E={at least one is defective}.
E   A  D   B  C    A  C 
Corollary 2.1
Pr( E )  Pr( A, D)  Pr( B, C )  Pr( A, C )
Pr( E )  Pr( D | A) Pr( A)  Pr(C | B) Pr( B)  Pr(C | A) Pr( A)

5 25 25 5 5 4 270 9
Pr( E )     
30 29 30 29 30 29 870 29
38
Prob. 2.23 Revisited
 Or, we define E={at least one is defective}, F={both are not defective}
EF
Theorem 2.2
Pr( E )  1  Pr( F )  1  Pr( B, D)

 1  Pr( D | B) Pr( B)
25 24 20 9
 1  1 
30 29 29 29
39
Prob. 2.24
 Two balls are selected sequentially (without replacement) from
an urn containing three red, four white, and five blue balls.
 (a) What is the probability that the first is red and the second blue?
 Let’s define A={1st is red}, B={2nd is blue}
1 5 5
Pr( A, B)  Pr( B | A) Pr( A)   
4 11 44
 (b) What is the probability of selecting a white ball on the second draw if
the first ball is replaced before the second is selected?
4 1
 B={2nd is White} Pr( B)  
12 3
 (c) What is the probability of selecting a white ball on the second draw if
the first ball is not replaced before the second is selected?
 Let’s define A={1st is white}, B={1st is not white}, C={2nd is white}
C   A  C   B  C   Pr(C )  Pr( A, C )  Pr( B, C )
 Pr(C | A) Pr( A)  Pr(C | B) Pr( B)
40
Joint/Conditional Probability
 For two events A and B, the probability Pr 𝐴, 𝐵 = Pr(𝐴 ∩ 𝐵)
is called joint probability
Theorem 2.1
Pr( A  B)  Pr( A)  Pr( B)  Pr( A  B) S
Pr( A, B )  minPr( A), Pr( B )  A
A B
B
Pr( A, B)  Pr( A  B)  Pr( A)  Pr( B)  Pr( A  B)

 For two events A and B, if the probability of A may depend on
B, then Pr(𝐴│𝐵) is called conditional probability, defined as
Pr  A, B  with Pr 𝐵 > 0 Pr B A  Pr  A, B 
Mutually exclusive
Pr A B  
Pr B  Pr A B   Pr  A, B  Pr  A
S
A B
Pr  A, B   Pr  A B  Pr B 
Chain rule (the product rule) What about this case?
Pr  A1 ,..., AM   Pr  AM | A1 ,..., AM 1  Pr  AM  1 | A1 ,..., AM  2 .... Pr  A2 | A1  Pr  A1 

41
Prob. 2.23
E   A  D   B  C    A  C 
Corollary 2.1
Pr( E )  Pr( A, D)  Pr( B, C )  Pr( A, C )
Pr( E )  Pr( D | A) Pr( A)  Pr(C | B) Pr( B)  Pr(C | A) Pr( A)

5 25 25 5 5 4 270 9
Pr( E )     
30 29 30 29 30 29 870 29
42
Prob. 2.23 Revisited
 Or, we define E={at least one is defective}, F={both are not defective}
EF
Theorem 2.2
Pr( E )  1  Pr( F )  1  Pr( B, D)

 1  Pr( D | B) Pr( B)
25 24 20 9
 1  1 
30 29 29 29
43
Basic Combinatorics
 In many cases, finding the probability of certain event, A,
reduces to an exercise in counting.
Pr  A 
Number of outcomes in A
Number of outcomes in entire sample space
 Principle of counting
 For a combined experiment, 𝐸 = 𝐸1 ⋅ 𝐸2 where experiment 𝐸1 has 𝑛1
possible outcomes and experiment 𝐸2 has 𝑛2 possible outcomes, the
total number of possible outcomes in E is 𝑛 = 𝑛1 ∗ 𝑛2 .
 Example 2.14
 Suppose we form a two-digit word by selecting a letter from the set
{A,B,C,D,E,F} followed by a number from the set {1,2,3}. All possible
combinations are enumerated in the following array
 A1 B1 C1 D1 E1 F1
A2 B2 C2 D2 E2 F2
  The number of total outcomes is 6*3=18
 A3 B3 C3 D3 E3 F3
44
Theorems
 Theorem 2.4 (Combined experiment):
 A combined experiment, 𝐸 = 𝐸1 ⋅ 𝐸2 ⋅⋅⋅ 𝐸𝑚 , consisting of experiments 𝐸𝑖
each with 𝑛𝑖 outcomes, 𝑖 = 1, … , 𝑚, has a total number of possible
outcomes given by 𝑛 = 𝑛1 𝑛2 𝑛3 … . 𝑛𝑚 = ς𝑚 𝑖=1 𝑛𝑖 .
 Theorem 2.5 (Permutations)

 The number of permutations of n distinct elements is
n! n(n  1)(n  2)...3  2 1 How many different
ways can a deck of 52
poker cards be shuffled?
 Theorem 2.6 (k-permutations)
 The number of k-permutations of n distinct elements is given by
n! How many different ways can
 n(n  1)(n  2)...(n  k  1) 5 cards out of a deck of 52
(n  k )! poker cards be shuffled?
45
Theorems
 Theorem 2.7 (Combinations)
 The number of distinct subsets (regardless of order) consisting of k out
of n distinct elements is
How many different sets of 5
n n!
   cards can be chosen out of a
 k  k! ( n  k )! deck of 52 poker cards?
 Theorem 2.8 (Partitions)

 Given a set of n distinct elements, the number of ways to partition the
set into m groups where the ith group has 𝑛𝑖 elements is given by the
multinomial coefficient,
How many different ways can a deck
 n  n!
   of 52 poker cards be partitioned into
 n1 , n2 ..., nm  n1!n2 !  nm ! 5 sets with different numbers?
46
Proof of Theorem 2.8
 Given a set of n distinct elements, the number of ways to
partition the set into 𝑚 groups where the ith group has 𝑛𝑖
elements is given by the multinomial coefficient,
 n  n!
n   
 n1 , n2 ..., nm  n1!n2 !  nm !
n1 n2 n3  nm
 n   n  n1   n  n1  n2   nm  nm 1   nm 
 Proof:             
 n1   n2   n3   nm 1   nm 
n! (n  n1 )! (n  n1  n2 )! (n  nm 1 )!
  m
n n! n1!(n  n1 )! n2 !(n  n1  n2 )! n3!(n  n1  n2  n3 )!
   nm 1!nm !
 k  k! ( n  k )! n!  n 
   
n1!n2 !  nm !  n1 , n2 ..., nm 
47
Examples
 Example 2.15 Combined Experiment
 In a certain state, auto license plates consist of three letters followed by
three numbers. What is the total number of possible license plates?
n  26  26  26 10 10 10  263103  17,576,000
 Example 2.16 Permutation

 A professor creates an exam for his course consisting of six questions. In
order to discourage students from cheating off one another, he gives
each student a different exam in a way that all students get an exam with
the same six questions, but in a different order. How many different
versions of the same exam can be created?
n  6! 6  5  4  3  2  720
48
Examples
 Example 2.17 k-permutation
 A certain padlock manufacture creates locks whose combinations
consists of a sequence of three numbers from the set of 0-39. The same
number is never repeated in a combination. How many distinct padlock
combinations can be created?
n! 40! 40! Valid: 2-13-27
   40  39  38  59,280 Valid: 13-27-2
(n  k )! (40  3)! 37! Invalid: 27-2-27
 Example 2.18 Combinations
 A certain lottery game requires players to select four numbers from the
set 0-39. The number cannot be repeated and the order in which they
are selected does not matter. What is the probability of a player selecting
the winning set of number?
n n! 40! 40  39  38  37 1
      27,405 Pr( the winning set ) 
 k  k! ( n  k )! 4!36! 4  3 2 27405
49
Examples
 Example 2.19
 In the game of bridge, a standard deck of cards is divided amongst four
players such that each player gets a hand of 13 cards.
 How many different bridge games that could occur? Partitions
 52  52!
    5.365 10 28
13,13,13,13  13!13!13!13!
 What is the probability to get a dealt that is completely void of spades?
 How many different sets of 13 cards that can be dealt? Combinations
 52  52!
    6.35 1011
 13  13!39!
 How many different sets of 13 cards that do not have spades? Combinations
 39  39!
    8.12 109
 13  13!26!  39 
  13 
   0.0128
 The probability of a dealt without spades
 52 
 
 13 
50
Exercise: Prob. 2.33 (1)
 2.33 I deal myself 13 cards for a standard 52-card deck. Find
the probabilities of each of the following events:
 (a) A={exactly one heart appears in my hand (of 13 cards)}
Pr(1 Heart)  Pr( H , H , H ..., H ) Order does not matter.
Experiment 1: select one card from 13 heart cards 13 outcomes

 39 
Experiment 2: select 12 card from 39 non-heart cards   outcomes
 12 
 
 39 
13  13 39! 39!
13
Pr(1 Heart )   12 
 12!27!  12!27!  0.0801
 52  52! 52!
  13!39!
13!39!
 
13
51
 (b) at least 7 cards from a single suit appear in my hand;
Pr(at least 7 cards in a suit )  4 * Pr(at least 7 club cards)
# outcomes of having m
 4 * Pr(7  8  9  10  11  12  13 club cards)
clubs out of 13 cards  4 * Pr(7 club)  Pr(8 club)  Pr(9 club)  Pr(10 club)
 Pr(11 club)  Pr(12 club)  Pr(13 club) 
13  39 
    13  39  
 m 13  m      
  i 13  i  
13
 4*    0.0403
(13-m) cards  52 
m cards
from 39 non-  i 7   
from 13  
clubs clubs   13  
52
 (c) my hand is void (0 cards) of at least one suit.
Number of outcomes  4  52  n *13 
Pr( void of at least one suit ) How to simplify    
the event? having void n suites  
n 13 
 Pr( void of one suit )  Pr( void of two suits)  Pr( void of three suits)
 39   26  13   39   26  13 
      4   12   4 
 4   13   4   13   4  13   13   13  13 

   
   
   
 1     2     3   52 
52 52  52 
 
 13   13   13 
       13 
39! 26! 13! 39! 26! 13!
4 6 4 4 6 4
 13!26! 13!13! 13!0!  26! 13! 0!  0.05126
52! 52!
13!39! 39!
53
Combinatoics Theorems
 Theorem 2.4 (Combined experiment):
 A combined experiment, 𝐸 = 𝐸1 ⋅ 𝐸2 ⋅⋅⋅ 𝐸𝑚 , consisting of experiments 𝐸𝑖
each with 𝑛𝑖 outcomes, 𝑖 = 1, … , 𝑚, has a total number of possible
outcomes given by 𝑛 = 𝑛1 𝑛2 𝑛3 … . 𝑛𝑚 = ς𝑚 𝑖=1 𝑛𝑖 .
 Theorem 2.5 (Permutations) n! n(n  1)(n  2)...3  2 1
n!
 Theorem 2.6 (k-permutations)  n(n  1)(n  2)...(n  k  1)
(n  k )!
 Theorem 2.7 (Combinations)  n   n!


n(n  1)(n  2)...(n  k  1)
  k!(n  k )! k (k  1)(k  2)...1
k 
 Theorem 2.8 (Partitions)  n  n!
  
 n1 , n2 ..., nm  n1!n2 !  nm !
54
Hint to Prob. 2.32
 I deal myself 3 cards from a standard 52-card deck. Find the
probabilities of each of the following events:
 (a) 2 of a kind (e.g., 2 fives or 2 kings);
# outcomes of two kinds out of three  52 
Pr(2 of a kind)   
# outcomes of three cards 3
 Experiment 1: Pick one kind out of13 kinds
 Experiment 2: Given a kind, pick 2 cards out of 4 13
 4
 Experiment 3: Pick one card from other kinds  
48  
2
 (b) 3 of a kind;
# outcomes of three kinds out of three  52 

Pr(3 of a kind)   
 Experiment 1: Pick one kind out of 13 kinds
 Experiment 2: Given a kind, pick 3 cards 13
 4
 
 3
55
Hint to Prob. 2.32
 (c) 3 of the same suit (a.k.a a flush, e.g., 3 hearts or 3 clubs);
# outcomes of three of the same suit  52 
Pr( 3 of a same suit)   
 Experiment 1: Pick a suit out of four suits
 Experiment 2: Given a suit, pick 3 card out of 13 4
13 
 
 (d) 3 cards in consecutive order (a.k.a. a straight, e.g., 2-3-4 or 10-J-Q).
3
# outcomes of three in consecutive order

Pr(3 in consecutive order)  52!
# outcomes of three cards in certain order 49!
 Experiment 1: Pick a specific order out of all 11 possible consecutive orders
 Experiment 2: For a given consecutive order, pick the 1st card out of 4 suits
 Experiment 3: For a given consecutive order, pick the 2nd card out of 4 suits
 Experiment 4: For a given consecutive order, pick the 3rd card out of 4 suits
56
Hint to Homework 2.44
 A certain gym teacher has a class of 20 students. He wants to
divide them into four teams of five students each in order to
have a class basketball tournament.
 (c) Neither boy (Tom/Bob) wants to be on a team with the class bully, Frank.
What is the probability that neither of them are on the same team as Frank?
 Solution (the key idea is to simplify the joint probability by using conditional probability)
 Let’s write the probability into four conditional probability below
Pr(T  F , B  F )
 Pr(T  1, B  1, F  1)  Pr(T  2, B  2, F  2)  Pr(T  3, B  3, F  3)  Pr(T  4, B  4, F  4)
 4 * Pr(T  1, B  1, F  1)  4 * Pr(T  1, B  1 | F  1) Pr( F  1)  4  0.614 / 4  0.614
# outcomes that T and B are not in Team 1 given F is in Team 1

Pr(T  1, B  1 | F  1) 
# outcomes that F in Team 1 17 
 
# outcomes that four out of 17 students (excluding T, B and F) are in Team 1   4 
  0.614
19 
# outcomes that F is in Team 1  
4
57
Bayes’ Theorem (Two Events)
 Theorem 2.9: For any events 𝐴 and 𝐵, such that Pr(𝐵) ≠ 0,
Pr( A | B) Pr( B) A : symptom (observable), B : disease (hidden)
Pr( B | A)  Pr( A | B ) : medical knowledge
Pr( A)
Pr( B | A) : disease diagnosis
Pr( B | A)  Pr( A | B )  Pr(B ) : disease population
Pr( A) : symptom likelihood
 Proof:
 Definition 2.5 Pr( A, B)  Pr( A | B) Pr( B)
Pr( A, B) Pr( A | B ) Pr( B )
 Theorem 2.9 Pr( B | A)  
Pr( A) Pr( A)
 Note:
 Theorem 2.9 is useful for calculating certain conditional probabilities since,
in many problem, it may be quite difficult to compute Pr(𝐵|𝐴) directly
whereas calculating Pr(𝐴|𝐵) may be straightforward.
58
Example
 A school of 200 students has 60% boys and 40% girls; all boys
wear pants; 35% of the girls wear pants and 65% wear skirts.
You see a student in the distance and can tell that he or she is
wearing a pant. What’s the probability the student is a girl?
Pr(Boy)  0.6, Pr(Girl)  0.4 Pr( A | B) Pr( B)
Pr( B | A) 
Pr(Pant | Boy)  1, Pr(Skirt | Boy)  0 Pr( A)
Pr(Pant | Girl)  0.35, Pr(Skirt | Girl)  0.65 Pant  Pant  Boy   Pant  Girl 
Pr(Girl | Pant)  ?
Pr( Pant | Girl)Pr(Girl)  Pr( Pant | Girl)Pr(Girl)
Pr(Girl | Pant) 
Pr( Pant) Pr( Pant, Boy)  Pr( Pant, Girl)
Pr(Pant | Girl)Pr(Girl)

Pr(Pant | Boy)Pr(Boy)  Pr(Pant | Girl)Pr(Girl)
0.35 * 0.4
  0.1892
1* 0.6  0.35 * 0.4
59
Theorem of Total Probability
 Theorem 2.10:
 Let B1, B2, …., Bn be a set of mutually exclusive and exhaustive events, that is
n n
Bi  B j   for all i  j B i  S   Pr( Bi )  1
i 1 i 1
n
 Then we will have Pr( A)   Pr( A | Bi ) Pr( Bi )
i 1
A   A  B1    A  B2   ...   A  Bn 
Bi are mutually exclusive   A  Bi  are also mutually exclusive

 Pr( A)  Pr A  B1    A  B2   ...   A  Bn 
n n
 Pr( A)   Pr  A, Bi    Pr  A | Bi  Pr Bi 
i 1 i 1
60
Bayes’s Theorem (Multiple Events)
 Theorem 2.11
 Let B1, B2, …., Bn be a set of mutually exclusive and exhaustive events. Then
Pr( A, Bi )  Pr( A | Bi ) Pr( Bi ) Diseases

Disease
diagnosis
symptom
Pr( A | Bi ) Pr( Bi )
Pr( Bi | A)  n
 Pr( A | B ) Pr( B )
i 1
i i
Pr( A, Bi )
Pr( Bi | A)  n
Pr( A) Pr( A)   Pr( A, Bi )
i 1
 Bayesian theorem is normally useful when Pr(𝐴|𝐵𝑖 ) is straightforward

while Pr(𝐵𝑖 |𝐴) is not available immediately and has to be computed.
61
Problem Solving Exercise: Prob. 2.46
 A communication system sends binary data {0 or 1} which is
then detected at the receiver that occasionally makes mistakes
Pr(0 received | 0 transmitted)  0.95 Pr(0 received | 1 transmitted)  0.10
 (a) Assuming 0s and 1s are equally likely to be transmitted, find Pr(0
received) and Pr(1 received).
n
Pr( A)   Pr( A | Bi ) Pr( Bi ) (total probability theorem)
i 1
Pr(0 received)  Pr(0 received | 1 transmitted)Pr(1 transmitted)

 Pr(0 received | 0 transmitted)Pr(0 transmitted)
Pr(0 received)  0.1* 0.5  0.95 * 0.5  0.525
Pr(1 received)  1  Pr(0 received)  0.475
62
 (b) Suppose a 0 is detected at the receiver. What is the probability that
the transmitted bit was actually a 1? Also, if a 1 was detected at the
receiver, what is the probability that the transmitted bit was actually a 0?
Pr( A | B) Pr( B)
Pr( B | A) 
Pr( A)
Pr(0 received | 1 transmitted) Pr(1 transmitted)
Pr(1 transmitted | 0 received)  0.0952
Pr(0 received)

Pr(0 transmitted | 1 received)   0.0526
Pr(1 received)
63
 (c) What is the probability that the detected bit is not equal to the
transmitted bit. This is the overall probability of error of the receiver.
n
A : error, B1  1 transmitted, B2  0 transmitted
i 1 A : error, B1  1 received, B2  0 received
Pr(error)  Pr(error | 1 transmitted)Pr(1 transmitted)  Pr(error | 0 transmitted)Pr(0 transmitted)

 Pr(0 received | 1 transmitted)Pr(1 transmitted)  Pr(1 received | 0 transmitted)Pr(0 transmitted)
 0.1* 0.5  0.05 * 0.5  0.075
Pr(error)  Pr(error | 0 received)Pr(0 received)  Pr(error | 1 received)Pr(1 received)

 Pr(1 transmitted | 0 received)Pr(0 received)  Pr(0 transmitted | 1 received)Pr(1 received)
 0.0952 * 0.525  0.0526 * 0.475  0.075
64
Hint to Prob. 2.49
 Researchers are investigating the physical development of children over time. In the
study, children are given a physical aptitude test at several stages in their
development. Let P be the event that the child passes the physical aptitude test.
Furthermore, let B be the event that the child taking the test was a boy, and G be
the event that the child taking the test was a girl. The computer storing the data
from this project experienced hard drive failure and only some of the data was
recovered resulting in the partial database shown in the table. Use your knowledge
of probability theory to fill in the missing items in the table.
n
Pr( A | B) Pr( B)
Pr( A)   Pr( A | Bi ) Pr( Bi ) (total probability theorem) Pr( B | A) 
i 1 Pr( A)
0.2775 0.55 0.4054 0.5946
65
Review of Theorems
Pr( A, B)
 Theorem 2.9: (Bayes’ Theorem)
Pr( A | B) Pr( B)
 For any events 𝐴 and 𝐵, such that Pr(𝐵) ≠ 0, Pr( B | A) 
Pr( A)
 Theorem 2.10: (Total probability theorem)

 Let B1, B2, …., Bn be a set of mutually exclusive and exhaustive events
n n
Pr( A)   Pr( A | Bi ) Pr( Bi )   Pr( A, Bi )
i 1 i 1
 Theorem 2.11 (Bayes’ Theorem)

 Let B1, B2, …., Bn be a set of mutually exclusive and exhaustive events. Then
Pr( A | Bi ) Pr( Bi )
Pr( Bi | A)  n
Pr( A, Bi )  Pr( A | B ) Pr( B )
i 1
i i
Pr( A)
66
 (a) Assuming 0s and 1s are equally likely to be transmitted, find Pr(0
received) and Pr(1 received).
n
i 1
Pr(0 received)  Pr(0 received | 1 transmitted)Pr(1 transmitted)

 Pr(0 received | 0 transmitted)Pr(0 transmitted)
Pr(0 received)  0.1* 0.5  0.95 * 0.5  0.525
Pr(1 received)  1  Pr(0 received)  0.475
67
 (b) Suppose a 0 is detected at the receiver. What is the probability that
the transmitted bit was actually a 1? Also, if a 1 was detected at the
receiver, what is the probability that the transmitted bit was actually a 0?
Pr( A | B) Pr( B) A : received bit
Pr( B | A) 
Pr( A) B : transmitted bit
Pr(1 transmitted | 0 received)  0.0952
Pr(0 received)

Pr(0 transmitted | 1 received)   0.0526
Pr(1 received)
68
 (c) What is the probability that the detected bit is not equal to the
transmitted bit. This is the overall probability of error of the receiver.
n Case I : A  error, B1  1 transmitted, B2  0 transmitted
i 1 Case II : A  error, B1  1 received, B2  0 received
Pr(error)  Pr(error | 1 transmitted)Pr(1 transmitted)  Pr(error | 0 transmitted)Pr(0 transmitted)

 Pr(0 received | 1 transmitted)Pr(1 transmitted)  Pr(1 received | 0 transmitted)Pr(0 transmitted)
 0.1* 0.5  0.05 * 0.5  0.075
Pr(error)  Pr(error | 0 received)Pr(0 received)  Pr(error | 1 received)Pr(1 received)

 Pr(1 transmitted | 0 received)Pr(0 received)  Pr(0 transmitted | 1 received)Pr(1 received)
 0.0952 * 0.525  0.0526 * 0.475  0.075
69
Hint to Prob. 2.49
 Researchers are investigating the physical development of children over time. In the
study, children are given a physical aptitude test at several stages in their
development. Let P be the event that the child passes the physical aptitude test.
Furthermore, let B be the event that the child taking the test was a boy, and G be
the event that the child taking the test was a girl. The computer storing the data
from this project experienced hard drive failure and only some of the data was
recovered resulting in the partial database shown in the table. Use your knowledge
of probability theory to fill in the missing items in the table.
n Pr( A | B) Pr( B)
Pr( A)   Pr( A | Bi ) Pr( Bi ) (total probability theorem) Pr( B | A) 
i 1 Pr( A)
0.2775 0.55 0.4054 0.5946
70
Row 1 Row 30
Examples
(11 seats) (40 seats)
 Examples 2.20
 An auditorium has 30 rows of seats. Row 1 has 11 seats while Row 2 has
12 seats, Row 3 has 13 seas, and so on to the last row, Row 30 that has
40 seats. A door price is to be given away by randomly selecting a row
(with equal probability of any rows and the randomly selecting a seat
within that row (with each seat in the row equally likely to be selected).
 Find the probability that Seat 15 was selected given that Row 20 was selected.
 Find the probability that Row 20 was selected given that Seat 15 was selected.
 Solutions
1 Pr( A | Bi ) Pr( Bi )
 How many seats in Row 20? Pr(Seat 15 | Row 20)  Pr( Bi | A) 
30 Pr( A)
 Using Bayes’s theorem to find Pr(Row 20|Seat 15) as follows.
1 1 n
Pr(Seat 15 | Row 20) Pr(Row 20) 30 30 Pr( A)   Pr( A | Bi ) Pr( Bi )

Pr(Row 20 | Seat 15)    0.0325 i 1
Pr(Seat 15) 0.0342
30 30 30
1 1
Pr(Seat 15)   Pr(Seat 15 | Row i ) Pr(Row i )   Pr(Seat 15 | Row i ) Pr(Row i )    0.0342
i 1 i 5 i 5 i  10 30
71
Row 1 Row 30
Independence (11 seats) (40 seats)
 What is independence?
 Recall from the previous example
1 1
Pr(Seat 15 | Row 20) Pr(Row 20) 30 30
Pr(Row 20 | Seat 15)    0.0325
Pr(Seat 15) 0.0342
1
Pr(Row 20)   0.0333
30
The event that Seat 15 is selected lowers the probability that the event that Row 20 was selected.
Event A={Row 20 was selected} is statistically dependent on Event B={Seat 15 was selected}.
 Two events A and B are independent if
Pr( A | B)  Pr( A ) Pr( B | A)  Pr( B )
 Pr( A , B)  Pr( A | B ) Pr( B)  Pr( B | A ) Pr( A)  Pr( A ) Pr( B )

Definition 2.6:
 Two events are statistical independent if and only if
Pr( A , B)  Pr( A ) Pr( B)
72
Independence Generalization
 Definition 2.7:
 The events A, B and C are mutually independent if each pair of events
are independent
Pr( A , B)  Pr( A ) Pr( B)
Pr( A , C )  Pr( A ) Pr(C )  Pr( A , B, C )  Pr( A ) Pr( B) Pr(C )
Pr( B , C )  Pr( B ) Pr(C )
 Definition 2.8:
 The events, A1, A2, …,An, are independent if any subset of these events
are independent, and in addition
Pr( Ai , A j ,..., Ak )  Pr( Ai ) Pr( A j )... Pr( Ak ) i, j ,...k , [1,..., n]
Pr( A1 , A2 ,..., An )  Pr( A1 ) Pr( A2 )... Pr( An )

73
Example
 Example 2.22
 Consider a communication network nodes A, B, C, and D, and links a1, a2,
a3 and a4, as shown below. The probability of a link being available at
anytime is p. In order to send a message from node A to node D we
must have a path of available links from A to D. Assuming independence
of link availability, what is the probability of being able to send a message?
Let Lk be the event that link ak is available.
Path 1 : A 
L1
B 
L2
C 
L4
D ( L1  L2  L4 )
Path 2 : A 
L3
C 
L4
D ( L3  L4 )
Both paths
are available
Pr( A  D)  Pr(Path 1  Path 2)  Pr(Path 1)  Pr(Path 2)  Pr(Path 1  Path 2)

 Pr( L1  L2  L4 )  Pr( L3  L4 )  Pr( L1  L2  L3  L4 )
 p3  p 2  p 4
74
Example
 Example 2.21
 Consider the experiment of tossing two numbered dice and observing
the numbers that appear on the two upper faces. Let the first die tossed
be red and the second be green. Red dice {1,2}
Green dice {1:6}
 A = {number on the red die is less than or equal to 2}
 B = {number on the green die is greater than or equal to 4} Red dice {1:6}
Green dice {4,5,6}
 C = {the sum of the numbers on the two dice is 3}. Two dices
{(1,2),(2,1)}
 Analyze the independence among (A,B), (A,C) and (B,C).

1 1 6 1
 A and B Pr( A)  Pr( B)  Pr( A, B)    Pr( A) Pr( B)  A and B independent
3 2 36 6
1 2 1 2 1
 A and C Pr( A)  Pr(C )   Pr( A, C )    Pr( A) Pr(C )  A and C dependent
3 36 18 36 18
1 1
 B and C Pr( B)  Pr(C )  Pr( B, C )  0  Pr( A) Pr(C )  B and C dependent
2 18
B and C exclusive
75
Mutual Exclusiveness vs Independence
 It is a common mistake to equate mutual exclusiveness with
independence.
 For example
 A = {number on the red die is less than or equal to 2}
 B = {number on the green die is greater than or equal to 4}
 C = {the sum of the numbers on the two dice is 3}.
 Analysis
 A and B are independent, but are they mutually exclusive? No
 B and C are mutually exclusive, but are they independent? No
A Strangers B A Foes/rivalry B
 Important Facts:
 A and B are independent means that they have no effect to each other.
 A and B are mutually exclusive means that they can never co-exist
(strong dependence).
76
Hint to Pro. 2.55
 Suppose two events A and B are independent.
 (a) Is it true that A is independent of ഥ
𝐵? If yes, give a convincing proof,
otherwise, give a counterexample.
 (b) Is it true that 𝐴ҧ is independent of 𝐵?
ത If yes, give a convincing proof,
otherwise, give a counterexample.
Pr( A, B)  Pr( A) Pr( B) 
 Pr( A, B )  Pr( A) Pr( B )
?
n n
Pr( A)   Pr( A | Bi ) Pr( Bi )   Pr( A, Bi ) (total probability theorem)
i 1 i 1
Pr( A)  Pr( A, B)  Pr( A, B )

Pr( A, B )  Pr( A)  Pr( A, B)
Pr( A, B )  Pr( A)  Pr( A) Pr( B)
Pr( A, B )  Pr( A)1  Pr( B)   Pr( A) Pr( B )
77
Row 1 Row 30
Examples
(11 seats) (40 seats)
 Examples 2.20
 An auditorium has 30 rows of seats. Row 1 has 11 seats while Row 2 has
12 seats, Row 3 has 13 seas, and so on to the last row, Row 30 that has
40 seats. A door price is to be given away by randomly selecting a row
(with equal probability of any rows and the randomly selecting a seat
within that row (with each seat in the row equally likely to be selected).
 Question:
 How the event Seat 15 is selected affect the probability of selecting each row?
Pr(Seat 15 | Row i ) Pr(Row i ) Pr(Seat 15 | Row i )  0.0333
Pr(Row i | Seat 15)  
Pr(Seat 15) 0.0342
 0 1 i  5 The event “Seat 15 is selected”

 increases the probability of lower
 0.9737 5  i  30 numbered rows to be selected.
 i  10
30 30 30
1 1
Pr(Seat 15)   Pr(Seat 15 | Row i ) Pr(Row i )   Pr(Seat 15 | Row i ) Pr(Row i )    0.0342
i 1 i 5 i 5 i  10 30
78
Example of Random Variables
 For the experiment of flipping a coin, the
sample space is S = {H, T}.
 We define the random variable X to be X(H) = 0
and X(T) = 1. That is, the sample space {H, T} is
mapped to the set {0, 1} by the random variable X.
 Assuming a fair coin, then we have Pr 𝑋 = 0 =
Pr 𝑋 = 1 = 0.5.
 The mapping is not unique and we could map the
sample space {H, T} to any other pair of real
numbers (e.g., {1,2}).
𝑋(𝐻) = 0
{H, T} {0, 1}
Sample space Real numbers
𝑋(𝑇) = 1
79
Random Variables
 Definitions
 Definition 2.9 Given experiment E, with sample space S, random variable
X maps each possible outcome, ξ∈ S, to a real number X(ξ) that is a
discrete (finite or countable infinite) or a continuous (uncountable) value.
Not
unique 𝑋
𝜉∈𝑆 𝑋 𝜉
Sample space Real numbers
 Definition 2.10 Probability Mass Function (PMF), 𝑃𝑋 𝑥 , of X is a

function that assigns a probability to each possible value of X.
Pr( X  x)  PX ( x) X : random variable

x : a specific value
80
Bernoulli Random Variable
 This is the simplest possible random variable and is used to
represent experiments which have two possible outcomes.
 0  H, 1  T
 These experiments are called Bernoulli trials and the resulting
random variable is called a Bernoulli Random Variable.
 We can associate the values {0,1} with two outcomes of the experiment.
If X is a Bernoulli random variable, its PMF is of the form
X ( H )  0  Pr( X  0)  PX (0)  1  p,
X (T )  1  Pr( X  1)  PX (1)  p
 For example, for the coin tossing experiment, we may map the outcome
H to the value 𝑋 𝐻 = 1 and T to the value 𝑋(𝑇) = 0. Also, we could
use the value Pr 𝑋 = 1 = 𝑝 = 0.5 assuming the coin is fair.
81
Binomial Random Variable
 Given a Bernoulli random variable Y with 𝑃𝑌 (1) = 𝑝, what is
the probability of k occurrences of 1 observed in n trials of Y?
 Out of n trials, how many outcomes with k 1’s (and n-k 0’s)?
k 1’s
1(1) , 1(2) , 1(3) , … . , 1(𝑛)

n
  (n-k) 0’s
k 
 What is the probability of each outcome with k 1’s (and n-k 0’s)?
Pr(an outcome with k 1' s)  p k (1  p ) n  k
 What is the probability of k occurrences of 1 observed in n trials
Pr( X  k )  PX (k )  (# of outcomes with k 1s) * ( the probability of each outcome with k 1s)
n
PX (k )    p k (1  p) n k k  0,1,2..., n
k 
82
Binomial Random Variable (Cond)
 Examples of binomial distribution
n k
Pr( X  k )  PX (k )    p (1  p ) n  k
k 
 Two parameters, n and p, control the PMF.
 The distribution is symmetric for p=0.5.
 n  1   n  1 
n n
 The distribution peaks at 𝑛𝑝 (why?) PX (k )          PX (n  k )

 k  2   n  k  2 
 Is the PFM of the Binomial Random Variable normalized?
n
 n  k nk
 Recall the binomial expansion a  b     a b
n
k 0  k 
n n
n k
P X (k )     p (1  p ) n  k  ( p  1  p) n  1
k 0 k 0 k 
 How about the case that when n approaches infinity and p
approaches 0 in such a way that lim 𝑛𝑝 = 𝛼?
𝑛→∞
83
Poisson Random Variable
 The Poisson random variable has the PFM function as follows:
 m   1
Pr( X  m)  PX (m)  e  4
m!   10
 A limiting case of Binomial distribution lim 𝑛𝑝 = 𝛼 .
𝑛→∞
 Where 𝛼 controls the PMF.
 The distribution peaks around 𝛼 (average).
m
 Is it normalized?
 
 m 
m
 PX (m)  e   e   e   e  1

m 0 m 0 m! m 0 m!
Recall the Taylor series 
 xm
exp( x)  e  
x
m  0 m!
 The Poisson random variable is extremely important as it describes the
behavior of many physical phenomena, e.g., queuing and traffic modeling.
84
Problem Solving Exercise Prob. 2.59
 Consider a Bernoulli trial where Pr(1 ) = 𝑝 , Pr(0 ) = 1 − 𝑝.
Suppose this Bernoulli trial is repeated n times.
 (a) Plot the PMF for a Binomial random variable, X, with 𝑝 = 1/5, 𝑛 = 10.
n k n  k 10   10  0 . 2 
k
10 
PX (k )    p (1  p )   0.2 0.8  0.810  
k 10  k
  0 . 1074  0 . 25 k
k  k  k  0.8  
     k
(The probability of k occurrences of 1 out of n Bernoulli trails) (k  0,1,2,...,10)
 (b) Plot the PMF for a Poisson random variable X with 𝛼 = 𝑛𝑝 = 2.
 m  2 m  2 2m
PX (m)  e  e  0.1353 (m  0,1,2,..., )
m! m! m!
 (c) Compare Pr(𝑋 ≥ 5) as computed by both the Binomial and Poisson
random variables. Is the Poisson random variable a good approximation?
Binomial : Pr( X  5)  1  0.1074  0.25k  0.0328
10 4
4
Pr( X  5)  1   PX (k ) 4
k
k 0
2k
k 0
Poisson : Pr( X  5)  1  0.1353  0.0529
k  0 k!
85
Problem Solving Exercise Prob. 2.59
 2
Is this Possion random variable

a good approximation for the
Binomial random variable?
What we do to the Binomial

random variable to make the
Possion random a better
approximation?
lim 𝑛𝑝 = 𝛼?
𝑛→∞
86
Problem Solving Exercise Pro. 2.60 (1)
 The arrival of telephone calls at a switch is modeled with a Poisson
PMF. That is, if X is the number of calls that arrive in t minutes, then
(  t ) k  t
Pr( X  k )  PX (k )  e
k!
where λ is the average arrival rate in calls/minute. Suppose that the average
rate of calls is 10 calls per min. (typo in the book, page 56)
 (a) What is the probability that fewer than 3 calls will be received in the first 6 s?
Average # of calls over time t : t
k
10 calls 1 1 1
t  min  1 Pr( X  k )  e
min 10 k!
Pr( X  3)  Pr ( X  0)  ( X  1)  ( X  2) 
2 2
1k 1
  PX (k )   e  0.9197
k 0 k  0 k!
87
Problem Solving Exercise Pro. 2.60 (2)
(  t ) k  t
Pr( X  k )  PX (k )  e
k!
 (b) What is the probability that fewer than 3 calls will be received in the first 6 min?
10 calls 60 k 60
t  6 min  60  Pr( X  k )  e
min k!
Pr( X  3)  Pr( X  0)  ( X  1)  ( X  2)

2
60 k 60
2
  PX (k )   e  1.63 10 23
k 0 k  0 k!
88
Geometric Random Variable
 Consider repeating a Bernoulli trial with two outcomes:
 Outcome 1 : Pr 1 = 𝑝
 Outcome 0 : Pr 0 = 1 − 𝑝
 If X represents the number of times the outcome 1 occurs

before the first occurrence 0 , then X is a geometric random
variable whose PMF is
PX (k )  (1  p ) p k , k  0,1,2...
Geometric series
𝑘 times
1 1 1 0
Pr(1 )  p Pr( 0 )  1  p
89
Problem Solving Exercise Pro. 2.65
 A roulette wheel consists of 38 numbers (18 are
red, 18 are black, and 2 are green). Assume that
with each spin of the wheel, each number is
equally likely to appear.
 (a) What is the probability of a gambler winning if he
bets on a red number showing up?
18 9 20 10
Pr(red/win)   Pr(green/loss)  
38 19 38 19
 (b) Suppose the gambler keeps betting on red until he
finally wins. Let N be the number of times he bets.
Specify the PMF of the random variable N. PN (1)  0.47
n 1
 10 
n 1
9 10 9 PN (2)  0.25
PN (n)    
 19  19 19 n PN (3)  0.13
90
 Highway A and Highway B merge to form Highway C.
Engineers found that the number of cars per minute that
travel each road can be well modeled as Poisson random
variables. Let 𝐾 =𝑁+𝑀 be the number of cars per minute on
Highway C. Find the PMF of K, 𝑃𝐾 (𝑘) = Pr(𝐾 = 𝑘).
( A ) n   A ( B ) m   B
Hwy A : Pr( N  n)  e , n  0,1,2,... Hwy B : Pr( M  m)  e , n  0,1,2,...
n! m!
 k  k
Hwy C : Pr( K  k )  Pr   ( M  m)  ( N  k  m)    Pr( M  m, N  k  m)
 m 0  m 0
k What kind
  Pr( M  m) Pr( N  k  m) of PMF?
m 0
k
( A ) m   A ( B ) k  m   B  (  k
( A ) m ( B )  m
 e e e A  B )
( B ) 
k
m 0 m! (k  m)! m 0 m! (k  m)!
nn
a  b 
   a k b n  k
n
m k (  )
k 0  k  e  (  A   B ) ( B ) k k
k!   A  e  (  A   B ) ( B ) k  A  e A B
 A  B k
    
m  0 m!( k  m)!  B 
1   
binomial expansion k! k!  B  k!
91
Common Random Variables
 Bernoulli random variable X
 The probability of each of the two possible outcomes
PX (0)  1  p, PX (1)  p
 Binomial random variable Y

 The probability of the number of occurrence of 1 from n Bernoulli trials
n k
Pr(Y  k )  PY (k )    p (1  p ) n  k k  0,1,2..., n
k 
 Poisson random variable Z lim 𝑛𝑝 = 𝛼
𝑛→∞
 The probability of the number of occurrence of an event that has an
average occurrence number 𝛼.
k
Pr( Z  k )  PZ (k )  e  k  0,1,2.....
k!
92
Geometric Random Variable
 Consider repeating a Bernoulli trial with two outcomes:
 Outcome 1 : Pr 1 = 𝑝
 Outcome 0 : Pr 0 = 1 − 𝑝
 If W represents the number of times the outcome 1 occurs

before the first occurrence 0 , then X is a geometric random
variable whose PMF is
Pr(W  k )  PW (k )  (1  p ) p k , k  0,1,2...
Geometric series
𝑘 times
1 1 1 0
Pr(1 )  p Pr( 0 )  1  p
93
Three Examples (1)
 Example A:
 If the probability of Tom being absent in each class is 0.05, what is the
probability he will miss k classes in a semester of 40 classes?
n k nk p  0.05

Pr( X  k )  PX (k )    p (1  p)
k  n  40
k  0,1,...,40
The time span is finite and discrete.
p  and n  lim np   Poisson random variable

n 
94
Three Examples (2)
 Example B:
 Given that Tom will normally miss 2 classes in a semester, what is the
probability he will miss k classes in a semester?
k 
Pr( X  k )  PX (k )  e 𝛼=2
k!
k  0,1,2,...
The time span is infinite (or very long) and discrete

95
Three Examples
 Example C:
 The arrival of telephone calls at a switch is modeled with a Poisson PMF
The average number of the arrivals is 2 calls per minute, what is the
probability of having k calls in a minute?
k
Pr( X  k )  PX (k )  e   2
k!
k  0,1,2,...
The time span is finite and continuous.

96
(  t ) k  t
Pr( X  k )  PX (k )  e
k!
 (a) What is the probability that fewer than 3 calls will be received in the first 6 s?
k
10 calls 1 1 1
t  min  1 Pr( X  k )  e
min 10 k!
Pr( X  3)  Pr ( X  0)  ( X  1)  ( X  2) 
2 2
1k 1
  PX (k )   e  0.9197
k 0 k  0 k!
97
 In a certain lottery, six numbers of randomly chosen from the
set {0, 1, 2, …, 49} (without replacement). To win the lottery, a
player must guess correctly all six numbers, but it is not
necessary to specify in which order the numbers are selected.
 (a) What is the probability of winning the lottery with only one ticket?
1
Pr( win with one ticket)  p   6.29 10 8
 50 
 
6
 (b) Given in a week, 6 million lottery tickets are sold. Each player has
equal probability to choose any possible number combinations and does
so independent of the selections of all other tickets. What is the
probability that exactly 4 players correctly select the winning combination.
 6,000,000  4 n
Pr(4 winners)    p (1  p ) 5,999,996  5.8  10 4 PX (k )    p k (1  p ) n  k
k 
 4 
m
 m
 PX (m)  e 
Pr(4 winners) e  5.8 10 4
m!
4!   6 106  p  0.3776
98
 A roulette wheel consists of 38 numbers (18 are
red, 18 are black, and 2 are green). Assume that
with each spin of the wheel, each number is
equally likely to appear.
 (a) What is the probability of a gambler winning if he
bets on a red number showing up?
18 9 20 10
Pr(red/win)   Pr(green/loss)  
38 19 38 19
 (b) Suppose the gambler keeps betting on red until he
finally wins. Let N be the number of times he bets.
Specify the PMF of the random variable N. PN (1)  0.47
n 1
 10  9 10 n 19 PN (2)  0.25
PN (n)    
 19  19 19 n PN (3)  0.13
99
 A roulette wheel consists of 38 numbers (18 red,
18 black, and 2 green). Assume that with each spin
of the wheel, each number is equally likely to appear.
 (c) Suppose the gambler keeps betting on red until he wins twice. Let Y
be the number of times he plays. Specify the PMF of Y.
PY (k )  Pr (1 win in the first k  1 plays)  (win in the kth play)
  k  1 9  10  k  2   9 
         
  1  19  19    19 
 
k 2
 10  81
 (k  1) 
 19  361
(k  2,3,...)
100
Prob. 2.48
 We are in possession of two coins, one which is fairly balanced and turns
up heads with probability 1/2, the other is weighted such that heads shows
up with probability 3/4 and tails with probability 1/4. The two coins are
identical looking and feeling so we cannot tell which is which. In order to
determine which is the biased coin we toss the coin 10 times and observe
the number of heads that occurred.
 (a) If 7 heads were observed, what is the probability that the coin flipped was the fair coin?
 (b) If 3 heads were observed, what is the probability that the coin flipped was the fair coin?
 Problem Analysis
 A={Fair Coin}, B={Weighted Coin}, C={7 out 10 are Heads}, D={3 out of 10 are Heads}
Pr(C | A) Pr( A)
Pr( A | C )  Pr( A)  Pr( B)  0.5
Pr(C )
Pr(C | A)  ? Pr(C )  ?
Pr( D | A) Pr( A)
Pr( A | D)  Pr( D | A)  ? Pr( D)  ?
Pr( D)
101
Prob. 2.48 Solution
Pr(C | A) Pr( A) 0.117 * 0.5
Pr( A | C )    0.3188
Pr(C ) 0.117 * 0.5  0.25 * 0.5
Pr(C )  Pr(C | A) Pr( A)  Pr(C | B) Pr( B) ( total probability theorem)
The probability of each

How many outcomes outcome with of 7
of 7 Heads out 10? Heads out 10.
10  1   1  10!  1 
7 3 10
Pr(C | A)      
    0.117
     7!3!  2 
7 2 2
10  3   1 
7 3
Pr(C | B)        0.25

 7  4   4 
102
Redo Prob. 2.32
 (a) 2 of a kind (e.g., 2 fives or 2 kings);
Pr(2 of a kind)
 3
 # outcomes of 2 of a kind 13 *  
 2 4 3 48
* Pr(one outcome with 2 of a kind)
52 51 50
 (b) 3 of a kind;
Pr(3 of a kind)
 3
 # outcomes of 3 of a kind 13 *  
 3 4 3 2
* Pr(one outcome with 3 of a kind)
52 51 50
103
 Imagine an audio amplifier contains six transistors. Harry has
determined that two transistors are defective, but he does not
know which two. Harry removes three transistors at random
and inspect them. Let X be the number of defective transistors
that Harry finds, where X may be 0, 1, or 2. Find the PMF of X.
 4  4  2   4  2 
       
 3   0 .2  2  1   0.6 PX (2)   1  2 
 0 .2
PX (0)  PX (1) 
6
6 6
     
 3  3
 3
432  3  4 3 2   3  4 2 1 
PX (0)   0.2 PX (1)      0.6 PX (2)      0.2
654  1  6 5 4   2  6 5 4 
104
Goals
 To introduce an Engineering Application of probability theory.
 To characterize discrete/continuous random variables by the

cumulative distribution function (CDF).
 To provide you some tips for Midterm 1.
105
An Optical Communication System
 Binary data are transmitted by pulsing a laser or LED that is
coupled to an optical fiber.
 When transmit a binary 1, the light source is turned on for T seconds.
 When transmit a binary 0, the light source is turned off for T seconds.
 The receiver must convert this optical signal back into a string of binary
numbers by using a photodetector.
 Two design questions to ask

 How to decide which number (0 or 1) is sent based on the number of
electrons received or what is the decision rule?
 How to a design a most cost-effective system with certain error rate?
106
Probabilistic Formulation
 We define a random variable to represent the number of
electrons counted during T sec. interval, and we describe this
random variable in terms of two conditional PMF (Poisson).
R0k  R0 R1k  R1
PX |0 (k )  e , k  0,12,... PX |1 (k )  e , k  0,12,...
k! k!
𝑅0 is related to device quality (electron leakage). 𝑅1 is related to the power consumption.
 R0 and R1 are interpreted as the “average” number of electrons observed

when a 0 is sent and a 1 is sent, respectively.
 It assumed that 𝑅0 < 𝑅1, (less electrons for 0 than those for 1).
 Given the condition that the photodetector receives 𝑘 electrons,
we use the following decoding rule:
Pr(1 sent | X  k )  Pr(0 sent | X  k )  1
k  x0  1 and k  x0  0
Pr(1 sent | X  k )  Pr(0 sent | X  k )  0
 How to find the threshold 𝑥0 ? Pr(1 sent | X  x0 )  Pr(0 sent | X  x0 )
107
Solution: Decision Rule
 Recall Bayes’ Theorem
Pr( A | B) Pr( B) A: k electrons received
Pr(1 or 0 sent | X  k ) Pr( B | A) 
Pr( A) B: 0 or 1 was sent
 Given a priori probabilities Pr(0 𝑠𝑒𝑛𝑡) = Pr(1 𝑠𝑒𝑛𝑡) = 0.5, then
k
R
PX |0 (k ) Pr(0 sent)  0 e  R0 R0k  R
Pr(0 sent | X  k )  PX |0 (k )  e 0
Pr( X  k ) k! k!
R1  R1
PX |1 (k ) Pr(1 sent) k PX |1 (k )  e
R  R1
Pr(1 sent | X  k )   e 1 k!
Pr( X  k ) k! Recall
 Then, the decoding rule (the threshold) is obtained as follows:
R1k e  R0 R1k R  R1  R0
k  R1 k  R0 ln k  R1  R0 k ln 1   R1  R0 k
R e R e   R1 ln R1 / R0
1 0 k
R0 e R0  R2 
108
Solution: Error Rate
 Then the threshold is obtained as
 ( R1  R0 ) 
x0    (floor rounding)
 ln( R1 / R0 
)
 There are two possible errors, and we can compute the probability of
error as follows (why?) k
R R0  R0
Recall PX |0 ( k )  e PX |1 ( k )  1
e  R1
k! k!
Pr(error)  Pr(error | 0 sent ) Pr(0 sent )  Pr(error | 1 sent ) Pr(1 sent )
x0 k x0  1
  x0
R0k  R0 R0k  R0
Pr(error | 0 sent )  Pr( X  x0 | 0 sent)   PX |0 ( k )   e  1  e
k  x0 1 k  x0 1 k! k  0 k!
x0
x0
R1k  R1
Pr(error | 1 sent )  Pr( X  x0 | 1 sent)   PX |1 ( k )   e
 Therefore, the probability of error isk  0 k  0 k!
1 1 x0 R1k e  R1  R0k e  R0
Pr(error)   
2 2 k 0 k!
109
Error Rate Illustration
Device quality
Power consumption
110

Stochastic Systems Chapter 1-2

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Stochastic Systems Chapter 1-2

Uploaded by

Copyright:

Available Formats

Chapter 1-2

Intro. to Probability Theory

School of Electrical and Computer Engineering

Prof. Scott L. Miller Prof. Donald Childers

 A non-deterministic signal has an element of randomness

“Concepts of probability have been

An outcome () is a possible result of an experiment.

An event (denoted by C) is a certain set of outcomes of an experiment.

 Many events can be defined from this experiment:

 The sample space consists of a collection of

 We can define events on a continuous space as well

Theorems Theorems Theorems

 Element 2: Probability of atomic outcomes

Atomic Atomic Atomic Atomic

Axiom 2.3a: If 𝐴 ∩ 𝐵 = ∅, then Pr 𝐴 ∪ 𝐵 = Pr 𝐴 + Pr(𝐵).

Can we represent 𝐴 ∪ 𝐵 by the union of some mutually exclusive sets?

Pr(𝐴) = Pr(𝐴 ∩ 𝐴 ∩ 𝐵) + Pr(𝐴 ∩ 𝐵)

Pr(1,1)  Pr(1,2)...  Pr(6,6)  1 / 36.

 To define the joint probability of two events.

 To define the conditional probability of two events.

 Corollary and Theorems

{HHH } {HHM } {HMH } {HMM }

Pr( A, B)  Pr( A  B)  Pr( A)  Pr( B)  Pr( A  B)

 Given a deck of 52 playing cards that are sufficiently shuffled

Pr( A, C )  13 / 52  1 / 4 C  A  C  A  C  Pr(C  A)  Pr(C )

Pr  AM | A1 ,..., AM 1  Pr  AM 1 | A1 ,..., AM  2 .... Pr  A4 | A1 , A2 , A3  Pr  A3 | A1 , A2  Pr  A2 | A1  Pr  A1 

Pr  A4 | A1 , A2 , A3  Pr  A1 , A2 , A3   (10 / 49) * (11 / 850)  11 / 4165

Pr  A3 | A1 , A2  Pr  A1 , A2   (11 / 50) * (1 / 17)  11 / 850

Pr( E )  Pr( A, D)  Pr( B, C )  Pr( A, C )

Pr( E )  Pr( D | A) Pr( A)  Pr(C | B) Pr( B)  Pr(C | A) Pr( A)

Pr( E )  1  Pr( F )  1  Pr( B, D)

Pr( A, B)  Pr( A  B)  Pr( A)  Pr( B)  Pr( A  B)

Pr  A1 ,..., AM   Pr  AM | A1 ,..., AM 1  Pr  AM  1 | A1 ,..., AM  2 .... Pr  A2 | A1  Pr  A1 

Pr( E )  Pr( A, D)  Pr( B, C )  Pr( A, C )

Pr( E )  Pr( D | A) Pr( A)  Pr(C | B) Pr( B)  Pr(C | A) Pr( A)

Pr( E )  1  Pr( F )  1  Pr( B, D)

 Theorem 2.5 (Permutations)

 Theorem 2.8 (Partitions)

 Example 2.16 Permutation

Pr(1 Heart)  Pr( H , H , H ..., H ) Order does not matter.

Experiment 1: select one card from 13 heart cards 13 outcomes

 Theorem 2.5 (Permutations) n! n(n  1)(n  2)...3  2 1

 Theorem 2.7 (Combinations)  n   n!

# outcomes of three kinds out of three  52 

# outcomes of three in consecutive order

# outcomes that T and B are not in Team 1 given F is in Team 1

Bi are mutually exclusive   A  Bi  are also mutually exclusive

Pr( A, Bi )  Pr( A | Bi ) Pr( Bi ) Diseases

 Bayesian theorem is normally useful when Pr(𝐴|𝐵𝑖 ) is straightforward

Pr(0 received)  Pr(0 received | 1 transmitted)Pr(1 transmitted)

Pr(1 received | 0 transmitted) Pr(0 transmitted)

Pr(error)  Pr(error | 1 transmitted)Pr(1 transmitted)  Pr(error | 0 transmitted)Pr(0 transmitted)

Pr(error)  Pr(error | 0 received)Pr(0 received)  Pr(error | 1 received)Pr(1 received)

0.2775 0.55 0.4054 0.5946

 Theorem 2.10: (Total probability theorem)

 Theorem 2.11 (Bayes’ Theorem)

Pr(0 received)  Pr(0 received | 1 transmitted)Pr(1 transmitted)

Pr(1 received | 0 transmitted) Pr(0 transmitted)

Pr(error)  Pr(error | 1 transmitted)Pr(1 transmitted)  Pr(error | 0 transmitted)Pr(0 transmitted)

Pr(error)  Pr(error | 0 received)Pr(0 received)  Pr(error | 1 received)Pr(1 received)

0.2775 0.55 0.4054 0.5946

Pr(Seat 15 | Row 20) Pr(Row 20) 30 30 Pr( A)   Pr( A | Bi ) Pr( Bi )

Pr( A | B)  Pr( A ) Pr( B | A)  Pr( B )

 Pr( A , B)  Pr( A | B ) Pr( B)  Pr( B | A ) Pr( A)  Pr( A ) Pr( B )