AT77.

02 Signals, Systems and Stochastic Processes

Asian Institute of Technology
Handout 10
1
Thu 23 Sep 2010
2 Probability and Random Variables
2.1 Random Variables, Probability Distributions, and Proba-
bility Densities
2.1.1 Fundamentals of Probability
The sample space S of an experiment or an action is the set of all possible outcomes.
Each possible outcome is called a sample point. An event is a set of outcomes, or a subset
of the sample space. For an event E, we shall use Pr{E} to denote the probability of E.
We rst present the axioms that a probability measure must satisfy.
2
Axioms of probability: Let S be the sample space and E, F S be events.
1. Pr{S} = 1.
2. 0 Pr{E} 1.
3. If E and F are disjoint, i.e. E F = , then Pr{E, F} = Pr{E} + Pr{F}.
The above axioms yield the following basic identities.
If E
c
= S E, then
Pr{E
c
} = 1 Pr{E}
Proof: Since E and E
c
are disjoint and their union is S, from statement 3 of the
axiom,
Pr{S} = Pr{E} + Pr{E
c
}.
From statement 1, we obtain the desired property, i.e.
1 = Pr{E} + Pr{E
c
}.

If E and F are not disjoint, then
Pr{E F} = Pr{E} + Pr{F} Pr{E, F}
Proof: Since (F E) (F E) = F and (F E) (F E) = ,
Pr{F} = Pr{F, E} + Pr{F E}
Pr{F E} = Pr{F} Pr{F, E}.
1
Course notes were prepared by Dr. R.M.A.P. Rajatheva and revised by Dr. Poompat Saengudomlert.
2
It is common to write Pr{E F} as Pr{E, F}. We shall adopt this notation.
1
Since E (F E) = E F and E (F E) = ,
Pr{E F} = Pr{E} + Pr{F E}
. .
=Pr{F}Pr{F,E}
= Pr{E} + Pr{F} Pr{E, F}.

If F
1
, . . . , F
n
are disjoint, then
Pr
_
n
_
i=1
F
i
_
=
n

i=1
Pr{F
i
}
Proof: The statement follows from induction. For example, consider n = 3. Since
F
1
F
2
and F
3
are disjoint, we can write
Pr{F
1
F
2
F
3
} = Pr{F
1
F
2
} + Pr{F
3
}.
Since F
1
and F
2
are disjoint, we can write
Pr{F
1
F
2
F
3
} = Pr{F
1
} + Pr{F
2
} + Pr{F
3
}.

The conditional probability of event E given that event F happens (or in short given
event F), denoted by Pr{E|F}, is dened as
Pr{E|F} =
Pr{E, F}
Pr{F}
Alternatively, we can write
Pr{E, F} = Pr{E|F} Pr{F}
A partition of E is a set of disjoint subsets of E whose union is equal to E. Let
F
1
, . . . , F
n
be a partition of S. From the denition of conditional probability, we can
obtain the Bayes rule, which is written as
Pr{E} =
n

i=1
Pr{E|F
i
} Pr{F
i
}
Proof: Write
E =
n
_
i=1
(E F
i
).
Since F
1
, . . . , F
n
are disjoint, so are the sets E F
1
, . . . , E F
n
. Hence,
Pr{E} =
n

i=1
Pr{E, F
i
}.
Using the denition of conditional probability, we can write
Pr{E} =
n

i=1
Pr{E|F
i
} Pr{F
i
}.

2
The Bayes theorem states that
Pr{F
i
|E} =
Pr{E|F
i
} Pr{F
i
}

n
j=1
Pr{E|F
j
} Pr{F
j
}
Proof: Write Pr{F
i
|E} as
Pr{F
i
|E} =
Pr{F
i
, E}
Pr{E}
=
Pr{E|F
i
} Pr{F
i
}
Pr{E}
and use the Bayes rule for the denominator.
The conditional probability can be dened based on multiple events. In particular,
we dene
Pr{E|F
1
, . . . , F
n
} =
Pr{E, F
1
, . . . , F
n
}
Pr{F
1
, . . . , F
n
}
It follows that we can write
Pr{F
1
, . . . , F
n
} = Pr{F
n
|F
1
, . . . , F
n1
} Pr{F
1
, . . . , F
n1
}
= Pr{F
n
|F
1
, . . . , F
n1
} Pr{F
n1
|F
1
, . . . , F
n2
} Pr{F
1
, . . . , F
n2
}
= . . .
= Pr{F
n
|F
1
, . . . , F
n1
} . . . Pr{F
3
|F
1
, F
2
} Pr{F
2
|F
1
} Pr{F
1
}
yielding
Pr{F
1
, . . . , F
n
} = Pr{F
1
}
n

i=2
Pr{F
i
|F
1
, . . . , F
i1
}
Events E and F are independent if
Pr{E, F} = Pr{E} Pr{F}
or equivalently
Pr{E|F} = Pr{E}
In addition, events E and F are conditionally independent given event G if
Pr{E, F|G} = Pr{E|G} Pr{F|G}
2.1.2 Random Variables
A random variable is a mapping that assigns a real number X(s) to each sample point s
in the sample space S.
If S is countable, then X(s) is a discrete random variable.
If S is uncountable, making X(s) take any real value in its range, then X(s) is a
continuous random variable.
The basic idea behind a random variable is that we can consider probabilistic events
as numerical-valued events, which lead us to a probability function. With this function,
we can neglect the underlying mapping from s to X, and consider a random variable X
as a direct numerical outcome of a probabilistic experiment or action.
3
2.1.3 Probability Functions
By using a random variable X, we can dene numerical-valued events such as X = x and
X x for x R. The probability function
F
X
(x) = Pr{X x}
is known as the cumulative distribution function (CDF) or simply the distribution func-
tion. Note that the CDF is dened for all x R.
It is customary to denote a random variable by an upper-case letter, e.g. X, and
denote its specic value by a lower-case letter, e.g. x.
The nature of the function F
X
(x) is determined by random variable X, which is
identied in the subscript. When the associated random variable X is clear from
the context, we often write F(x) instead of F
X
(x).
Since F
X
(x) indicates a probability value, it is dimensionless.
Some Properties of a CDF
1. F
X
() = 0
2. F
X
() = 1
3. If x
1
< x
2
, then F
X
(x
1
) F
X
(x
2
).
4. Pr{X > x} = 1 F
X
(x)
5. Pr{x
1
< X x
2
} = F
X
(x
2
) F
X
(x
1
)
An alternative description of the probability distribution of random variable X is
provided by the probability density function (PDF) dened as
f
X
(x) =
dF
X
(x)
dx
NOTE: A common mistake is to think that f
X
(x) = Pr{X = x}; it is not always true.
Some Properties of a PDF
1.
_

f
X
(x)dx = 1
2. f
X
(x) 0
3. F
X
(x) =
_
x

f
X
(u)du
4. Pr{x
1
< X x
2
} =
_
x
2
x
1
f
X
(x)dx
Over all, the PDF f
X
(x) or CDF F
X
(x) provides a complete description of random
variable X.
4
2.1.4 Continuous vs. Discrete Random Variables
Roughly speaking, a continuous random variable has a continuous CDF. A discrete ran-
dom variable has a staircase CDF. A mixed-type random variable has a CDF containing
discontinuities, but the CDF is not necessarily constant between discontinuities. Fig-
ure 2.1 illustrates dierent types of CDFs.
CDF PDF
continuous
discrete
mixedtype
Figure 2.1: CDFs and PDFs of dierent types of random variables.
Since the PDF is the derivative of the CDF, a continuous random variable has a
continuous PDF. However, the PDF of a discrete or mixed-type random varaible contains
impulses due to the discontinuities in the CDF.
PMF
For a discrete random variable, let X denote the countable set of all possible values of
X(s). We can then dene a probability mass function (PMF) as
f
X
(x) = Pr{X = x}
where x X. Note that a PMF is only meaningful for a discrete random variable. The
same notation f
X
(x) is used for both the PDF and the PMF; it is usually clear from the
context which type of function is referred to by f
X
(x).
Example 2.1 : Consider rolling of a dice. The set of sample points of this probabilis-
tic experiment is S = {1, 2, 3, 4, 5, 6}. The natural denition of an associated random
variable is
X(s) = s, s S.
5
The corresponding PMF is
f
X
(x) = 1/6, x {1, . . . , 6}.
The corresponding PDF is
f
X
(x) =
1
6
6

i=1
(x i).
The corresponding CDF is
F
X
(x) =
1
6
6

i=1
u(x i).
Figure 2.2 illustrates the PDF and the CDF for this example.
0
1 2 3 4 5 6
1/6
2/6
1
1/6
Figure 2.2: PDF and CDF of the result of a dice roll.
6