Probability and Random Processes

DEEPTANSHU
1 Fundamentals of Probability
There are several phenomena where a detailed deterministic analysis is not possible
(quantum mechanics) or not feasible (impact of solar radiation and meteors on radio
waves through the ionosphere). Unpredictability gives rise to randomness. But often
random events have statistical regularity, captured by relative frequencies of events. As
with Newtonian mechanics, probability theory is justified becuase it predicts the results
(usually relative frequencies) correctly.

1.1 Definitions
There are 2 undefined entities (similar to axioms), in terms of which all other terms are
defined - Random Experiment and Outcome of the random experiment.

a. An outcome ( ) is also called an elementary outcome or a sample point.

b. is the source of randomness/uncertainty in the experiment. Can be thought of
being picked by the Goddess of Chance (Fortuna) - beyond our control.

Def. Sample Space set of all possible outcomes of the experiment.

Sample space depends on what the probabilist is interested in. For the same experi-
ment of tossing a coin, may be {H,T} (the face that shows up), N (the number of turns
it takes while it is up), R+ (the velocity with which it lands).
Def. An Event A is said to occur if A. Formally, an event is a F measurable set.
An event can be thought of as a subset of or a set of sample points that is of interest.
It corresponds to a result of the experiment - often more useful than just the elementary
outcome.
Def. A collection F0 of subsets of is called an Algebra if
1. F0
2. Closed under Complementation: If A F0 , then Ac F0
3. Closed under Finite Union: If A F0 , B F0 , then A B F0
An algebra is also closed under finite intersections. However, an algebra is not pow-
erful enough to capture all events of interest. As a motivating example, consider the
experiment of tossing coins until the first head turns up.

= {H, T H, T T H, T T T H, . . . }

1
Probability and Random Processes

Define an event A : total number of tosses even.

A = {T H, T T T H, . . . }
Then A can not be constructed by the rules of the algebra defined above (complement
and finite unions). The rules of an algebra prove insufficicient for dealing with an un-
countable union of elementary outcomes in this case. Hence the need for a (slightly)
stronger structure to build a rich enough theory.
Def. A collection F of subsets of is called a algebra if
1. F
2. Closed under Complementation: If A F, then Ac F
3. Closed under Countable Union: If A1 , A2 , F, then
S
i=1 Ai F
S
Note that i=1 Ai shouldSnot be interpreted as a limit. It is just the set of elements
presents in at least one Ai . iN Ai is a better notation.

algebra is concerned
S only with countable unions. It says nothing about un-
countable unions like xR Ax .
De Morgans Laws are applicable to countably infinite unions and intersections.
So, F is closed under countable intersections.
There are algebras that are not algebras. But every algebra is an algebra.
An event A is called a null event if Pr(A) = 0. A null event is different from the
impossible event . Randomly selecting a point inside a unit square always yields
a null event, but not the impossible event.
Power set of is denoted by {0, 1} . If is infinite, its power set is too large to
assign probabilities reasonably to all its members.

(A) Probability Measure on the measurable space (, F) assigns a numerical value to

each event. Pr : F [0, 1] satisfies
Pr() = 1 , Pr() = 0
(Finitely Additive) If AB = , Pr(A B) = Pr(A) + Pr(B)
P
(Countably Additive) If {Ai } satisfy Ai Aj = i 6= j, Pr (
i=1 Ai ) = i=1 Pr(Ai )

1.2 Borel Sets and Lebesgue Measure

Standard approach in the discrete setting - assign probabilities to singletons and define
events in terms of countable unions of singletons. This approach fails for uncountable
sample spaces. Any interval of R is an uncountable union of singletons - not possible
to sum probabilities over singletons as in the discrete case (rules of algebra do not
specify how to deal with uncountable unions).

2
Probability and Random Processes

1.3 Conditional Probability

Pr(AB)
Pr(A|B) , provided Pr(B) 6= 0
Pr(B)
Conditioning can be visulaized as moving to a new sample space B from the original
. New events are A0i = Ai B.
Note that if A0i A0j = , Pr(A0i A0j ) = Pr(A0i ) + Pr(A0j )
Theorem 1 (Total Probability Theorem). Let {Bi } be a partition of , Pr(Bi ) > 0 i.
Then, for any event A,

!
[ X X
Pr(A) = Pr (A Bi ) = Pr(A Bi ) = Pr(A|Bi ) Pr(Bi )
i=1 i=1 i=1

1.4 Independence
A family of events {Ai : i I} is independent if for all finite subsets J of I,
!
[ Y
Pr Ai = Pr(Ai )
iJ iJ

Let C be an event with Pr(C) > 0. Two events A, B are conditionally independent given
C if
Pr(A B|C) = Pr(A|C) Pr(B|C)

1.5 Special Results

1.5.1 Continuity Lemmas
Lemma 2. Let {Ai } be a non-decreasing sequence of events (A1 A2 A3 ).

[
Pr(A) = lim Pr(Ai ) where A = Ai = lim Ai
i i
i=1

Express A as a union of disjoint events A = A1 (A2 \ A1 ) (A3 \ A2 ) .

X
X
Pr(A) = Pr(A1 ) + [Pr(Ai+1 \ Ai )] = Pr(A1 ) + [Pr(Ai+1 ) Pr(Ai )] = lim Pr(Ai )
i
i=1 i=1

Lemma 3. Let {Bi } be a non-increasing sequence of events (B1 B2 B3 ).

\
Pr(B) = lim Pr(Bi ) where B = Bi = lim Bi
i i
i=1

Proved by taking complements and using the previous lemma.

3
Probability and Random Processes

1.5.2 Elements occuring Infinitely Often

Let {Ai } F, Bn =
k=n Ak . Then, Bn F n. Also, n=1 Bn F. Define

[
\
\
A= Ak = Bn
n=1 k=n n=1

4
2 Random Variables
A random variable is a X : R satisfying { : X() x} F x R. Such a
function is said to be F-measurable.

Since X is a F-measurable function, the distribution function FX (x) = Pr(X x)

exists for all x R.

2.1 Tranformation of Random Variables

In the discrete setting (like Binomial, Poisson and Geometric random variables), it
is obvious that if y = g(x),
X
Pr(Y = y) = pY (y) = pX (x)
x:g(x)=y

In the continuous setting, consider U Unif[0, 1]. X = F 1 (U ) where F is a valid

cdf.

FX (x) = Pr(X x) = Pr(F 1 (U ) x) = Pr(U F (x)) = FU (F (x)) = F (x)

Let Y = g(X), where g is continuously differentiable with non-vanishing deriva-

tive (i.e. derivative is continuous and non-zero).