ENCS 6161

Probability and Stochastic Processes

Concordia University
Course Material 01

Probability models
Design of a system: involves making choices
from various feasible alternatives.
Criteria of choices: cost, reliability and
performance.
Qualitative evaluation of these criteria cannot be
made on actual implementation and experimental
evaluation of alternative configuration.
Why not? Costly in terms of labor, equipment and
time.
2

 Model: is an approximate representation of a physical

situation that explains the observed behavior using
simple and understandable rules. The rules can be used
to predict the outcomes of experiments involving the
given physical situation.
Mathematical models: used when the observed
phenomenon has measurable properties. The model
consists of a set of assumptions about how a system or
physical process works.
The assumption are stated in the form of mathematical
relations involving parameters and variables of the
system. These mathematical relations allow to predict
the measurements that would be obtained if
experiments were performed.
3

Deterministic models versus probability models

Deterministic models: conditions under which the
experiment is carried out determine the exact outcome.
Each repetition of the experiment gives the same
outcome. In deterministic mathematical models, the
solution of a set of mathematical equations specifies the
exact outcome of the experiment.
Example: circuit theory models.

Probability models: A random experiment is defined

as an experiment in which outcome varies in an
unpredictable fashion when the experiment is repeated
under the same condition.
4

An example of random experiment

A ball is selected from an urn containing three identical
ball labeled 0,1 and 2. The number on the ball is noted
and the ball is then returned to the urn.
The outcome of this experiment is the number from the
set = {0, 1, 2}
The set of all possible outcomes is called the sample
space.
The outcome or observation of this experiment is a
number from the set = {0, 1, 2}. The outcome of this
experiment cannot be consistently predicted correctly.

Statistical regularity
A useful model must be able to make predictions
about future behavior of a system. In order to be
predictable, a phenomenon must exhibit regularity
in its behavior.
Many probability models in engineering are based
on the fact that averages obtained in long
sequences of repetitions (trial) of random
experiments consistently yield approximately the
same value.
This is called statistical regularity.
6

Relative frequency

=

: number of times the experiment is repeated under identical

conditions.
: number of times in which the outcome is the event
( = 0, 1, , )
By statistical regularity, we mean that () varies less and less
about a constant as is made large:
lim () =

is the probability of the event .

This equation provides a key connection in going from the
measurements of physical quantities to probability models. The
probability of an outcome is the long-term of times it arises in a long
sequence of trials.
7

 The conditions under which a random

experiment is performed determine the
probabilities of the outcomes of the
experiment.
Example: Place an identical fourth ball with a
number 0 in the urn experiment
2 1
1
1
0 = = , 1 = , 2 =
4 2
4
4

Properties of relative frequency

A random experiment has possible
outcomes. Sample space: = 1, 2, ,
: number of occurrences of the outcome
( = 1, , ) in trials.

1. 0
Dividing by :

0 1
10

2. The sum of the number of occurrences of all

possible outcomes must be

() =
=1

() = 1
=1

Occurrence of events are associated with the

outcomes of an experiment
Event: an even numbered ball
= 0 + 2

0 + 2
=
=

= 0 + 2 ()
11

Example
3. Let be the event: or occurs, where
and are two events that cannot occur
simultaneously.
= + ()
Dividing by :

= +

12

Development of the theory of probability

Difficulties with the long-term relative frequency definition of
the probability:
i.

The use of relative frequency to define probability would

rule out
the situations in which experiments cannot be repeated.
ii. We never perform an experiment a infinite number of
times, so we never know the probabilities exactly.
i. It is not clear as to when and in what mathematical sense
the limit
= lim ()

exists

13

Axiomatic approach to probability

theory
a) The theory is not based on any particular
application or particular notion of what
probability means.
b) Any definition of probability assignment must
be consistent with the relative frequency
interpretation and must satisfy the three
properties.

14

 The modern theory of probability assumes

that:
A random experiment has been defined and a
sample set has been specified.
A class of subset of called events has been
specified.
Each event has been assigned a number [] so
as to satisfy the following axioms:
1. 0 1
2. = 1
3. If and are events that cannot occur
simultaneously, then = + []
15

 The theory of probability does not concern

itself with how the probabilities are obtained
or with what they mean. Any assignment of
probabilities to events that satisfies the three
axioms is legitimate.
It is up to the user of the theory ( the model
builder) to determine what the probability
assignment should be and what interpretation
of probability makes sense in a given
application.

16

Building a probability model

From a real-world problem that involves randomness, one
needs to develop a probability model for the problem. The
theory of probability requires to identify the elements of the
axioms. This identification involves:
1.
2.
3.

Defining the random experiment inherent in the application.

Specifying the set of all possible outcomes and the set of events
of interest.
Specifying the probability assignments of the outcomes from which
the probabilities of all he events can be computed.

The challenge is to develop the simplest model that explains

all the relevant aspects of the real-world problem.

17

Example
We would like to test a telephone conversation to
determine whether a speaker is currently speaking
or silent. We know that on the average, a typical
speaker is active only 1/3 of the time and the rest of
the time he/she is listening to the other party or
pausing between words and phrases.
This physical situation can be modeled as urn
experiment in which we select a ball from the urn
of two white balls (silent) and one black ball
(active speaker).
18

Simplification
i. Not all the speakers are the same.
ii. Not all the languages have the same silence-activity
behavior, etc.

The usefulness and power of this simple model

becomes clear when we begin asking questions that
arise in system design such as:
What is the probability that more than 24 speakers
out of 48 independent speakers are active at the
same time?
This question is equivalent to:
What is the probability that more than 24 black balls
are selected in 48 independent repetitions of the
above urn experiment?
19

Basic concepts of probability theory

Two types of sample space .
a) Discrete sample space
When the outcomes (sample points) of an experiment are
countable.

1. Countably finite:
1: Select a ball from an urn containing balls numbered 1
to 4. Let balls 1 and 2 be black and balls 3 and 4 white.
Note the number and color of the ball selected.
1 = 1, , 2, , 3, , 4,
There are only 4 sample points. Since the outcome
consists of 2 observations, 1 is 2-dimensional.
20

2. Countably infinite
2: A block of information is transmitted repeatedly over
a noisy channel until an error-free block is received
at the receiver. Count the number of transmissions
required.

2 = 1,2,3,
The samples are discrete (countable) but
infinite in number.
Since in a discrete sample space, sample points
is countable, they have one-to-one
correspondence with positive integers.
21

b) Continuous sample space

is uncountably infinite. Outcomes are numbers
that can assume a continuum of values.
3: Pick a number between 0 and 1.
3 = : 0 1 = [0,1]

4: Measure the lifetime of a given chip in a specified

environment.

3 = : 0 = [0, )
22

5: Pick two numbers at random between zero and

one.

5 =

, : 0 1, 0 1

6: Determine the value of an audio signal at .

6 = : < < + = (, +)
2 , 3 , 4 and 6 are one-dimensional.
1 and 5 are two-dimensional.
23

Events
We are usually not interested in the occurrence of a specific outcome, but
rather in the occurrence of some event, i.e., whether the outcome satisfies
certain conditions.
The events are subsets of . The condition of interest defines a subset of the
sample space.

1: The ball is even-numbered and white

1 = 4,
2: Fewer than 10 transmission are required
2 = 1, 2, , 9
3: The number selected is non-negative
3 = 3
4: The lifetime of a chip is more than 1000 hours but less than 2000 hours
4 = : 1000 < < 2000

24

5: Pick a number between 0 and 1 and then a number

between 0 and

5 =

, : 0 1, 0

6: The absolute value of the voltage is less than 1 Volt

6 = : 1 < < 1

25

Two special events

i. Certain event : consists of all the outcomes,
and hence, always occurs.
ii. Null event : none of the outcomes satisfy
the conditions that specify a given event. For
example, = A black ball number larger
than 2

26

Event classes
: set of all possible outcomes of a random experiment.
Events: Subsets of
Probability theory requires that we state the class
of events of interest. Only the events in this
class are assigned probabilities. We expect that
any set operation on events in will also produce
a set that is also an event in .
i.

Events of a discrete sample space (i.e., the number

of samples are finite or countably infinite)

: All possible subsets of

27

ii. Events of a continuous sample space ( i.e.,

is the set of real number or an interval of real
line ). cannot be chosen as the set of all
possible subsets of and still satisfy the
axioms of probability.
In this case,
: the class of events is obtained as a set of
compliments, and countable unions and intersections
of the intervals of the real line of the form (, ] or
(, ]

This class of events is referred to as the Borel

field.

28

Example
Let = {, } be the sample space consisting
of the outcomes of a coin toss experiment. All
possible events of :
= , , , ,
We can generate all possible subset by taking
possible values 0 or 1 of and

Subset

29

Example
Let be the finite sample space:
= {1,2,3, , }
If we allow all possible subsets of to be
events, the class of events is called power
and it is denoted as = 2 , since it has 2
members.
We can index each of all the possible subsets
of with a binary number 1 2
30

Axioms of Probability
The relative frequencies satisfy a number of
properties. Any definition of probability should
satisfy these properties.
A probability law for a random experiment is a
rule that assigns probabilities to the events of the
experiment that belong to the event class .
The axioms of probability formally state that the
probability law must satisfy certain properties.
: A random experiment
: Sample space
: Event class
31

Probability law
for each event , a number [] called the probability of
is assigned such that
Axiom I: 0
Axiom II: = 1
Axiom III: If = , then = + []
Axiom III: If 1 , 2 , is a sequence of events such that
= for all , then

=
=1

[ ]
=1

Axiom III needs to be replaced by Axiom III in order to

handle experiment with infinite sample space.
32

 Corollary 1
If is complement of , then = 1 [].
Since = , by Axiom III, =
+ [ ]
Since = , by Axiom II, 1 = =
= + [ ]. Therefore, = 1

33

 Corollary 2
1
From Corollary 1, = 1
Since from Axiom I, 0, we have 1

Corollary 3
=0
Let = , then =
From Corollary 1: = 1 = 1 1 = 0,
from Axiom II.

34

 Corollary 4
If 1 , 2 , , are piecewise mutually exclusive, then

[ ] 2
=1

=1

According to Axiom III, this result is true for = 2. We prove this

corollary by induction.
Assume that the corollary is true for , that is,

[ ]

()

=1

=1

+1 =
=1

{ +1 }

=1

=1

35

 Thus, the union of events 1 to is mutually

exclusive with +1 . Therefore, by Axiom III,
+1

=
=1

+1
=1

+ +1

()

=1

Using (i) into (ii):

+1

+1

=
=1

[ ]
=1

Thus, the corollary is true for + 1.

36

 Corollary 5
= + []

Venn diagram
Decompose as the union of disjoint events

37

 =

= + +
= +
= ()
= +
= ()

()

Substituting from (ii) and (iii) into (i):

= + []
This corollary can be extended to the case of 3 events:

= + +
+ []
38

Corollary 5 can be generalized as Corollary 6

Corollary 6

=1

[ ]
=1

+ 1

+1

+
<

[1 2 ]

The corollary can be proved by induction.

From Corollary 4, it is clear that

+
39

 Corollary 7
If , then []

= + [ ]
Since 0

[]
40

 We need an initial probability

assignment for some basic set of
events from which the probability
of certain other events can be
computed using the axioms and
corollaries.

41

Discrete sample space

The probability law of an experiment with a countable
sample space can be specified by giving probabilities to
the elementary events.
a) Finite discrete sample space

= 1 , 2 , ,
Events class : All possible subsets of
Elementary events consist of individual outcomes of
the experiment { }.
Thus, all distinct elementary events are mutually
exclusive. Therefore, the probability of the event
is given by
= 1 , 2 , ,

= 1 , 2 , ,

= 1

+ 2

+ + [
].
42

 The probability of an event is equal to the sum of the

probabilities of the outcomes in the event.
Thus, the probability law for a random experiment with
a finite sample space is specified by giving probabilities
of the elementary events.
A probability assignment of particular interest is the
case of equally likely outcomes.
In this case, the probability of elementary events is
given by
1
1 = 2 = = =

Thus, the probability of any events that consists of k

events, = 1 , 2 , , :

= 1 + 2 + + =

43

Example 1
An urn contains 10 identical balls numbered
0,1,2,..,9. A random experiment involves selecting a
ball from the urn and noting the number of the ball.
Find the probability of the following events:
(a) [], (b) [], (c) [], (d) [] and (e)
[], if the events , , and are given as:
: The number of ball selected is odd.
: The number of ball selected is multiple of 3.
: The number of ball selected is less than 5.
44

=
=
=
=

0,1,2, , 9
1,3,5,7,9
3,6,9
0,1,2,3,4

(a)
= 1,3,5,7,9
(b) = 3,6,9

= 1 + 3 + 5 + 7 + 9
= 3 + 6 + 9

(c) = 0,1,2,3,4
(d)

= 10 = 2

= 10

= 10 = 2

5
3
= + =
+
3,9
10 10
8
2
6
3
=

=
=
10 10 10 5

(e)
= + + +
5
3
5
2
2
1
1
9
=
+
+

+
=
10 10 10 10 10 10 10 10

45

(b) Infinite discrete sample space

The sample space is infinitely countable

= 1 , 2 ,
The event class can consist of all possible
subsets of . The events can now be of the
form = 1 , 2 ,
Axiom III implies that probability of can be
obtained as
= 1 , 2 ,
= 1 + 2 +
46

Example 2
A fair coin is tossed repeatedly until the first heads
shows up. The outcome of the experiment is the
number of tosses required until the first heads
occurs, find the probability law for the experiment.
The sample space for this experiment is given by
= 1,2,3, .
Let the experiment be repeated times.
: The number of trials of in which the th
toss results in the first heads.
47


,
2

If is very large, 1 since the coin is fair.

This means that a second toss is necessary

about 1 = times and again we expect

2

22

that about half of these, i.e., 2 = = will

result in heads, and so on.
Thus, for large , the relative frequencies are

2
1
=
=
= ,
= 1, 2,

48

49

 Therefore, a reasonable probability law for this

experiment is
1
=
2

= 1,2,

These probabilities add up to unity. Using

1
geometric series with = ,

=
1

1
2

=1

50

Continuous sample space

1. Outcomes of an experiment are numbers that assume a
continuum of values. The sample space is the entire real
line or some interval of the real line.
2. If we allow the event class to consist of all subsets of ,
the class would be too large, and it would be impossible to
assign probabilities to all subsets of .
3. It is possible to assign probabilities to all events in a
smaller class that includes all the events of particular
interest. This is called Borel field, .
4. The Borel field, , contains all the open and closed
intervals of as well as all the events that can be obtained
as countable unions, intersections and complements.

51

5. Let 1 , 2 , , , be a sequence of
mutually exclusive events that are represented
by the intervals of real line. Then, by the
Axiom III

=
=1

[ ]
=1

where each [ ] is specified by the probability

law. For this reason, the probability laws in
experiments with continuous sample spaces
specify a rule for assigning numbers to intervals
of the real line.
52

Example 3
Pick up a number between 0 and 1.
= : 0 1 = [0, 1]
This set is uncountably infinite

53

 Probability law : the probability that the outcome falls

in a subinterval of is equal to the length of the
interval.
, =
01
Axiom I: 0
Axiom II: = 0,1 = 1 0 = 1
Axiom III: if 1 = [1 , 1 ] and 2 = [2 , 2 ] with
0 1 1 < 2 2 1
Then, 1 and 2 are disjoint
1 2 = 1 + 2 = 1 1 + (2 2 )
0,0.25 = 0.25 0 = 0.25
0.5,0.75 = 0.75 0.5 = 0.25

54

 If 0 is any point in , then

0 , 0 = 0 0 = 0
since there are uncountably infinite number of
equally likely outcomes in any interval in [0,1].
Let be the outcomes that are at least 0.25 a way
from the center of the interval, that is,
= 0, 0.25 0.75,1
= 1 2
Since 1 and 2 are disjoint, by Axiom III,
= 1 + 2
= 0.0.25 + 0.75,1
= 0.25 0 1 0.75 = 0.5
55

Example 4
The life of a memory chip is measured and found that the proportion of chips where
the lifetime exceeds t decreases exponentially at rate
= (0, )
If we interpret the above finding as that a chips life time exceeds t decreases
exponentially at a rate , then we obtain the following assignment of probabilities
to events of the form (, ):
(, ) = > 0
Axiom I: 0
Axiom II: = (0, ) = (0) = 1
Since , = , (, ) and the two intervals on the right side are disjoint, by
Axiom III:
, = , + ,
Thus, the probability of the lifetime of a chip in an arbitrary interval (, ] can be
obtained as
, = , ,
=

56

Example 5
Pick two number and between 0 and 1. Then,
= , : 0 1, 0 1
Find the probability of the events:
(a) = > 0.5
(b) = > 0.5
(c) = >

57

If we assume that all the pairs of numbers in

the unit square are equally likely to be
selected, then it is reasonable to use the
probability of any region inside the unit
square to be equal to the area of .
1
2

1
2

(a) = , (b) = , (c) =

1
2

58

Conditional Probability
We are interested in determining whether two events
and are related in the sense that the knowledge of
occurrence of alters the likelihood of the occurrence
of .
The conditional probability [|] of event , given
that has occurred is defined as
[]
=
> 0
[]
Knowledge that event has occurred implies that the
outcome of the experiment is in the set . In computing
[|], we can therefore view the experiment as now
having the reduced sample space
59

60

: number of times the experiment is repeated

: number of times the event occurs
: number of times the event occurs
Frequency of interest :
/ []
=
=
( )
nB
/
[]
If
[] []
=
=
=1
[]
[]
If =
[] []
=
=
=0
[]
[]

61

Example 6
A ball is selected from an urn containing two black
balls numbered 1 and 2 , and two white balls
numbered 3 and 4. The number and color of the ball
is noted.
The sample space:
=

1, , 2, , 3, , (4, )

Assuming the four outcomes to be equally likely,

find (a) [|] and (b) [|], where
= black ball is selected ={ 1, , 2, }
= even numbered ball is selected ={ 2, , (4, )}
= the number on the ball >2={ 3, , (4, )}
62

 = 2, = 0.25
= = 0
[] 0.25
=
=
= 0.5 = []
[]
0.5
The knowledge of did not alter the
probability of .
[]
0
=
=
=0
[]
0.5
The knowledge of implied that has not
occurred.

63


=
=

=
= []

These two equations are useful in finding

probabilities in sequential experiments.

64

Example 7
An urn contains two black balls and three white
balls. Two balls are selected at random from the urn
without replacement and the sequence of colors
noted. Find the probability that both the balls are
black.
This experiment consists of a sequence of two sub
experiments
: event that the outcome of the ith (i=1,2) draw is
black
: event that the outcome of the ith (i=1,2) draw is
white
65

Probability of interest

2 1 = 2 1 1

1
=
4

2
1
=
5
10
66

Example 8
A user inputs 1 with a probability or 0 with
probability 1 and the corresponding signal is
transmitted. The receiver makes a decision about
what was the input to the system based on the
signal it received and makes random decision
error with a probability . Let
: event input was i, i=0,1
: event receiver decision was j, j=0,1

Find , = 0,1; = 0,1

67

0 0
0 1
1 0
1 1

=
=
=
=

0 0
1 0
0 1
1 1

0
0
1
1

= 1 1
= 1
=
= (1 )()
68

Theorem on total probability

Partition of
1 , 2 , , : mutually exclusive events whose
union equals the sample space

= 1 2
The sets 1 , 2 , , are referred to as a partition
of

69

 = = 1 2
= 1 2
Thus, any event can be represented as the union
of mutually exclusive events. Hence, by applying
Corollary 4, the probability of can be obtained
as
= 1 + 2 +
+
(1)
By applying the relation of conditional probability
on each term of the right side of (1), we have :
= |1 1 + |2 2 +
+ | [ ]
70

Example
An urn contains two black balls and three white
balls. Two balls are selected at random from the
urn without replacement and the sequence of
colors noted. Find the probability that both the
balls are black. Find [2 ], where 2 is the
event second ball is white.

71

Let :
1 : first ball is black= { , , (, )}
1 : first ball is white= { , , (, )}

Thus, 1 and 1 form the partition of the sample space .

By using the theorem on total probability:

2 = 2 |1 1 + 2 |1 1
3 2
2 3
3
=
+
=
4 5
4 5
5
This probability is the same as the probability of
selecting a white ball in the first draw. This makes
sense, since we are computing the probability a white
ball in the second draw under the assumption that we
have no knowledge of the outcome of the first draw.

72

Example
There is a mix of good and bad chips.
Assume that the fraction of the bad chips is
and that of the good chips is 1 . The life time
of good chips follows the exponential law with a
rate of failure and that of bad chips also
follows the exponential law with a rate of failure
is 1000. Find the probability that a randomly
selected chip is still functioning after seconds.
73

Let
: event chip still functioning after seconds
: event chip is good
: event chip is bad.
= and =
Thus, and form a partition
By the theorem on total probability:
= +
= 1 + 1000

74

Bayes Rule
Let 1 , 2 , , form a partition of a sample
space . Suppose that event has occurred.
What is the probability of event given that
event has occurred?
By the definition of conditional probability:

|
=
=
[]
=1 [ ]
75

Interpretation
1. Suppose that we have the knowledge of a priori
probabilities of events s that form a partition of ,
i.e., we know [ ]s before the experiment is
performed.
2. Now suppose that the experiment is performed, and
we are informed that event has occurred.
3. The a posteriori probabilities are the probabilities
of the events in the partition, i.e., [ |] and given
by the Bayes rule.
The Bayes rule tells us as, to how the probabilities of the
events s are modified with the knowledge of the
occurrence of the event .
76

Example
i. User inputs a 0 or 1 into a system, and the
corresponding signal is transmitted: 1 with
probability and 0 with probability 1 .
ii. Receiver makes a decision about what was
the input to the system with the probability
of the random decision error.
Determine which input is more probable given
that the receiver has output a 1. Assume that, a
priori, the input is equally likely to be 0 or 1.
77

Let
: the event that input is , = 0,1
: the event that the receiver output is ,
= 0,1

78

 Note that 0 and 1 form a partition of the sample space . Since

1
the two input , 0 and 1 are equally likely = , i.e., 0 =
2
1
1 =
2
Using the theorem on total probability:
1 = 1 0 0 + 1 1 1
1
1
1
=
+ 1
=
2
2
2
Applying Bayes rule, the posteriori probabilities are obtained as
1

1 0 0
2
0 1 =
=
=
1
1
2
1
1

1 1 1
2
1 1 =
=
=1
[1 ]
1/2
1

Thus, if < , then an input of 1 is more likely than an input of 0,

2
when a 1 is observed at the output.
79

Example
A manufacturing process produces a mix of good
and bad memory chips. The life time of good
chips follows the exponential law with a rate of
failure . the life time of bad chips also follows the
exponential law but the rate of failure is 1000.
Suppose that the fraction of bad chips is , and
therefore, that of good chips is 1 . In order to
weed out the bad chips, every chip is tested for
seconds prior to leaving the factory. The chips that
fail are discarded and the remaining chips are sent
out to customers. Find the value of for which 99%
of the chips sent out to customers are good.
80

Let
: the event chip still functioning after seconds
: the event chip is good
: the event chip is bad.
The events and form a partition of the sample set. Thus,
by the theorem on total probability:
= +
= 1 + 1000
By applying Bayes rule:

1
=
=

1 + 1000
We would like to have = 0.99
1
= 0.99

1000

1 +

81

 1
= 0.99

1000

1 +

1000
1
1
=
1=

0.99
99
99
999

=
1
1
99
=
ln
999
1
1

Assume that = 20,000 hours and = 0.1, then

= 48 hours. Thus, after testing a mix of chips with
90% of good chips for 48 hours and discarding the
failed chips, 99% of the chips sent to customers are
good.
82

Independent events
Two events and are defined to be independent if
=
(1)
Since
= |
(2)
using (2) into (1),
| =
3
Thus, if the events and are independent, the knowledge of
the occurrence of event does not alter the probability of .
Similarly, since
= |
(4)
Using (4) into (1),
| =
5
Thus, if the events and are independent, the knowledge of
the occurrence of event does not alter the probability of . 83

 Note that if 0, (3) implies (1), i.e., independent

of event and .
Similarly, if 0, (5) implies (1), i.e.,
independent of event and .

If two events and have non-zero probability

and are mutually exclusive, then they cannot be
independent, since if they are independent
=
Since and mutually exclusive, i.e., =
= = 0
i.e., = 0

Thus is impossible, since and are nonzero

84

Example
A ball is selected at random from an urn containing
two black balls, numbered 1 and 2, and two white
balls, numbered 3 and 4.
= black ball selected= 1, , (2, )
= even-numbered ball selected= (2, ), 4,
= number on a ball is > 2.= 3, , 4,
a) Are the events and independent?
b) Are the events and independent?
85

(a)

1, , (2, )

= (2, ), 4,

= 2,

1
=
4

1
=
2
1
=
2

Therefore,
= =
4
Thus, and are independent.

2,
1/4 1
| =
=
=
=
[]
2, , (4, )
1/2 2

1/2 1
=
=
=
[]
1
2
Thus, the knowledge that event has occurred does not alter the probability of
the event
Reason:
The proportion of the outcomes in that leads to the occurrence of is equal to
the proportion of the outcomes in that lead to occurrence of

86

(b)
=

3, , 4,
1
= =
2
= = 0

1
=
2

1
= 0 =
4

0
| =
=
=
=0
[]
3, , 4,
1/2
1
=
2
Events and are mutually exclusive. Occurrence of
implies that event has definitely not occurred.
87

Example
Two numbers are selected at random. The events ,
and are defined as follows:
= > 0.5
= > 0.5
= >
(a) Are the events and independent?
(b) Are and independent?

88

(a)

1/4 1
| =
=
= = []
[]
1/2 2
Therefore, the events and are independent.
Reason
The proportion of the outcomes in that lead to the
occurrence of is equal to the proportion of the outcomes
in that lead to occurrence of

(b)

3/8 3 1
| =
=
= = []
[]
1/2 4 2
Therefore, the events and are not independent. The
knowledge that event (i.e., > ) has occurred
increases the probability of from 1/2 to 3/4.
89

Conditions for events , and to be

independent

i. Events , and are pairwise independent, i.e.,

= ,
= ,
=
ii.The knowledge of the joint occurrence of any two events, say and ,
should not affect the probability of the third:
| =

that is,

=
Using pairwise independence: i.e. = on the right side:

=
Thus, three events , and are independent if the probability of any pair or
triplet of the events is equal to the product of the probabilities of the individual
events.

90

Example
Two numbers are selected at random
from unity interval, Events , D and
are defined as:
1
= >
2
1
= <
2
1
1
1
1
= < <
> >
2
2
2
2

91

1
= ,
2

1
= ,
2

1
= =
4
1
= =
4
1
= =
4

1
=
2

1
= = 0 =
8

Hence, the events , D and are not

independent.
92

Independence of events
For a set of events to be independent, the probability of an event
should be unchanged with the joint occurrence any subset of other
events. This leads to the following definition of independence
involving events.
The events 1 , 2 , , are said to be independent if for
= 2,3,4, , ,
1 2 = 1 ] 2 [
where 1 1 2
Thus, for a set of events, one has to verify that probabilities of all
2 1 intersections factor into the probabilities of the
individual events. Therefore, the application of the definition of the
independence is cumbersome because of the necessity of verifying
so many conditions.
However, the most common application of the independence
concept is made for the events of separate experiments. Such
experiments are referred to as independent experiments.
93

Example
A fair coin is tossed three times and the resulting
sequence of heads and tails are observed. Find
the probability of the elementary events.
The sample space
= {, , , , , , , }

Since the coin is fair, the outcomes, or , of

a single experiment are equiprobable, i.e.,
1
1
= = , = = 1
2

94

 We assume that outcomes of the three tosses (three sub-experiment)

are independent.
1
= =
8
1
= =
8
1
= =
8
1
= =
8
1
= =
8
1
= =
8
1
[] = [][][] =
8
1
[] = [][][] =
8
95

Sequential experiments
Many random experiments are sequential
experiments. The sequential experiment consists
of a sequence of simpler sub-experiments. These
sub-experiments may be independent or
dependent.

96

(a) Sequence of independent experiments

A random experiment consists of performing the experiments:
1 , 2 , , . The outcome of is an tuple: 1 , 2 , , , where
is an outcome of the experiment . The sample space of the
sequential experiment is the Cartesian product of the individual
sample spaces s denoted by = 1 2 . The sample
space consists of the -tuple.
Because of the physical considerations, it is usually possible to
determine when the sub-experiments 1 , 2 , , are independent,
in sense that the outcome of one sub-experiment cannot affect the
outcomes of other sub-experiment.
Let 1 , 2 , , be events such that concern only the outcomes
of the kth sub-experiment . Then, if the sub-experiments
1 , 2 , , are independent , then events 1 , 2 , , are
independent. That is,
1 2 = 1 ] 2 [
Thus, it is possible that determine probabilities of all events of a
sequential experiment.
97

Example
Ten numbers are selected at random from the
interval [0, 1]. Find the probability that the first 5
numbers are less than 1/4 and the last 5 numbers
are greater than 1/2.

98

 Let 1 , 2 , , 10 be the sequence 10 numbers; then the

events of interest are:
1
= <
,
= 1,2, , 5
4
1
= >
,
= 6,7, , 10
2
Each selection of a number is independent of other selection
1 2 10 = 1 ] 2 [10
1
= ,
= 1,2, , 5
4
1
= ,
= 6,7, , 10
2
5
5
1
1
1 2 10 =
4
2
99