Professional Documents
Culture Documents
processes
Instructor: Intekhaab Siddiquee
2016 SPR
Contents
1
Probability Theory
Combinatorics
Counting methods
Introduction to Communications
2 / 93
Contents (cont.)
Uniform distributions
Bernoulli distribution
Geometric distribution
Binomial distribution
Multinomial distribution
Negative Binomial(Pascal) distribution
Poisson distribution
Expectation
Bernoulli random variable
Geometric random variable
Poisson random variable
binomial variable
Variance of a discrete random variable
Introduction to Communications
3 / 93
Introduction to Communications
4 / 93
Definitions
Definitions
Set: A set is a collection of elements. One way to define a set is by explicitly listing all the
elements belonging to this set. Sets are represented by capital letters and the elements are
enclosed by curly braces and separated by commas. E.g
A = {, , , , 5, 7}
When an element belongs to a set it is represented by the symbol , e.g.
A
the symbol meaning belongs to. If an element does not belong to a set, it is represented by
the symbol .
/ Example
8
/A
Below are some examples of commonly used sets:
Instructor: Dr. Intekhaab Siddiquee
Introduction to Communications
5 / 93
Definitions
1
Set of integers, Z = { , 3, 2, 1, 0, 1, 2, 3, }
Closed interval on the real line, e.g. [a, b] is the set of real numbers x such that a x b,
where a and b are real numbers.
Open interval on the real line, e.g. (a, b) is the set of real numbers x such that a < x < b,
where a and b are real numbers.
The second way to define a set is by defining the property of the elements of the set. In this
case, they are represented the following way:
Introduction to Communications
6 / 93
Definitions
1}
Subset: If all the elements of set A belong to set B, then A is a subset of B. Set B is the
superset of A. This is mathematically represented as A B, where the symbol means
subset of.
Example: Suppose A = {1, 2, 3}, B = {1, 2}. Then B A.
Null Set: A set that contains no elements is called a null set or an empty set. It is usually
defined by the symbol . i.e. = {}.
Note: A null set will be a subset of any set.
Universal Set: It is the set of all elements in the context that is being considered. In
probability theory, a universal set is called sample space. Universal set is usually
represented by the symbol S.
Instructor: Dr. Intekhaab Siddiquee
Introduction to Communications
7 / 93
Definitions
Example: Suppose A, B, C, D, F are all possible grades of an exam. So the universal set is
S = {A, B, C, D, F }.
Equality of sets: Two sets A and B are said to be equal if A B and B A, i.e. they
have the same elements.
Example: Sets A = {1, 2, 3}, B = {2, 3, 1} are equal while C = {1, 2, 4}, D = {1, 2} are not.
Introduction to Communications
8 / 93
Definitions
Venn diagram
Venn diagram is a useful way to graphically represent sets. This helps to visually
understand relationship among different sets. A set is represented by a closed region. Two sets
having some common elements are represented by overlapping regions. A subset is
represented by a region completely inside the region representing the superset. A universal
set is usually represented by a rectangular region inside which all the sets reside.
Introduction to Communications
9 / 93
Definitions
Introduction to Communications
10 / 93
Definitions
Introduction to Communications
11 / 93
Definitions
Introduction to Communications
12 / 93
Definitions
Introduction to Communications
13 / 93
Definitions
1.
2.
3.
4.
5.
{1}, {2, 4, 7}
6.
{2}, {1, 4, 7}
7.
{4}, {1, 2, 7}
8.
{7}, {1, 2, 4}
9.
{1, 2, 4, 7}
Introduction to Communications
14 / 93
Definitions
Introduction to Communications
15 / 93
Definitions
(Note: A B = A B)
Instructor: Dr. Intekhaab Siddiquee
Introduction to Communications
16 / 93
Definitions
Example: Suppose that the universal set S is defined as S = {1, 2, , 10} and A = {1, 2, 3},
B = {x S : 2 x 7}, and C = {7, 8, 9, 10}.
(a) Find A B
(b) Find (A C) B
(c) Find A (B C)
(d) Do A, B, and C form a partition of S?
Solution:
(a)
A B = {1, 2, 3, 4, 5, 6, 7}
(b)
A C = {1, 2, 3, 7, 8, 9, 10}
B = {2, 3, , 7}
thus:
(A C) B = {1, 8, 9, 10}
Introduction to Communications
17 / 93
Definitions
(c)
A = {4, 5, , 10}
B C = {2, 3, 4, 5, 6}
thus: A (B C) = {2, 3, , 10}
(d) No, since they are not disjoint. For example,
A B = {2, 3} 6=
Introduction to Communications
18 / 93
Definitions
Introduction to Communications
19 / 93
Definitions
Introduction to Communications
20 / 93
Definitions
Introduction to Communications
21 / 93
Definitions
(1)
A (B C) = (A B) (A C)
(2)
Introduction to Communications
22 / 93
Definitions
De Morgans law
De Morgans law, sometimes called De Morgans theorem, can be stated by the following
two formulas, given n sets A1 , A2 , , An
A1 A2 An = A1 A2 An
(3)
A1 A2 An = A1 A2 An
(4)
Introduction to Communications
23 / 93
Definitions
Introduction to Communications
24 / 93
Definitions
Introduction to Communications
25 / 93
Definitions
Introduction to Communications
26 / 93
Definitions
Introduction to Communications
27 / 93
Definitions
Inclusion-exclusion principle
For two finite sets A and B, the following is true
|A B| = |A| + |B| |A B|
(5)
(6)
As a general case,
|n
i=1 Ai | =
n
X
|Ai |
i=1
n
X
i<j
|Ai Aj |+
n
X
i<j<k
Introduction to Communications
28 / 93
Definitions
|A1 |
=5
A2 = {1, 2, 3, 9}
|A2 |
=4
A3 = {2, 4, 9, 10}
|A3 |
=4
A1 A2 = {2, 3, 9}
|A1 A2 |
=3
A1 A3 = {2, 9, 10}
|A1 A3 |
=3
A2 A3 = {2, 9}
|A2 A3 |
=2
|A1 A2 A3 |
=2
A1 A2 A3 = {2, 9}
|A1 A2 A3 | = (5 + 4 + 4) (3 + 3 + 2) + 2 = 7. Now
A1 A2 A3 = {1, 2, 3, 4, 7, 9, 10} and the cardinality is 7 which verifies the principle.
Introduction to Communications
29 / 93
Definitions
Example:(problem solving ) Suppose in a party we have guests with white or red shirts and
black or non-black shoes. Below are the combinations:
- 10 people with white shirts and 8 people with red shirts;
- 4 people have black shoes and white shirts;
- 3 people have black shoes and red shirts;
- Total number of people with white or red shirts or black shoes is 21. Need to find the number
of people with black shoes.
Solution: To simplify, we designate people of red shirts as set R, black shoes as set B and
white shirts as set W. Now from the given conditions, we have the following:
|W | = 10, |R| = 8, |B W | = 4, |B R| = 3, |W R B| = 21
and need to find |B|. From exclusion-inclusion principle, we can write
|W R B| = |W | + |R| + |B| |W R| |R B| |W B| + |W R B|
21 = 10 + 8 + |B| 0 3 4 + 0 (|W R| = 0)
From
above, we get |B| = 10.
Instructor: Dr. Intekhaab Siddiquee
Introduction to Communications
30 / 93
Definitions
Functions
A function f is a rule that maps an element from a specific set, called the domain to an
element of another set, called co-domain. Mathematically represented as
f :AB
If x is an element from the domain, the corresponding element of the co-domain to which x is
mapped, is represented as f (x), The set of all the elements of the co-domain that are
obtained by the mapping of all the elements of the domain form a subset of the co-domain and
is called the range of the function.
Introduction to Communications
31 / 93
Definitions
Definitions
Random Experiment is a process by which certain uncertain is observed. An outcome is
the result of the random experiment. Set of all possible outcomes is called the sample space.
Example: Tossing a coin - sample space is {H, T }.
Example: Rolling a dice - sample space is {1, 2, 3, 4, 5, 6}.
Example: Tossing a coin two times the sequence of heads and tails is to be observed sample space is
S = {(H, T ), (H, H), (T, T ), (T, H)}.
When a random experiment is repeated, each experiment is called a trial.
An event is a subset of the sample space. If, e.g. the event is the set of outcomes of even
numbers when die is rolled, event is the set {2, 4, 6}.
Intersection and union of events are also events. A B occurs if either A or B occurs.
A B occurs if both A and B occur.
Introduction to Communications
32 / 93
Definitions
Example: We assume that any outcome between 1 and 6 is equally likely when you roll a dice,
P (1) = P (2) = P (3) = P (4) = P (5) = P (6). Because sum of probabilities of these outcomes
comprising the universal set must be equal to 1,
P (1) = P (2) = P (3) = P (4) = P (5) = P (6) =
1
.
6
Introduction to Communications
33 / 93
Definitions
P (A B) = P (A) P (A B)
P (A B) = P (A) + P (B) P (A B)
n
[
!
Ai
i=1
n
X
i=1
P (Ai )
P (Ai Aj )+
P (Ai Aj Ak ) + (1)n1 P
n
\
!
Ai
(9)
i=1
i<j<k
(8)
i<j
Introduction to Communications
34 / 93
Definitions
P (sj )
sj A
When sample space is finite and the outcomes are equally likely,
P (sj ) =
|A|
|S|
1
1
1
1
+ + =
6
6
6
2
Example: In a roll of dice two consecutive times, what is the probability of the sum of the rolls being 11?
Answer: The sample space in this case will have 36 elements, each element is of type (ai , bi ) where ai {1, 2, 3, 4, 5, 6}
and bi {1, 2, 3, 4, 5, 6}.:
A = {(6, 5), (5, 6)}
Hence the probability of the event under consideration is
P (A) =
|A|
2
1
=
=
|S|
36
18
Introduction to Communications
35 / 93
Definitions
Conditional Probability
Conditional probability: If A and B are events in a sample space S, then conditional
probability of A given B is the probability of A after outcomes in B has been observed. So,
we can write the following two equations:
P (B)P (A|B) = P (A B)
(10)
P (A)P (B|A) = P (A B)
(11)
(12)
P (A|B) =
(13)
P (AB)
P (B)
P (AB)
P (B)
=
=
P (B)
P (B)
P (A)
P (B)
= 1.
Introduction to Communications
36 / 93
Definitions
3
9
1
, P (A2 |A1 )
3
2
8
1
, P (A3 |A1 , A2 )
4
1
.
7
Hence
Introduction to Communications
1 1 1
1
=
3 4 7
84
37 / 93
Definitions
Independent Events
Two events A and B are independent if P (A|B) = P (A) or equivalently, P (B|A) = P (B).
When events A and B are independent, we see that
P (A B) = P (A)P (B)
Three events A, B and C are independent if P (A B C) = P (A) P (B) P (C) and any
two of the events are mutually independent, i.e.
P (A B) = P (A) P (B); P (A C) = P (A) P (C); P (B C) = P (B) P (C)
n events A1 , A2 , A3 , An are independent if
P (A1 A2 An ) = P (A1 ) P (A2 ) P (An ) and any combination of n 1 events are
mutually independent.
Introduction to Communications
38 / 93
Definitions
Introduction to Communications
(14)
39 / 93
Definitions
1
2
5
= 0.96875
To note that the probability, in fact, is 1 minus the probability of not getting a head in any of the
tosses.
Introduction to Communications
40 / 93
Definitions
X
i
P (A Bi ) =
(15)
P (A|Bi )P (Bi )
Introduction to Communications
41 / 93
Definitions
Introduction to Communications
42 / 93
Definitions
Bayes Rule
Bayes Rule provides a way to find P (B|A) from P (A|B). From equations. 12 and 13, the
following can be obtained (on condition that P (A) 6= 0):
P (A|B)P (B) = P (B|A)P (A)
(16)
P (A|B)P (B)
P (A)
(17)
P (B|A) =
If B1 , B2 , B3 , form a partition of A, law of total probability can be used to get the following
equation:
P (A|Bj )P (Bj )
P (Bj |A) = P
P (A|Bi )P (Bi )
(18)
Introduction to Communications
43 / 93
Definitions
P (B3 |R) =
Introduction to Communications
44 / 93
Definitions
Combinatorics
Introduction to Communications
45 / 93
Definitions
Counting methods
Multiplication principle: Suppose number of outcomes in a series of r experiments are
n1 , n2 , , nr respectively. Then there are total of n1 n2 nr possible outcomes in
the series of r experiments.
Example: How many different words containing three different letters from English alphabet
can be formed?
Answer: Suppose the experiments are 1. choosing the first letter, 2. choosing the second
letter and 3. choosing the third letter.
Outcome of first experiment (possible ways to choose the first letter)=26
Outcome of second experiment (possible ways to choose the second letter)=25
Outcome of third experiment (possible ways to choose the third letter)=24
Hence, the total number of outcomes, i.e. ways to form the words is 26 25 24 = 15600.
Introduction to Communications
46 / 93
Definitions
Sampling
Sampling: Sampling from a set is the process of choosing an element from that set.
Random sampling is the process where each element of the set has equal probability of being
chosen. Two kinds of sampling are possible:
a. sampling with replacement (no restriction on number of times an element from the set may
be chosen) and
b. sampling without replacement (an element, once chosen, is removed from the set).
If ordering matters, i.e. a1 , a2 , a3 6= a1 , a3 , a2 , e.g. the sampling is called ordered
sampling. If order does not matter, the sampling is unordered.
Introduction to Communications
47 / 93
Definitions
Introduction to Communications
48 / 93
Definitions
n(n 1)(n 2) (n r + 1)
(n r)(n r 1) 3 2 1
n!
(n r)!
(19)
Introduction to Communications
49 / 93
Definitions
Permutation of n elements
Suppose we have a set of n different elements. We want to find out how many ways they can
be ordered. Obviously, the total number of ordering is given by
(n)(n 1)(n 2) (3)(2)(1) = n!
(20)
n!
.
0!
Example: Number of ways 10 elements in a set can be arranged to get different ordered sets
is given by
10! = 3628800
Introduction to Communications
50 / 93
Definitions
n!
Prn
=
r!
r!(n r)!
(21)
n
.
r
n
X
n
k=0
ak bnk
Introduction to Communications
(22)
51 / 93
Definitions
Combinations...examples
Example: Suppose we have a set A with 10 partitions A1 , A2 , , A10 . How many different
subsets of A are unions of 3 partitions of A?
Answer: The number of subsets as union of 3 partitions from A will be given by
C310 =
10!
= 120
3!7!
Example: What is the probability of drawing 3 cards from a deck of 52 cards such that at least
one of them is an ace?
Answer: Total number of ways 3 cards can be drawn from 52 cards is
52
.
3
Suppose we
separate the 4 aces and draw 3 cards out of 48. Total number of ways this can be done is
48
.
3
This is the number of draws from 52 cards that will have no aces at all. So the number of draws
having at least one ace is
52
3
52
3
48
3
22100 17296
= 0.2174
22100
Introduction to Communications
52 / 93
Definitions
n
nk
(23)
= 2n
(24)
n + 1 n n
=
+
(0 k n)
k+1
k+1
k
k
m + n X
m
n
=
Vandermondes identity
k
i
ki
i=0
Introduction to Communications
(25)
(26)
53 / 93
Definitions
n
k
pk (1 p)nk
(27)
Suppose we have n elements and we want to divide these elements into r groups containing
n1 , n2 , , nr elements in the groups. The number of ways this can be done is given by
n
n!
=
n1 , n2 , , nr
n1 !n2 ! nr !
(28)
X
n1 +n2 ++nr =n
n
r
x n1 x n2 x n
r
n1 , n2 , , nr 1 2
Introduction to Communications
(29)
54 / 93
Definitions
Introduction to Communications
(30)
55 / 93
Definitions
nk
n!
Pkn = (nk)!
n
n!
= k!(nk)!
k
n+k1
k
Introduction to Communications
56 / 93
Definitions
(31)
(32)
xRX
P (x A) =
(33)
xA
Introduction to Communications
57 / 93
Definitions
(x , y)
Number of Outcome
P(X=S)
(1,1)
1
36
(1,2) , (2,1)
2
36
3
36
4
36
5
36
6
36
5
36
4
36
10
3
36
11
(5,6) , (6,5)
2
36
12
(6,6)
1
36
Introduction to Communications
58 / 93
Definitions
Introduction to Communications
59 / 93
Definitions
Introduction to Communications
60 / 93
Definitions
Uniform distributions
Uniform Distribution: A random variable X that assumes the values x1 , x2 , , xk has a
uniform distribution if it assumes each of the values x1 , x2 , , xk with equal probability. The
random variable X is said to be uniformly distributed with the following probability mass
function:
P (X = xi ) =
1
k
1ik
Example: The random variable X that takes on the value of a single dice roll has a uniform
distribution, since X can assume any of the 6 values in the set {1, 2, 3, 4, 5, 6} with an equal
probability for each. This probability is 1/6.
Introduction to Communications
61 / 93
Definitions
Bernoulli distribution
A random variable X is said to be a Bernoulli random variable with parameter p, written as
X Bernoulli(p) if its PMF is given by
PX (x) = 1 p
for x = 1
(34)
for x = 0
otherwise
Introduction to Communications
62 / 93
Definitions
Geometric distribution
A random variable X is said to be a Geometric random variable with parameter p, written as
X Geometric(p) if its PMF is given by
PX (k) =
p(1 p)k1
for k = 1, 2, 3,
otherwise
(35)
Interpretation of k is the following: It is total number of trials to get the first success, if the
success probability of a trial is p.
Introduction to Communications
63 / 93
Definitions
Binomial distribution
A random variable X is said to be a Binomial distributed random variable with parameters n
and p, written as X Binomial(n, p) if its PMF is given by
PX (k) =
n k
p (1
k
p)nk
for k = 1, 2, 3,
(36)
otherwise
A binomial distribution may be considered as number of ways the population can be divided
into two groups of k and n k each.
Introduction to Communications
64 / 93
Definitions
Introduction to Communications
65 / 93
Definitions
Multinomial distribution
In a multinomial distribution, a set of variables X1 , X2 , . . . , Xn have a probability mass
function defined as
N!
P (X1 = x1 , ..., Xn = xn ) = Qn
i=1
n
Y
xi !
xi = N and
pi i
(37)
i=1
pi = 1.
Multinomial coefficients provide the number of ways N elements of a set can be grouped into
n groups having x1 , x2 , , xn elements in them (x1 + x2 + + xn = N )
Example: How many ways cooking items may be divided among 11 people so that 6 of them
get to prepare the main dish, 2 of them desert and 3 of them the appetizer?
Answer: It can be done in
11!
6!2!3!
= 4620 ways.
Introduction to Communications
66 / 93
Definitions
PX (k) =
k1 m
p (1
m1
p)km
for k = m, m + 1, m + 2,
(38)
otherwise
Introduction to Communications
67 / 93
Definitions
Poisson distribution
A random variable X is said to be a Poisson random variable with parameter , written as
X P oisson() if its range is RX = 0, 1, 2, 3, and PMF is given by
PX (k) =
e k
for k RX
otherwise
k!
Introduction to Communications
(39)
68 / 93
Definitions
Expectation
Let X be a discrete random variable with range RX = {x1 , x2 , x3 , }, where RX is finite
or countably infinite. The expected value of X, is defined as the weighted average of the values
in the range.
E[X] =
X
xk RX
xk P (X = xk ) =
(40)
xk PX (xk )
xk RX
Introduction to Communications
69 / 93
Definitions
Linearity of Expectation
Theorem:
E[aX + b] = aE[X] + b for all a, b R
(41)
(42)
Introduction to Communications
(43)
70 / 93
Definitions
E[X] = 0 (1 p) + 1 p = p
Introduction to Communications
71 / 93
Definitions
E[X] =
X
k=1
xk PX (xk )
kp(1 p)k1
k=1
=p
Introduction to Communications
72 / 93
Definitions
E[X] =
X
k=0
xk PX (xk )
k=0
= e
e k
k!
k
(k 1)!
k=1
= e
k=1
= e
k1
(k 1)!
(1 + +
2
3
+
+ )
2!
3!
= e e
=
Introduction to Communications
73 / 93
Definitions
Introduction to Communications
74 / 93
Definitions
(44)
x2k PX (xk )
xk RX
p
V ar(X)
Introduction to Communications
75 / 93
Definitions
(45)
Introduction to Communications
(46)
76 / 93
Definitions
(47)
Example: Suppose a coin is tossed 3 times and X is the variable representing sum of the
number of heads in the tosses. The possible outcomes are in the set {0, 1, 23}. The
probabilities are the following:
P (X = 0) = (1/2)3 = 1/8
3
P (X = 1) =
(1/2)3 = 3/8
1
3
P (X = 2) =
(1/2)3 = 3/8
2
3
P (X = 3) =
(1/2)3 = 1/8
3
Instructor: Dr. Intekhaab Siddiquee
Introduction to Communications
FX (x) =
1/8 x 0
1/2 x 1
7/8 x 2
1
x3
77 / 93
Definitions
Introduction to Communications
(48)
78 / 93
Definitions
(49)
d
[FX (x)]
dx
(50)
FX (x) =
(51)
fX (u)du
(52)
fX (x)dx = 1
Introduction to Communications
79 / 93
Definitions
Examples
Example: Let the probability density function of a random variable X be defined as
fX (x) =
cebx
x0
otherwise
where c and b are positive constants. Find (a) possible values of c and b, (b) CDF, (c)
P (1 X 3).
Answer: (a) From the property that
Hence =
c
b
fX (x)dx = 1, we get
cebx dx =
c
=1
b
= 1, i.e. c = b.
Introduction to Communications
80 / 93
Definitions
ceby dy = 1
FX (x) =
cebx
= 1 ebx
b
cebx dx = eb e3b
(using CDF)
(using PDF)
Introduction to Communications
81 / 93
Definitions
Expected Value
Expected value of a continuous random variable is defined as
Z
E[X] = X =
(54)
xfX (x)dx
The expected value of the sum of two random variables X and Y is the sum of their expected
values:
X+Y = X + Y
Introduction to Communications
82 / 93
Definitions
(55)
g(x)fX (x)dx
Z
=a
Z
xfX (x)dx + b
fX (x)dx
= a X + b
Introduction to Communications
83 / 93
Definitions
2
V ar(X) = X
=
(x X )2 fX (x)dx =
x2 fX (x)dx 2X
(56)
For independent random variables X and Y, the variance of their sum or difference is the sum
of their variances:
2
2
2
X+Y
= X
+ Y
2
2
2
XY
= X
+ Y
Introduction to Communications
84 / 93
Definitions
fX (x)
fX (g 1 y)
= 0 1
g 0 (x)
g (g y)
(57)
x0
fX (x)
y
x
fX (x)
fX (g 1 (y))
= 0 1
g 0 (x)
g (g (y))
Introduction to Communications
85 / 93
Definitions
yb
.
a
We can write
fY (y) =
1
fX
a
yb
a
Introduction to Communications
86 / 93
Definitions
fY (y) =
n
n
X
X
fX (xi )
=
0
|g (xi )|
i=1
i=1
f (g 1 (yi ))
X0
g (g 1 (yi ))
(58)
Introduction to Communications
87 / 93
Definitions
(59)
1
ba
Introduction to Communications
88 / 93
Definitions
Normal Distribution
The probability density of the normal distribution is:
2
fX (x | , 2 ) =
(x)
1
2 2
e
2
(60)
is the mean of the distribution. The parameter is its standard deviation with its variance
2 . A random variable with a Gaussian distribution is said to be normally distributed and is
called a normal deviate.
Introduction to Communications
89 / 93
Definitions
x x2 /(22 )
e
,
2
x 0,
(61)
where is the scale parameter of the distribution. The cumulative distribution function is
2
F (x; ) = 1 ex
/(2 2 )
x [0, ).
(62)
' 1.253;
2
2
X
=
4 2
' 0.429 2
2
Introduction to Communications
90 / 93
Definitions
Introduction to Communications
91 / 93
Definitions
f (x; ) =
ex
x 0,
x < 0.
Introduction to Communications
(63)
92 / 93
Definitions
F (x; ) =
1 ex
x 0,
x < 0.
(64)
The mean or expected value of an exponentially distributed random variable X with rate
parameter is given by E[X] =
1
.
1
.
2
Introduction to Communications
93 / 93