EECE667: Pattern Recognition  

Hwk 1 Solution  

Problem 1: Expectation
1) Suppose that the two variables x and z are statistically independent. Show that the
mean and variance of their sum satisfies

E[x + z] = E[x] + E[z]

var[x + z] = var[x] + var[z]

Solution:
Since x and z are independent, their joint distribution factorizes p(x,z) = p(x)p(z), and so

Similarly for the variances, we first note that

(x + z – E[x+z])2 = (x – E[x])2 + (z – E[z])2 + 2(x – E[x])(z – E[z])

Where the final term will integrate toz ero with respect to the factorized distribution p(x)p(z).
Hence

For discrete variables the integrals are replaced by summations, and the same results are again
obtained.

2) If an integer between 100 and 200 is to be chosen at random, what is the expected
value?
 3) Suppose that X is a random variable. E(X) = μ, Var(X) = σ2, then what is the
value of E[X(X − 1)] =?

Answer: E(X) = (100 + 101 + ... + 200)/101 = 150

Problem 2: Measurement and Data

1) Compute the 2*2 covariance matrix for the data matrix X.

µx(1) = 2.5
µx(2) = 6.5
The following table shows the mean centered data:
X(1) X(2)
-1.5 -1.5
0.5 0.5
-0.5 -0.5
1.5 1.5
-1.5 1.5
1.5 -1.5
0.5 -0.5
-0.5 0.5

2*2 covariance matrix is now given by X’X / N

10 0
0 10
8

1.25 0
0 1.25
 2) Pick all that apply: X(1) and X(2) are:
(a) Independent (b) Dependent (c) Correlated (d) Uncorrelated

Non diagonal entries of covariance matrix = 0.

Hence (d) uncorrelated
NOTE: Covariance matrix does not tell anything about independence. For instance
assume a uniform distribution. To test independence P(X(1), X(2)) * P(X(2))
Consider the first value, X(1)1 = 1 and X(2)1 = 5

P(X(1) =1, X(2) = 5) = P(1, 5)

From table =

P(X(1) = 1) =

P(X(2) = 5) =

P(X(1) = 1) * P(X(2) =5) =

Hence they are not independent. Since no information is given about whether the data is
uniformly distributed or not, no comment can be made about independence.

3) The correlation coefficient between X(1) and X(1) is

(a) 1.00 (b) 0 (c) 1.42 (d)Infinity

Since non diagonal entries are 0, correlation coefficient between X(1) and X(2) has to be 0.

4) The correlation coefficient between X(1) and X(2) is

(a) 1.42 (b) 1.25 (c) 130 (d) 0

For any data, this is always equal to 1. The correlation between a variable and itself is always 1.

Problem 3: Probability and Bayesian Theorem

1) Suppose that A is an event such that Pr(A) = 0 and that B is any other event. Prove
that A and B are independent events.
Solution:
Since the event A^B is a subset of the event A, and Pr(A) = 0, so Pr(A^B) = 0. Hence
Pr(A^B) = 0 = Pr(A) * Pr(B).
2) A box contains three cards. One card is red on both sides, one card is green on both
sides, and one card is red on one side and green on the other. Then we randomly
select one card from this box, and we can know the color of the selected card’s upper
side. If this side is green, what is the probability that the other side of the card is also
green?
For each of the cases below, answer what box would you open so as to maximize the
chances that the box you open contains the prize. Support your arguments by computing
the probability of the prize being in box A and C.

Here are 2 strategies according to which box B was chosen to be opened:

In this strategy if you first pick a box (in this case A) with a prize, then one of the other
two boxes is opened at random. On the other hand, if you first choose a box that has no
prize, then the empty box that you did not pick is chosen.

In this strategy one of empty boxes is chosen at random (independently of whether you
initially pick a box with a prize or not).

3) In a certain day care class, 30 percent of the children have grey eyes, 50 percent of
the children have blue eyes, and the other 20 percent’s eyes are in other colors. One
day they play a game together. In the first run, 65 percent of the grey eye kids were
selected into the game, 82 percent of the blue eye kids selected in, and 50 percent of
the kids with other colors were chosen. So if a child is selected at random from the
class, and we know that he was not in the first run game, what is the probability that
he has blue eyes?

Solution:
Assume B: blue eyes; O: other color eyes; G: grey eyes; NF: not in the first run game
| . .
P(B|NF) = | |
0.3051
| . . . . . .
Problem 4: Gaussian and Bayesian Distribution

a. Find p(x), p(y), p(x|y), and p(y|x). In addition, give a brief description of each of
these distributions.
Solution:
Problem 5: Risk: Consider a classification problem in which the loss incurred when
an input vector from class Ck is classified as belonging to class Cj is given by the loss matrix Lkj,
and for which the loss incurred in selecting the reject option is λ. Find the decision criterion that
will give the minimum expected loss. What is the relationship between λ and the rejection
threshold θ?

Solution:

A vector x belongs to class Ck with probability p(Ck | x). If we decide to assign x to class Cj
we will incur an expected loss of ∑ | , whereas if we select the reject option we
will incur a loss of λ. Thus, if

|
Then we minimize the expected loss if we take the following action

, ∑ | ;
Choose
, .

Problem 6: Entropy Calculation:

1) Find entropy of X if : X= 1 with probability of p and 0 with probability of 1-p
Solution:
H(X) = -p log p – (1- p) log (1-p) H(p).

In particular, H(X) = 1 bit when p= 1/2. The graph of the function H(p) is shown in
Figure 2.1. The figure illustrates some of the basic properties of entropy--- it is a concave
function of the distribution and equals 0 when p =0 or 1. This makes sense, because when p = 0
or 1, the variable is not random and there is no uncertainty. Similarly, the uncertainty is
maximum when p = 1/2, which also corresponds to the maximum value of the entropy.

2) Show that the entropy of a univariate Gaussian is

given by: .

Note that:

3) Let (X, Y) have the following joint distribution:

The marginal distribution of X is (1/2,1/4,1/8,1/8)and the marginal distribution of Y is (1/4, 1/4,
1/4, 1/4 ). Find H(X|Y) and H(Y|X)

Solution:
H(X|Y) ∑ |

= 1/4 H(1/2, 1/4, 1/8, 1/8) + 1/4 H(1/4, 1/2, 1/8, 1/8)

+ 1/4 H(1/4, 1/4, 1/4, 1/4) + 1/4H(1,0, 0, 0)

= + 2 0

= 11/ 8 bits.

Similarly H(Y|X) = 13/8 bits and H(X, Y) = 27/8 bits.

Remark: Note that H(Y|X) is not equal to H(X|Y). However, H(X) – H(X|Y) = H(Y) – H(Y|X), a
property that we shall exploit later.

4) Let A = (0, 1) and consider two distributions p and q on A. Let p(0) = 1 - r, p( 1) = r, and
let q(0) = 1 - s, q( 1) = s. Find the mutual information D(p||q) and D(q||p)

Note that :
Solution: