EE376A: Homework #1

Due on Thursday, January 21, 2016

You can hand in the homework either after class or deposit it, before 5 PM, in the EE376A
drawer of the class file cabinet on the second floor of the Packard Building.
1. Entropy of Hamming Code.
Consider information bits X1 , X2 , X3 , X4 {0, 1} chosen uniformly at random, together with check bits X5 , X6 , X7 chosen to make the parity of the circles even.

X5
X3

X2
X1
X7

X6

X4

Thus, for example,

1
1
1

becomes
0
0

1
1

That is, 1011 becomes 1011010.

(a) What is the entropy of H(X1 , X2 , ..., X7 )?

Homework 1

Page 1 of ??

Now we make an error (or not) in one of the bits (or none). Let Y = X e, where
e is equally likely to be (1, 0, . . . , 0), (0, 1, 0, . . . , 0), . . . , (0, 0, . . . , 0, 1), or (0, 0, . . . , 0),
and e is independent of X.
(b) What is the entropy of Y?
(c) What is H(X|Y)?
(d) What is I(X; Y)?

2. Entropy of functions of a random variable.

Let X be a discrete random variable.
(a) Show that the entropy of a function of X is less than or equal to the entropy of
X by justifying the following steps:
(a)

H(X, g(X)) = H(X) + H(g(X)|X)

(b)

= H(X).

(c)

H(X, g(X)) = H(g(X)) + H(X|g(X))

(d)

H(g(X)).
Thus H(g(X)) H(X).
(b) Show that if Z = g(Y ) then H(X|Y ) H(X|Z).
3. Data Processing Inequality.
If X,Y,Z form a markov triplet (X Y Z), show that:
(a) H(X|Y ) = H(X|Y, Z) and H(Z|Y ) = H(Z|X, Y )
(b) H(X|Y ) H(X|Z)
(c) I(X; Y ) I(X; Z) and I(Y ; Z) I(X; Z)
(d) I(X; Z|Y ) = 0
The following definition may be useful:
Definition 1: The conditional mutual information of random variables X and Y given
Z is defined by
I(X; Y |Z) = H(X|Z) H(X|Y, Z)
X
P (x, y|z)
=
P (x, y, z) log
P (x|z)P (y|z)
x,y,z

Homework 1

Page 2 of ??

4. Entropy of time to first success.

A fair coin is flipped until the first head occurs. Let X denote the number of flips
required.
(a) Find the entropy H(X) in bits. The following expressions may be useful:

r = r/(1 r),

n=1

nrn = r/(1 r)2 .

n=1

(b) Find an efficient sequence of yes-no questions of the form, Is X contained in

the set S?. Compare H(X) to the expected number of questions required to
determine X.
(c) Let Y denote the number of flips until the second head appears. Thus, for example, Y = 5 if the second head appears on the 5th flip. Argue that H(Y ) =
H(X1 + X2 ) < H(X1 , X2 ) = 2H(X), and interpret in words.

5. Example of joint entropy.

Let p(x, y) be given by
Y
X

1
4

1
4
1
2

0
1 0
Find
(a) H(X), H(Y ).
(b) H(X|Y ), H(Y |X).
(c) H(X, Y ).
(d) I(X; Y ).

6. Infinite entropy. This problem

the entropy of a discrete random variable
P shows2 that
1
can be infinite. Let A =
(n
log
n)
.
(It is easy to show that A is finite by
n=2
bounding the infinite sum by the integral of (x log2 x)1 .) Show that the integervalued random variable X distributed as P (X = n) = (An log2 n)1 for n = 2, 3, . . .
has H(X) = +.
7. A measure of correlation.
Let X1 and X2 be identically distributed with positive entropy, but not necessarily
Homework 1

Page 3 of ??

independent. Note that H(X1 ) = H(X2 ). Let

=1

H(X2 |X1 )
.
H(X1 )

(a) Show that 0 1

(b) Show that I(X1 ; X2 ) = H(X1 )
(c) Show that = 0 iff X1 is independent of X2
(d) Show that = 1 iff there exists a one-to-one function g such that X1 = g(X2 ) with
probability one.

8. Two looks.
Here is a statement about pairwise independence and joint independence. Let X, Y1 ,
and Y2 be binary random variables. If I(X; Y1 ) = 0 and I(X; Y2 ) = 0, does it follow
that I(X; Y1 , Y2 ) = 0?
(a) Yes or no? Prove or provide a counterexample.
(b) If I(X; Y1 ) = 0 and I(X; Y2 ) = 0 in the above problem, does it follow that
I(Y1 ; Y2 ) = 0?

9. Markovs inequality for probabilities.

Let p(x) be a probability mass function. Prove, for all d 0,
 
1
P (p(X) d) log
H(X).
d

10. Smallest Typical Set

We have a memoryless source U , i.e., U1 , U2 , , are i.i.d. U , where U takes values
in the finite alphabet U. Let un denote the n-tuple (u1 , u2 , , un ) and p(un ) be its
probability, i.e.,
p(un ) = ni=1 PU (ui )
Let > 0 and for every n let B (n) U n be an arbitrary set of source sequences
satisfying |B (n) | 2n(H(U )) . Prove that:
lim P (U n B (n) ) = 0.

n
(n)

In words, the typical set A (defined in class) is essentially smallest (on an exponential
scale) among the sets that have non-negligible probability.

Homework 1

Page 4 of ??