The noiseless coding theorem

When both the information channel and communication system are error free, the principal function of the communication system is to represent the source as compactly as possible. Under these circumstances, the noiseless coding theorem, also called Shannons first theorem, defines the minimum average code word length per source symbol that can be achieved. A source of information with finite ensemble (A, z) and statistically independent source symbols is called a zero-memory source. If we consider its output to be an n-tuple of symbols from the source alphabet (rather than a single symbol), the source output is a block random variable. It takes one of Jn possible values, denoted i, from the set of all possible n element sequences A = {1, 2, Jn }, where each i is composed of n symbols from A.

The probability of a given j, is P(j) and is related to the single symbol probabilities P(j) = P(aj1) P(aj2) P(ajn) (i) where the additional subscript is used to index the n symbols from A that make up an j. The vector z denotes the set of all source probabilities {P(1), , P(ajn)}, and the entropy of the source is
jn

H(z)= - P(i) log P(i)

i=1

which upon simplifying yields H(z)= nH(z) Thus, the entropy of the zero-memory information source (that produces the block random variable) is n times the entropy of the corresponding single symbol source. Such a source is referred to as the nth extension of the single symbol or nonextended source.

As the self-information of source output i is log[1/P(i)], it seems reasonable to code i with a code word of integer length l(i) such that log[1/P(i)] <= l(i) < log[1/P(i)] +1 Multiplying by P(i) and summing over all i gives
jn jn jn

P(i) log[1/P(i)] < = P(i) l(i) < P(i) log[1/P(i)] +1

i=1 i=1 i=1

or H(z) < = Lavg < H(z) +1 where Lavg represents the average word length of the code corresponding to the nth extension of the nonextended source. Dividing by n H(z) <= Lavg/n < H(z) + 1/n which, in the limiting case, becomes

This states the Shanons first theorem for a zero-memory source which reveals that it is possible to make Lavg/n arbitrarily close to H(z) by coding infinitely long extensions of the source. The efficiency = n H(z)/Lavg

Problem: A zero-memory information source with source alphabet A={a1,a2} has symbol probabilities P(a1)=2/3 and P(a2) =1/3. What is the efficiency of the nonextended and second extension of the source?