id > Hd
Sis the staf nt (n} ae mul ease, los fom the
dtcorem of oa probabity tat the mcondvoned probity of 4
costs
v(e}~ Sein trte| ma = "SP réeimo dp arte)
and the unconditioned probability ofthe event eror, denoted fit
Hg = 1 Ie) aur
For he womsag ca in Fig. 235
alee nent
nlm = = 2)=
ant
Reh = Pinte eh do
trim fein ap
ssrcgneryeme
He PED ate fencer of Magen cle ine nator
|
aa
‘The probability of rors tereore
PLC] = Pima ee ae
ee
[Equation 2.184 can be expresed in terms of the fonction OC ) of
Bq, 2.50 by making the change of variable 2 =(p — in the fst
Jnagel and = (p = aie in the sooond: then
19} «rimao(®@=4) +rtmao(?=)
ne pti of ay Ht ess, = =
em +3) and te evr roby jus OU — 592A.
apa probes, Before a tansision occur, hea prot proba
bit Bm] ofeach mesiage mis bow a he cer. When a olage
Fp rzsed thea posta! probaty of each mesage my te
teers Pm, [f= pland the optimum reaver dies infor of ht
rresage for wich the a porartprbabliy is grete..The canna,
egal cmon ean ls coh
Tn the stnenee of 2 chante, the "9
es)
]
[E3x>4]
‘hws we may immediatly invoke Fo, 2165, wih d= 1 — d= be TE
‘oauumeltntp Ol and N= 1, tbe Chern bound yk
noe (ETT
-(GresT
(wt. 105 rwoeaauery maxon
On the other hand, substcating
yields
ap
1p) _ 01009)
pe < PP O19)
0) < ASP = Rw 008.
“The comparison betwoen the strength ofthe two bounds is more dramatic
ite tiple N (039. The Chernoff bound s then cabo to yield 2.2% 10°,
‘whereas the weak law bound is vide by thre to yield 0.014
nthe weak aw bound
Central Liet Theorem,
‘We noted in connesion with Fig. 2.7 that he binomial density function
(hati, th density function ofthe semple mean
re
make
in the particular cate for which the fx; are tatinticlly independent binary
random variables, each with mean 3 and Vviace 2) exhibits an envelope
that becomes simultancously narrower end more bellshaped asf
inerenen The face that the envelope bacomes narrower is atebuue to
the normalization factor 1/M in Fa, 2467: as M increases, the mean
Fema remains constant, whereas the variance og? = o¥/M decreases,
‘Wie are interested here in investigating the tendency ofthe envelope to
become bellshaped. Consequently, insted of m, we consider the reited
random variable + defined by
fees
Waa en® 2.168)
eae,
‘With this normalization = 0 and 0 = 6% so that both the mean and
the variance of = remain constant sx NV ineeats. The Behavior of the
‘envelope ofp, a6 increases i evidenced in Fig 285.
"The bellshaped tndeney stated in Fig. 2.45 for the binomial
istibution is an example of a much more general group of theorens,
Called collectively the contra dol theorem, one statement of which retds
fs follows:
Let (y) denote a set of stray Independent, 20ro-mean random
riches. each with the came deny function py, = Py and fate
arlance 0. Define
Le
jaa (2.1688)
1 The ptr i hore tte ce ical he inde y them, Ts
fester eed tere re ede Refocse 50a)
cereal
|
corns. tar THOR 107
at
1 tt
Lay 4
Genayeoaxe te
Wo pnonenahvers
(epnokentbveo Wr o1e=03.v=40
FRpue245 ‘The Nat bin deny faction semmaad o 260-084 aod mm
[rien [Gc ermtas erm
oth ap e680)
or, when 8 = 420,
) cas
tim [ped da = o|
Since the choiee doesnot ffi the right-hand side, the integration interval”
‘OF Eg. 21700 may be eogen either to include or exclude the points @
dnd
‘Discusion. The ceneal Limit theorem does not imply that, isell
approaches the Gaussian density function; it does imply that the stegral
Epic between fed nits proaches a value given by te inepal of the
Gausitn density function, The distinction is clear if we consider p,
torte binomial; for any 1, 0 mater how large psa su of impulses
td therfore never approximates the encoth) Geusin density funtion.
"The cenel limit theorem is operationally weful in estimating such
ey or
when is finite but very large aad [ao is relatively small constant
CGniependent of N}. Quantitative evustion ofthe words “very lage”
Sid Telatively small” depends on dhe details of the original density.
Function pif py Haef ie Gaus, the central init theorem exact for
fay N aod alo An equally tivialcousteresarple isthe binomial ease:
iPtach y, assumes only the valbes —1 and 1, and if is any mumber
eater than Vi,
{Siar ]-oxe(2)
probabil in whi ae, prows with sue
LSyovie) ery
Incsinet
fi Sn>]="
the ose me of pan
dae ]~of) am
nsminseninascntosnsinenepial
cctnaL Ur Tena 109
{a dubious, regardless of hov lege we tke N, Consder, for example, &
‘Sof inary random variables (2) in which for exh i assumes the
Values 0 and L with equal probability. With 9, 2b of = i and
af, we obtain from Eq. 2.171b
LE aat+ofs)-avm eine
af
We have already sen (of, Bg 2.121) that the Q fonction Behaves ex
Ponenvally #8 Qwerty a0.
ors
igeefeem eam
whereas the exact expression is
Hideo) of
ee enters
ports
eo
yetj=rywecm. a7)
_qrons with Nand becomes enormous when Nisiarge, On the other hand,
i realy veriied thatthe Cheroff bound aprss with Eq. 2.173, which
{sin accord with our carr statement thatthe Chernol! bound is ex
‘oncotillytgit,Thus the Chernofl bound should be wed in lieu of the
‘Sata mi ealelation inate suc, ag in whic the Tit of tte
goon Eg, 2.708 meee i N
“Argument. No appricaton ofthe validity ofthe central limit theorem
cat be gained from he Fllowing arguments
Tet M0) denote the characterise function of any one of the N
fdencaly istibuted zero-mean random yarlsbes (x and tet MA)
fdooue the characteristic faction of het nocmaized sum =. Then M0)
fand Mo) ae selated by
= owls} —liter l
Dba 8
M,(0) 3 Ele" 110 ragakanry mizony
In wich we have wed the fc thatthe mean of a rout of aisaly
independent random vrais i the product of ther means.
‘Now let us acme that py sch Sat every moment (7) n= by
2ycocr is nite Then, in ecordans witk Eq, 214b, M6) may BS
‘apes in the ponerse expansion
monte ont Be Pee. ans
Since ~ 0 aad FF = owe have
Moya 8 S440) eT)
where (8 sotinvous funtion that approaches the constant (—i}6)
Seo approactes zo.
rom Egy 2.174 and 2.1786, we bare
iano= win) =a —
aa)
‘me tpeina may be apne a te oe e
nd+H=w
7 eam
which converges for any complex variable w for which jy] < 1. Since we
{he intrest inthe inias N+ co, we ray take Wsoiaty lege hat,
fans fod
54+ Gall
Applying Eq. 2.17 10 Ba. ra fave
ofits a)
Lita eee
paso ua
o-etinnnentnai
si
inane sin
AEvERSIN:E TRANSFORMATION OF RANDOM YrETORS 111
Sines the exponential faction is continuous i fellows that
jin Moy a7)
‘We recognize thatthe iting form of A) the character fuseion
of a zero-mean Gaussian random variable with variance oy
Tee Iannis tbe Sy ton
In rlernia [miorma arm
{is Gaussian, As we have already sean in connection with the binomial
Aistibution, sucha claim ie fale! ‘The operations of limit aking and
fntepration in Eq, 2179 eanao, in general, be interchanged,
‘Although the density function of » dees tend to Gaussian if py is
sufcendy smoot, the general cental limit theorem statement that the
{itribution function converges to Gaussian form hinges onthe atonal
“mocthing” tht i niodced by integrating the density function, to
tthe distribution function F.
{APPENDIX 24 REVERSIBLE TRANSFORMATION OF
RANDOM VECTORS
“The changeotvariables transformation considered in Eq. 2.78 ie a
special case of a reversible transformation of vectors. A transformation
o> y, with both Xana y k-dimensional vectors, sealed reverie is
‘e-to-onethat i i the iverse transformation y+ also exists fr all
Sand y ofinteres, For example, let
YEU... AED (Ata)
‘where each ofthe [0 is «funtion of k variables that i, each signs
1 (difret) number cay ya), (0 vector x(o)- The transformation is
‘eversible if thee exists anther set of Functions fg) such that
X= GO) 2)--. 0). eat)
1 is convent to express Eqs 2.1 inthe more conse form
yet) pan
9) = at. a2)
We now relate, top fora reversible wansformation in which the
partial derivatives 3f/@% and 2p, exist for all and j, 1-61, ke
First we determine the probably distribution funedon F, and then we 112 pmonamunry THEORY
diferente Fo obtain p,. By defnon,
116)=[esoie ox)
‘where /is the region 7
Ale) < Br fle) < Brees nffe) SB QARB)
“oking the derivative OY(28, 2° 4) ofthe right-hand side of Ea.
2A;3e to obtain p(B) is complicated by the fact that J not simply
“xprese in terms ofthe variables of iteration. This difiuly ean be
voided by making the change of variables
yo Me. aaa)
“Then it follows frm the existence ofthe inverse transform g that
a= a0. Aa)
“The region of integration Fcan be expressed simply in terms of y as
I= (768) (Ade)
Since aC) may be substituted for ain the integrand of Eq, 24.3 the
‘nly problem sn performing the change of variables of Eq. 2Asa i to
{ela thediferental volume clemeats de and dy. The relationship is
de = Us, QAse)
whore Uy is te abgolute value ofthe Jacobian Jy) sssciied with
the transformation g. The Jacobian, by defiition, is the determinant
x Sg o> Su
Yn Jao In|
non| . ease)
a ten --* Ja
‘with lente
Jy = Ps FAD & @ASe),
wr eng vii of 2A BTA
140)= [ patonusole
af Po fowoneéoin one
aes
eveRSIBLE TRANSFORMATION OF RANDOM VECTORS 113
“Taking the partial derivative is now teva, and wo obsin the sited
‘elation betseen p, and p, when random vectors y and x are celated by
‘the 1:1 wansformations = ffx); x = #0)
4B) = pulat®)1 8 ean)
Forther insight into the elation between py end py may be gained
ty sealing te findamental interpretation of the probability density
function: py i the function which, When evaluated at a point a aad
sulipied by the volume AV, of « small pion AV, incuding the point.
{yds the probability that xl ie In the rego. But, 7 Has In the
region Mp then y = fx) must le in & coresponding region Aly of
‘Volume AF", whic contsine the point b= fla). Thus
PAD AY, = psa) AY (2A)
Since a = g(), we ave
PAW) SY, = plat) Ae east)
‘Of course, AV, je notin general equal to AV,; indeed, feom Ea, 24.5
ay,
a =H (Ase)
Substtating Ea, 2A.e in Eg, 2A.Sb yields
Ad) ~ ple) easay
which is consistent with Eq. 287.
"Asn ample ofthe use of F247 condor the polar wansformation
xory given by
(A.0)
As shown in Fig. 24.1, the inverse transformation is
= gO = Hc08
30 O and
Eg" my ~ mT = AN AT) = AAA,
dnvohing Bg. 3A.28 and 34.30 we have
WAP = (TAA ALATA
G
Using the wolknown result for determinants® that
ATL = IAL
‘and that for any two Ge K) matrices
leo)
ic11D1,
whic implies
SWAT
‘we have
penton
is 7 TAT
Accordingly, may be ween concisely 3s
~ayh 630)
Fguation 3.90 the genteel form ofthe nonsingular Gaussian density
function, We abeere tat dependsooly on th mean vector my and the
‘covariance matrt A, That Eg. 3.90 represents the most genera form of
‘the nonsingular Gavtsan density function follows from the fac that any
such set of Gustin variables nay be obtained by mats transformation
1 _
21) = RR HE AF
ld
‘me Gauss PROCESS {TL
‘from a statisialy independent st. AS final point, we note t
(Py and M, are Fourier ansforms of each other, the eighthun
By, 390 the inverse transform of
Myo) = exp (— BA + Im), Gan
‘As an example of Eq. 3.9, consider the two-dimensional case with
ee a(S Tp ne
oe
aa eo
won se Saal, 7)
tm, = 0.0, vehwe
aad sax fom at Out ( 1, +) (B)]
sap aa“
‘hie density function has aieady been studied in det in Seton 3.2.
dom process x), and let x0) = (6), 244), -- 200)
ents the random variables obtained by sampling 2) a the set of ime
Festants (4). Ie the variables (9 ate josly Gausian for every finite
st of observation instants ff) then >) i clled a Gaussian proces.
"The conditions thst process must met in order to be Cawsian are
stringent. The one-dimensional central Lm theorem of Chapter 2 has
besn uted, however, to argue thatthe output of ier impulse mole
source, observed at any single Sime, can bo adaquately modeled mathe-
‘mately by » Gavesian random variable, More genealy, tbe multe
‘aa cenzl imi theorem juli te assumption that k output samples
‘observed at any set of times (4} cam be adequately modeled by k random
‘arabes tht are joinly Gaussian, The important condition for the
‘alcity of sch a mathematica! model that the values ofthe observed
"amples depend on the sum oft large number of relatively adopendeat
Dovsurbasions. Since there are many cicumstaness, such as thermal 1 RANDOM WAvEFORDS
noise in resistors, diftsion noise in tansstors, spontaneous emission
ole in pavers aad intergalactic nose im rao astronomy, ia which this
‘onuiion set, Gaus procests sr of the utmost practical (swell as
’
fathematicl) importance.
Specification of Gauslan Processes
“We have sen that an arbitrary random proces is considered species
Sand ony its tales implied fr determining the joit density Function of
$Sampic taken at any finite set of ime instants ((),7= 1,2...» Ode
‘of the important propertics of set of jointly Gaussian vaiables, say
"Geuste sss fs te that pe depends only on the mean values
Eps) Sm, = GF AD 0.9,
and the set of eovariancest (=, 2-4 RE Jb Devos a i
whieh
hy 2 El, = BE) FD)
= Fa — 7, oo)
sve see that
‘how the
La
‘4 Gatssian pres 0) com
15 % denote the random variable), 1
Sy species
“The mean function. To be wb to specify my for any seo it
uit neconary sad solcient that we Kcow function m0), called
‘the man faction of), defined by
my © Bite 693)
For example, since =, andy denote the random vatiales cbtsined by
Stmpling the proses times Zand fy respestvely,
R= mt) G4)
Rams. G4)
‘The couriance function, Siisly,In order to beable to spsily the
sel of covariane®s (i) for any set oF instants (itis necessary and
Shifcient that we Know a function 2,9 clled the coarlance funtion
‘of 1), defined by
244,92 B14) — mK) = mE. 95
{Mee in tet anne eh ema =
SPECICATION OF GAUSSIAN PROCESS 173,
“Then for samples 2 and taken at ands, rspactvely, we have
hu Lhto 6.95)
In imerpreting 0,8, ne think of observing each sample function
(1) of the process fst ak some particular tine? and again at some
times, a8 showa ia Fig. 218. The product of these two samples (vith
means subtracted) is o(o,1) — maOe(e,s) ~ ma} The covarinee
function 27.) describes how the expected value of this product, over
Shecsenle of pens wn vanes essen of hpi
guy 38 lotepetion of he cova fun. Forti peice coke of
PES a tat ue tei 0, Les bea ond eae Steed Oy
xo) Ec) Seen. The
Mo) 2 11-87) = 48
fey=d— Oh 174 RANDOM wacronns
Example. Suppose iti known that a random process 0) is Gaussian
aad)
ng) = sin, G96)
2a. 2 ee 0.968)
‘Tet ferns wo ample aan a ie = tant
of Sig eet
odie etetal,
2) putin ada =
‘The vector ofthe means
m= (mAsdom£ea) = @.-D-
1 follows from Eq, 3.47 (oF Eg. 390) that the joint density function is
i
Pane =
xeol-aacal
“The Corrdaton Fanetion
In dition to the covariance function of @ random proces 20), We
frequently encounter the covrelation fancion, denoted 3,3), and
elined as
840,914 El 0) = THT 69
From Eq, 3.95 we se that (3) nd Zs) re relate by
£46.) = I= MOD = IO,
Au+ns0+0])
a) = Ha) — mA AR) — mH + mA mds)
ls) = RUG) — mM 6.98)
Toied Hak all three of these functions —say
infty 2.) snd Bef 3) may aso be defined for any process 0) that
noe Gavssian ese, however, knowledge ofthese
ply thatthe pres is completely specie.
For example, consider the twa random processes (?) and 2() shown in
folios that. a Gaussian process completely spcied by knowledge of
so
Rai ee
Ry= | become Loeyt)
aye yout) dab
2
Spesrbany caus moc 195
Fig. 219; in both cases every sample function isa constant, and itis
‘dese that
mdi) = mf) = 0
Rode Roa he
‘But, for any observation instant ¢
Priegla) = HB — V2) + 282) + Ka + VD},
whereas
Pans) = He = 1) + He + DL
‘Stationary Guusian Processes
“To be stationary an arbitrary random process must be sue tha all
invariant to any tanlation i time gin. For
pint density function p, of samples {5}
1,2,---k depends only oo the
{and covacianes {2}. Thus sonata by wansaton Tn time
a only if
mg) 8 Ble = Ela, + TNE m+ D
16.99)
sd aso
Ay = Pint) = He) -B
A Ela, HAE MBA Ll) O99)
for allt all 7. In partoular, if we choose T= ijn Bq 3.996,
‘we ave
Ay = Bla, — 1) 300)
oe al gt which implies that
2h Fhe 59) 6.999)
{or ll ands. The covariance Function must depend only on the interval
(=) between observations and noc dizecly on these observation
{asians themsclves. A Gaussian proves i stationary whenever Eqs. 3954
fad care atid. In order to simply aoalon, i is conventional to
top the second argument in Eq, 398e and write Zt) instead of
Sram OREN TUTTE OE
to be stationary are then
fo) = = 4 comiant 2.1003)
£409) = EM 9) 6.1008) cx
f
TF
2
176
&
Photek
Hono
1
”
Figure 419 Two dot random prone wth test in and ova Function
CCAUSSAN PROCES THROUGH LINEAR FILTERS 177
‘An example of «covariance function meting this constion s Eq. 3.260,
civen that Eq Tei said, «requirement equivalent to Eq 3.1005
(with th same notational convention) is
Alt.) = 80-9. 6.1006)
"Notice that thexe conditions are ot sucent to gusranzesaionariness
fo eandom process that ae aot Gaussian. The following random
process acount Tlf contan fvesampe pois andleteach
Point be asigned probably f. Let s() be the process whose sample
Functions x0, 7) are
Homi) = —Vicos!,
ay.) —VEsins,
Hei) = VR eos + in), 6.10
Hoyt) = (008! = si08,
Hoyt) = (Gin! — 6081.
1 can bo verified by diet ealaltion that
W=1Esou9 =o; toralte
TW =1E eo 91045) = Seosce
HR) = EE Hg Xo 8) = $000
“Thus the conditions of Eg, 3100 ae met
‘On the other hand iit eagy to show that a) isnot stationary. For
insanee consider the two random variables = and x, obained by ob
serving) at times 5 —Oand f= 9/4, We have direct trom Eq, 3.101
Pio = Ha + V3) + He) + Me = VD + Ka D E+ DT
pale = UR + 1+ Ke = 2+ 20
Ths PH Pay
‘Gaasian Prooeses through Linear Filters
‘We hae argusd in Sections 32 and 3.3 that filtered impulse noise
tbezomes Gaussian when the number of impulses per sscond becomes
Targe, More preccly, me roguire that the average numberof impulses
‘caring ducing the efcive duration of the Site's impuse response Be
laege es indiated in Fig 316) and thatthe arial times ofthe impulses be
‘utslantally independent of one another. In genera, follows from the 178 naxbow wavirons
arti
‘ous por sand
ot
yet
Ss
gue 120 The smouting Hs cade, The el ox may be cons
tier asa gh rw pe ona ad eee craton do wo
sin tasede {the elntye tion of my Stor epaoe 0) soalsng
lpi ceo utr ofeach» Sher ako mpl ad oD oe ot
Witee Guasuinsr ad
same aegoment thatthe ouiput of 2 second iter connected in cascade
behind the ist as shown in Fig. 320, also becomes Gaussian. Al that is
‘eure i thatthe eerie duration ofthe overall impulse response of
{he pair of ites in cascade should again be slconly long.
“The preceding arguments suggest hat the outpusof any linear fer i @
Gousian process wheneer is inputs Gaussian proces. Although formal
proof of ths fact mathemaaly involved, the observation that the
{HE EXPECTATION OF AN INTEGRAL 179
input and ouput process x) and 4) in ig. 3.20 are rated by
wor[loM—ae am
Posi utr want Ainge 4
Mim Fata) He — 4) Bay,
‘we note tbat the consusion that y) is Gaussian is consistent with the
Property that a weighted stm of Gaussian random variables is Oausian.
‘Ae mentioned in connection with Eq, 3.10, Eq, 3102 i an example of
specifying a new random process by means of applying a stated operation
[convolution with] to the sample fusctions of «given proses. The
relative mathematical cae with which Gaussian nose can Be banled in
‘eommnication problems sems from the fact tat a Ostet input to
ea fier Yields a Gausian output. This of course, mot tue for Non
Gaorsan input,
25 CORRELATION FUNCTIONS AND POWER SPECTRA
‘We have soon that th random process at dhe output ofa linc ltr is
Gaussian whenever te input is Gaussian. Since any Gaussian proces is
specie by ils mean and correlation functions, the effet of the linet
Ser cna Gaussian input is desrbed cone
pletely bythe eet ofthe linear filter onthe _ 269 1
Inean and correlation functions. We no¥ so
consider how to ealelate thew functions; — fat brawl
the results re valid whether o not the aput
Sea Fie 3.21,‘The ra poe
(Shy wh canis fentone
‘Sey ) ewe om pasing
‘The Hxpectation of an Integra Sean ear oh
In Fig. 821 we show a linear fee Ke)
whose inpit isan arbitrary random proces =(). The sample fonctions
(ofthe random process (atthe fite outpa ae related to the sample
Tonetions of 40 by the contusion integral
woof 2(04 4) hia) da; allovin 2 G.103)
From Eg, 393 the mean funtion of 0s
wo=suer=e{[xone—ae], 09 180 RANDOM waverones \
Simplifcation of Eq, 3.104 is straightforward when the number of
points in the simple space 0 is Ante. Let us asume that thee are k
Points (0), 4= 1,2,.--, 10 each of which i asigned probability P.
Then
Peso
mio Reflseum—oe a.
‘By interchanging the onder af summation and integration in Eq, 3.1058, we
‘obtain
myo] SPxorad ne — da
* eM ade, (@:105b)
Under these conditions the autocorrelation function of 9) may be
‘obtained by smile procure:
992 TVD
[fh some oaef” aon 9a]
200
sononi 94” Hon DME~ Bab
PJ [senso Me — 9 Me aes
6.1069
-Aginintachangig te ner of ite suninatonandinegation weave
8400)=[7 [7 [Sectors pat 9m ~ naa
Fi farow-oe-pae ome
‘The mathematical ives avolved iain
and eo i
‘af
ee a emer aon
mton [nto cus
aor~ [i aemme—ome—Paaes, (109
remain valid, however, whenever the double integral of Eq. 3.108 eit
frit and 2"
ng the order of nt
rowan specu 181
For ecither of these equations do we rogue that o() be Gaussian,
‘When xi)—henee vf)—is Gaussian, however, evaluation of thse two
nepal completly species the process).
Power Spectrom
Important addtional insight into the effect of living a random proc-
ss: which again naed not neces be Gausia, can be gained from
Eg. 3.108 in the specu in whic 3) depends only onthe interval
(Coa betwcen the sampling instante rand stn patiouay, if his eon-
‘ition i sats, we shal fd it posible to investigate the distribution
tfmean power i) ava function of frequency. Accordingly, nthe est,
ofthis section we shall sume that
0.9) = 0), 6.20)
ore
and the notations that of Bg 3.100
"Equation 2109, when subsituted in Eg 3.108, implies that
auuoe[t [ome pme aie péea
sg ego
Fi flau-rte—nmonoansr sey
Since the righthand side ofthis equation depends only on (¢ ~ 3), we see
that whenever 342) 8 a fonction only ofr =f — soa is 3):
Rita) = Rl) Rts) = HO. 2.1105)
Equation 3.110 can be simplified if we iatoduce the Fourie transforms
of Ryo) and Ro, say Lf) and 8,00:
bles ys = 5, ps = B, we obtain
Afi
se C Reh" dry (.itla)
sine aoeme a om
1 allows by inverse transfor th
awpe[ amen a our
and
awo=[" sce at ou) 182 xaNow waveronss 4
‘When + is substituted for f= sad Eq, 3.112 is used to express
Ae + u — 9) ners of 8, (/), Ey. 3.110 becomes
n= ff suneieernannn arava
=a of ner af” now a
“The integral on vis reognied asthe Fte’s transfer function HY), and
the integral ony is eecognized as 11*(). Thus
n= [0 supine ar 6.19)
nd, comparing E3113 with Eq 3.123, we Rave
87) = SUP ou
‘We may interpret Eq. 3118 as follows. Fist, we note thatthe mean
square vale of the filter ouipur process y@) is independent of Une
whenever Ba. 3.100 is satisfied:
FD = 860.9 = Ale = HO. ens,
[Noxt, we consider sf) as an ensemble of voiage or current waveforms
“pplcd scoss a Ll resistor, so that yA, 2) i he instantaneous power
‘lipated in the resistor at ime «bythe wavtorm associated with sample
pointe. We therfore interpret 74(0) asthe expected vale of the power
(isipat in the reetor at any insant.
From Eg 3.1128 and 3.114 we have
a0=[ swf swmnra 09
two now let 1) bb pr ft, shown ia Fig 22, for which
nal ESIC aim
“we obtain, : “
so-Punasfsuna eam
We sall son se tht 8) alvays an even function of frequency.
Since Eq, 3.117 implies thatthe mean power delivered by 2(0) im any
‘arrow frequency Bond of width Af centered on f* is approximately
28,") Af. as shown ia Fig. 3.23, 8(f) describes the distribution of
“Hieon s Gaus input prose,
the eine although Eqs 3.118 and 3.
owen srscrnuat 183
Ho
Aca gmares f
Figure 322 tial bangpate Ser. Alou not physcay rae (he np
pons not oly nro fac <0) Hat ng ts a0 st Co
pepo of ami
scan power with frequency im the process 7). For this reason 8,(/ is
‘led the power dems futon of 30.
Wide seme saionariness. It sential i he decivation of Eq, 3.117
that (0) = Sr
formation of Bg. 3.1
8.G) is nt defined,
‘Since knowledge of 8,(f) atthe output of iene iter implis knowl
‘cannot be made and the power density fucetion
edge of (0), Eq. 3.104 (logether with the tltion between tbe mean
Fantions The Sear ora near
‘leron a at pes When oO) nak wast
ll permit ustocalslate the total
‘an squat nsanfeneovs power Out of the filter. The abit to do thls
fnd to fall about the power dansiy ofa random proces is suficintly
fmportant ins own ght that processes 2() which meet the conditions
‘mo = constant unis)
lus) = Kfe~ 9 e118)
so
see
+l?
gue 123_The mean poverty) aay dt wt Yon
Sirhaqalouemonse 184 naNbow waverOR 1
ave given the spocial name widewese starionary} Stationarnes, as we
‘dened it in Seton 31 is often called srictsensesaionrinesin order
1 avoid posible confusion.
[Any stficvsenge eisionary process is widens stationry, but the
‘converse ot teue. The proces of Es. 101 isa counterexample. 4
widecensestaonary Gausian proces salvo srctsense stationary, sine
All of the conditions of Eqs. 3.100 are met
Propertesof 8, (fend B.(3). Since the power deasty function 8,()
ofa wide sna lationary random process () isthe Fourier transfor of
the corrlation finetion 3), the properties of the two functions are
Intmatly related, Fist, we nte that J) area, even Fuetion o
A) = 6.19)
“This follows from the definition of Eq, 397; 2) seal, and
Ai) = Rfe— 9 2TH)
= HAD SE A(e— 9 = Af).
Equation 3.119 imps that 8,(f) is rat, enn function off. We
prove this by observing that, sace (seven and sin 2rd,
[svosnagea
+ seoet ara [a(n -
sr=[aermrn ou
Since the righthand side of Eq, 3120s an even function off the proot fs
complete.
"Rent, we claim that 8,(/) mus also be & son-nepaiv Fantion
84>: forallf enn
“This is cleasly a necesary condition for the interpretation of 8,(/) as
‘omer density tobe meaning. Proof follows by noting that ig. 3121,
frere no fue hf, and could be chosen forthe rectangular filer ia
Fig. 324 such that
[isinarce em
‘Fin my tens, proce ehing ony S186 acl wide stony.
owen seccreune 185
0
a t i a
igue34 oot onrdson tht power doy fon eanat beneai.
Bt, rom E3117 and he ovenes of 8, (this integra is oe ba be
expected lve ofthe square oft ilar outa a) and ths Eq. 3122
‘rou be fn contradstion to th fat that A) mast Be non-egative
“The fact hat Bf) x nonoegative der not imply that) lao
son-egtve It dos ipl tet the contain function of aay Wide.
ue sonny process) sass te inity
FRO
- Goa)
-EENEES
ees.
we ings
> (reat
ehh (+28)
“The derivation of Eg. 5.78 is completely independant of te particule
set of transmitter vectors (6; the result valid for ony set that mets
the signal energy constant of Bq. 5.62. Thus the negative statement of
the channel capacity theorem is proved. Since traasmssion with set of
20 yectors having 1Ycomponeats includes the possibility of & tran
Imisions, each wth a set of 2°" yotors having Ne components, the
theoretn hols true fr any transmission stategy-Wesse that te probar
bility of communicating a Dlock of RT —= NR, bits without any eror
batsoever must approach 20 as 7 grows larpe if Ry exceeds Cy
“This proof is described as "sphere packing” I is @ negative proot ia
that no claim is mide about tho existence of signals {4} such that the
Gdeson regions [i actully ave spheres of equal vad. Clery,
goometty dos not permit, ‘The eseace ofthe argument issirply thatthe
SScked spheres” ieaizton implies « bound on performance that no
realizable set of signals ean surpass
Proof of the postive statement, We now prove the positive channel
capacity statement that if Ry < Cy ten for any positive number « (10
Imater how smal) there exits & lege enough value of Wand a set of
[A= 2°% signals (5) sued thatthe attainable probability of error is ess
than «, The poof # complicated by two facts: rt, jo genera itis
hot posible 10 exhibit expicily such a set of signals {s), and, second,
then such ase ould be exbibited che calulation of its probability of
ror would be enormovsly dict, These complications, which we
ncouotered before in connection with Ry may again be ceumvented by
Consvering not just one communication sysiom but rather &whcle
nvemble of systems, cach consisting oF a tansmiter, channel, and opt-
tum receiver, Ax before, we construct our ensemble in such & way that
PROOF OF THE CAPACITY THKOREM 333,
the mean probability of eros, FL], may be easily ealulnted, We prove
the theorem by showing that PTE] < « for suliendy large Ns the
‘ensemble must then contain individual systems for which the probability
fof eroriv alo les than «
‘Specification of Codes for the Basomble of Syotoms. The capaci
theorem forthe Gaussian channel concerns normalized N-dimensionsl
signals (5) each of which stises the average power constraint
I whe Odi
‘We overbound Pl [ml for cach system inthe ensemble, hance over
ound PE, a by seglestng the Tat that, may sf be decoded exe
relly (Seeatse of codewords in Sy othe Shan even if Eq. 6.103 i not
atid, Thos PIE [xis bounded by the probability tat atleast one
‘Of the 28" EL vectors in 5, a move probable cause of ethan isthe
teanomited segment sy
all yin 5, 6.109
sequent pecooine 28
‘The remaining sep isto recogize that tis lst probability is closely
related v9 our mean probably of eror bound on PLE] my] with a BSC
4nd a Ke-dipt block priy-check code having 2" equally ikely mestages,
‘The ony mathematical dsiaction that with a cosvoutioal code hare
are only 2° EL vectors that can case an etor, whereas inthe block-
ode cate there are 2" Without further ado, we have
FIR Te] <2 <2
a stn, (6.104)
in which
vane (6.1086)
and, from Ea, 6.95,
= 1 = logs (1 + 20 PL (6.10%)
“This completes the proof 1
64 SEQUENTIAL DECODING
Although in principle both block and convolutional codes alord a
TB) that dzereases exponentially with K, we have not yet adresse the
‘cial problem of actualy building decoders that achieve such exor
Pevformance. Specifically, the suboptimum decoder considered thus far
Fp not realizable for large A became ls procedure for decodiag cach
sucresive input bit %, f= 1,2,..,, involves comparson ofthe
ressived mestage span yr with 24 Klbranch codeword segments The
[Adoption of» “sequential” prosedure for detrmining each, evades this
‘xponctil blow-up and permits ws to spesy decoder that achieves
‘exponentially emall error probability Whe rewaining realizable even
sho Kis args
Ta thissoation we introdvce sequential decoding by a heuristic discussion
of ity applcaion #9 the binary symmetric channel, We then detail
cle decoding algorithm due to Fano.™" The algorithm i extended to
‘ore general chansels and anslyeed mathematically in Appendix 6A
Enginceing spplcatos ae dicused i Section 6
Th its simples: form a Sequential decoder proceeds in much the same
‘way aa our suboptimum decoder, Hoth deide on each suscesive esage
“Input bitin turn, one alter th otar indicated in Fig. 637. Por both
the problem of docoding i equivalent to the prablem of decoding x,
provided thatthe th starting node is correct. The two decoders die:
Aisne, however, in how the deiions fs) are determined 426 PLINENIATION OF CODED SYSTENS
‘Tree Searching
“We have slay remarked shat the convolutional coder input x may
be regarded ata et of instructions tht direct the transmitter along some
path through the code toe, Let yy represent the fist 1 — Ke digits
Encountered slong that path and let yw denote the fst N noise digits IC
‘ve assume inal thas the BSC i hostess, 20 that ym ~ 0, then
iy Dima). In tis tvial cane a decoder provided with a replica of
the encoder can eal race ou the st K branches of the path designated
bby x. The decoder state atthe first node of the code ree, generates both
branches diverging therefrom, and follows the one that agrees with the
fst edits of e- aving thus been directed to a particular seconlevel
node of the code tre, te decoder again geecates both branches diverging
therefor and fllowe whichaver branch agrees with the second w digits
of er toa thintlevel node, Continuing inthis way, the decoder rapidly
‘etrmins the Sze Kegs of 2 The procedure works without difclty
{slong asthe two branches diverging from any node of the code tree
‘jer by at eat one dig [is clea fom Fig. 6.32 that sucha diference
tnay be guaranteed by connecting the fost reise stage to the fst
tmodul-2 adder, that, by making gy = Ie
“When the BSC is noisy ais notin general O and the procedure jst
eseribed i not euficient evento decode the ist message digit. But
‘simple modifeaton is appealing and may be used to decode with
igh laity. IP nether branch stemaing fom an intermediate node
tolnides with the corresponding e gts off the decoder first follows
whichever branch agrss est. Clearly, when more than #2 wansitions
‘our in the transmission of «branch, sucka decode inlly proceeds
{oan incorrect node. Having onee made this mistake, however, in
‘subsequent branch comparisons the decoder is unlikely to find any path
‘emming from this incorectaode which agrees well with the remaining
Gigs of yr. For example, wih the truncated K = 4, v= S code tree of
Fig. 639, assume
x=O.1.0) (6.1059)
{= (0010, 10111, 0900, 00100), (6.1056)
0 that the transite vector it
ay = (11H, 10101, 01101, 11011) (6.106)
and
af = (OUOF, 0010, 01101, 1111), (6.07)
ln this case, as shown in tho figure, the decoder follows the coxeet path
to node (a) thence the incorrect path 10 node (2). But none ofthe paths
‘nee seancimno 427
—aane |
Lou
reine re ed
ff
sown _[f
‘nh
5:2 tpt
Fige639 Theta ofan geet tml pnt
{Ee perinont tenn Porteous ee
soa cere om ay crt ee Sept
extn bjo ae) ages ih in et 0 may ine
las does the correct path x When ois propesy chosen with gar tof
BSC transition probability p, the effet of @ wrong turn is hkely to be
readily noticable a5 the decoder attempts to penetrate deeper into the
ode tre,
"The idea of sequential decoding isto program the devoder to act much
as driver who occasionally makes a wrong choice at a fork nthe road,
‘but quickly discovers his eror, goes back, and tes the other. The
ecoder’s objective isto construct path K branches long extending all 428 rtzuentaTion OF caDeD sera
te way through the truncated code in Fig. 639 to one of its 2 terminal
nodes At soon as such a path is found 3 is determined in accord with
{he st branch of that path. The obeeved N-digt span of the received
Sequences then sited veg tothe righ, as indicated in Fig 637, and
the entire decoding procedute ( eltrated to detsrmine An etror in
evading 2 resuls I and only if'a weong toro atthe fst node is not
recoglaed before the decoder penetrates K branches beyond it.
iar 640A cuenta retin
\We now describe how 2 sequential decoder rtognias weong turns in
devoding =, Let us assume thatthe decodor has peneuated I branches
Jato the code re, 0-<1 2 (21
‘we have left a new code consisting of a least 2+ signals for each of
‘which
PIE] md <22. (6306)
“Moreover, the rate of this expurgnted code in bits per densa,
6308)
‘is very nearly equal tothe original unexpurgsted sate, KIN, when K is
large. The dificult with the expurgation procedure is tha one needs to
know the {PE mia order to apply st: a8 already pointed out many
limes, in general we cannot hope to calulte al ofthese conditional
probabilities when X large.
Orthogonal and simplex codes. Patiy-shsk codes may ao be wed
to generate orthogonal and simplex signals, with N= 2® aod 2" — 1,
respectively. Iti partculay interesting that with thi technique each
resting signal veto, say 61s binary; tat,
B= GuSarceesidi Fe ONE
sue tyEx or — Jimi forall ij
‘To.se how to generate such Signal sts, consider the case K = 2 and
Nm 2Pad We take =O and choose the paritveheck coder
‘eonnestion vectors and fo be
halorg
fal 100.
with
(631) 1382 MpLEDENTATION OF CODED SYSTENS
“Then, in accordance with Ea. 628, the {,} are
nao 000
nero
nelio
wool
°
o
6,
oa)
“The comresponding binary vectors (5) ase
aaVEGl -!
an VEG, 1
aaVAGh +h
s=VBCH +h Hi, -0.
ts apparent tha the dt product of any to vector, 38
Bo = NED (632)
“Thus these vectors (form an orthogonal st, and each has length
NE.
“Tae reaon forthe orthogonality of the {s,) becomes clear when we
consider the struetute ofthe (5). Esch f coms of lernae groups of
{and Inf the groupe ae of length 2; inf the groups © of enh
2 this because ofthis that each vector yg dlls tom every other veto
4a exacly N/2 coordinates, whieh fact in turn implies orthogonality
between the
‘We now prove for every Kand N= 2% that ifthe coder eonnetion
seotrs an stenate groupe of I's and O', with f having groups of ent
DP the reuling ender genrats a set of 2© orthogonal vestors. Let
{hgh 12... denote the connection vetors for the ase (¥ = 2,
eo baad it Ge b= 2,--- E+ I, denote the connection vectors
for the ame (W = 2, K== 41), The alternate grouping implies
GA
B=
BGs oa)
PARETYSCHEEK COOKS 383
in which we use the notation
292 Gstin eet tating) (636)
“The (+ 1h connection vector i
Bes © aso B04... (6330)
Bis Fos
Proof of orthogonaiy by indueion. Asam that the st of veto
(6) orthogonal for K'= k. From Eqs. 628 and 6.33 the et of signals
ERS ECE Tg cn te wien es of ea for
Shas
Se = G08)
sin = Cas)
Equations 6.34 follow from the ft tat since 2, by convention isthe
least aigicant digg enters into the determination of Sy but not
Into the dtrminaion of forall <2 1. Tho ff nea
Sos fo ange the sgn of heft 2° components ofthe signal vetor
that would result 1 were not insted.
From Eg, 6.336,
(2) Cu) = (0-40) + 6220-
By vie of the orthogonality (asm for the induetion) between the
2eomponent vectors (3) we therefore have
G05) (645) = PEI + MEd = BE Boy (6350)
(oes tt) = MEd # BEd = PE hy (639)
GoD Cand DF Sat MEd =O (635)
“Thus the orthogonality of he signal yectos in the case K = k guarantees
the orthogonality in the cio K'= 1 Sioe we aye seen that he
theorem is tle Tor K = 2 the proot Is compete. That the theorem i
ao tre for K'= |i obvious by inspection. {Tis convenient to begin
the induction argument with K'= 2 Beeaue of the fught ard into
the suture ofthe ))
“The adrentage of gznerating orthogonal waveforms ia this way is
‘obvious rom an engineering pot of view thir generation is rately
Spe. OF cous, this is also rue of shot pues poitioned in time 9
thal they do not overlap. With paycheck waveform, however the
problem of high peakypower requirement is avoided, e tsteated in
Figo3.
"To obtain aot of 2 simpox waveforms i only nersary ¢9 molly
the coer jst describe by dseting the Nh tae ofthe eter, avg
fort 0,12, om set wrunastanon oF cone seTO
tent 2 — 1 stages unchanged. This corcesponds to deleting the st
Rehan oath the fine ouch fhe (0) was so hat
Jere Sioa tallow latyy = Ofori; thstrunating th code
{nga = ao oe ore te
Wei ato Cape imps never ne ound fom
Serta by nada ht Sd nt als P))
78) Off CLL
Dao
{l ayo
u
HE
ove
av
L L "
° a 7 z
o
ice 69 Peak pods requis wi ic inary and puke psion enhognat
Sign ot
Proof that the set of signal vectors resulting fom this trncaton does
form a simplex is tila, Latag {6) denote the original orthogonal
‘ipnal vectors of length 2 and fa) the truncated set, we observe that be
Sih component nays contributes the term (FE,) 10 53, THUS
shes) = 60081 En
Diy: fort
_ 0" De a
ari ford xt
Panare-cHecx cones 385
Since Eq. 6.36 dees to she simplex dfinsion of Eq. 499 when 24 is
‘Menitid with M and AE, with 2, the profs complete
“The encoder for simplex vectors inthe cave Km 3, mx 251 = 7, in
shown in Fig. 610, ‘The conection vectors are
helororod
helloo1io0
Belt ttoog an
1 is interesting to note tbat cach column on the rghtband side of
Ege, 637 represents diferent one of the (" ~ 1) distinct nomaull
pavtycheck connections. Itcan be shown tha this is true forall:
Spl codes with 2" words can be geneated by pforming all distinct
Figure 610 Party chat contr fo sng = 5 = =
YG, 98, O24 8052, Ooent, ay
‘non-null party checks on a sequence of K masage bits. An especially
simple implementation ofa simplex code isa bi sit register connected
ina maximallength feedback configuration" Aw example is shown in
Fig. 61
Discusion. Much stuly bas been devoted to party.check coders,
partially for the binary signal exse. Catalogs of optimum, that is,
‘minimum PIL), binary codes have been compiled for many cases in
‘which ether Kor 17~ Kar both are smal, The known techniqes for
Finding optimum codes are esentilly those of exhaustive enumeration
and evaluation and usually cannot be applied when both X and (N — K)
fare lege. No general algorithm is known for consicucting explicit codes
{ar which it can be proved thatthe probability of eror is overbounded
by Fa. 66.
Avs glace i i starting thatthe eso probabilty averaged overall
(@%, binary codesis, a general, smal than te behavior of te best code
2 386 tmPLaneNTATION OF CODED s¥sTES
for which the eto probity can be ealculted. Ta somosenseitappears
that isthe abrence of simple sructure that makes a code goed. Uafor-
tunately, however it (s aot posible to caulate the eror probably of
spol large codes that are not highly steuctre.
or ie
ce teat tt
oes
G0 aghdal B85) ade
oy esta asses | sees
Figur 61 Movin eg hit reper ezoer fo gee (=> 7 = 3
Ye Geursr es 9808 93521914 Bene OD.
62 RECEIVER QUANTIZATION
‘We have been staying the problem of building @ transmits that is
capable of eficenlly communicating one of
M2 (6389)
massages even when NRy it are. For the adie white Gaussian noise
‘channel we have observed that is not difcult to consteuct an ensemble
in whieh the tanemitee are esily implemented and for which the bound
Pc teat (6380)
is said with am Ry that neaely optimum.
‘The problens of implementing an eficient rediver 4 not so easly
resolved, The bound of bq, 6.380 was derived under the assumption that
‘ich member ofthe ensemble of communication systems has an optimum
fesse, Optimum rsrivers for signal
Fe Ota 1 (639)
fhe been studi in Chapter 4, As iluseatd in Fig. 612, one imple-
‘mentation i bank offers matshod to the (()} flowed by eects
CIVER QUANTIZATION 387
that compute the M dot products
rye Brie (6-400)
and determine for which ithe decision variable
BeBe Hie FOL MT 6406)
{is maximum, (In Eq. 640 and throughout this chapter we assume equal
prior message probabilities)
a
I
~~ = . “
at Vs | in re
: re et
i) a
salen Pi
Fee 612. As opin eine tition: = (ress
‘Claly, the complesity of implementing the bank of matched filters
ingstated in Fig. 612 grows no faster than lineal with N. Indeed, a6
tated in Seaton 6.1 in connection with modulator design, the complexity
's independent of Wi the {p,(} are chose to be mnverlaping time
translates of ingle waveform of durstion +. Ax showa in Fg. 613, in
this case we can use a single matched filter apd sample is output at times
Pefe hd N
“Ga the ote hand there remains the problem of caleulting the set of
ecson variables (.} At fist glance it might appear that high-speed.
Agia computer could resolve this dificult. But fr large Theis note03,
from Ege. 6.40 the number of elcuations involved in deterinig the
1) 8 NAL. For R= 1000 bisice and T= 3 see, we have
A= 2 a 2 108,
‘which fn a serial computer would alow 10 nanosecond for computing
‘ach sum, One cannot tie with exponential growth, 388 iMPLEMEWTRtION OF CODED sxsTEAS
‘An aerate isto eleulate the {gi i time parallel rather than in
timesequenee, perhaps by resstor Weighting networks and summing
Tastes, ut this would fequite approximatsly NAF resistors, and ex-
poeatil growth jn number of componoats 5 no move attractive thaa
‘Sone growth im speed of computation, In general, the only recourse
{eto sept erocrver hat ses than optimum
‘Once we ae reconciled to some losin performance, the problem i to
termine resiving procedures with acteptable degssdstion. In th |
Framework speia-purpose dig computers, called decoders, wsime &
ei gel pd sit) |
ee ie Tinie *
Moweemo }
ne
wo An ws seseien
Tah Sea, ‘sn
deka
pe 642 Optioum eter rintion fora tine rans etosoen st (9)
a=
oa)
=fisonte- nom
—Firone—u-oae
Froonoaan
DMEAWRE OF DEGRADATION — 389
ole of ental importanee, primarily because ofthe gret Rxibity with
which they proces data.
‘When a decoder is ured, the performance degradation aries fom two
sources. Fist, the vector ra the output of the matchedsier book has
Components (that are defined on a continwum, wheress a digital com-
Daler operates only with dscste numbers. Thus some form of amplitude
{Guanization is ustally introduced ahead ofthe computer. Second, the
‘pumber of computations demanded ofthe computer must be restricted to
{growing no faster than linear with the sigeal duration 7. The frst
Source of degradation is comideed inthe remainder of this Section; the
scoond is considered in Section 6
‘Measure of Degradation
Wis evident that tansfrming the component vector rinto a discrete
‘estr stable for computer processing san frevorible operation and
fn general degrades the attainable eor peformance. It is intuitively
reasonable thet this degradation wll be small if the quantization
cexttemly ne. On the oer hand, coarse quantizaton's desirable because
[t dcretsee the memory requirements, hence the cost of the desoder: if
‘ach component of ris guaatie into one of @ levels (Qa power of 2),
‘log, Q bits oF memory are required to store the quantized vector in he
compte,
"The appropriate enginering balance between sjstem cost and per.
formance sant be ajuiged without some quantitative measure of the
fflect of quantization on the probability of eror. An especally wef
measure of degradation n'a coding situation i provided by the exponent
in the random coding bound,
Heretofore th exponential parameter Ry athe bound PH] < 2-¥'%-)
has been determined only for an ensemble of communication systems
uliing parity-cheek coder, transducers, and opvimum (unquantize)
receiver We no consider the parameter Ry’ in coresponding bound
PIB] < 2-"%'=M for an ensemble of systems with the same transmits
but wit ecivers having the strate of Fig 6.14 in which quantize Q
it inverted between matched filer and dscoder. Te decoder itself is
‘ssumed tobe optimum inthe segs tat it Setermines, rom the quantized
‘etor®’ and knowlege ofthe signals), which message has maximum
1 posterior: probably Pf, |= r, (5). The dilleresee between Ry
find "provides a menningel measive atthe degradation ue fo quan-
Seaton
| Meth tha he et quanto ofeach componet oe may ao be
{erode dcr Sossder inure Seo pct 10) 390 mieLswaeAtION oF copED sestENS
vata} ON
era
|»
aa} NE
t
Siete tem?
gue 14 Qeanind reser, The doin opt ves Got
“The Quaatized-Chammel Model
With an ave white Gusan noise chaos ich component oof
she osm’ veo se corpte bythe adloon of aatically
indepndone Gaunian noke vane, Thon, fe) denae the tan
Inter lps wien ym th ik component of the (giant)
Fosved eto dss by the denny fincton
Poly |8, = a) = weer (oat)
As ila in ig 615, he quater map, no an outa com-
neat at cant asune unary vale on te et he Bt
si
Figme.15 Inpucoipu lon oF quiet, The ena a erepning t=
rena
‘Tur quaxrzen-cuasnet Monet 391
late 9d
ip 16 "The naneon probably geese fhe hed ion
resticted to being some letter of the quantizer oupur alphabet, say
{ogy h= 12,70. Given that isa, we denote the probability that
1's byby the Symbol gu:
an 2 Phe) =o] 5,= a 6a)
‘As shown in Fig 6.16, the value of for any particular quantizer is the
Integral ofthe Gaussian deasity function of Eq 6.41 over the ht
‘ation interval. The set of probubilos {gay P=, 2,--.,4,
4,2,..., 0. specifies the probablisti connections betwee the taasmiter
aipaber (a,) and the quanti output alphabet (6). The fg), called
transition probabties, ay be conveniently displayed in a diagram (ee
Fig. 6.170) when A and Q are small and inu matrix (See Fg. 6.178) when
A aed Q are large.
‘The components of the Gauesian noise vector which the channel ad
to the signal vector sare staistialy independent, If we sssume that each
o
gue 617 Teasion plobabity dag and mata: @ A= 3. @= 4,
‘new: he aman ech tow sumo nu Ie he nee of ry al pose 282 impusaeeraTioN oF CODED SYSTENS
matched fier output ry j= 1.25-
‘quantier, etch components i ae
Atonsiton probate fy} Thus i
I, i subjected to am ietial
independently bythe same set of
(9-00)
Cored
then
Pie my|saal=TT Re =7|ym ek (648)
Here, each a, may be any member ofthe transmitter alphabet {aj} and
cach 7, may be aay member ofthe quantizer output alphabet (6). For
example, I
= Cote 0d
= Cr bn bd,
‘then in acondanos with Eqs. £42 and 6.43,
Pe = y|s = al = Gamo
Caleuaton of
‘We re now in postion to calculate « bouad on the mean probability
of eor for sn ensemble of quantized reeter communication aystems
{We assune thatthe connection vectors for the paiy-check encoders are
[EL and statistlly independent over tbe ensemble. Consequently the
‘gna {x are statically independent by pis:
P= PIP alP =P anki (6440)
‘Furthermore, the components of eschsgeal ae statistically independent,
M%
(6486)
aefiriaan te8M
stro ne oh
tecadens Hs
“The {pi depend on the choice of Nand ofthe digitsbanalog transducer
of Bg 68.
Formation of the bound, “The desivation ofthe bound
Py creme (645)
is formulated in manner denial to that we have encountered before,
[CALCULATION OF my) 393
We stat with the union Bound
P|) <3, Fao 640)
By virtue of Eqs 6.4 the expectation of Pie] ovr the ensemble ot
‘proms independet of he subsp ad Hence
Pais PAT: all ks (tay
and
PIB) = FET ml < MPI. (647)
Since M = 2, we nee only show that
Pa <8 (640)
to obtain the desied bound of Ba, 645
Yenficetion of Eq 6.47 and evaluation of Ry’ remain to be done, We
real thet Pays] the probability of error when s, ands, are used to
‘Communicate one of to equllyUkely mesages. Let and B be two
‘cots with components in the tranemiter alphabet (a) ety bea veetor
‘ith components in the roeve lpiabet 0) and assume forthe moment
tat
ane Bovey (68)
“The optimum devoder in this wosmessage case makes an error when ms
the mnie input and r= yi ested ard only if
Pie = y[s)=2 > Pl = |= Bl (649)
Equivalent forms ofthe error condition are
inPien
Pie =y/a=8)
Yes =71] 2d
po. 0)
Be Piri s0= Fa “
We simpy oan by denne
2AM srlu=ad ory t2,....N (51a)
Py = Ys| Sep = Bd
and
st) 34 naetmuevranion or coDeD sesTEss
Given y= 4,55 <8, and = y, an optimum reckiver in out wo
‘message cave computes the number Zand ses st = m if Z <0. Thus,
‘when m =m am eror is made if and only iF
Z>0. 652)
Ensemble averaging. Over the ensemble of codes and channel noise,
wwe recognte that sy sy andr are random variables, with 3, and 6,
Taaging over the tamsmiter alphabet (a) and r/ ranging over the quan-
tet alphabet (by). Hence =p which i uniquely determined for sated
ales of hn say md 1/8 also a random vatable. Ia parca, fom
gr 642 and f51 ne have
Iguled
16hee
“The probability assignment for follows from the statistical independence
oft and sy. We have
Pl = ay Say = ty 0’ = ma
= Phy = a) Ph = a) Pl?
uP 654)
Filly, by virtue of as. 6.43 and 6.440 we not that the random variables
Gehije tees Ns ae statistically independeat, hence that Zis «sum
of satay independent, identically distributed random variables.
‘We shall hound Pls] using the technique fist introduced in
etvation of the Cherno bound in Chapuar 2 If ve define the us
fusetion
= ty = aor/ = = (5,= If
639
[mt = ad
1 Zo
1-19, 200, eo
th, from Ea, 6.2, we have
Pica = PF) =, (055)
here the average is over the jolat ensemble of codes and channel
Cistorbaness
Beatson ofthe bound. Divect evaluation of f(Z} is wo, in gener
posible, For the special case of no quantization, which Was described in
‘Chapter 5, the coresponsing bound on PE] was expressed in terms of
the funtion QC rater than the unt step function fl). The averaging
‘vas cally carried out after the substitution ofan exponential bound on
|
Ccuicutamon oF RY 395
x20
1
igus Exporetil veto othe ul ep.
the O( function. A similar strategy, which we now adopt, st over-
oud the unit step /(2) by the exponential.
"Ae shown in Fit 618,
POV&e, Foran 2>0. 658
Subsituion of Bg, 66 in a, 6836 yields
FR 7D <7
{oo ($e)]=*Lite]
=[- a>0. 6s
in which = danotes any one of the ideotclly distributed, statistically
Independent random variable Defining
(6580)
Ry) 8 Ione
FAR < 288; Apo. (sm)
In the derivation of Eq. 658 we have exploited the fact that the
random variable [3 hens the random variables {e"), ae statistically
Independent, this enables us to equate the mean of the product of the
{ey and the product of ther meant Indeed, the motivation for adopting
the exponential bound of Eg. 66 & that it permits exploitation of ths
ft
“The bound of Eq. 6.58 i valid for any A> 0, We now choose the
pacameter in auch way thatthe bound is 3 tight ws possible. From
654,
we have
x (210
EBPranawce (2108!)
~ ERE nation! 9 386 wmLEMEWTANON OF CODED sysT=NS
whete the ice w and run from 1 to A and runs fom to Q. We
Uherefore sek the value of for which
SP )=TZE vara aida —tnaa). (6600)
Since he righthand sie of 9, 6.60 symmetrical in he indices wand
the solution to Eq. 6:60 is
anh (6008)
hich, ofcourse sais the condition 2 > 0. Ifwe define
Ry & RéDmae = RSW) = “Loe SES ruralTata), (610)
4.658 beanes
Pec 8" 61d)
Saiation of a, bin E47 ys died
BUR < 27M, (6.628)
Dy vite ofthe symmetry of Ea, 6 6a with respect to the indies and
the expression for Ry may also be writen
n=nbaS [Soe]! i
Discusion. Equstion 642 provides abound on FUT that is val for
any st of probabties(p}. For a given st of tapsiton probabilities
fat the (2) may be optimized by us ofthe formulas of Appendix SC.
‘Aehough Eq, 60 as Boo derived wih reference tothe quantized
‘dive white Gaussian note chanacl i valiity doesnot depend oo the
‘specie mechanism that produces the tation probalies (qu? For
Any eicrete channel deseribed by the diagram in ig 6 17e ot the matrix
in Fg. 61% wo may communicate one of M = 277% messages by means
fof a puriy-checkencoded signal et} wth component in (a)- As ong
seach component of affected independent bythe eansiton robe
Wiles tga the ensemble average eror probably i bounded by
4,602
Increasingly Fine Quantization
‘We now consider the limiting behavior of Ry asthe quantization grain
‘becomes incressngly fine. By 30 doing we obtain an exponential ound
sepcbbies te mot sty hsonitontg, > Oa Sy
+ Seth
INCREASINGLY move QUANTIZATION 397
‘on PR] that eppis to unquantind-eciver vector channels that ate far
‘more geveral than those disturbed solely by additive Oxuasian noise
Indeed, the conditional probablity density fneton of any unquantized
‘eccived signal component given the ansled inal componeat
ray be quite arity. We need only require that this conditions
18.
APPENDIX 7A. WHITENING FILTERS
‘The fiter O{f) in Eq, 7.7 was determined by a genral mathod which
alvays guarantees that both GU/) and G-X7) ae physically relizable
whenever S,(/) ca be waten arto of to polyoma in ft
syn eM o p= = B=
MO kp ‘Gn am ca)
In By, TAs (2 he tof comple rot ofthe narrator polyoma
N(f), assumed to have degree n, and {n,} is the set for the denominator
rijeomil DU) ated thaw epee The set (tbe as tnd
{ad the poles, of 5G). Sines 8G) = 8L-P)= 8) @ power
4316p otal yl pone prot mit
Yer tateg | Streeter oe 562 wrontasr cman. HoOeLs
specrum seal nd even, both rs and pols are symmetrically sted
out the eal and imaginary ate, a shown in Fig, 7A. Sts convenient
to number the rots o hat edi-oumbered roots bave pose imaginsy
juss ond even-numbered rots bave negntve imaginary pecs. For
nip inthe power spectrum of E770,
NY =ft+4—=G- 104i b= Ge PB
DAP ei = GH IAL+ ID) meinem “1.
2a Roser gf) = EMD.
Figs 0D.
“The whitening filter, G/), ray now be easly specified. Define
Ny = (f= P=) “°F Lod oe)
MO=G- OF WU br OAD
Dolf= P= ml = 1 ted a4)
DAN = (P= WKS = dF Hab oAS
4 eye PLD aa,
No
“Thus the numerator of G(/) incorporates the upper half-plane poles of
S.(p) and the denominaor incorporates the upper alf-pane 2er of
SU). Cleey,
aan,
|
convexny 565
so that Js denied asthe power density 2/2 at the ouipt of the
whitening filer:
"For the specirum of Eq, 7.7,
oy f= ett
S=R 2
retains tobe shown that both filters Gf)and O-) = No(UDoAN)
are realizable. Consider afer tanserfention ofthe fers
ijv-wo
Ty - m0
Leving s = ff, we may express te fer characteristic in terms of the
‘complex frequency variables a5
Tyler to TIfe= pet
Ti i)” TL e= en
I ye 0+) has postive imaginary part, the plane pole j2mn,
‘~2eb + [Bea has negative eal part ad lis inthe fethalsplane All
enominator root of both G{/) and O-WJ) satisfy this condition of
postive imaginary part Thus all he pole ofboth Sites fll inthe let-
Estas iene th are eae Firing wit Gi) ere
reverible operation.
Roots of 3y(/) on the real axis are sways of even multiplicity: the
situation may be handled matbematicaly by assigning conxcatve indices
to any such root, Ofcourse, these roots correspond to loses resonances
‘and do not eeur in practice.
‘A more substantive issue is that any 8,(/) encountered in practice
oes 1 200 a3 f-> eo. Bocas ll physics ulimatly bacome
‘apactive at high frequencies, idea whitening is impossible. Th dificly
is resolved by recognizing that GL) need only “whiten” the noise over
the frequency band contuaing most ofthe transmitted signal ener.
(payrnn,
APPENDIX 7H CONVEXITY
‘A function fis said tobe “conven” over [0, =) fis second derivative
satisfies
als>0.
O60 te) 564 nonzaer cM sooeLs
Toe fain fi sone” oer (0 iF
SG) PO; alla >0. (TB.1b)
1th equality sign in Hg, 78ln (PB) is nt permite, fs aly
ones’ for sie conane. A conver fincton ovr f,)
flawed Ie Fg. 7Bla tnd 2 sly Comer faneon over (0, i
Fe 731,
hoo
ha °
iy cone
o
ie 73 A connec sa sty come ation on
Theorem. Lt fe}, 2.5 M660 se of real mumbers sujet 10
the contains
> IAN ded ona)
ond x
Back 2)
end ttf be died as e
fm RSs. a)
comer 565
Uf emoes or 0,2 800
{<5 3/0 =s@) 784s)
pts, aa
ss0
‘Whenever Js steely conve, the eqeal sgn holds in Eq, 7B. if and
only ify i= 1,2,
22) +)
‘ea + a
igre 702. Increase in po whens ,
fis concave, =f s conver, and the statements of the theorem apply
withthe inequalities revered. We henceforth rect our attention to
‘convex functions.
Proof. Since the function f must be continuous inorder for f* toi
and each ofthe (a is restrced tothe close interval (0, x1,
sa t3 ye
4
rust take on a maximum and a minimum es the fx} ae varied, Let be
Strict convex. We now prove by a geometrical argument that f i=
‘maximum when =, = forall . Tor sume that some set (2) ot
satiying = all produces the maximum. At least one pair of he
(just be unequal, sa 2,2. However iis lar fom Fig. 73.2 that Soren ops
tepsng htm nd 2B 297 ees fii hanging
Ser Hae te asumpton th ay staf ot ing
Fae a ator te maximus es Yo + eontion ad
Fa aft th ncomie
oc mine sat Oe at = KB
och ii, Be es Babe nea a
See en eee ha ceeh wens
a ~S rll
a +40
1 snes
gue 782 Desezin pore wie tat woof th emo,
wherever cones, but ot sity 20 he posi ex hat
“pont da sain sqrt off tay cho fh
Ser ck hat enh fact ideo. 8a eso the 6
Tene eet a ace he sme mas
eee SP am toa poe random vrale inves Hie
na shane aan pret, we now lace 3 co
MoS on pp me tnn om 3p For isan y= Be 8 don
Soak at ?
[frou nso
73s)
{forcosen vt
noi spc ponve nec Ober at win eae
flO) < J) + (@— FH «PO
XTGION TO RONCONVEE FUNCTIONS S67
Thus
FO<7O + @=Bf@ -f0. Be)
fis red conver, the equality wil hold If and only iF
he) = tha = 2). Bn
Example 1. In ¥9, 7386,
Pte) = QeV2E LID,
ws o
ert Conary 6 ot
Ce
ier aisa pose ranom vibe, ine tn
4S O/T: 20
is strictly concave as shown in Fig. TB, we have
PIB) > O@WIETND, as)
ith equality if nd only ia = 3 with probability one.
0 eas
[Extension to Nonconvex Functions
‘Wenow extend Eg, 78.6 to aonconvex functions. Consider the fusetion|
{defined in Fig. TB-S0 over [0 0]. We ean construct [rom f a nique
conven function f* such that /-< J forall in (0, co]. We do so by
Sarting athe origin and following f tothe fist point, saya, at which 2 568 pmoaTaNT ciLANNEL MODELS
to
o
fey
0
o
Fie 84. Coirston of come exeboun 6 9).
line get wo fats tng off some e> a6) The
Siiguinc osetiog tan) ae pro and te poss
‘Sine Tepes ean. TB.
"he ony cipro ink cosas of feces wien Hoe
tarp toate ong ls iow fat sae aegis Ta is
SEE spent isu gin psig hough he og ad
[EXTENSION 70 RONCONYEX FUNCTIONS 599
tangent to fat some pint, say. several pints of wngeney are posible
the one yielding maximum slope is chosen, as shown in Fig. 7B.
Since fYou might also like
