LANGUAGE

AND

THE

THEORY

OF

INFORMATION

C. L. Hamblin Thesis for Ph. D. University of London October, 1956.

) (r,

'

ABSTRACT

of communication, in terms the is defined information of the concept of transmittediover a messages sets of of statistics has This no relation concept channel. communication to the logical concerned. of the messages content The concept defined closely sentences to analogous for of the of can be content of logical language in logical a manner a technical of information concept

In

the

technical

theory

in place that except probability of logical of "information concept with analogy, of "correlation" "confirmation" affinities or of one

probability a concept empirical The be related substituted. must has a similar logical transfer" both to the logical measures and to by another. measures of

"dependence" sentence

concerned "sentences"

The scope can be enlarged of a logical

the logical of all concepts by applying to not merely language as ordinarily understood

but

also to "unsigned in this sense defined

sentences" thesis.
the

or "questions"

in

Comparison of information logical and

of "information

technical transfer" that

concepte the with the latter

analogous

concepts

indicates

basis for a as can serve that the logical concepts as the more fundamental. the with ordinary concept uses this of words by conclusion the

definitions

of

the to both

former,

and

are therefore Comparison of

be considered sorts of in everyday reinforce concepts

concepts revealed "information" and "inform" showing that the logical

are closer

to usage.

,,

iv)

page 1.
2.

Introduction
The technical theory of communication

1 18 to the information 76
115 144 170

and measurement 3.
4. 5. 6.

Logical measures concept

The logic of

allied

questions and language

Information Language

and information

References

188

1.

INTRODUCTION

"Send for containing

etc. is etc.

our illustrated

brochure, on how to ..... it

six
It

pages
is

of information
it its'unit

no novelty,

seems, is

that for

"information" practical

a measurable

quantity;

purposes

the

page,

the

line,

the

chapter,

the perhaps,

paragraph is and as

or the. book. information. poetry

a rule, textbooks except of in in

Not everything Novels, less.

to impart information is

in books, example,

for

contain

little, does not,

in

books
aim

Speculative
information, is not of

philosophy
and always specific is directories found even

scientific packed, or a list

the

closely heats par

when there formulae.

a table Information

useful

excellence

encyclopaedias, the printed

telephone pages at the

and timetables; of the a pocket daily diary,

beginning in

and under

"Today's

arrangements"

newspaper.

Until suggest everyday that the

recently,

no one would which appears

have dared in such

"information"

contexts

can be measured: in

any more fundamental

I
A

sense.

Information,

it

would

have been, said, which about the long its

has

imponderable attempts Today, verdict cavalier. to

differences speak with

of quality precision in

frustrate quantity. this so

although-we still

may find

run that to be quite

stands,

we cannot

afford

It
and criticise of information. in

is

the

purpose

of

this

thesis
known found

to
as the

set

out

a, theory It fields is

which

has become which

theory

a theory as widely

has

ready

application physiology concepts tools in

separated

as physics, some of to its

and linguistics, and technical the hands of terms the

and already are beginning

be ordinary

engineer.

In a certain is rate hardly a, scientific. theory theory theory. in

sense the at all.

theory It is

of information not at any

a scientific

the, sense that are

the

theory

of the, atom or the theories;, refuted it is

of relativity

scientific or

not

a hypothesis There it true it is

to be confirmed no "crucial

by experiment.

experiment" is no,

which, can demonstrate "classical" obsolete,; to goals The sort the theory except previously of theory

or false, replace

and there or render short

which

will as it

in, so far attainable to which

may suggest

cute routes.

only it

by more devious is

may be likened such as the or

rather of the

"purely

mathematical" theory

theory of classes

theory

matrices

or the

(particularly)

3
theory analogies And perhaps to liken it to it is

of

probability. too far

not

stretching theory

a philosophical

like

Plato'stheory,

of forms.
The aik. "of theories of this' of second empirical framereformulations in time new stands class

is

not

the bt

prediction rather the

or

systematisation of

facts works; of

provision

new conceptual laws in sort but

they

give'us-not and perhaps

new naturgl a language., The first

old'-ones,

which of

nos

may be expressible.

theory

or falls of theory stands us with

by its

empirical

verification; susceptible extent

but

this

sort it

needs none and is rather

to none --to which

or falls

by the

it-provides

a convenience This

or economy of thought muot'not of course first

and expression.

be interpreted sort do not involve

as meaning new concepts about. old a theory off trees,

that

theories

of the give

or do not

us new ways of speaking Einstein apples needed, fall

facts. of

Neither

Newton nor to tell

gravitition yet both

him that

gave us new and original sort. Yet in both

ways of these, cases

describing we might the

events say that

of this the

aim of simplifying was subordinate accurate. Newton's

and clarifying to the aim of of

d$scription

of facts

making their

such description activities Einstein's of ---

The other

side

invention

of the

concept in terms in

of force, of the

redescription space,

of dynamics ---

curtature

and so forth

does not

\4
itself involve an appeal out to facts, and could Thus for theory in principle much

have been carried of the mathematical had been laid-down

before Einstein

separately.

example

framework

of the

of relativity

by Riemann and others

it theory 36), with of his

many years
equations. an n-dimensional put quite fruitful forward, in, spite physical

clothed

relativity in

Riemann's manifold.

generalised (see ref.

measure was first

when it ,

was. considered of the fact that It

significant it had, of

and important

had no immediately it is true, certain

application. to, the led large" But minor to of the its to geometry

applications and it

figures about

on curved the geometry regions

surfaces:, of

speculation and "indefinitely

"indefinitely of physical these space. relatively

small" stemmed as from not the

importance applications language of of

so much from fact that it of

contributed "flat" of etc., space, "unbounded expanded in such

physics.

The concepts curvature, geodesics,

spaces

positive of

and negative (generalised) of

finite" the

spaces, descriptive

power

mathematics

and physics

a way as to bear the advent

fruit

many years theory.

canes, that

later

in

facilitating

of relativity
In other

of

planetary

motion

for

example,

accurate the

description invention

of

the

phenomena

concerned apparatus. always

preceded

of a good descriptive extent at least, this must

And to, some slight case; the theorist

be the

must have something

on which worth that opens of it it

to

build. will

But usually in

a new concept be so for come existing in-a-new, one.

which

is

really both it concept motion, ofhalf ,

while

a double, field If

reacon;, and, that

contributes up: poobibilitiQs contributed provided

Newton'

force also

to

the

subject' for the

of-planetary; development

a framowork

'. dozen a other

departments

of physical

'theory.
in terms in use. seems to abstract of

A new `concept existin conjunction In the first in , ones; or it maybe se$ of

may be definod introduced

undefined its

with'a case, the

rules of

proscribing its

no problem the

application clearly remains

arise;

second,

concept

until

an interpretation
it for is the,

is

put

upon it.
that

As a point
these two

of

methodology procedures clearly

no doubt -introduction

desirable of

a concept At

should the

be

distinguished

'and kpt

distinct.

same

time,

it

must be said in practice

here

that

the not

difference so great

between as it at

them is
first

very

often

appears.

There is that the "existing

are two reasons concepts" not in just

for terms

this.

The first a new they

of which

concept

may be define&do in-scientific rules in

grow on trees; either their

are themselves, concepts concepts definitions with

contexts,

undefined or use of on

(or axioms) governing terms they of others. hang is itself

defined

The chain

on which

hung either

some set on the unwise

of undefined

"primitive" ordinary

concepts language,

or' perhaps which In it is case

everyday to accept

pegs of as fixed

an unchanging. of the of order is concepts

either concerned

questions are not

of interpretation to be ruled out

as definitively they may in tail

answered; some may

and when a new definition cases wag'the legitimately dog.

appended afresh

be raised

the ----it

Under these that to the

circumstances

may be. important were an to that

to demonstrate independent
of the older

new concept

has as it claim

right

existence,

an equal

inhabitants.

Something
In connection originally some, of it their. with the

like
theory

this
of

happened
"matrices". of ordinary the

in mathematics,
Matrices numbers; order

were but that

defined

as arrays

properties -: to

so upset consider in in

accepted

became

fashionalbe

them

alternatively number 12) that etc., (in

as being field. matrices, obeyed general) and that with It

themtselves was shown under the lawn

"numbers" (b)r Cayley

an "abstract" 1858; of ref. addition except multip; in

suitable of

definitions arithmetic of

elementary

that'of certain

the

commutativity could

ication; connection could the be algebra

matrices

be used

rotations

of, 'co-ordinate'axes as angles. of two-by-two with that Cayley also

and hence showed with that

considered of a. class

matrices of

complex and it

elements was

was coextensive

quaternions;

later

shown

that with Although to

the that

algebra'of of in a class

complex of

numbers with field in detail

was real it it; is may

coextensive elements. impossible. easily of be

matrices

such

a complicated connections

trace that

causal the

seen

way wasopen'for as carried definition be paralleled "abctractl' In 13) out

a re-examination by Whitehead of real numbers and

the, basis

of. "number" that-the easily

Russell;

and also

so obtainedicould definition arithmetic recent of is context, first their

by an independent entities whose more

non-commutativo, that of matrices. (ref.

a relatively introdueeaihic and only

Eddington

"E-numbers" points out

by abstract possible

methods,

later One

matrix

interpretation.

is

tempted

to conjeetre as matrices field. In this

that

real

numbers are

are alternatively:, "numbers" of

expressible some abstract

whose elements

caso,

of course* beginning the the

it

may well

be

said the

that tail

the

dog was already

to wag before circumotanc'os procedure are in of

was attached. exemplify and the

Nevertheless the fact that

sufficiently definition not mutually

"abstract"

procedure

practice

exclusive. There is also, however, between another the fact which

somewhat clouds It is

the

distinction to express,

two procedures.

more difficult

and much more difficult

to is

exemplify. clearly the

Briefly, introduced theorist reservations context context in

it

is

that

even in

when a concept an interpreted may well that not the legitimare in Thus to say not the the two

by definition (and., scientists about under which it,

system, have the

generally) to the effect is might

mental particular

consideration the concept

most, general ately; be, usod;

and'even, they

when the are or treguently name the

reservations implicit concept.

made ,consciously,, 'words situation concepts

uned, to. describ may arise in

which in

we. may want different ,

that aro

B,, defined and ,A of to the

contexts,. indifferent we shall a limited the concept B when point: have in the any one. need set

exomplificationo and in up in order

name concept to do, this 0 with becomes and the to

fields; to of set

be able

abstraction properties

a concept which

axiomatic given another. introduce (or of at its

A when given when we our minds

one interpretation More anew least germanely concept

concept

the, present

A we frequently in our terminology)

presuppose

possibility definition

subsequent of it will

generalisation; be at best

and hence a provisional

we give

Perhaps-this could have defined then "force"

will in

do as an example: of the that very only

Newton

terms

two kinds gravitation

of force and that "force"

required

by physics, But the

due to

due to contact. would

uee of the word procedure;, it

seem inconsistent

with

this

suggests sorts of

that force. to

there It

are

at

least to

in later

principle generations that

other of were a generof the

was left with magnetic)

physicists other alisatin generalisation the very'first.

establish (e. g.

certainty which the

there

phenomena of the

permitted

concept;

but in

possibility concept

implicit was And we still generalisations

Newton's do not

from there Newton

know whether The fact that

may.not'be in'. fact which

to force

come.

chose causes

to

conceive

as an undefined these they

"something"

accelerations empirical merely as "the

gives air;

generalisations are not But two (--there if "generit

a straightforwardly alisations" had been which for'the reason been have but defined

new "applications". relation of their between masses" ---) of physics would or is

bodies to ignore is no have

exists

on account forces that

moment to

duo to the

contact

suppose

course

would simply

different; added

subsequent the, words "or

physicists their If charge, a concept will not than more

their first

poledefined

strength, in a narrow

or

....

"etc. this

context,

preclude an initially specific

a latof broad one.

broader definition

definition, precludes

any more a later

So much is What I want form, though to say,

obvious

as soon as it in

is

said..

however, a form

can be put it

a different not necessarily sometimes

it'is

in which It is

will this.

command universal

agreement.

What is

more use

important is the

about

a concept in (Thus which

than it

its is

precise expressed, "force" its

scientific in has a ordinary It are development; is

terminology

ordinary penumbra contexts the the if its either named". speaking, sometimes cases

language. of to

a word connected

like with

associations express

use

in

moral'or in

physical ordinary

compulsion). usage scientific too that

associations driving the force

developed behind use we shall is "wrongly of

aconcept's a word always differs

scientific use it

much from to insist is "wrongly

normal that

be inclined or thatit

defined" matter, concept

Now it to this to for

is'an'easy the that the

methodologically and leave it But at in that; other give up

I

re-name is all

can be dbne. concept of would the

simply

re-name a good

be to concept

the'c'earch

dofinition

properly

called

by the

original
"To

name.
with the same example: as a sort and if attempt of to this limit is the

continue

concept refinement can see

"force" of that to

might the'concpt in the

be represented "cause"; run any set at

of/JY4al is it likely possible than those be poorly so, by we.

long

definition to that

any particular It havo list, in is

phenomena in of principle cause "force"

be unsuccessful. motion any to should finite figure:

least sorts

other

named in loft free

and the hypotheses

word,

must even in

' .. __ be it

confirmed

or strongly

disconfirmed

hypotheses

--

covering

77

these only

other possible

contingencies. definition

(cf. of

psychical which

forces).. is

The

"force"

consistent

with

this

extra-physical (or virtual

requirement motion) it

is causes;

one in

terms

of

the motion
or a similar

i. e. Newton's,,.
could who has ''a
t

one. named the

No radically "force". of the i. e. for what

differont'concept And any scientist

be properly feeling out to for study

use

word

"force", will that

and who sets be guided he is as

dynamics, feeling

forces, it is

much 13y his as by the

studying

current

definition.

Such an account, and we must not Where there example, the is overlook an elaborate will

of course, other elements

is

oversimplified, in the
Y

situation. for

mathematical sometimes this

formalism,

theorist

be driven is-no-doubt
of the

more by the the

modern more confined esoteric to

momentum of his
in some of the But of

mathematics:
more even "abstract" the technical have of

case

departments terms of albeit

physics. departments a more

science class

"usages", people.

restricted

These so far as to permit for as the may follow no further.

points the

are

important of

hero

only of

in

deduction and for

a principle

methodolygy In so far or

science, principle from

scientific

criticism. without we need

may itself other admitted

be admitted facts,

them, press

them

study

the

intuitive study of his what

properties some-modern in, their

of

his

conceptual would

framework. call the

He must "logic"

logicians everyday

concepts

linguistic

setting. . can never

there'-could But, we can departures. critically, Too radical it will lead

This

does not for if

mean, of course, concopts

linguistic as ,a matter

that

he

flout

usage,

did
or of

not

change

be no advances, at least require should

scientific. principle and involved. in that that

from with

usage

be made consciously of the issues,

an apprediatian will

a departure to

be self-stultifing.. and it

misunderstanding,

may be retrograde

in
It

that
is

it

loads

research
function of this

in wrong
of sort;

or unproductive
criticism it should

directions..
to be added,

an important mistakes

scientific though

correct

that
ibility

the

scientific
to avoid

critic
such

has himself

a special

respons-

mistakes.

Historical
necessity occurring of for some such example

examples
principle wherever it

in

support
sufficiently possible

of the
abundant, to. say that

are is

"such-and-such -(or of etc. his ). "frustrated! theory" Such in

an author's ", --charges of

unwieldy

terminology --"the

delayed" acceptance of science", at (see

or ,"Precluded") (or "impeded the have his for example for

progress been the

levelled calculus

Newton

respect

notation

___- `- 1

e. g. at

ref.

37, Peirce 2, of

P" 39); in p.

at

Priestley with Eddington

(ref. his

7, work

'15);, P. in symbolic

'

C. S. (ref. theory

connection 519); at

logic his

in (ref.

connection 41, p. 137).

with

physical

fundamentals

In the
been P.

field

of scientific
Berkeley

criticism
(eee e. g.

they
'ref.

have in
44, p.

particular

levelled'at

87 and.

158)In such the matters it the is more of course very easy

to

be wise

after

event; without

so as criticisms re#oaling have obviated any

can be made ad hoc, general the never rules of

necessarily which Precise could rules like

procedure

situations be insisted

concerned. upon in

can naturally Nevertheless

cases

these.

it

is'possible

to critic choices

indicate

the

kind

of question ask himself, cases

a scientist and the

or scientific procedural
He must of this ask

can relevantly in him to open

are the main of

of difficulty.
of current are usage retained

"What

features these

word? ",

and

"Which

features

in

the

formal

thoory?

";

and if (1) (2)

a serious

divergence word some the in or

at courses. account thesis of (chapter issue

appears in the or

he should formal (3)

either or

choose qualify theory

explain none

a different the word in to

the of

theory, alter
(4)

way,
last

the

formal

conform,
points

resort

strenuously for place a later accepting for

and his This is

reasons not the but

these

a more chapter

detailed of this

principles;

"1

6)

may be, regarded technique.

asa

practical

essay

in

the

requisite

critical

The concept introduced of already into modern

of

"information" by definition in particular

has, been in it terns leans foundnot in so

science

accepted on the theory theory

concepts; of

heavily ations firm like distinct

probability.

The logical however, are

of that. the

the

of"probability, to there of take are the then at

we can present

afford one;

for least

granted two

a, case

quite

interpretations

probability

cal: culus,

as : Jell uses
with ,

as a plethora word

of philosophical'disputes (and. allied words)

over in connection
1 i
_

of the
induction,

"probable"
confirmation

and prediction. of information

An uncritical in'. terme anew, of. though

acceptance-of probability without But

a definition could raise

all`theseproblems except

r a i

contributing this Thus is

anything unrealistic; our

a new terminology. no definition will be to is discuss the

naturally

sacrosanct.

one of

tasks

adequacy
the the formal

of

current
conditions of of

definitions,
which

given
we should and

on the
wish to other less two

one hand
impose hand well a founded) may on

concept set

"probability", formal If conditions we find legitimately

on the (albeit

similar for in

"information". some circumstances

that

the

concepts we must

be linked,

be prepared

to

treat

the

relationship,

from

a logical

point

of vier,

as a reciprocal light
all

one. fields.

Ideally

we should

hope to

shed-some
Not

in
of

both
the

applications

of

the

"informin.

ation"-concept

are'of

a class.

Wo 'must., distinguish

particular
theory the

between
and the

(for

want

of better

terms)

the

"logical"
:

"technical"

theory,.

The, former of a logical in

concerns language, 1935, and more..

"logical'

content", by

of, a sentence Popper (ref. (ref. I

as first

defined:

37, 'p'. '67) p. , (ref. 405) 26,27). and

as elaborated recent theory in 1928, writers follows

by'Carnap

other The

such, as Kemeny ideas first in put

"technical" -(ref. 23)

forward

by Hartley

and developed found significant

1948 by- Sha.nnoin (ref. application It is to the our field tasks.

38,39), of to,,, -,,

and has electronic

communication. the close

one of

demonstrate two theories,

analogy

- which' has

exists. been (ref. course only 1).

between'

these,

an analogywhich

imperfectly Subsidiary'

explored concepts. are the

by. Bar-Hillel which are

and Carnap drawn in in the

of

the

comparison in (of bears "rate in of

those-of context

"correlation" of by a logical evidence): the

(particularly language) each of

as applied

and'"confirmation" these concepts of

a hypothesis some analogy-to transfer of

technical'theory's from In transmitter

concept to

information" system. between break

receiver of a full

a communication analogy, we are however, requiredlto

the

development

the

logical

and technical ground; it

concepts will

some new logical

11

be, shown that

are of of framed a logical what

certain
naturally

of the
not

requisite
in terms

logical
of

concepts

most

"sentences" but or in terms-

language call

as usually "unsigned

conceived, sentences"

we shall.

"questions".

Chapter
of with thiss relevant featuroc of

2'is

an introductory
theory. in

presentation
A comparison 3, and of

the-technical is the 4. '' between the attempted

r`, 4

the i.logical'"theory continued. with' ,

chapter

introduction-of

the-Logic

"questions"-

ii! ', chapter

The analogy' theories be explored the without that is primarily asking

technical one;. whether two fields.

}logical and. and. it it isSuch-. -, dan

a !'forna1 the question in the in

same concept

appears raised

a question that there

is-explicitly are two --one other, ---

chapter

5., r'It"'seems title see, of, there concept respective is

claimants and actually, connected

for.

the - proper as we shall

"information" at of least

with

the

physical their

"entropy"

and we must

adjudicate

claims.
Thertremains, point logical simple requiring, concepts mathematical systems a "message", languages. discussion. are defined models with of however, Both the one outstanding technical of relatively former for on what artificial in more and

in-terms reality,

the

communication constitutes logical

definite the latter of

restrictions for certain

Questions

application

general This In

contexts situation id at

are of

not

necessarily typical it of in is

straightforward. of all any formal readily present that natural no system.

course

the

case that

issue,

however,

too

assumed intrinsic

such

extensions and

application

difficulty,

particular

languages croatod at
In

(such for in

as English) or for by the

Shannon's written 34), are

can bo fitted logical employment

work (ref. and which

into

the (or

pattern, both)

"nessages" principle

languages

leant
this

of known techniques.
40) on tho by auch

connection of (ref.

"redundancy" Mandelbrot

English, both of

some work seem to

involve in

a presupposition, liable to

relevant. But that the

Such work we must information any unit ...

extremely case

misinterpretation. thinking is the

see what in more

can be made for natural than "the languages page, in

ordinary fundamental This is

measurable lino, 6. the

in chapter

" etc.

attempted

chapter

It thesis which in

will

be seen that firstly so to in speak, that

"language" it is

entere

this

two roles; is,

the medium in be it or the natural "code" the word

information

carried, language in

language

or an artificial system; itself at

logical

of a corsmunicution "information" discuss

do.

and secondly of it

that

is

a piece the job

language,

and we to

its

success

has been selected

2.

THE TECHNICAL THEORY OF COMMUNICATION AND MEASUREMET1T.

This presentation and-is. tively relevant Only to for of here of the

section technical only known. of the

is

designed theory of the is

as an introductory information, theory is comparaof

necessary

because It of

new and. little portions is

largely (ref.

a review 38,39). must

work

Shannon

an outline original of

presented, or or to for

and reference works more of

be made nature

the

sources theorems

a text-book treatment 19,21,

proofs special

detailed refs.

topics.

(See

especially

t#6).

The major for from the most part the "discrete"

in the later

applications "continuous" of

of the signals

theory as distinct

involve signals

(typically)
will however signals, is here less

telegraphy.
not and dealt than extend

Our interests far the beyond theory*of

chapters of discrete signals

theory continuous

consequently with full to

somewhat justice.

summarily, Shannon's

and with measure

perhaps of

information

"subject

a fidelity touched at

criterion" all.

(ref.

38 or 39,

ch"

5) is is

not also

Somewhat summary treatment

7777

r 19

accorded since points of the its it

to

the

"theory for

of us

reception" is largely with

of in

Woodward certain

(ref.

46),

interest raises.

logical

We conclude between of

a brief

discussion entropy, the problem (ref.

analogy

"information" the technical

and physical theory to

and the of

applidation

Maxwell's

demon in

thermodynamics

by }3rillouin

5' 6).
Certain
theory deferred.
7

results
into

connected
later

with

the
and

technical,
are

fit

more

naturally

chapters,

2.1 ation Lotus the are beat

The roots seen first in

of certain

the

technical simple like the

theory systems of

of

inform-

telegraphy: at of

consider transmitting on it,

a system is

following: with

end there and of any

a battery of multiple of

a number switch,

terminals the to the

some kind one of of

permitting voltages and at

connection the input

a number telegraph

different wires;

end of

a pair is at

receiving of

end there the of switch symbols

a viltmeter. the transmitting as the the

The various end are of the the its marked

positions with

some set

such at

letters

alphabet, same set scale. a battery the switch

and the of symbols

voltmeter marked'in at

receiving

end has places

appropriate transmitting to

around

To each voltage, is

symbol which to

the is

end corresponds the wires when

connected and this

turned

it;

voltage,

received

Zo

at

the meter,

sets, the

meter

needle
to

to point
over

to the.
this system

symbol.

We-imagine

messages

be sent

as sequences 2.11
be'a This maximum is

of the

available

symbols. reasons
such of

' will necessarily

For various
speed only system time to at which

there
a device

can be operated. but takes its successwire

so not of the

because itself. settle

human limitations, The meter down at needle of

because

appreciable an ive-positions; itself-has It S ttling voltages

each the

and more characteristics time" can-be and limit made to

importantly which the succeed rate

telegraph

introduce at. which

an electrical different depending

I

one another,

also limits
W4ys.

on the

power available

from they

the

battery.

These in various
have some

are-not-absolute;
Nevertheless any

may be extended
system will

practical

such limit,
imately quite second 2.12 (--exactly that

and it

will

be possible

to

specify
in

approx-

and as we shall ---) it the

see later,

some cases per

maximum number Let us call

of symbols this does "r".

can handle. This quantity,

howevor,

not

alone

tell

us how fast important

for

messages is the

can be sent number of

i. e. the at

over

the wire. symbols

of possible

Equally
available

elementary
number the

transmission, switch

(non-equivalent)

positions

transmitting

end,

and the

corresponding

number of

significantly

different

2i

positions suppose'that alphabet, a "space! ';

of. the tho and that then

roter symbols the there

needle used zero will

at are

the the

receiver. letters is used of to

Lot the

us

position

indicate at the

be 27 switch

positions

transmitter meter
"signalbi 2.13

and a corresponding at'the

lovols", Now it code fact

27 positions We call
it known by

of tho the number of

needle

receiver.
and denote is well

this
"n".

that than

messages two

can

be transmitted-in levels. Bacon (This

with

not

more

signalling by Francis cipher").

was implicitly his example, "on" and famous

recognisedt "bilateral only the are two "dots"

when he invented code, for call

The Morse which

employs "off";

levels, and as Thus and a

wo might of

"dashes" compounded a "dot" "dash" such

standard of

Morse

practice of these

regarded levels.

time-sequences one elementary one time-interval and two and symbols, additional If

two

occupies throe;

time-interval, is allowed

between for two and of

two a

symbols,

time-intervals we regard "1" code Thus for the "on"

"letter-space", as elementary "off", alphabet

so on. putting the

lovels "0" the for

we can regard into these

Morse

as a code we have

symbols.

"dot"
"dash" letter-space

10
1110 00

Z'Z i.

whence

for

example

the

letter

"A"s

or

"dot-'lath",

becomes:

10111000
and the letter "Q", or "dash-dash-dot-dash", becomes 1110111010111000.

2.14
sentation code aural is that arises

The complication
of for course ready from

in
the

this
fact

method
that

of reprethe Morse for

wan'designed or visual of the

manual

transmission case,

and

reception. binary code Here

An analogous used each in letter

however, with alphabet two sequences of the

connection of the

modern

teleprintor.

occupies'five signalling five; remaining each

elementary levels, letter there is

time-intervals. are

Given

25 = 32 possible a sequence, spaces,

allotted usedifor

and the stops and other

sequences

are

purposes. type,
2.15

This

is

clearly

as economical

a code,

of its

-as could

be designed.
Notice, however, that whereas in our

original

system

(with of the

27 signallind; alphabet in

levels)

we could timewe

send one letter interval, in the

each elementary (with two levels) (and in

teleprinter

system per

must use five on the

2.16 the rate at

time-intervals many more).

Clearly which to got

letter

Morse,

average,

a realistic over

estimate

of

messages

can be sent

a channel

we must take into

account

the number n of signalling

levels,

or

(in

a general number a symbol affoct3

sense) the

elementary rate, with the such

symbols. that a choice of in

To

see how this transmitting between per

we notice making rate r

corresponds Doubling two

n alternatives. would

symbols place;

second

mean making

choices

of each previous
to making a single doubling of

single
choice the rate

choice,
between r is n. n is n to

and this
nxnn

is

equivalent
n2 altctnatives; sgunring

that so the the

equivalent In general, to power; the

to

number rate of step

alternatives

increasing raising and it rate the is at of

r by a factor alternatives from this to

equivalent the nth that

number short

saying

maximum

transmission

of mensagea is signalling
the nm

proportional levels,

to tho. lobarithm using

-

of the number n of
(characteristic of log

the

identity

logarithm mm log

function): n.

We call
capacity

the
of

maximum rate
the channel,

of transmission
and write it

so defined

the

Cr
2.17 measures is to that of with this two It remains quantity. signalling

log
to fix

n.
on a set the it of units for

Since levels, of define other

simplest

system

seems natural in terms or (for in of this,

express

characteristics

systems unit"

and we correspondingly short) "bit" as the

a "binary of information

amount

involved

a choice

between

two alternatives.

(We shall

later

require

..

z4

that

the

alternatives of in a system "bits per

be equally, as in the if base

probable). above formula

The will that since

information be the logt 2-1

capacity measured logarithm this 2.18 (with gives

second" to

we specify 2; for

be a logarithm 0=r for Analogously

a two-level

system. "natural units" units" (with

one can use e) or "decimal the c. identity logo only to

logarithms to

to base loge

base 10).

logarithms

From logg

bare is

b by a scale logt 10 factor. bite,

we see that Thus or

measures unit

changed equivalent

one decimal 3.32.

about

2.2 whereby information a design " His 'the

Hartley capacities

in of

1928

(ref.

23)

proposed to its

"a measure

cliffernnt and

systems suggested

transmit use as system. "inform-

may be compared", parameter is for

any projected what

communication called the

measure

effectively

we have

ation

capacity"

in

the in

case of telegraphic pronciple

levels. the

systems, signals

somewhat

generalised
are that would in , not

to covor
to

cases where
It could

confined this to

discrete

be imagined capacity since of possibilities

without have of

restriction

information infinite; number number in

be considered between choice of

potentially a finite a real

place

a choice the

we should

have

a"continuum.

_I

i. ,

2S

Such a postulate in practice (or there other

would will

be unrealistic, always

however, limit that can that

since to be can and levels but be the

be an upper quantities) discrimination close number of the of

voltages employed exercised thus the

physical to two the

and a limit between notion not of of

voltages

together; signalling

a finite

reappears, as a feature 2.21 adjacent acies in

as a feature physical Limitations

"code"

employed

limitations. on discrimination between inaccurof elect i

voltages the

nay

be imposed used; other or

by unpredictable by the presence or "thermal from

ecqipment signals

interfering trical storms,

from or and of the

equipment, by

etc.; origin

ultimately of magnitude various in

noise" principally circuit

of molecular

depending the

on the. temperature and the black-body

parts-of the space

radiation

surrounding

it.
2.22 riyquist

(Sve. ref.

36).
Drawing to some extent proved Q .nee on the that work the of

(ref. in (

36,36a), _this to more

Hartley rereral.

information

capacity channel dependent magnitude

z)

of a communication
limitations and This is on the is a result

subject chiefly of the

realistic on the

physical power available

"noise"

interference,

which

has been derived and we shall

in

a still deal with

more general it-below

context 2.6).

by Shannon,

(section

z6

2. j comparison of of C-r (in any of

In-the different

work

of

Hartley

and Nyquist, the

the possibility in terms

systems

presupposes of

some sense) desired logt n set sets

efficient of

coding

a message

fundamental limit

symbols. to where But it the the does to effect rate

The formula, at of which inform-

an upper

information ation under a, rate does not it is the

can be transmitted, "binary choice". it This is

unit not attain

specify such it all,

what of

circumstances transmission.

possible means of in

that at

give

us a definition not to toll

"information" sort of

because of

does is

us what

transmission at the

a message rate.

be regarded

as transmission

maximum 2.31 to This to the is

To answer problem the of the

this

question of the of

we must different of

return codes.

comparison of rate not

starting-point of aerage which are

work

Shannon, of

leading inform-

a definition in full channels capacity".

transmission

ation "at 2.32

necessarily

transmitting

Let

us reconsider

the

telegraphic

systems

introduced ignore
the

above.

We can for

the

moment conveniently sedond,

as

r,

the number of
of a system

symbols
per

per

and express

capacity

symbol

C= (We shall tacitly

logt

n. the number of symbols per

assume that

Z7,

second let

is

constant that

and the

sane

for

all

oyotenis). say, of

Now a and that

us imagine of

a message English)

(consisting, is to

passage any

ordinary

be transmitted be performed and receiving connecting referred to such

coding

and decoding by special of

required

will

automatically apparatus. witting channel; of the

transmitting the system is

he part and receiving and we shall

trans-

equipment conoider than of

as the

0 to the

be characteristic and is of For the channel

channel

rather

transmitting that number it

receiving characteristics symbols purposes 2.33 both of require 1 bit of

system,

on the that

assumption the

determine levels is

elementary present number. code a capacity introduced between 4

signalling channel

available. defined the

the

completely code nand

by this

The Morse a channel symbol; n-

teleprinter with system of do the

with the 22, to

21 i. e.

per

alphabetical i. e. what

earlier and 5. systems tantamount rate of

requires

a capacity extent

We can now ask: utilise to the asking

respective This of us is the to

capacities for of codes.

they

require? definition enabling

a suitable information,

transmission different

compare

2.34 Before
a few must

Such a definition we proceed

preliminary take.

is

provided let

by Shannon. us notice
a definition

to

give

it,
about

however,
the form

points

such

t 7

In capacity information information C but such C has rate rate that, to

the

first been symbol,

place, conceived

since

the

channel

already per R to at C by This

as a maximum expect general cases, message that in bits it the less it or actual than can be of be

we should in ideal of

be a quantity 13 zk in choice of

least suitable means,

made equal coding measured 2.35 depend but also not

system. in the

course, e. g.

must per

same units Secondly,

as C, and most of

symbol.

importantly, the coding to

R will system employed,

only on the

on the, nature nature this by it share: of the of the

message

be transmitted. Although than which short occur it is the the codes

We can demonstrate the Morse code is

by a simple less

example. efficient

comparison embodies it

teleprinter latter to the does letters in

code, not

one principle relatively likely to that

gives

alphabet

most designed

frequently to transmit.

the That

messages this

(English)

gives

a considerable

saving

in

the

case it is

of typical not perhaps

messages

is

sufficiently that

obvious,

though
of of the the

so obvious
lead

application
in the case here, least with

same principle teleprinter is the

could code.

to. a saving 14). efficiency of the with

(See ref. of the

The point of messages at

however, some codes which of the they

dependence

on the are

characteristics Thus "T",

used. "E",

a message etc. will

a preponderance as regards

letters

"A"

compare

.3

2y

transmission same number so: long teleprinter R is must to

time of

very

favourable but is two

with

a message in the "Q",

of

the etc.,

letters code the

preponderating but in

"Z" the

as Morse code permit

used; are

cases of If the

indifferent. of two such

quantity it of make it must'"

the

comparison on the

systems

either

depend

detailed message be of ovei,

characteristics (which would or it

some particular insufficiently be an average In the latter

contemplated general value case of relative it the to taken will

much value) a set of such

messages.

depend in

on the the

statistical set chosen;

characteristics i. e. on their

symbols

message

frequencies.

2.36'
depends that rate which the (a) is of of it of is on the should the

But thirdly,
message it

if

the

information
to

rate
require

seems reasonable to define of to the the

be possible

"information coding operation it. over forms, which sequence it

message" out

independently. on it in in (b) In order effect the each merely another

carried

transmit in-two in

channel. the sent original over

The message form, the channel, consists for and

appears coded case in it the

form is'a

symbols; one set of

coding symbols The will is

substitution not necessarily of any rate the the if feequentwo

(though

one-for-one). sequences attention cies of

statistical in

characteristics some way, at

be-related paid to

correlations

as well the

as to

single

symbols.

Consequently

information

3n

rate form

should of the

be definable message, the

in

some uniform such (It

way for it

either remains

and in

a way that course

unchanpediby to per 2.37 information , simpler Shannon messages symbols. given whether notion calls take

coding. of the

may of

be necessary of symbols

account before

different

average

number

second

and after we are in the

coding). led from the to notion of the

Hence rate of

xlxxkzk

a channel information. source",

the. -somewhat of what of of be their

rate i. e.

an "information by the

a set

specified

statistical rate in since

parameters a channel it makes coding will will

The information same definition, the of

by the

no difference or after; be its

we consider property under those

messages the

before

and a basic invariance including

measure class of

defined coding

a certain contemplated

operations,

above.

2.4 a measure 2.41 theory to of

It

was Shgrmon'B which convenient first

merit satisfies in

to

have these

diecovred requirements. Shannon's and comprises of latter the

information It is

presenting "discrete" The former number the by

distinguish

between systems.

continuous" cases where,

communication as above,

there elements

is or is

a finite symbols; carried

information-carrying case in which the

information

some quantity

which

can

take

any in in our

one of practice original the

a continuoue is frequently of

range

of'values. conventitn.

The distinction For example,

one of

example

an "alphabetical" used are

telegraphic chosen scale.

system,

27 signalling is essentially

levels

arbitrarily If in this

on what case the of of

a continuous are regarded as

27 levels system, user

an unaltorablo from built But for the and point

feature of view

the the

as is the will the stage,

appropriate system is

once

operating, purposes, of levels

"diacrote" e. g. at

analycic determining design

be appropriate. maximum or determining

other number

usable

the

the maximum rate considerations

will 2.42 systems first. be relevant.

at which arising out

symbols of the

can be transmitted, "continuous" theory

We shall

treat of

the

theory

of

"discrete" systems will

The theory briefly Let us in

"continuous" 2.6. first that

be discussed

section

suppose

we have

a set

E of mutually about
P(E3) We look

exclusive

possiblo except
of the

events their

E1, E2,

.....

En

which
(j for -

to know nothing
1,2, ... n) of

probabilities
in some situation. involved

occurrence amount of

a measure

"choice"

in

selecting

the

event
of

which

actually

occurs,
we are with

or in
respect

other
to

words the

a measure occurrence

how "uncertain" the given such situation.

in

Calling

a measure

H(E),

we observe

d J

3Z

that These

it

will are

be a function subject 0 to

of

the

probabilities

P(EJ).

inequalities P(EJ) <1 the set of events to be

for

each

j;

and we assume whence we shall P(ES)

exhaustive,

have

-.

-Since of is the

the

events

are

mutually

exclusive E8Ek of

the two

probability such events

Boolean or

"intersection"

zero,

P(E jEk) o2, "join" is the what amounts Ei v Ek, the P(E Jv to the i. e. same thing, the

0 the

(j

'

k) of or Ek, the

probability of Ej

occurrence probabilities P(E of j)

sum of

individual Ek)

P(Ek)

(j Yj

= k). of

The probability an event E is given PES) by

the

"complement"

1 -P(E3).

2. k3 on the measure 1. probabilities 2. then it should n (since

The following H(E): It -

is

a possible

set

of

requirements

should

be a continuous

function

of

the

P(ES). If all the probabilities increasing likely events P(E ) are function there equal, of is the more

be a monotonic with equally

number

33

choice,
is

or uncertainty,

when the number of possibilities

increased). 3. If is the choice down should determining into two the occurrence choices, average

of the

an event original of let E2 v

broken H(E)

successive the

be the

sum of to of

values Thus Elv first Now let the tive P(Ei)/Z

the E' ...

measures set

appropriate consisting E+l these El, ... chosen whose the v two, Ei

such the

choices. possibilities and let the H(E'). of

be the v E,

two

and

...

v En,

choice el

be between be the set

giving ; if will

a measure the first

alternatives set of P(Ei)

has possibilities where H(el) in

been

this

be an exhausare 1 to j; now and and alternative. to the

probabilities is from

summation these of

we can define similarly We average probabilities of our the first

using the :a

probabilities; the second

11(02) these of

case

measures occurrence, alternatives i. e.

according the

their

probabilities; thus

and second is = H(E' )

respectively:

requirement H(E)

+
+

P(El
P(E,

v ...
+1 v ...

v Ei)

H(el)
H(02). measure other for terms the

v En) is and the the

The first corresponding give second the

terra

on the with the value

right-hand first of

side

choice, xmamul

average

measure

choice.

3t

2.44 only is of measure the form

Shannon II(E)

proves

(ref. these

39,

p.

19)

that

the

satisfying

three

requirements

H(E) where of 2.45 the of constant a base of for the K is a positive

-K

P(Ei) This

log

P(Ei) is the basis

constant.

result

subsequent It K it the

theory. will be seen that whatever the value choice of.

may be made unity logarithm; with choosing A convenient 2, H(E) of then choice

by a suitable fixing

and that a definite the units choice being

on a definite base. H(F) and in the "bits". case of is

value merely to

K together to

logarithmic in is which K-1

amounts

be measured. base is equally above. the

logarithmic One "bit" of this two

measured in cf.

amount probable

involved

alternatives:

the

use

unit

Thus

we have

TIt )
2.46
the maximum

i
the
of

PCB) 1og2 P(E )

is
the for

(bits).
fixed,

If
value

number n of alternatives
11(E) i. e. 1062 n occurs P(Ei) (bits), value is when 1/n all

probJ.

abilities This with and zero; gives

are

equal, 2.

each in

H(E)

and is of unity

accordance is zero, rest

requirement occurs there

0

The minimum the as it P(Ei) were,

H(E)

when one of is-here,

and the

no choice

or uncertainty.

35

2.47 is This its is additivity virtually

An important in the

property case of

of

the

measure choices. 3 above, Suppose but

independent

specified more

by requirement clearly as follows: exclusive exclusive joint events a member of the

may perhaps we are given

be put

E a set/of

mutually mutually set of

events events EiFj of

El, F1, (i

... ... 1,

Em Fn. ...

and another We denote .. m, j=1,

set

F of

by "EFL' the .... to n).

Choosing both also

EF is

equivalent of of F. If

choosing

a member given that of

E and a member probabilities of events in the in is i, j) of set E and F,

now we are of that E are the

events

independent of

those involved

we find EF is

amount the

choice of

the

sum of

amounts the

choice

involved requirement (all logt

F separately.

Thus P(EiF

independence P(Ei)P(F P(EiF

whence

H(EF)

J)

PCEjF j ) P(Ei)P(P j)

P(Ei)P(F3 j -Z P(Ei)P(F3)(1og2 P(E1)

) logt

+ logt

P(FJ))

a-zZ

P(E)P(F4) 102; P(Ei) 2

-' P(Ei)P(F) logt P(Ei) 10 P(FD) 2 ) logt p(Fj)

-Z 11(E) In a similar case,

P(E1) +

-Z J

P(F

H(F). where. two such sets E and F are not

36

independent, measure denote We can relative based "the then to

it

is

convenient

to

introduce

a relative Let "P(F3, Ei". in F Ei)"

on relative of the

probabilities. event of F, given

probability define

event involved

"amount -

choice

E" as follows:

H(g,
From the

E)

formula

P(E1FJ) j

1002 P(Fjt

Ei).

probability P(EiP

P(Ei)P(Fj$ quite generally for

Ei) any two sets E and F

-there the

follows formula

H(EF) In the special

H(E)

H(F,

E).

case of independence H(F, E)

important..

we have simply

H(P).
later when we dome to consider

;.These "noisy"

formulde channels.

are

2.5 type above of will

The application telegraphic already communication be clear. it is that

of

this system

measure

to

the considered exact more

we have

Before necessary must

we introduce to examine

definitions, closely 251 to consist of (i) (ii) the

however, assumptions Shannon five

be made about a communication

the

system.

considers (ref 39,

system

parts

p.

4): -

an "information the "transmitter"

source", (or "encoder"),

. _...

37

(iii) (iv) (v)

the the the

"channel" "receiver" "destination".

(of (or

given

"capacity"), and

"decoder"),

We shall-'start ation , rate -of first source", at the which measure in the of "with it a view to

by

considering

the

"informof the

obtaining

a measure

"produces" 11 defined

information. above, however, represent

The application involves the us

problem

of

how to

statistical

properties 2.52 case we are

"messages". An "information source", anything of in capable which finite the of discrete producing we may set. _, out of be

_,

considering, of symbols

is

a time-sequence take Certiln account interested to be drawn kinds as of in of

some sort,

from source,

a pre-determined however, In (or

may be ruled

no interest. "random"

particular

wo shall or

"statistical",

"stochastic")

sources
of this

rather
sort,

than
however, in in

"determinate"
are not

ones.
necessarily the=selves, state of were

Distinctions
to but be taken rather with be --be

as distinctions as distinctions respect capable say the to of them.

the our Thus

sources assumed if

knowledge known to

a source

producing in the

only

one particular of 'r ---

sequence it since would

figures

expansion information

regarded-. sequence without

as having could reference in

zero principle to it.

rate,

the

be calculated Alternatively,

independently however, the same

 .. _w. 3$,

source-would -, } about given this it .

be ' considered it

random

if

all

that of

were digits

known with

was. that'. equal) the

produced

a sequende for would which each not

(say

probabilities. rate to. sources , ofour

digit; be zero.

and

in

case

information apply.

Similar,

remarks

emit

recurrent

or periodic: 2.53 source_is, "-orf the produces. "; involves set 2.54 from it is the

sequenco. The-state specified:,.

assumed statistical

knowledge

of

by means'of set. of

parameters that it

sequence,

or

possible of auch

sequences, statistical which it

The, specification certain primary

parameters is well to

assumptions

out` explicitly. - point The fundamental of view from of our of property present possible a single old of a "message", is that if

approach, messages. symbol. it S1,

selected

a set of about only it

Thus

message,, a consisted the , important. from symbols. fact the are

SL say, had been .... Sn further

E

be that symbols

selected If the

set not

of

possible

equally

probable, with set their

we must each; of

specify-the-probability the symbols represent alternatives, A set

associated an exhaustive tho with Thus is sum of specified the that

and

since

mutually

exdlusive oust

probabilities is about in

be unity.

probabilities we assume an ensemble

called'an the possible

ensemble.. messages

knowledge they form

..

31

this
2.541 of

sense.
More of generally, if a message speak consists of it as each N

a sequence from long. deduce

N symbols, of

we shall possible symbols

selected symbols we could

an ensenble If successive the

messages were

independent, of the the ensemble of be of e. g. that pousible we

statistical long but from this

properties those will. high in English of not

of messages single-symbol the care.

N symbols messages, (Witness

ensemble

generally

the of etc.

relatively letters ). Thus

frequency text, assume each

certain "TH", are

combinations "ING", "TION"

we must with

given

the

probability

associated

TZ-symbol messages, specification indirect. 2.55 we permit extend directions. assumption; probability Nevertheless to of to

message; and this of

a1thou ;h si:; ce there are aN such the may be a very large number, the probabilities in practice will

be

A theoretical the messages in is to both naturally

simplification be doubly the infinite,

results i. e.

if to

infinity This and it of each

positive rather

and negative an artificial to specify the

now becomes message of

impossible the ense; ble it

individually. possible in terms

with the

certain theoretical for

assumptions concepts this case; the theory

remains

formulate limiting

required

processes

and. practical in the came of

applications

can approximate

4o
"sufficiently attaches 2.551 of doubly
.....

long" to this

massages.

A special and it is the

importAnce one adopted. as consisting

procedure, Let us thus

picture of
sjO,

our form
s

ensemble

infinite
53-2, ,

messages
s_l'

the
Sill

J2"

.....

with

some appropriate Now such unchanged 'all to the the

specification an ensemble of is

of

statistical stationary Thus shifted, occupies ensemble, if we say, the if

parameters. if it is

called

by a. change of

origin. ensemble each Sjk the

imagine x places position

sequences right.,

the that

such occupied

originally

by S-, k+x

stationary,
and the will or the

still
massages case

consists
still have

of the
the

same messages

as before
This positive that do not of the

samo probabilities. x, whether is

be the negative. statistical

whatever

the idea

number of

The basic

stationarity of the If say: the

characteristics sequence progresses.

sequence we think

change

as the

sequences means do not 2.56 that

as time-sequences other with things time. being

we can equal

stationarity probabilities

change

Amongst

stationary

xwj=

ensembles

there
called but

is

an important
A full idea is

sub-class
definition a very that

of

ensembles
cannot one; of its

which

are
here,

ergodic. the basic is

be given an orgodic it is

simple

ensemble of the

one such

every

member

typical

ensemble

as a whole

as regards

statistical

41

properties.

The literature

on the

theory

of

ergodic

ensembles
covering reference. (ref. 15);

is
the

extensive;
case of

for
ensembles

definitions
of discrete (ref. without A full of/ 24+).

and basic
sequences or to is

theorems

may be made to a brief 39, "p.

'rechet

17) proofs

Fuller given by concerns

treatment 15-).

Shannon mai#ly functions

(ref. the

discussion sof

which

more is

general

theory (ref.

continuous

given

by H6pf

The socalled

"ergodic
of in

theorem",

to

the
can

effect

that

"almost
on to with

all"

members

an ergodic time, to

ensemble cribed

be relied tion,

go through, the frequency proved by

any. - pre: the'

convIlu-

proper

ensemble

as a whole,

was first

Birkhoff
by the others

(ref.

3) and has been subsequently

Wiener of the (ref. word 42a). "almost"

gonerGlised
incidentally' above

including use

(Notice in the

technical

formulation zerd').

to mean "except

for

a set

of total

probability

Speaking ergodic process simple of when which example it is is in

very

broadly, in terms

an encemble of a statistical one. the

is

definable a certain suffice is

sense here to

a uniform indicate

will that

kind

restriction

implied.

2.561
different

Suppose two biassed

characteristics. The

dice,

A and B. with
of all possible

ensemble

sequences ,, similarly-that

of tosses all of

of die

A is

an ergodic sequences

ensemble, of tosses of

and

possible

4z

die those sort

B,

provided normally

only

that

the to

conditions apply when

of

tossing of i. e.

are this

understood to. illustrate

examples theory, of

are

used

probability and independence being

randomness, tosses present ensemble a, the of of b (the

stationarity, latter condition

successive in the

unessential construct two

instance).

We can however of the

a non-erodic Lot Let tosses with ensembles sequences

as a combination positive

as follows. sum is unity. of

be two probabilities

numbers

whose with

associated by a,

sequences associated the which of two is this

A-be tosses

multiplied of

and those

B by b.

Then putting

together The overall obtained with (not

we have

a new ensemble properties those a, zero) b; which e. g. Thus are only it and not if) is of

non-ergodic. ensemble can be

statistical by averaging factors

the but

constituent there will

ensembles, be subsets different with the subsets as a whole.

weighting of propability

have

uniformly

statistical original of

properties, ensembles. which (and

those will

corresponding be appreciable of the ensemble

there typical

sequences If

2.562 to ergodic

we restrict permissible of the

our to

considerations alter our properties of the

ensembles, somewhat,

terminology of

speak

statistical of those is

message-sequences from all" which

themselves they are are permits

instead drawn. of to This

ensemble "almost Ergodicity,

because

sequences that is,

typical us

the

ensemble. ("almost always")

equate

`.

,3

relative probabilities ensemble 2.563 the

frequencies derived apriori. Let us,

in

a particular consideration

sequence of

with the whole

from

for Sj

example, at

represent t ... in

by'E

i(t)"

occurrence the

of

symbol

position are S1, that as the

a sequence, is at

where

fundamental Notierthe t is basically of the set

symbols probability defined of

Sn and t

an integer. podition probability) this dent position of t).

81 should measure have this ergodic is

occur (total Si in

sequences stationarity is

which

(and But if

assuming the

indepenalternat.

ensemble

we can

ively

represent
of the

this

probability
Sj in the

as the

limiting
under "almost

relative
considerall"

frequency ation,

symbol will

sequence same for

and the of

result

be the

sequences

the

ensemble.

2.564
Shannon generated such that further

For the
restricts

purposes
his

of

communication
to

theory
ensembles

considerations i. e. each than random

by Markov the

processes, of more

processes on of

probability but not

symbol a finite it. of

depends

a finite the

number, which

number, Such

symbols

immediately by giving

precede a table the

a process

may be represented Thus Si, in at the

probabilities. of M-1 aymbol

above-notation t+M time 0 given that

probability

-ml backwards,

the preceding/symbols,
... ,S J1I will be

enumerated
writton

are

Sim

Jm_2

P(E m
and will relative of values

(t+n)

(t+i) (t)E E. il ID_

of for in a i1arkov t; all the

....

EUm-1 (t+n-i))
of such

be independent probabilities, of Jos ... of

and a table poocible range 1 to of

combinations n would n+1. concitiute

a specification

process the

order

Equivalently, P(E J0 (t);; i1 (t+1) .....

absolute

probability

E Jm (t+m) )

could

be tabulated,
are

since

this

and the

above relative
ratio by is summation.

probability P(E4 m (t)),

mutually is

derivable; obtainable

their from either

and this

2.565
sequences sequences his claim is of

Shannon affirms
sufficiently practical

that
to In

this
cover order

class
all to

of random
messagesubstantiate

general interest.

he gives in

examples

of with

"synthetic" the

messages

constructed of ordinary

accordance

letter-probabilities of 26 letters a random probabiility: -

English. one can which all

Assuming first for

an "alphabet" reference appear with

and a space sequence in

construct equal

symbols

XFOML p, XKHRJFFJUJ YDQPAA. PMZAACIBZLHJQD Next with their appropriate OCRO IILI EEI .

ZLP6WCPWICCYJ FPJEYVKCQ GII-

taking

letters

independently -

but

probabilities: RGR NLIIELTIIS

EU LL NB NESERYA TH

ALHEN 1TTPA OOBTTVA NAH BRL. IS', each letter is allowed to depend on

4s ,
the preceding one: ARE T INCTORE. ST BE S DEAMY ANDY

ON IE. ANTISOUTINYS L.CEIN D ILONASIVE TOBE SEACE CTISBE. At two preceding the next is

TUCOOW AT TEASONA1E FUSO TIZIN

stage, taken into

the

dependence -

on

letters

account:

EEY CRATICT' PROt IN 110 IST LAST W, PONDENOr OF CEE. The reoeLlblanco increases by carrying more striking at each the stage, to English OP DEMONSTURE3 OF ' THE REPTAGIN IS

BII $ GROCID REGOACTIONA

text

clearly do so more

and could further.

be made to Similar

process

and perhaps using words Wo shall (section 5).

examples

can be constructed instead on this of subject

as the however

fundamental have more

symbols to say

letters. below

2.57 definition section of 2.45 of

Vie are information above, information assumed in

now in for such per

a position a singlo a way as to

to

extend

the as in a definition

"choice", provide

average

symbol

of

an "information process. apply but to the roups groups

source",

representable is to

as a Larkov necessary single to the is symbols limit to

A11 that previous of are 2.571 definition and to to not

symhols; allowed

proceed infinitely

as the

become Thus let

long. the

us

consider

group

of

m successive

symbols these

starting symbols

at should

position be, in

t. order,

The probability S, S,....

that S, is

Um

P(EJO(t)EJl(t+l) Since this is, independent P(jd', where for of p is-a , simple ill

.... of ....

Eim(t+m)). t we may write jm) function per symbol of In the this j. group Now it simply

numerical

the m+l

average

information we define

symbols -Hm+l. " =

2. JO i1 ..... jal

P(Jpt

ill

...

Jm)

10g2 9(i01
,, and for the final definition lim. of

il,

...

JO
rate

average per

information symbol).

(bits

;. to ensure that

The ergodicity such a limit

We might have Hm+l the of ,C Hm. Hm, note

of the exists
in

ensemble

is

sufficient

"almost
that

always".
we shall larger Markov the ml;

2.572 always than

passing that

i. e.

H is. never hand,

any

of is of

On the ml, say,

other the

when the of

process sequence and for difference 2.573 that of

order

convergence

H will

be rapid than H. about and, the of in

as soon 2m1 there limiting function

as m exceeds will value 11 is be little H.

m greater between

The form a function used

the

essentially

statistical

thermodynaxaica

47'

and associated 11, also para.

with 2.9 of is in

the below). the

concept

of

entropy. alternatively

(See

ref. calls name when

Shannon

H the such

"entropy as this

message". to the

A less be preferred noisy

committal since it is

general

wd later able to

come to give the

consider word

channel a more

decir-

"information" usage,

restrictdd to equate

connotation. the 2.574 for rate a source in two.

Technical

however,

tends

The above. definition also serves

of

information of

rate

definition a as since, alter

information see,

a noiseless does not

channel as a rule

as we shall the H, when measure. for the

"coding" 2.575 of is

The maximum value symbols, successive symbols additivity will occur

of

a given ensemble

set

fundamental such that

symbols

are equally measure

independent probable. for

and the In

fundamental view of the (as

everywhere of the of n

independent all the the rate H. are

choices here C of

a consequence and His using Shannon logt the

para.

2.1+7)

equal, the

as above.

Thus

capacity that symbols. and

channel

maximum

information

can be achieved 2.576 receivor In order of to

the

permitted of as the

fundamental transmitter

speaks

a communication perform operate

"information coding

transducers". operation, only, such but

an effective not

a device

must

on single

symbols

'48

on groups transducer symbols received rust state" of that from turn

of

symbols; have

this

means of

that internal

a non-trivial stare group we "finitenumber such moves in where is

oust

some kind until

"remembered" be can and ready this for store

a complete To be practical, we speak with the of

coding. finite;

assume

transducers, states input state the not

i. e.

transducers of the

a finite and it state input possible

internal tui0 one

(conditions govern

store), which of

symbols to

way in changes of

another

and the (The set

govern need

output. be the and they

possible set of

symbols output operation It is

same as the need not

symbols, of

be one-for-one). or

The irreversible.

a transducer if with there its

may be roversibl exists ouput will another restore channel, and the

reversible when In fed

transducer the original the receiver trans-

which input. mitter should

a noiseless clearly inverse. One of

communication be reversible

should be its

the that

concequoncec a finite-state of cit.

of

the

definition

of

information increase

rate the 7,

is

transducer (This This entropy a reversible being is so,

cannot

entropy loc.

a message. P. 27). the

Shannon's a reversible The measure coding

theorem

transducer H is thus

must invariant

leave

unaltered.

under

operation.

2.577

The most important

fact

about

transmitter

opoygtion; Shannon's Thia_rstates-r. of-. -capacity,: capable. only, if

is, thowever. 'contained' "coding that, -theorem" ' given'a *(-for-

in

what

is

known

as

the'-noisoless of rate

,ch'aniiel)..

4source'

ii `and a 'channel . tranedticer' if and a'" suitable of

C; ' there

exists-'a :

finite-state for

'of, acting H- is terJIt.. at less

s transmitter -than:

the hystem ". Thus realise with

C. td or equal"*; possible channel to

transmit

is; always the full

a rate

transmission 2.578 value

capacity. H and its' maximum ,

1 is:. called is

The tdifference the"L redundancy' atLL itaIfullx',

between of ' the

message .1'. When the that it messago ' i`s' pssible redUndanoy which

a: channel

working.;

capacity means -

in : tt Chas zero +reddndancy. `toa. design from given;: original 2.579

. ,.P..

'. This

a tranuzitter"'iwhich

will,

elimihtethe =and, "a-recciver11 restore it

froIa message:; a .!

a--givens "source, =measaSe "will

the;: non-redundant, form. (,, : A particular

c. r-w rd

to -its,

case
.yw,

of
w

interest,
ix J-...

tw"3

to

which as

we shall
rim

return, If

is

that

of

a natural
.. r..

language suggested process,

such

English.

English as the to estimates information a,space), that.

messages output what put

can of their at (for 2.4 text

(as

above) then, is. puts 26 It

be represented it is pertinent of

a Markov average about

ask

redundancy 50%. This of symbol.

A number the

this

average and means

rate at about

an alphabet bits could per

letters '--also,

any. English

be 'reversibly-

'-

,.

_'

S'

coded

into

about

half A special

its

length. investigation ch. 5 below) conducted suggests by that i. o. it X30/

Shannon the

(see

ref. of even

40 and EnGlioh 90`/0. could If

redundancy

higher, be rauch may this were the case

and-perhaps mean that tenth of

would one-

a text its

be reversibly

coded

into

length.

2.58 bad. Although in

Redundancy it is almost

in

a message an axiom a channel of

is

not

necessarily

cottmunication should be used

enSineering

practice

that

to its

full

capacity, consider

our view "noise".

general of

of the

case alters theory

when this

we come to
leads of the to

In Shannon's
rx capacity.. of the

a new and more rate fundamental (and

definition This is

information second

channel

dfinition

theory. by a channel of always of distortion and the case a

The distortion on the simple messages recoding; to passing for through example

introduced it symbol This at might Sl

consist might sort

be changed however need where 2.581 is as follows: not the

S2 and vice always

versa.

could further

be corrected us. place of input the of More in

the

receiver, is

concern takes

important a random

racoding

manner. that is in,

Our model At the

noisy the

channel, channel

a sequence

C `

s
mado up of sequonce a symbol given the symbols the S1, Sn; .. " T1, at ... the Tn,. output Each a time

made up of is transmitted generally) (t) 0 ...

symbols one is a cet of 0

received; probabilities (t) ...

and we are

(most P(E

im

(t+rn)Fk

Fkm(t+n))

for
and

the probability
simultaneous that the

of transmission
reception ensemble of of

See. sequence of jp
Tk of ... Qm joint Tk . We events process. set up an

8 im

sequence

assume EjF.

sequences

can be regarded In the

as generated noiseless between the

by a Markov case we can

exact

correopondence symbols is a k'

transmitted basis.

and received That is, for

fundamental every j there

on a one-to-one such that 1 =0

P(Fk(t),

E(t))

for

k=

k'

otherwise,

and for

every
P(Ej(t),

k there
Fk(t))

is

a j'

such that
1 for j= j'

=0 In the noisy channel

otherwise. it will not always

be possible
following to which

to set
definition a channel For

up a correspondence
and theorems may be used simplicity here to the for

at all.
the

The
extent

indicate

transmission.

2.582 confine symbols, ourselves of both

of

exposition where

we shall successive sequences is are

case

transmitted each

and received received

independent,.

and where

symbol

independent

of by

all

transmitted vice a

symbols versa).

except This

the

current that

one

(and

implication only

means (The

we need

consider to

L3hntantt. is the completely definition that

generalisation

dependent the of Hm). set

sequences lines Thus of of

ctraihtforward, of II above as the

following limit simply

we aczume

we need

be given

the

probabilities ii of 1, ... n; k=1, of ... Si ) n' and

P(E jFk) of events consisting reception Fron obtain sets of

the of

transmission

cimultaneouo

Tk. set of for probabilities the transmitted we may symbols

this

probabilities

alone,

and for P(ES)

the

rcievod
k 5P

syibolc P(EjFk)
(E. Fk) .

alone.

Thus

and

P (Fk)

2.583
is defined R(E1, F)

now the

information

rate

in

the

channel
P(

77 jkP

P(EjFk) (bits

k * 10u2 s P(E )P(FD) per symbol).

This, function
(i) "noiseless it entropy is equal P(E4) in to

has the
It the the

following
is

properties:
the

channel In this is case H(E) equal to, the

a maximum when indicated at which the is above.

sense entropy P(E1),

transmitter, in turn

logt

HH(P) at

the

receiver.

--- ----

._--

S3

(ii)) when P(E Fk) symbols

It

is

a minimum for are

and all

equal j

to

zero i. e.

P(Fj)P(Fk) received

and k, of

when the transmitted.

independent

those

(iii)

If

wo write

"EF"

for

the

ensemble

of

joint

events

EjF.,

and consequently

If (EF)
for the . This "joint R(, F) entropy,

Z Z.

27

y
we find H(E) +

P(E3Fk) 1062 P(rjpk)

I1(F) H(EF).

may be written

R(E, F)
where Hi(E) xH(EF) transmitted be considered destroyed message

if (E) II(F) relative of the

iiF(r)
is the "entropy of the received", of and may information

to :the the

as a maasure noise in

amount

by the In

channel. we can define H(E) relative to the

a similar

manner -

HE(F) as the "entropy of the

H(EF)

receivedimessage

transmitted".
2.534 greater entropy be careful entropies the other than of Notice H(E), the not signal. to treat that i. e. it the This any is possible to have H(F) the

channel illustrates and every

may increase that statement rates. how is it

we must about On that

as a statenent hand, wo are

about prompted

information to aale:

54

the

entropy

can be increased?

Rosghly

speaking,

we sae

that

entropy
(if

16 somewhat akin
it occurs) of is the

to

"randomness";
the by random the

and the
element in tho

increase

due to nessaGe

superimposed channol. 2.5135 channel has two

on that

noise

It alternatively inputs,

is

possible

to

represent

a noisy transducer which

as a finite-state the transmitter wo can source call

one from source the

and one from a noife source. in the

an independent Mathematically, exactly only the difference

which noise

may be specified source it is specified;

way an information is that

we consider as the source less

as interference to receive.

and the If of the

transmitted entropy find + of that H(IT). however,

signal the noise in

one we want is than for H(N), or

we shall to the rate terms of in

course H(E) channel, in

H(F) The can

equal

Sum the H(N)

formula only

information in

be rephrased

certain

special

cases.

2.586 ation rate in

In the

terms

of

the of

above noise,

definition Shannon for

of proves the noise

informan

presence of the

acmdmSman analogue case. present faultless There is

coding

theorem when

noiseless is

one difference; in general It

random get

we cannot

expect

to

completely ----

transmission.

can be shown however

and this

is

still

remarkable

--

that

by appropriate

SS

coding

the

frequency

of

errors

can

be

made

as

small

as

desired, served

provided as regards

only rate

that of

suitable trancmioeion. 0 of of

conditions

are

ob-

The capacity defined sources the as the capable maximum of case ac before, is no simple value

a noisy R, taken to the the

channel over the all

is possible In

serving this

as inputs reduces to in C.

channel.

noiseless

same simple noiseless case

definition there 2.587 effect rate that

but formula

except for

The General it is possible to

coding to attain

theorem

is

th

the at with a source equal a

transmission capacity, Given than or

as close

as desired small

the

channel of errors. less

an arbitrarily of to of of rate Ii, then exist its loss

frequency and only if

if

11 is

C there sending error

a transmitter output over the

and a receiver channel specified with

capable a frequency

than This

any

positive has

value. corollaries. the This transmitted value

2.588 In the noisy n bits that if must

theorem the

some interesting C is noiseless equal to less than case. C the

channel per

capacity of than the or

(logt means

symbol) leas

H is contain

sequence

a certain is In necessary general is

redundancy.

Exactly for that (though

how much redundancy any the not given function all kinds channel. of of

can be calculated then, to we may say noise

redundancy redundancy

counteract equally

are

efficacious).

To some extent the in consideration an English the

this of text

general a natural

principle language; by

is

obvious simple

from misprints precisely obvious. to

can be corrected (the structure)

inspection makes them

because In

redundancy text

an unrodundant of reading.

each misprint

would

give

rise

a change 2.589 of in R(E,

A further F) as a suitable channel is

justification measure provided by of

of

the

choice rate consid-

information following p. 37). under

a noisy

the cit.

eration us

(Shannon's that, in there

theorem addition is

109 loc. to the

Le

assume

channel

consideration, what the point is errors over

an "observer" is

who can

see both

transmitted due)to

and what noise

received, data enable

and who notes to the receiving

and transmits channel" to

"correction a

the

receiver

to

correct

the

errors.
The information lost by HE). is over case: small the the it fraction channel has amount correction is possible of at the due to noise, according

to

the

above

theory, that

is

given this

Thus of

we should information channel. to errors this correct if

reasonably that This all and needs is but only

expect to

be supplied to be the

found

an arbitrarily if the correction

least

capacity.

x. 56 that to are signals

We shall necessary which in vary

now deal the over above

briefly theory

with when it range theory from

the is of of that

changes applied values.

a continuous extent, the

To a large uously discrete are, the varying case signals by means two basic the

continof the There In of

can be obtained of a simple

limit-process. involved. a finite of number

however, first

limit--processes of that

place, symbols

assumption way to case of

elementary

gives in the

a continuum and most other

(one-dimensional applications, as for In, the of of example second but in

telephony two to

possibly applications the of of is

with

or more (ref. the

dimensions, 16)).

optics of second

place, number

assumption per

transmission way to signal. two generthat

a finite the

symbols

gives

transmission It

a continuously to treat start of

varying these

2.561 alisations the that about of analogue is, the

possible

separately. of the

We shall transmission consider once possible It would of and

by considering a single the only, form symbol; implies the value

we shall

first

what once values

theory of

transmission, whose

some quantity

(onea to such of to go on values,

dimensional) to consider

continuum. the to transmission the

be possible sequences of of

analLibgously

transmission proceed,

sequences however,

xxzk the case

symbols.

Wo shall

'

of is

a constantly a function of with

varying time). this

quantity In part the of

(i. later the

e.

a quantity chapters

which we shall in its

be concerned broadest brief

theory exposition

only is

aspects, and without

and the detail.

present

hence

2.62 probability This should given If is

Let density a function be in by a small

us

suppose

that p(x) the (

we are of

given

the x. x is

function such that

some quantity that its range

probability j+ Ax) of

region p(x4) AX.

now we imagine up into

the

entire small

range regions,

of

variation have

of

broken

stich with

we shall

a case 2.42. the }

corresponding (For simplicity has

a discrete the

ensemble, range of finite,

as in so

para. that

we assume a finite

ensemble

number

members).

Thus

suppose

there

are n such regions,

boundaries j) Q, x discrete fT (j are - 1, case ...

and that
xl, n). for ... xn;

their
the

respective
probabilities the formula

left-hand . are p (x for the

Applying the

we find Z P(x4)

entropy (P(x) Ax)"

63C 102

This

becomes

H=-Z
Now in of the intervals tcr11 becomes x the

P(x3)

ax

1092

-P(x J)

the

logt
size the

`x*

limit

as vie decrease their

and increase

number,

Cirst

----------

--

P(x) but than the second term expect infinite, continuous boeomen as the

10 52 p(x) infinite. number it is of

dx This is

no more

we should

"fundamental

symbols" for to

becomes of

but

an embarrassment it would appear

a theory make the

signals for if

since making the

measure

Ii useless signals,

distinctions signals to the differ inbration

between

different in

even of

considerably problem.

other

ways

relevance

The practical second term, and use simply

solution the first

is ono

to

ignore

tho

(ac

Ui' abovo).

This
giving

moans that
us

longer no can we
measure. only

regard
In in most

our formulae
caoon of

as

an absolute we are

interest, between

however, entropy formula

interested In rate

a difference we have seen

measures. for the

particular, in a noisy

that 2.583)

the

channel

(para.

can be written R(E, F) =g H(E) 1I (E)

and it
finite the II(E) the

can easily
in the limit,

be shown that
and will of the

this
have

rate
the

will

remain
if i. e. makes finite.

same value entropies, it seems,

finite and infinite

formulations I? F(E), capacity are

individual Noise,

adopted. of the

continuous

channel

2.63 caso of signals

Now let which

us turn vary

our

attention with

to

the We

continuouoly

time.

might taken in

note in

here itcr

that

the

"continuous" word. sense, i. e.

raust the

not

be

usual

mathematical f is "continuous

sense A contin-

which

one speaks of time

functions". specified

uous of its

function values values to

completely

by means since all

over

any

finite

time-interval, using e. g. Taylor's

other

derived be may postulate rule of out, probability

theorem; of signals plausible

and thus would

mathematical in ordinary theory

continuity contexts, in the time

be to

any

application

domain.

2.64 use of a body

Elootronic of mathoL1atics

ongineoring associated

has made considerable with Pourior

anily
very the

is.
general

A mathematical
conditions, of

function

of time,

c an. under

be reproeented elementary of (w t+ the c functions form

as made up by such as sine-

superposition i. e.

functions,

functions sin

of

different the of f,

amplitudes quantity the where The

and different known

values as the

of

W and freequalunderThus

lp , qucncy" to 2

4) being

"angular (It is

sine-function ff is the

concerned. "frequency" called

as normally the "phase".

stood),

quantity

C9 is

it

has become customary

of any given

to

speak of the

"frequency
is the

comvirtually

ponents" completely and phase

function; by giving

and a function the values (The of

specified of all such

amplitude and

components.

amplitude

phase

are usually

given

together

in

the

form

of a"single

6/

complex (which spread called ically of to

number).

The assemblage in over number, a range of the

of

frequency even

components typically is mathematprocess ts

may be infinite continuously the it is "complex obtained

and of the

frequencies) function; by the is

spectrum" from

function The process

"Fourier the

transformation". of a complex

analogo. its in radio

analysis pure

musical

sound

into

constituent communication stations or "wave

tones, in

and finds the in

application of "frequency

also

representation different

different bands"

as operationg bands". (This is

not of

the

place

for

detailed A short theory

mathematical account is given with in

treatment special rel. 46;

Fourier

transforms. to ref. inforration 42).

application see also

2.65
of functions of of

Special
time

importance
whose spectrum are

attaches
covers

to the
only to as

class

a finite "band-

range limited" which

frequencies. functions,

These and have

referred

one important of the theory

property of information by Shannon

makes

the

application

particularly (ref. 39, p.

straightforward. 53); -

As expressed

the

band

specified discrete

of time a function from 0 to W cycles per by giving its ordinates points spaced l/2V7

If

f(t) second at

is it

limited is

to completely of

a series apart.

seconds

..

_.:;. ________

-.;

_2

Thuo

in

the

case of to

of

band-limited

functions measures is

of

time

the we

generalisation do not per than Given points values all the need

information

immediate: of symbols not more

consider

"tinfinite an on the avera, of the

number e there function at

second",

because

are per

2W independent the values of

values the

second. set of

function apart all

a discrete deduce

spaced of the

1/2W seconds function at carried set this of

we could plint; function values (ref.

the

other by the

and hence can be regarded! in given. has

information when this In a useful

as given 2.66 introduced variable called has

discrete

context

Gabor Any parameter

13,19)

terminology.

independently of a eyetoia to its band in 77

information-carryinc a "logon". Thus second; we calculate by 2'1. of the

a function and to the

limited calculate averggo

2W logons

per rate

average

information per to logon a symbol of

information is analogous differbandsymbols

and multiply in

Thus symbols;

a "logon" and the

a sequence betveen and that'of in the

only of of are

ence limited is

principle functions whereas from

information discrete case "logon" in

theory sequences

that

discrete set is the

the of

symbols the theory

chosen

a finite functions

of range

band-limited of 2.67 real

a number

a continuous

numbers. Development of the appropriate information

63

measures parallel of

for with

this the

case. proceeds discrete case.

for

the

rest-,

closely ensembles

We introduce

band-limited functions

functions, for

represented'by of sample given

probability ordinates in para. functions Thus function of the an information source of the

density = functions 2.62 in if for

sequences For the

concerned. we use

reasons of these

above place p(xi, of

integration of the

density

summations xn) of is

discrete probability in

probabilities. density the output of

...

a sequence the by

n ordinates or

source, i given

entropy

information

rate

P(x1,

...

xn)

1062 p(xl,
dxl ...

.. 0 xn)
cxcn. (bits

.
par logon). inform-

The formulae ation 2.63 for this case, of rate in

for

relative

entropien, channel, follow

and hence the

for

a noisy

same lines. again The however, in communas a

The coding and requires the applications the is

theorem

may be proved comment. theory,

no further of of the

generality deserve particular ication, yardstick systems, in too, \that

mention: what and the for in the

concept

"coding"

includes in radio serve

known

as "modulation" measures of different their It with thus

information comparison in (see

modulation effectiveness should be mentioned, belong

particular noise

assessing ref. 25).

countering

many problems

associated

telegraphy
i

A --,

d-

to fact with

the

theory

of

continuously quantiitio2

varying (voltages are

signals, etc.

since ) associated

th.,

the

physical

telegraphicatrans2ission rather than discrete.

essentially

continuous

quantities

2.69 band-limited applications, frequency any long case run band all functions since is

The restriction matters more the or less little

of

the in

theory

to

engineering of in the approg-

sharp

limitation and because can dot; res wide in of

common practice; occurring to in practice

sicnals

bo represented as band-limited,

a :sufficient in a very

imFation

albeit

band.

2.7

The

theory-

of

rocent:

ion

2.71 no far the such the emphasis design of

In has

our

account on the

of

information "ghat such hardly

thoory is and less

been

question for of

idoal

communication ".

system

n purpose

(channel)? is: beat In the

A question

practical cystcra, recoivinG system

importance what is the

"Given that case "almost" the for

a non-idoal

comnunidation at the codod cannot pronent

can be achievocl of a non-optimally perfection,

end? ". perfection, and the

even best

be expected a probability nescages;

receiver the

can do is different the

distribution hence

possible theory. All

a new element

enters

that

need

be said

here,

however, to

can

be said

very

briefly,

and we confine

ourselves

genoral

observations.

2.72

-Lot

us

simplify

our

notation

al

little.

We shall

(j "E " the symbols - 1, ... n) for events use the transmission or of M symbols drawn from some of/sequences consisting
set, there which are we need just not further specify: of we this

fundamental suppose that

n pousible

sequences

length.
1, ... n')

Simil arly
for of

we shall

use the
of whether one.

symbols
the

"FJ"

(j
of

events the

consisting

reception the

sequences fundamental refer since fixed those large,

Came length, or to a different the is

from

same need to

sot

We do not of our

explicitly stationarity set of

time-behaviour

sequences, that upon, very infinite: some e. g.

assumed:

we may assume is N is fixen

successive at ta0.

time-instants The number the

sorting so that

assumed it is

we approximate make no explicit

case

where to

and we shall

reference

a limit-operation.

Let messages
reception

us assume that are known.

consists

the

statistics

of of

in
of

the
any

channel
signal

Nov; the
in

problem
kaotiving

simply

P(Ej, F1}
for all j and k, i. e. the relative for probability each possible of the case of possible
a received

transmitted
sd quence.

sequences

-i E
If statistics P(Fk, thoorera B3), of are tha Bayes, (as is in most the is likely) of the P(EJ), by channel 2(rk) the and known result viz. torn

given

simply

inversion

P(E , F)
To apply
of the

--------needs one .
the

this
transmitted

forrtaulra

to

know

the

distribution
of the

signals,

distribution

receivedi
. rclative

oi6nalo,
to each

and the
transmitted

distribution
signal.

of received.

signals

2.73 of probability on the concppt. subject

This theory. if

fox^.nu1a is However, it in

almost

a triviality is thrown

: ono now light tome of the

we view

information

We can the in rate a noisy Fk) the due direct dovired to H(E) H(E) -

expect (cf. The

to

rocoivo

infornation for the

at rate

the. formula of the

channel). at the

entropy is

distribution this represents

P(Ej, exactly lost most to

receiver

HF(F). has it lack been

But

information non-optimum possible

which coding; form our

irretrievably in the duiative

reprocents of information

the

information Entropy

H(E). figures ax I1(E) easurea i3 our in

2.731 conceived. but of (rof.

can

a eenae of

be

46 p.

49) Thus

not prior

"information" ignorance

"ignorance".

'!

_. 6 1
Il

of

what

willof

be transmitted, what has been

and. Hr(E) transmitted..

is

our

posterior that

ignorance

The fact

the

coding

is
value

leas
of

then
the

ideal

will
i. e. the

be reflected
the received

in

non-zero doco not

latter;

signal one. This

completely

determine

transmitted

in

turn

means that
the transmitted probabi; is 2 JO to itieo

all

that
signal (P(E for the of

can be said
is j, that Fly), vqrious this loot; Oet

at the
aro 21.,

receiver
such after transmitted

about and

there where

such

liticz

reception, of ynals is equal

known)

po: sib1o of

Tho entropy tho information unity

probabilities if one of would the the

only rest

probubil.

ware

and the

zero

entropy
2 74 .

be zero

and the
Thin

transmission
interpretation i: due to the it of

faultless.
of vloodaard of the coca (ref. Shannon, to some in indbviduel given is of 46), in that

non-optimum. involve: it is

codinE;, one departure

which

from

practice to the

neceecary not only but the

in

formulating values the

consider information rate of just

extent the

average also ensemble.

channel, of

information Thus since the

signals in

account

some degree. o incorredt,

we have

considered

the

entropy

of the
received

transmitted
signal to

signal

i; von some
as 11F(E). taken all over

indovidual Thia all is

be representable quantity, as over

strictly

speaking received

an average as well

porsible

signals

possible

trmmsmitted

ones.

A consistent,

though

to

some extent

,___

misleading .. Qasuro to of

account information V for any

could for Eifinal of

be given

by

introducing signals P. for

a euga1 Weacould the then

ind. ovidual of probability

1002

introduce distribution. signals speak lost 'do the 2.75 is not

a nevi= sort

relative of received

entropy z signal; about

of probabillitioc _iven a particular

tranonitto&I and hence the information does not

more in

reasonably

and directly case.

a particular although

Woodward, introduce cit. p.

however, with 13). the

this,

he does

some qualifications

ieasuxo

indocated'(loc. If what

is

required but si. a. al,

at

receiver

a probability of the the only fron v&lue illustrate

distribution transmitted thing the or that

some doinite alboit in an incorrect practice F, ) its gives is from. not ref. it. in 46 a device, general

indication one, is to then

can be done x'(13,

calculate probablo to

distribution

most

mod.o . that

"Woodward however ouch i. o. a calculation non-reversible; it is (See into

examples actually the

entropy-destroying, of most probable

sequence to

readings

possible p. 60).

reconstruct if the

a distribution. receiver has. built

Thus

which .. signal,

automatically the

selects ac=t

the mozit probable rate is reduced

transmitted below the

that

information

maximuln "guesswork

obtainable. destroys

:aloodward information".

summarises;

by saying

The ideal

receiver

may in

any ease perform

i,.

. -....

,..

__

r_

____..

' .;
it destroy

.=,69

an irreversible

operation,

since

must

what

it

can of the
operation,

received
destroy the tute

"noise".
any more

But
than and of

it
it

must not,
need of

for
the i

ideal

information its output-

from must

transmitter, the fog.

on this lity,

criterion distribution.

a probab

Even this of the

trivial 2.76 the above analysis and there of a iian of

is not

possible : tabistics,

without except
compute from

knowledge in the

transmitter
ecn3e that

and channel
a receiver The theory hass somotines is at clearly . the

mi 01.t Which

Simply emerges called.

P(FD).

boon

"measurement between the.

theory", situation channel process. misleading. process, cone and idea

a resemblance end of measurenents however,

receiving,

a communication on a natural

and that

a man making can, of

The re3eJablance In the case apply to

be oxtromoly of a natural one has "transmittr"

measurenent theory to

ono cannot what

the apply

unless the words

meaning

"channel"; of

and unless of the

one has "channel

some independent statistics". (loc. to cit. radar, here. of think the of The )-and. has Thus position the and

source

knowledge of the

application othors always if of true the radar to

theory of

by Woodward

processes

measurement,

e. g.

depended. is used

on some special to obtain it tkris

assumptions an estimate possible to

some distant value ectiniate of

object, the ilstanca

as a'transmitted 6igual"

signal" perturbed

obtained

as a "received

7o

bZynoise

in

the

"channel"s

but

this

model

only

sac=

-reasonable
in general

because
performed and in

the measurement
more any case accurataly it is

of distance
in atill other

can ba
way. -, than uncertain

by radar;

as a rule

what meaning ability

of are

to attach object's
theory-,

to the

concopt

o. apriori other
for only

probfeatures

of the

distance.
that of or

Certqin
"coc1ing" present the this

cormunicotion either

inotanco, in a drastically of

cozxp1etoly form. is to For

absent those

modified roceotion"

reasons for

name "theory theory.

bo preferred,

The nhVmica1 2.81 briell field of account of 6f

theory We conclude this

section (rot. 5,6)

with in

a the

work

by Brillouin More to

thormodynamics. theory notably theory of

extensive has 18); Signals,

applications been but made by this some

co ;nunication writers, the

physics (. ef.

other involves our

Gabor

largely outside

continuous

and is

scope. trillouin has of put forvrard a nuiibor of

considerations an akin He coins to the (and

in

favour Lterely

regarding analogous for this

physical with) "nogativo quantity.

entropy

not

information. entropy",

word

"nogontropy" with

and querts

"information"

(That positive entropy is

Shannon equitoc not inconsistent

information with this if

with we

 7(' :'

remember

that

the

"information

content"

of

a message

can also
to it.

be conceived
Sac ref. '11).

as the

"prior

ignorance"

with

respect

2.82 the case of "Maxwell'

Brillouin a demon",
in charge

treats

in

some detail to lla: cvzell' s

two and

who according
of a door

paradox chambers operate fast of that the and the

can be placed of the gas, door

between

observe in such

moleculos

approaching; discriminate the " (ref

him,,

a way as to thus

between

slo-7r nolcules, It is the

decreasing by Szilard impossibility, of is entropy largely

entropy . 40a) and that

system. "demon" of

was proved a physical increase account in terms of

the

principle

is

thus

not

violated. of Szilard's

rillouin's proof

a rofgrbishing of the theory

a version

of

information.
In terms : horn can of firstly secure of is a simplified that, the model entropyto that that tho in of such a

system that of

Brillouin the "dc. on" (in the

decrease the amount

proportional choices)

information about this

terms

binary

ho can order

obtain to of

molecules;

and secondly he must a similar increase amount.

obtain tho

information by at least

ontropy

system

2.821
proof
4`

Lot
first. We suppose in order

us treat
that to the "roe"

the

second part

of the his
This

demon raust the

supply

own illumination

molecules.

lZ

involves per since orders

the

exec

dituro where

of TSic

at the has

least

one quantum of of the very Thus 3oltzmann's the

hd - kT chamber,

observation, the ambient

temperature fluctuations must at

radiation

this least the

and the

roflocrod

signal

be as great entropy conctant. is

as tho increased,

background by at

fluctuations. least k, i. e.

The by "usinr, up" let

ontropy of

decrease information that there

that is is

can be obtained calculated a clip-tuber of of as gas

one bit us

fojlowc: which door simplify chamber information molecule is slid

suppose into

can be divided (which the

two halves frictionless).

by means

a sliding

may be assumed example of our respect place with the let

To further the as in the of If say, in the door loft-chamber the;

us ascure

that

consists in with into

a single possession to the the

molecule, is the

and that location door. in,

sliding molecule of could with of the

the the the

hand has

eompartmant, boon halved; to its

"volume" gas

gas

and the original

be isothermally a gain of of useful system. the

expanded work This

volume decrease

and a consequent decrease of

entropy

the

entropy

can be calculated

from

gas law 2.83

of

(per kT pV =

molecule) Brillauin

as

k loge that

2. if one bit
2

concludes

information

is

taken all

as equivalent that is needed

units

o ontron; ,

to (minus) k log, . to save physics

,73

fron
of

Lw re11 Is paradox
enbrop7 to a 1av Thero in a cystew, Gas; of

is

alteration
e of

of the
entrop

Jaw of increase

increae.

r ninu3 of of the

information. entropy

may,

ho cayo, the only

be a decrease of the

as in but

case when

cNanpie io

win; lo--=olecu1e

information

being

uce. up in
in. crcazO oxponsc of or

compensa icn.
in: roriiiatian a proportional

Sinila:
about

1y there

may be an
but only at the

a system; e of

incrca:

entrap y.

2.84 I, thatin is the physics. no precise concept It of must

Thus

there

are

ground. e for may be found however,

thinking useful that there and context

"information" be emphasisod, between is

connection as the latter channel.

physical interpreted is

entropy in tho

"information" of the which channel probable or vice

iunication a comm. ensembles are

This are quite for forms

basically

because symbols

considered probable, physical sense Thus in

different; in far

equally

example, which that are of

a communication from equally

may take in any

approaching we shall follows: of the

thormodynamico; with rathor the

versa. theory the

be little our

concerned concern of is

priysical to

what

elucidato

connection

theory

xirtl=
to

communication
in the

with
next

a logical
chapter.

theory

of

information

be outlined

2. r3

In

its

more

direct

applications,

the

tccb

i a- theory

of infoxLlation

has been illuminating

and useful that progress

in

a remarkable in this

degree; has to design

and there stopped. electronic criteria

is

no sign

direction

The customs have yielded

applications system grettly

New

communication cohere no.ne into

such to the as

Arc o before
new typal of

and have
s"3:3tcic. and

stimulRted
systems of

research
modulation deal

iepulso-code"

"delta"

have

o-'rod o groat

tr coSeveral or n.jrtial
on. the

methods, elimination
throihahold

of of
of

roT,: Aon con~np. of

en.

(i.

e.

elimination transmicsion

redkindancy)
becoming

speech

;; cep

noer?. nG possibilities

for

soma types

of

channel.

(gor

;; eneral

re

erencos

to

technical of

work

in

these

fields theory"

sec e. g,. the symposiuuz,

theory (ref.

proceedings

the, "Coiunication

London 1953).
quite

Coiunication directly of to optical of spatial of work time has time systems the function also). been done

can be applied 16) is in which

instead a twwobe

a function

` nossace" (which

tyypically colarso

diuensional a function amount where practical system. of the

may of

A considerable in to connection tiith telvision,

redundancy savings

appears could

be sODle '95'% and gro by a suitable coding

be effected

It information in tens of content the

has of

boon

suggested

(.ref.

33) . that is

the

simple to

lino-drawings

moasurable

signals

a eenvo-operated

drawing

nechanicm.

In physiology,

estimates

have been made of

the with

capacities prosthetic in and it study

of

sense-organs, Studies fibres to brain,

partly

in

connection mechanism information in connection of

devices. nerve appears of the

on the used

transmission concept, with the

have be of

the

importance "storage

whose

capacity"

is

a subject

of controversy.
Coding techniques computation, for the assume where inversion importance methods of In involving

the

case

of

automatic (e. g.

random are in

sequences use.

matrices)

76

LOGICAL MEASURES ALLIED

TO THE INFORMATION

CONCEPT

Having

seen

how the

concept

of

"entropy"

operates systems
to

as a measure and related

general

of

information

in

communication we now turn

of "information" logic.

engineering
examination in systems of

problems,
measures of abstract

a more

and related

concepts

In a logically
theory they these represent considerations some justification of of its the technical

ordered
should

account

of information
since

come first, of the

assumptions and indicate

and definitions the limits

theory,

applicability. Eiraft consider generally what

We shall

can be meant by the

statement logical of or

"logical

content"
Our aim is

of a sentence,
to major elucidate definitions rate of the

proposition. of theory, the first that

standing

of of

the

information

the

information

a source. We shall measures associated with then go on to consider certain

relations

between

sentences,

--

--

------_

. ya_

'i , .6.

77

;ti .'

statements "correlation", will assist

etc.

and associated "confirmation"

with and the

such like. the that of

concepts These second the

as measures

in of

placing information channel.

in

perspective theory,

major

definition rate in

information

a noisy

3.1

Logical

content

3.11 has a measurable

obvious one;

The idea "content"

it is

that in

a logical

proposition and
should

is
not

many ways a trivial

what logic measure of

though

obvious modern

be employed.

The beginnings

in

a theory

of

the

"content"

of propositions
who says "If (ref. 45,

might
para. followa than the

be traced
5.14): from -

to

Wittgenstein,

a proposition says latter. more If

then less from

the than

latter the

another, former, the former

p then "

A tautology nothing.

from q and q p follows they are one and the same proposition. follows from all it propositions: says

Wittgenstein any numerical although definition later of at

however

does not

suggest says". theory

"how much a proposition a connection with the

he hints

of probability. By the any numerical thing, measure term "logical vague says". content" but we ahall mean simple

of this

essentially

"how much a proposition

. 78

3.12

Before

we can

set

up a measure

of

quantity

we must have a scale; between. two sorts

and "strong" We say that

and it

is

convenient which we might

to distinguish
call "weak"

of scale,

respectively. there is a, strong scale . of

objects
if the given

or entities
any two or is relation that or its of

of a certain
them it is

sat
possible

in

a certain
to are say

respect
which is

"higher" It the such

"lower" quite "is although inverse all.

or whether common, howel'or,

they to of" or

equally relations class bears set-, say it that of the

"high". (e. g. men) relation does there not is

find the

an ancestor everyobject to In

in

entity of the

some others this case

do so to a weak It

we shall

scale. is only when there of is a strong ( in and set scale the up

that simple

we can sense

speak of

unequivocally

a measure measure)

a one-dimensional

a metric. In (ref. 29, p. 34) "I there are of this on the maintain, of connection subject then, of we may quote "probability": in what follows, between Keynes that the

some pairs

probabilities

which no comparison is of magnitude that possible; we can say, nevertheless, of some that the one is pairs of relations of probability less, it greater is not and the other although to measure the difference possible between them; and that meaning type special a very of to a numerical can be given in case ..... comparison a of

members

79

magnitude".

Keynes is that
but

saying, only

in

terminology, our of probability,

cases.

there
that

is
there

in`general

a weak scale
in

may be a strong'-scale

special

The same point 32-40). covering

3.13. it is necessary There

has been discussed a general theory of ref.

by Popper "partial 4).

scale of into the

(ref. ordering"

37 seens.

(For

such cases

see also

To convert to will enlarge.

a weak the

a strong ordering ways in

one

scope

relation. this of

usually

be different wish on the it with to

which

can be done. "seniority" or we could

Thus

we might

base

a measure relation. a number status, simplest to

"precedence" supplement based

ancestral any sex, of

To do this of additional

relations

on age,

marital the of in X"

income, shift mean of

profession all would born world

or what-you-will. be to before would redefine X", which

Possibly "ancestor at least

"anyone

a nonstrong

relativistic scale 3.14

make all

depend

on the

of. time-order. Now from says" Wittgenstein's we have less been only than account one hint q if "how an from been'

of of

much a proposition ordering q. relation:

p says might to have

p follows to have

Wittgenstein here sense

supposed of'Iaterial

referring in the

a relationship

implication" but this will not

of'Whitehead

and Russell;

do,
-----

in virtue
-- - -- ------

of its

definition

which

makes the

truth

or

falseho"od

of

a proposition

expressing

a material

implication
of the

statement

dependent
which

on the
the

truth

or falsehood
is

statements

between

relationship

asserted, In place that

(or

rather

than

merely

on their

logical

form., noed true

of the notion deducibility.:

false) being

of material two sentences

materially

implicationiwe'. can be both

imply

'of
both

and hence interdeducible.

one another is governed

without

Dedticibility

by the

logical

form
in of the

of the
case

statements
of a formal for the

(sentences)
system system; in

concerned,
terms of

and definable explicit rules

inference

by definition

"elementary two no relationship.

sentences"

can stand

in

a deducibility

When we require this "D (q; concept p)" for

"D(q;

a symbolic Popper, from

for ...

notation ref.

for

we shall "q is
pl,

(following deducible
... pl, pn)"; p2,

37h) Write,

D" 1
"g pn" " with

or more
is deducible is "q" to etc. be

generally from the

p21 of

conjunction

This "p",

understood

as a syntactical

symbol,

as variable 3.15 of "logical

syntactical Now it content" is of

names. our intention sentences of to base a measure a logical them. relation language The of the

on the relation right

deducibility

relationships is

between

of deducibility sort, but in it gives it is

a transitive

us only not in

a weak scale general true

of the of two

sentences,

that

TTTT Si
sentences p, q that one, is deducible from the that other. any

We. can

reasonably

postulate

measure

of logical

content

must be consistent Thst'is, from

to that

with it

the

deducibility down that

p is greater If. D(q;

relationships. if q is
than p)

we c'an'lay the

deducible
or then equal C(p)

p then
of C(q). q.

conttent

of ,
-

Or in'symbols:

It

followz

that i. e.
D(q; p)

interdeducible

sentences

have equal

content,
If,

and

D(p; refer

q) to

then these

C(p)

"

C(q). as

We shall

requirements

the

"deducibility For

criteria". some purposes of the then this is we should of these p) also like to i. e.

require ,.If but

the

converse find

second D(qi not

criteria, q),

C(p)

C(q) that

and D(p;

we shall

always the

practicable. deducibility

3.151
relation in

How can we supplement

order It to obtain a strong

scale? that we might But similarly the equal, relation apart put this

might

be sup&sed p and q are if -

C(p) would

= C(q) imply

whenever C(p)

unrelated.

a C(r) C(q)

p and r wore C(r), all

unrelated, of q to r.

and hence We should

whateuor contents

end with

from

those

of tautologies

and contradictions.

8t

3.2

Logical

probability

3.21 ,

The first

explicit

measure

of

logical

content-was p. in 67). terms

put Popper of
,

forward

by Popper

in

1935 (ref, should

37, bo measured

proposedi. that

"content"

"how much a proposition

in the terms of the logical concerned.

de nies",
probability This

and interof proposal the has

preted negation received deal

this

of

statement of

a measure it in

general

acceptance, But first it

and we shall is necessary

with

some detail.

to

flay something

about This

the notion

of

"logical

probability".

concept
of such of 37g)

has appeared
"truth

in many versions.
in a the Popper can be definition

Broaely, logical

it

is

a measure i. e. a family (ref.

possibility" as permits universes". the measure

language, of

a language "possible that to

enumeration haw himself introduced of a logical

shown without

recourse

separate this

and prior provides

language,

and that

a natural

and simple
following of the

basis
account,

for

the

theory

of probability.
a simple terminology omitting

In the
version and most

however, follows (ref. of task his in

we present mainly the

theory of

which

methods of the

Carnap

8,9,9a), symbolic hand.

though logic (For not other

features to the

immediately accounts see

relevant ref.

22,29,37d,

37f)-

The simplest put forward by Carnap is

of the

relevant the

concepts "L-range"

what he calls

13

of

a sentence;

and this

concept

is

most

simply

defined

for

a sentential

calculus
which

with

a finite
call

number n of
pl, p2, """ pn"

elementary

sentences,

we shall

3.211 for-, continual polato

symbols

a note

dispose of the necessity order-to it is well to intercomments on notation, Our here on the conventions adopted. In

language with are not symbols of the logical them; but rather names for as such they which we deal, the language to be belonging in which considered as can (metalanguage, language), English, viz. syntax wo write in so far except quote marks etc. are unnecessary for by ordinary usage for other as they would. be called and

reasons.
The logical language, of logical however, is form of an expression by the evinced of placing the

constants, to it; which we use to refer for-logical constants symbols selves. Symbols names, i. e. in question. formulations suffixes, of these of in with

sometimes brackets

in the expression etc. thus we might say that serve as names for them-

integer

suffixes of for with or "k"

names of particular Variable names are represented the letters

sentences as required by symbols "i", "j", in

are the

constant language

syntactical letters as , or one the range explicitly

usually with

variability the context.

an integer suffix being indicated

addition, implicitly or

As variable of the language e. g. suffixes,

names

for

complete symbols

expressions

we on occasion Ilp", rtq"

use

without

`'..

3.22 are taken bility

The elementary as independent hold

The

sentences

pl,

p2,

"""

pn

in

the

sense that

no deduci-

relations

between

them.
of a sentence any can be defined, a table F) for

'L-range"

in

terms

of

its

truth-table, showing its

For

sentence (T or

can be prepared

truth-value

every

possible

combination
elementary

of truth-values
and its cases in

of its
'L-range which it may has

constituent be defined the value

sentences; of

as the T. Bor

proportion

example,

in

the

case

of

the

sentence

Pi v p2 . pl F F F F T T T T Five out of

p3 p2, F P T T F F T T eight

we have the p3. F T' F T F T F T entries is 36. in pi

table: v p2 . T' T F T F T F T the

p3

last

column

are

T,

the ' L-range whence

The value 0 (contradictions) cases, sentence

3.23

of the

L-range

can, vary In

from

to 1-(tautologies). it is unrelated A restatement

An elementary

intermediate of the clearer.

however,

to

the will

truth

concerned.

make this
and its

sentence

negation

together

form

a basic

pair.

A conjunction

' "-..

chain

containing

just

one member

of

oach

basic

pair

is

a state-description. as it were, a possible sentence

A state-description universe. there are For

specitiesi a language with

n elementary

2n Possible

universes.

A theorem states that in

about

the

sentential sentence equal

toi

calculus (--contradictory

any non-contradictory any case have Z-range

aas terms or is equivalent

sentences
be expressed chain the

to

zero --)

can

a disjunction This is called

whose dinjunctive

are normal

state-descriptions. form of the sontnco. the

Assuming is equal form possible 4 of or state-

no repetition to of the the number sentence 2n. It

of of

equivalent terms in

terms, the

L-range normal the total

disjunctive by the

concerned, is thus equal with

divided to the

number

proportion,

desci'iptions might the say, sentence the

consistent proportion

sentence;

as we in which

of

possible

universes

would

be true..

Carnap "L-range" for the

originally

intended under It suitable is

the

notion

of

"explication" an as concept of probability. of probability possible concerned calculus,

definition 5.15).

circumstances related favourable other of the with

clearly

to the to

definition

as "ratio of Laplace early

of the and the

the (equally)

cases" in the

mathematicians probability
Wittgenstein's (ref. 45 para.

development identical
truth-grounds"

and is

virtually

as "ratioof

_9

91

Carnap for in an exact a logical only numerical language slightly is only not

himself definition auch from of

has

laid of

down a proposal logical probability which 9', appendix). for our purpose of the s$ntences Thus red", y

as that the

considered (ref.

differs

L-range importance there are such

The difference and concerns of

great where

cases

"families" as e. g.

sentences predicating if there

a similar various

structure, colours of

a particular of will the

object. forms have "A is L-rang

are

elementary and "A is

sentences blue" each

"A is (like

yellow" any other

e; ementry

sentence);

but

provision

must and can be made, for these sentences are

the

sake of consistency in this; for combination, or at least the members of three

would

when for allotting

considered less than

them probabilities the sum of is

if

such that, such a sentences

be

such probabilities not

they

family
quoted, allotted

greater
are

than

unity.

Thus the
exhaustive,

considered of 1/3

probabilities

each.

We shall ation logical

3.24 which terms in however of rolative leads

however

ignore

this

minor

complic-

for

the most part probability is

in what follows, the

and assume that

same as L-range.
definition same calculus of occurrence of

A rival to the

of

probability --of is that

--in

frequency sequence of

an event class.

an empirical

events

a certain

As usually

stated

this

definition

applies

to

the

probabil-

ities easily

of

events

(of to to

a given the

type);

but

it

could their penny

equally

be applied e. g.

sentences euch

expressing as "The

occurrence,

a sentence

falle

heads".
,.

The two definitions concepts

[: `
t:

of probability radically

define distinct;
of prob-

which
many

are under
most

aom0 conditions
interesting

' yot

in

of the

applications

ability For

theory

there

ie. a close

relation

between only in
5)

them.

the moment wo shall

be interested
tatet

logical
discuss concept

probability; practical must

but, we shall casee in into which

(chapter

the'! celtivo-frequency" with it.

be brought

relation

3.25 whereby to any measure

There based

is

a mathematical

technique nay be reduced

on probability language.

based one

on a symbolic

This

consists as if

in its

treating
probability in terms

a sentence
were of

of a given
its

probability
form

due to

logical elementary

when expressed

(hypothetical)

sentences.

To obtain
logical any foram we first number of

an expression

of the

requisite
to

express figures;

a as a binary i. e. in the

dedimal, form

desired

O. blb2

...

bN

where

each bj

is

0 or 1. last
or

Without

loss

of generality
we assume only

we may assume the

that is not zero

figure
unity,

to be 1 (where
cases

corresponding respectively).

with

contradictory

and tautological

formulae

Now taki:

6g N krpothetical

elementary

i:
sentences Pl where If the pl, p2, """ (P3 are to pN C we write ..... out the P. U) expression ..... )))

88

CP2 spaces

(PIT-1

be filled the first

by logical apace, second

connectives. and if space, will to the 1 we and

b1 is "v".

0 we write Similarly bN-l. (in

". " in for Then

write r

b2 and the the sentence

so on up to have L-range

so formed equal

the

sentential

calculus)

probability

For
we shall get

example

if

we take

aa

21/32

0.10101_

pl whose L-range
Proof. expression the for

(p2 v

(p3 v (p4'* .

P5))) to have the same value.

of the with

may be calculated
Let p which sentence is p, "p(j)"

denote or

the

part

conjoined _1; i. e.

disjoined expression

elementary

the

Pi
with with figures prove (1) its the spaces

(Pi+1
filled

...
with Lot

..

(PIT-1
logical "b(j)"

) PN)..
connectives denote of the a. in last accordance N-J+l

prescription. of the that binary the

decimal

expansion P(p(3))

We shall holds for

relation

= 0. b()

j-N, provided

(2) and it holds

(1) P(P (N) )

that for

this j-k1.

relation

holds

for

j-

k-i

We have P(PN) --

P (N) 14 -

a pN,, 0.1

and

bN

1.

Thus

O. b(N).

(2)
fixed or or k greater Pk-1 O. lb V P(k) k.

Let
than I.

P(p (k))
Now as is P(k_1)

O. b(k)
is

for
Pk-l' Is, of

some'
P(k) O. Ob(k) P(k) we

depending Since Pk-1

O. b(k-1) indPndent,

have 2Pk-1 p(k)) " 89 P(Pk-l)P(P(k))

O. b(k) and 1(pk-1 vP (k)) 0. Ob(k) 2(pk-1) + yi + + P(p(k)) O. b(k) 3zx O. b (k) PCPk x 0. b(k) P( I

0.1

0.0b (k)

0. lb(k).
Thus It follows
N.

2 (P (k-1)) that
Since

0. b (k-1) 0. b(j)
and 0. b(1)

. all 3 from
our result

P(p(3))
P(1) "p

for

1 to

- a,

is

given

by the

case

ja

1.

3.3
3.31 measures

Popper's

measure
Logical

of content
probability, of we said a sontence; (para. it is 3.21),

the

"truth-possibility"

the proportion

of state-descriptions

or possible

universes

`.1n

consistent definition

with of

the content -

sentence

in

question. above, S

Popper's namely

as introduced 1. PW

C (p) correspondingly is the. proportion

measures of

"flasehood

possibility"; or possible

it

state-descriptions

universes

inconsistent

with

the true

sentence sentence

p. or tautology
and hence has I

A necessarily
11 is consistent with any

state-description

content false with fact

zero. sentence

At the

other

end of the unity, This false since is

scale, it is

a necessarily inconsistent with the

has content

all that

state-descriptions. from is a necessarily deducible.

Any elementary

consistent

sentence

or contradiction

any sentence

sentence

(if

we ignore in para.

the

case 3.23) two in

of

"families" content those appears first

of 4. in

sentences It which divides it

as mentioned state-descriptions appears it and directly is

has groups,

into and those with of the

which of

it the

negated; group,

consistent with those

those second.

inconsistent

A conjunction elementary conjunction (2m 1)/2m. sentences has

ppk content 4,

of

two

distinct general has a content

and in

of m distinct The highest is (2n of

elementary content _1)/2n, the that

sentences

a non-contradictory n is this the number

sentence of

can have

where language;

elementary

sentences

value

,_

is

attained

by any state-description.

3.32 sentences given , in, the

If

we let

the'number the it is

n of content still "and

elementary of the any case

language

increase,

sentence

remains

unaltered; has

that-any.,

elementary-sentence

content'A,

so on.

Hence it
finite consider

is

possible;
of

to- remove the

restriction
provided this` which ones; would

to

a
we

-number, only

elementary

sentences, Even

that restriction can be thus a

finite for

expressions. infinite cases an infinite chain. in the infinite

can be removed considered.

expressions of finite

as limiting in

state-description of an infinite appears,

language However

consist

conjunction since will

a certain each to that

anomaly

language unity, is is not oqual

state-do3cription of with in a contradictory anything so far

have

content This but to it

sentence. we have said, like

inconsistent

contra-intuitive the para. converse 3.15, q). i. e.

as we should criterion C(p) m C(q)

require to D(q; in p)

deducibility If

referred then

and D(p;

If

this
by

is

laid

down as a requirement
language, from since

then

it

is

contra-

vened cannot

the

infinite

a contradiction

be deduced

a state-description.

3.321 that
ition. n$ then though

It this
If

is

suggested

by Kemeny (ref. by simple

all the large limit,

27) definfinite even

situation
a given it

can be circumvented
inequality as holding holde also it for in

we define

we can no longer

demonstrate

numerically.

92

This

point

is'of

more

importance-in

a functional

calculus,

which
3.4

to, consider proceed]. we .

the --Extension-to

briefly.

functional

calculus

3,41, been framed the of in terms

The dofinitione 'of a 'sentential is but however in'torms

so far

given

have Much of in terms;

calculus. framed not

worlk on this a sentential'

subject calculus

of a functional
This is

calculus, then case for by to

usually'with with ' Carnap's' and also and'Carnap theory. Thus

one-place, definition with

predicates. of logical

'probability, of this work

example, Bar-Hillel infornation

developments some (ref. 1) with special

reference

we have ... Fm, has FSak, these

h language

with

m one-place a,, a2l

predicates .. an. Such of

F1,

F21

and n individuals in effect (j - 1, all

a language the form

mit elementary ... m, k-1, ... in n).

sentences It is

required

that

shall

be independent

the the

name sense as before, calculus corresponds sentences.

interpretation

so that with

for

practical

purposes with

a sentential additional

calculus point
existential

Mn elementary
concerns sentences. the

The only
of

universal

and

3.42 consists hich W in assuming

The simplest that the

way of

considering describes

these a universe .

language

has exactly

n individuals

each of which

has a name

93
in are that the all language; the i. e. that in the the individuals universe. It al, a2, ... an

individuals

follows

we may write (x). Fjx .. and similarly . __ . F, al v, - F jag v ... vF jan can be considered of like
follows

Foal

F, a2 . .

...

Fat .

for

each

j;

(3 x). F jx Thus universal as simple elementary

sentences

and existential Lunctions,

sentences (truth-functions)

logical

the

sentences
in disjunctive

and can be expressed

normal form. It

other
that

L-range for this

or

logical

probability exactly wo allow limit,

and content

can be defined

language If

as before. the number n of individuals approach approach

to the
zero.

increase content

without unity,

universal

sentences sentences

and existential

3.43 other restrictio4s

Kemeny on the

(ref.

26,27)

has of the

shown

how

applicability

measure

may be removed.
Firstly, the elementary to sentences the it is not necessary it is to assume nec-

independent; concept

only

essary

redefine chain is calleda

"state-description". just one member But not all of if each the maximal.

A conjunction basic pair

containing maximal are not

conjunction. independent,

elementary

sentences

conjunctions

will

be consistent.

In

this

case

a state-

94

description

is

any

consistent

maximal

conjunction.

Content

sentence of a

is

still

defined
with

an the
it.

proportion

of, state-descriptions

inconsistent

Secondly, that 'every' individual

universe of this the

it

is

not

necessary the

to assume If

has a name in
with "number N (greater of

language.
n)

we assume 'a we can language the is speak in

than

individuals of in the of'

possible

models" content sentence in general have

universe, of models

and define such that the

terms in

proportion false. N;

question on the

The content thus"a

measures

depend

number

universal

sentence

will

content

)2. (21t - 11. . individuals a measure with, of

Kemeny envisages remains content the finite, we allow above but

that that

the for

number n of namedi the purpose without of limit;

N to

increase regarding

however,

convention the definitions

inequalities.

Finally,
quite calculus functional may arise. it will not this naturally with to higher

can be extended
e. g. a functional

languages, predicated,

two-place etc.

a second-order complication, language content, procedure. however,

calculus, For in would

Additional in

example, general amount

an "undecidable" to to calculate a decision

be possible in effect

since

56 3.51

Additivity %

and

synnmetry above that when content is

We noted

measured-in

a sentential

calculus

an olomontary

sentence
.A

9c
has content yi, and the might arise: conjunction is there of two has content such such that that 4.

The question we can take the conjunction

a measuro and

elementary of two ,

sentences elementary

as units, sentences

measures

trice

as much as either,

alone, part

and so on? of the question for

any

The second request

sentences; elementary

is

for

a moasuro which
i. e. which sentences C(p jpk) p is

is-, additiv
such that for

elementary
two distinct

and p. C(p j)

wo have + C(pk). to_have equal

Since

we consider

all

elementary

sentences

vilue, of

this

is

equivalent

to

rgquiring

that

the

content be

a conjunction

of k elementary The logical

is 1/2.

eentencea, probability
Hence the

ohould

proportional
of such

to k.

(as I, -range)
simple expedient

a conjunction

is

suggested

of writing M(P) This 'wDf will

to

1062 P(P)' be called the lo{; arithnic

we have had

measure,

as

opposed

the

linear

measure

so far.
(ref. view 1s of

It

has been introduced

soo also ref. 2 of 27). the It

by Bar-Hillel
has the

and Carnap
in sentential

property, (in the

the

base

logarithm, that distinct

calculus) has content

3.52

any con k.

ction

/3lementary k of

sentences

llore

generally,

we might

squire

an add.itivity

condition

of

something

the

following

forms

.
C(pq) " which It this = are C(p) + C(q) for. -any (in two sentences to p and q

,, 9:-. ..

independent out that too, sense., the if itxx

some sense

be defined). satisfies in the of the

turns

logarithmic we use k

measure "independent" It is

condition

probability

a theorem

probability
`vo have

calculus

that

independent and q are when p

P(pq)

P(p)P(q).

If

wo use the
of

logarithmic
the measure

definition
for such

of content
independent

the
sentences

additivity is immediate.

There
in terms of a logical in is (but tha

is

no simple
the

way of expressing
conditions Absence nor of for interA either A in terms with

language above sense.

independence dedueibility sufficient p or

neither not

sufficient

necessary. is that

necessary)

condition or

q should

be a contradiction can of course

a tautology.,

complete of the 3.53 measure d6sirable, the

condition numbers of

be expressed consistent

state-descriptions

respective

sentences. For some applications requirement some not. would of the content be linear

the

additivity and for

seem to and

The logarithmic

measures

are

totoloRicallg that holds

equivalent, for
deal 5.

in

the also

sense that holds

of

any inequality
for these the other.

one measure
with

We shall in chapter

some applications

neasures

The reasonable by no means introduce distinct exhausted-by these

possibilities measures. is

are

however now

We shall topologically

a tuxther from either

possibilityywhich of the measures

soafar

introduced.

3.54. logarithmic sentence that have

C(-P)

In measures, is related sentence, to,

the the

case of the content not in

linear

and of a with, we

of the nogation general the

(p)

but

identical measure

of the

ltself.

For

linear

i-C

and for expression For

in

the

logarithmic involving

measure logartthmic

a rather

more complicated functions.

and exponential be discussed

be interested, is symmetrical

some purposes,
the next to chapter}, content we shall though it of is

however,

(to

more fully
in a measure as between

we might which such it better Since will

allied

measures call

p and --p; measure; criterion measure

a measure not not the to

a "sign-symmetrical" the it deducibility as a measure

because perhaps

satisfy describe important

"content".

most

of this
of the

class
entropy

is
of

closely
a source, and

analogous
we shall such

to hannon's
call such

measure
measures

"entropy"

measures,

donot

a measure

"H(p)".

The most obvious

is to take of some kind p and .-p. of n Thera or are

way to
average

aocur
of the in

sign-symmotry
non-symmetrical which this

measures

many ways

might

be done.

We shall

however

only , be concrnd

with

one,

namely metic

that

which the Df s where

results logarithmic P(P) C(p) '(P)

when"we

take

a weighted Here (1 -

arith-

mean of H(P)

measure. --

we have P(P)') 1092 (1 - P(I

10152 P(P) + P(-P)

C(P) "-

C(-P).

1062

P(p).

This
being It is equal to 1. for it this for our

measure
It is P(p) the

is, a xaaximum for

about -1 value

P(p)
this

-,

symmetrical =0 and P(p)

point. continuity Other

undefined that of see, with with

but 0.

requires properties Wo shall conform conforms

have

completing will that or

moasuro example, additivity

be discussed although deducibility criteria it

in does

chapter not it

4.

criteria which

certain

a2alogous

wo shall

then

consider.

3.6

Logical

correlation

3.61 theory its which deals

There with

is

a large like

body of mathematical "correlation". with correlations measures

ratio"

concepts

In

9g i] 1

most usual

applications

we deal

between
"correlation up,

two random sequences;

coefficient" used considerably or

numerical
"correlation in statistical

of
are set

and are

analysis.

We shall between

here

consider in

what wo 3aiGht mean by "correlation a logical language. This is a

sentences"

. !

natural

extension

of

a consideration

of

measures

of

"content",
a sort of

since
mutual

a correlation
content of natural the

is

is

some respedts
concerned. of our

like

sentences extension

Another

investi-

gation

is

into

measures We shall 3.7.

of

"confirmation's

(of

one sentence of confirm-

by another). ation in

consider

some measures

section

3.62 relationship their between

Let two

us

start

by

considering p and q in zc

the terms sections of

sentences previous in

probabilities.

As in "probability" as before suppose the that

we may interpret although wider. again Let us

this

context is and P(q)

as in

"L-range", fact = b.
i

interpretation P(p) aa

It probability special
is than it is or

does not

of course pq is

follow ab,

that

the in the

of the case of
less loss

conjunction

except

than than

"independence" it ftatover of p and q. or at most equal to one of them,

either a or b, /but Let may be greater P(pq) the product. us write = c.

3.621 us to write
truth-function P(pvq) P(--p) P(p-q) P(-p-q)

These three down the value

of p and q.

numbers

a, b and a enable of any example

of the probability Thus we have for

&+b -c

1-a
awC

I-abfC
4

1o. ... p

ti

and so on.
i. e. given (whore p, of

We can also
"Pp (q)" q"): -

write
stands

down relative
for "the

probabilities,

probablility,

Pp(q) Pp(-q) P_, (q) P and the so on. usual pvl All (pq) these identities laws

c/a (a (b c/(a c)/a c)/(l +bare - a) a) easily to proved of from the

probability involved). Now it

(subject

existence

fractions

is

clear

that

if

we are

to

speak

of

a moasure q, it

of

"logical

correlation" in terms

between of the

two sentences appropriate

paand

must be definable a, b and c. A first quantity (i)

(ii)

quantities 3.63 to use the .

suggestion has the

would

be simply

c.

It

propertiestb.

0<c<a,
ca ab in case

0<c<,
of

independence.

c>

ab

indicates (iii)

positive If
c=ab=a,.

correlation, then c-

c< ab - b;

ab

negative. and

a1

if

b-1

then

(iv) o ab=0.
(v) q entails p then (vi) If

If

a-

or if

bA0

then

p entails c tt b. If p entails

q then

c-a;

and

if

-q

then

c -. 0.

irr
(vii) c=a - b. If p is equivalent to q then

3.631 intelligible of tion case value "scalIag" with (ii). in

All

these

properties measure,

are but

intuitively

a correlation for

some kind in like connecthe

seems called That is,

in

particular perhaps

we should

of independence of

to be repreronted with

by some particular the using

the

our meacure unconnected

probabilities evalues called

of p and q themselves.
or c/ab, which for The

This. suggests
independence take has

ab
0 and. 1.
a

respectively.

second

of

these

been

the.

"coefficient Its its

of dependence" is is

by Keynes

(ref.

29,

p.

151). finite;

minimum value maximum value tend to

0, when c-0 infinite, to the

to

and a and b are approached

being

as a, b p
of

and. c all
approximately both very

zero

same order
the find,

(i. e. with

equivalent small).

q but later

probabilities that

We shall

some importance to this one.

attaches

to

a measure

topologically

equivalent

3.64 seems implied

be If symmetrical wo write

One requirement by the word "correlation"

the for two "the

on the measure is that it

which should
q.

as between "C(p, q)"

scntencos correlation

p and

between

p and q",

we might
C(p,

require
) c1. this _ as the C(q, p). requirement";

We shall

refer

to

"Byrmetry

or more explicitly

as the

"sontence-symmetry

requirement",
.j

"""' lOt_ .
z

to

distinguish

this

kind above. say

of

symmetry When the the

from

tho

signis

symmetry fulfilled

mentioned we shall

requirement is

that

measure

"sentence-

symmetrical". All in this sense. the ablve measures for

of

are

symmetrical of
b ands c

The condition
in should b and vice Perhaps the terms

sentence-symmetry
the quantities a,

a measure is simply

expressed that for it

remain versa.

unaltered

when a is'

substituted

simplest

examples relative

of

nonprobabilities

sentence-symetrical p( q) c/a and

measures Pq(p)

are, the c/b.

When wo consider.

such measures symmetry

3.641 this context

we shall

indicate

their

lack e. g.

of

sentence-

by our notation

by writing
also of the of

"Op(q)".
use It two in in may sentences. such

We shall for the to concept

have

further

"sign-symmetry". extent to which i. e.

be that are

we wish

neasuro

related

independently

their

signs,

a way that extent

this

-p would If in

be counted addition

related

to q to the

same

as p.
implies

we have sentence

symmetry

C(p, We can test sign-symmetry

q)

C(-p,

q)

C(p, in it and

-q) terms

C(-p, of a, b,

-q). c) for

a measure by testing a by

(expressed whether 1-a

has the e by

same value b-c. None {

when we replace of the

above measures

has sign--syamotry.

'c,

i*, -.

/o.
3.642 call-"sign-perisymmetry". mo by Professor sentence alters popper). the measure more usual ' (This oasis what

-}

we shall to of a

name was suggested the sign way,

Here' changng in

a doterminate ) f(C(p, q). function of is

i. e.

where that which f (f (x)) includes

f(x)

is x.

a numerical

x such a property with

Sign-pericymmetry as a spcial

sign-symmetry

case,

f (x) f (x)

X. -x

But more generally k (x) l/x. or

To'test-for

we ,tind

cases

with

sign-periiymmetzy, 1 - for, a and

we transform b-c for

the

measure

by writing

and then

aoe whether
of the have original. tho

the

result
(If

can be expressed
it.,, csn, the function

as

a function must in fact

required measures

propeZty)", so far the if C (p. montioned, corroaponding q) and cab C(-p, -q) only

Of the cab is is C(-p3 q).

sir, m-poricynraetrical; f(%) q) This is -x. C(p, measure "positive" of Thus -q)

function we find ffi C(p,

ab -c indicate

can thus or

whether in accord

a correlation with the ueual

"negative",

moacuroa

mathematical

statistics.

3.65
"product-nonnt" The meacuro which

Wo can actually
tochniqu results directly is due to

apply
to the

the

mathematical
of sentences,

case

Kemeny and

Oppenheim

(ref.

28).
. -

/o
The usua]A. application of this technique

is

to

a case like
connection population. to be

the
there

followings
is between

Wo wish
height of and

to

see,

say,
in

what

and weight for

a given are found

Measurements xl, x2, yl, subtracted ... y2, xn

height the

n people

corresponding

weight of the

measurements height is

yn. * .. from the

The mean value height measurements

and similarly
this done,

fn-the
wo have

weight

measurements:

supposing

2 Xj Y
is computed, namely (1/n)
This is a suhl of

0 o.
of the measurements

Then the mean product

Xjyj"
such that we count Positively

terms

the
or

cases where height

both below we whould with is average, expect weight,

and weight
and neratively it"to

are both
the if

above average,
remainder; height tends and

hence to zero scaled and 1, the or xi

be positive if the It

increase if there in

negative relation. its

reverse, is finally are

no average

such

a way that by the and the

limiting

values

-1 of

by dividing and the 3j,

root-mean-square product-moment is

values coefficient,

estimated

correlation,

__ dos

-a

3.651

If

we apply values x with

satisfy

this with

to

two sentences their p, truth-states. and y

for and y'

p and q, we must associate Let

with zero

us associate
positive q.

the value
Then have to to

positive
the

condition values x'

mean we shall

associate

with

their

negative

truth

states

such that

xP(F) + x'P(-p)
9P(q) or xf y' Now tho _ -XI _yl product+ ) 3''P(-Q, a a =b omont

0
0

can be computed,

equal

to

xyP(pq) + xy'P(p-q) + :: 'yP(-pq) + x'y'P(-P-q)

and similarly the scaling iactor$. cab The reuult is that wo find

4
This as the a scale and 1.

1-.

)b(1-b)
is fact more cmakes or ab, the less the

3.652 same measure that range it has -1

is

above-mentioned factor It with is which

except of and maximum and the

limits

sentence-symmetrical (x) 39. aab for _ -x. Its the p equivalent

silrn perisymmetrical mininum values to occur q,

when the latter

former to

for -q.

p equivalent It is zero

when p and q are

independent.

lo 6

3.7

Confirmation

3.71
sentence-symmetrical, then we turn to

A concept
less "confirmation", since

of

"correlation"

is

customarily

commonly

sign-symmetrical. would equate great

sign-symmetry it would ignoring cases in the

seem to
"r

be inappropriate, with

virtually the

confirmation logical simple the

dicconfirmation, of of to these the two

asy=etry application

most?: typical i. e.

"confirmation" Seneralisations. of

concept,

application

universal

The relevance case of confirmation is nore

sentence-symmetry

in

the L=

problematical.

Although:

in
is -

typical
in it

cases

the

evidence

which
related from

confirms
to the

a hypothesis
hypothesis -we

no sense may even

symmetrically be deducible

the

hypothesis

should apriori.

not

wish

to rule find

out in

sentenco--synmetrical fact that extent

measures

We shall feature

sentence-symmetrical in confirmation-

measures
theory.

to a certain

3.72
those It is just considered

A measure
is put

very
forward

closely

related
(ref. of

to
37e). x to

by Popper "the events. power It

symbolised y", little (writing where

"tE(x,

y)",

read

as

explain makes events

x and y symbolise if p, q).

Measure

clearly of

difference them
popper's .

wo use

sentences

instead

reduces

to

/07

E (p

q) ,

c-----

ab

c+ab but it has no

It

may be seen to be sentence-symmetrical

properties.

sign. -symmetry

As an elaboration suggests of (rer. 37e)zxx c -- ab

of of

thi:

measure

Popper

a measure

"degree

of confirmation

q by p" as follows; O (9. ) m

--------c+ab

c (1 + )
nor satisfy signthe follhas

This

measure

has neither It

sentonce-symiaetry is designed to

symmetry oviing been list

properties. of properties in

(where

Popper's into than,

symbolism our equal own): to,

translated (i)

a direct is

manner rsater

Cp(q)

or

less oi',

th=

zero

dopending q.
-1 0 If

as p supports,

is

indepandont

or undor11ineL
(ii) (iii) (iv)

= CFCp(q) Cp(P) p entails C(P) q then

<

Cp(p) <1

<1.

- P(-p) Cp(q)

Cp(p) a

a C(F)

" (v) If p entails -q then Cp(q) - Cp(-p)

C2

-'L

(vi) Cp(q) Then of for approaches any explain of q. riven

Lot

q have q)

a high ---and

content lot

--

so that q. power

E(p, p,

p support with the the

Cp (q)

increases with

q to

p,

and therofore

scientific

interest

/
(vii) greater as Pr(p) than, is (viii) (a) equal greater If If to, 0(p) or than, C(q) than to, p then 0. then Cr(p) C5(q) or less

is depending than Pe(q).

less equal

q entails Cp(q)

(b)
incroaco toGether. (c) increase together.. (ix) If (a)

for

any given

q,

CP(q)

and

C(p)

for

any

given

p,

Cp(q)

and

P(q)

-q

is

consistent <0.

and

entails

p then

Cp(q)

(b) increase together. (0)

increase 3.721 C(p) of logical P(-p) together. It

for, any given

q,

Cp(q)

and

Pep)

for

any

given

p,

p(q)

and

P(q)

will

be noted with clear

that

the

definition

corresponds It of is

Popper's that

definition Cp(q) iaay be

content. as a sort Provious

considered

relative work directly Cp(q) with (see

content-measure. e. g. in ref. . of 9) had tended

to in

treat particular

"confirziatiog" equating oho; that of his (viii). the such

terms Pp(q). this would

probability, Popper and practisatisfied gives

examples illustrates cally are

to

inadequacy a definition

of

view, satisfy ones

none (vi)

requirements:

the

main

and

. 10

3.712
0 (q)
bocause notice it that conflicts this

Popper

explicitly

rejects

a definition

=--0+

c-

ab
ab

(1 + c)
(iv). We might

with rejected

the

requirement is

definition

sentence-symmetrical,

though with 3.72

much

the

grounds

for

its

rejection

have nothing

to do

:3ontcnce- , yrmtry. It
simpler measure

is. interesting
Cp(q) except c/ab

to notice
satisfies if --the i. e.

that
all

the
of

Popper's themselves

requirements are

(viii)Oe),

requirements; if the

reasonably

interpreted

numbers
if and Simplicity of

representing
C(p) is

limits

of ranges
asp equal would which thus does to

are

suitably
the

altered,
1/a. dropping

redefined

1l/P(p) favour not

considerations (viii)(c), one.

requirement important

seem a particis not of of course the it essence should is no

ularly

The revised showing that

measure it is

sentence-symmetrical, of not very a confirmation be. great of This

measure example

(pace

(viii)(c)) shows us that

that there of

further between

difference sentential

the

co. copt

confirmation

and that

correlation. Popper, would of however, intimates for for This to is reasons because laws.

(Professor that the the measure of hie his list c/ab

be unsatisfactory confirmation,

purposes to his

theory of to

external he wishes

requirements. be applicable

measure

universal

11
se have P(p) Thus =0 the and consequently c/ab other would

IZ

p is

such

a law, i. e.

P(pq) would

a 0; always of the

a-c-0.

measure with and this

be indeterninate, interest applicion in the

together theory;

measures preclude

desired).

(At any of our present

general connections other in character: between

the

same time,

this in
still

does not so far

point and

invalidate are

considerations,
i. e. we can the where concepts these

as they
out

concerned, special points

consider apply).

applications

do not

. 8
3.81 c/ab

"Information

tranofer"
If in the place of the "linear" measure

eve 'were

to

use

"logarithmic"

(topologically

equivalent)

one 1002
C

vie should satisfy

secure Popper's

one very roquirement C(I))

interestin (iv) lob

; result. Nvo need 1 to

To define

Pi "
of content as

1082 i' (P)

This defined is exactly in the the logarithmic section. Popper's for this requirements measure, it is measure

previous All of

except appropriate

(viii)(c) modifications, It ranges

again

hold and in

with

addition infinity,

sentence-symmetrical. c=0 and a and

from

minus

when

are to

finite, the

to

infinity It

when is

a, zero

b and c all in the cane

tend of

to

zero,

same'order.

independence. 3.82 concept in ghat the of the Analogy technical chaptor said with the of "information" as introduced with as to q, and

theory suggests

information consistently this with quantity respect recall

previous been the

that

has

we may interpret p givoc clear

measuring vice from vorsa.

information To make thin 2. In

vie raust

a definition

chapter

a communication probability of Tk, symbol the of

channel, the joint

whop via denote ovont consisting

by of

"P(BjPk)" the

the

transmission of symbol is

G, and

simultaneous rate

reception over the

averaC, o information

channel

defined

R
This

P(F, Fk)
is

lo; ,
ovor the

F(EF,
P(E

) -

Mtn per
symbol) oymboln

j)P(Fk) the

expreesion

an averogo, of

all

transmitted

and received, 1062

quantity

P(rJ Pk.) P(r, 4J)P(Fk)

which involved

latter in

may be cohceived a single In the joint sane channel,

as the event

information

transfer

BiFk. in a logical quantity analogue

way,

of

the

communication

the

, ti

$.r

', fly.

C(P,

q)

c 1002^'

may be considered
when a sentence

as the
p is

information
and

which

is

tranoterred
q rocoived,

transmitted

a sentence

or vice

versa. Just Whether

diacuc,

there
later. in

oxioto

a valid

analogy
however

of

this

kind

we shall

We shall neantime with between in

consider of "the

ouraelveu information or "the

justified one

thu

speaking to ".

sentence

gives

respect

another"

information

transfer

yentance

3.9
3.91

Entropy-typo
If proceeu the relation of tho in

measure
as above in order between to in section 3.5 wo use

an averaging measure get of

get

a sign-symmetrical wo once this more time i. . in the

sentences

an analogue of rate

Shannon's econd a noisy

entropy

neasure, dofinitioa, as quoted

an analogue that of the

fundtiiental channel,

preceding

paragraph. We take a weighted of just truth-states introduced. avorage, of This over p and gives all q, of us the:

possible logarithmic the rather

combinations measure elaborate

formula

R(p,

q)

Y(P<1) lOg2

P(pq) P(P)P(q) + p(p-q)log

PCP q) 2P(P)P(-q)
P(-P-q) P(-p)P(-q)

P(-pq) t P(-. pq) 1002 P(-p)P(q)

sr u

P(-p-q

)1092

--

Y? I

113.
The formula " p" the by first are "tp" wifficiently and vice verca to. =i, gjSn--oy=otry obvious simply and the properties if wo notice amounts second to and of this that replacing;

interchanging fourth.

and third

It

is

of

course

also of

wentence-s7rmetrical. the signR(P,

In view
R(pI where H(p) -P) is tho If we have case if R(p, p and q)

etry

we find

p)
introduced p, q is is

11(p)
in para. or 3.54. contradictory the

neasure one of 0.

necessary also in

This

general

q are

independent.

3.92 fled
but the ne,

This in
aloo next

being

so, of the

wo can now foel entropy

jucti-

opoalcing,
of the

not
entropy

moroly

of a sentence,
In

transfer

between

sentences.

chapter, describing

however, this

we shall meaoux e.

introduce

an important

way of

3.94

In

this

chapter

wo have

introduced

four
theory. C (p)

iaeasures
Firstly, --

of importance
we have

in
the

connection
logarithmic we have

with

inforiaation

content-noasuro a sign-sy n etrical the:

1a&2 P(p). obtained frone

Secondly, this

measure

by an averaging

operation,

entropy-type
have two

measure H(p).

zn
of relations of

Than we:
betcteen dependence

corrospoizding the

measures

sentences:

1oGarithtic

coefficient

I/4

C(p, R(p,

q) q).

log2

c/ab;

and finally

the

"entropy

transfer"

We single of the But which their theory it could make analogy of to,

these

out

from in

the

others with,

because

and interest

connection in chapter

information also be

as eApounded that they

2.

claimed more yet, is

have than fully

properties the others

there rather We have not this

important however, our task in

mentioned. these

dotailod next chapter.

properties:

the

._j; .

4.

THE LOGIC OF UESTIONQ

The various of they "correlation" all claim to in chapter

dofinitions 3 have this

of in to

"content" common; n3ontonces I2ussllian) basically of the

and that of

be measures of orthodox we shall

relevant (basically not

a logical kind. In

language this

chapter but the

introduce certain or

new measures; ----- principally ones --in

wo shall sign--,

intororet ynrotrical throws

measures

"entropy-typo" on their in showing

a now way which

ooxie light consists

properties.. that 'by the

The re--intorprotation measures concerned to

can be greatly not circles, sentences"; incidentally of

illuminated "contences" but or rablar (as we the

considerin

thorn ftndorstood

be measures in logical

as usually of ghat

wo might see) to

call

"unsigned This

shall

"questions*. be Soneraliscd

permits

measures

somewhat.

4,1
i norwri the "ig

A zigi-zymnc1rical
il of the sentence

neasuro
coxcecned.;

virtually
that in.,

it

has the

same value

for

a sentence

p as it

has for

lf

the is or

negation that briefly, it is

-p.

Now part

of

the

definition

of true

"sentence" or false;

something something

which which

can be either has a "sign".

When signwe seem

symmetrical

measures

are

considered,

however,

to be giving setting
proceeding concerned would

with

one hand and taking with

thorn in

away with

the

other;

up ontitie3
to is uce

a certain
such

property

and then
the property it

a way that

irrelevant. and measure it siiapler is

Procedurally, to think of

therefore, the entities

be bettor our

on which after

defined unnecessary

as themselves to specify

"unsigned", the symmetry

which

becomes the

properties 4.11 of entity

of

measures. This involves were sotting represent call is up a now sort aLlbig; uously such like is an entity what not itself addition

which or its

can as it negation.

a sentence an "unsigned logicians a sentence, of 4.12 at are this like a "sign".

We can In a way

sentence". might but call it

this

a "sentence becoxacs

f, orru" ; it

one when

completed

by the

A suggestive situation "binary is the following: (By two

alternative "unsigned a "binary possible question but

way of

looking

sentences" question"

questions". with just

we mean a question "yes" which and can "no"). itself

answers, is its not

e. g.

Chus a binary '. bs ;, rue or or false;

something

possible plus

anstiwors

can be true

false,

and hence

a question

i7
is to sentence

answer

analogous

an unsigned

plus

sign.

4.13
convenient corresponding symmetry notation with

This
for

analogy
us: the

moreover
unsigned "? p".

suggests
sentence The sign-

p can be written in the notion ?-p'is of

inherent

a question equivalent in

can now to ?p, applicable

be expressed whore to we use

by saying the word

that

"equivalent" careful

a new sense, will

questions:

a more

definition

be given

below.
We shall also sometimes denote a question

by the

letter

"Q",

with

or without

a suffix.

4.2
moan not simply a

By an answer
"sign" (as "yes"

to

a question
or "no")

we shall rather
?p.

but
to

a statement.

Thus

p and -p

are

both

answers

Emphasis
because need not it

on the word,, nstatement"

might be thought that

is

here

necessqry
to

only

an answer but might

a question

be a complete

statement

sometimes

be something can reply

but merely

less;

e. g. that giving
noun.

it

I am asked my name I is not a statement

to point

by simply
a proper

my name, which
Here the it is

necessary

out is

that

in

the

context

of x/question the assertion

to questions

giving that it
have

my name is
all

equivalent
in

to making
general

my name;
the

and that

answers

logical acterised implied

characteristics as "true", by other

of "false";

statements they

(they can imply

can be charor be say

statements;

and so on).

We might

11S

that'"yes", forms ulati, of

"no" answer;

and

similar in

formulations practice indicates abbreviations they stand the

are

coded

and that

verbal

a form-

on of

a question permitting what

frequently such

an appropriate to for. be used

"answer-code*, and 4.3 about questions (such possible specifying

statements In laying not

down genreal confine ourselves "? p") these

principles to '! binary"

we need as those

questions just two

represented though Genreally, set

which'have constitute an

answers; case. "any

especially a "question" exclusive as

important tobe statements", This

we can define of mutually referred requires to

exhauttive such

statements first,

being however, it

"answers".

d4finition we should

some justification; reasonably three main well points with

show that conceptions. be at issue

corresponds There here: are (i)

ordinary might

which

that

a question
(ii)

is
that

representable
the possible

as the
answers

set
to

of its

possible

answers,

a question

are together question

justify

exhaustive,

and (iii) exclusive.

in turn.

that

answers

to

a to

are mutually
each of these

We shall

attempt

4.31 what counts to knowing

(i) as an answer what the

We want

to

syy that is

knowing

to a question is.

equivalent point the which point made a

question here: is

The only with

needs demonstration

connected

!1q

above

that

an answer

to

a question merely which giving

must

be considered of is a statement. Luxembourg? a continent, "

as a statement, Suppose could and the

and not "In just such

as a part continent the were about

question

be answered suppose that to

by

name of not

an answon a statement; set of

considered the location of

equivalent Luxembourg. "Asia", since "In etc. they what

making Then the would not

possible what

answers the question to the

"Europe", was, question

specify well Ecuador? that

could continent the

equally is

be answers ". So long are to in

as we accept, statements, we question "Luxembourg clearly or to any couldn't

however, shall are is the in say

principle the

answers answers is

that

possible

the

first

statements

"Luxembourg

Europe",

Asia"

and so on; to the question

and these about to the

as a set Ecuador,

be answers question not

equivalent

one asked.

4.31L it is possible what

case

It

might

be said

in

some cases

that

to know. and understand the possible

of the question

a question For
continent is

without

knowing
in the

answers
"In

are.

example,
is without or

what

luxembourg? being able

" one might to remember are. that

know what the Such names a case,

a continent of the

continents is is

how many there since the fact

however,

irrelevant

the

word

"continent"

a generic

name for whether

-..... _ .

Europe,

Asia

etc.

is

logically

independent

ob

anyone knows it
------

to be.

.... >-s__-

hLo

4Q3 of possible answers

(ii) to

The necessity be exhaustive is

for

the

set

illustrated

by the whiih

classical is a logically answers

possibilities.,

"Have you stopped improper. "yes"-and

beating just

your

wife? ", the

question "no" do not

"In

because all

indicated I,
l19 ical

cover
which it

the

The question like this tooq

continent presupposes procupto "In be which

i'Luxembourg? t at Luxembourg is If,

" is

because

must'be'in only we were

a continent; because asked the it

and the happens question to

X Bition true. continent

unimportant instead,

is

Honolulu?

" we should

be forced!

invent

the

supplementary which is

answer required
into

"No continent in order

at all",

i. e. the set of

answer
possible

to make the
sot.

answers

an exhaustive

4.33
answers following continent or Asia, must also

(iii)
be mutually Suppose

To see that
exclusive, on being to

the

possible
the which Europe, that I had now Asia, I

consider asked "In "Either

examples is or given

Luxembourg? Africa". a proper It

" I were might answer,

reply

easily in This the

be objected sense that might or it

had not not given

a complet'answer. way: The answer

objection

be put or not e. g.

another

"Either answer, by

Europe, because proper are

Africa" exclude the

cannot

be a proper excluded

does answers, mutually we mean

and is. not "Europe". is

other

answer

Complete simply one of

answers. the

exclusive,

and this

things

.sr.

=., , ,

izr

by "completeness".

4.4
reasonably in accord

Our definition
with usage;

seems,

then,

to be
of course

and we cannot

demand more. which

4.41 ,, to a question, be

It

hae one or two immediate

consequencea

are

of

some interest.
First, in accordance let us notice with This that answers to our that definition, (with an answer can one

never proviso)

self-contradictory. from the fact if all it. with

follows must

be mutually were be, possibility answer.

exclusive; sblf-contradictory deducible of

because

an answer the other

some question would the

answers concerns

from

The proviso one and only

a question

one possible

4.42 If a question
is

We can however has one and only

prove

as follows: answer, that

one possible

answer

a tautology. Thus consider p21 """ a question pn" If with these the n

possible are

answers

P1 their

answers

exhaustive, pl

disjunction '7 Pn

v p2 '7 `

must be a tautology. ?p with being If possible the familiar

(A particular p and -p, of the

case is the

the

question then pv) one'

answers "law

disjunction middle", with just the

excluded

however

we now conoider pl say, it

a question is clear

possible

answer,

that

disjunction

llfntY

zZ

will

reduce

to

the

single

statement

pl

itself.

Hence

pi

must be a tautology.
Moreover, since with of all tautologies answer with are will only logically be

equivalent, equivalent.

all

questions

a single the

We shall

speak

question

-one possible
4.43 theorem would

answer

as the
We might

"empty
note serve is to

question".
in passing that this of which

alternatively i. e. possible a tautology answer

as a definition a statement

"tautoltogy"; is the only

some question.

"A in is the following

converse form: then it If is

of the

theorem to

can be proved a question answer. be mutually

any answer the only

a tautology, follows

possible

This

from

the

reqqirement

that

answers

exclusive.
4.44 strain case of the just meaning Every of the question word has an answer. to cover We can the

"exhaustive" when it

one possible

answer,

specifies

that strain

the it

answer to is

should the set: sense. question

be a tautology. case where such a set the

But we cannot set of possible not be "exhaustive"

cover

answers in

a null

could

any reasonable is the

The "emptiest" with a single the

possible answer.

question 4.45

Thus we find

logic

of questions

'-

differing If the

to "empty" that there

some extent question we have is no

from is

the

logic with

of

statements. tautology, the If as it contradicwe were, we

compared to

the with

we find tion: imagine into get with denial

nothing

compare

"contradictory statement addition of

question". turned,

a contradictdmy! by the with

a question not the of a question

a question-mark, answer but

no possible answer

a'question from the

one tautological the contradiction.

which

results

4.5 of of statements questions. have,

Certain however,

other their extent, some of

features analogues in the (In to the fact,

of

the in

logic logic

the

To a certain with

wo can build if it the

"question-calculus" a so-called seems more "sentential appropriate

properties this latter context

calculus". to refer

as the

"statement-calculus"). 4.51 contains it first Thus for We shall say that a questio4 Q1

Q2 when from each another question is possible to deduce an answer to example " is and the question in of latitude the "In which

answer to the the second. is the

continent "What is

Ecuador? latitude peak? ", could

contained longitude from

question

Ecuador's

highost.

mountain figures we the

because the

and longitude (and this not just

deduce

continent

in

case of the
....: ..

correct

answer,

but

in

the

case of any

12+-

possible 4.52 such that

they are

answer). When two questions each contains

equivalent.

Q1 and Q2 are say that

with the

the
This

other
use is

wo shall
consistent

use of the ?p is
4.53 of two

word "equivalent" to ?-p.

Similarly

above,

i. e. in

saying

that

equivalent

we can the as it which of two question were

defLine which asked

the is

join asked and

questions. two

This

is are

when whose at and

the

questions are in

together, questions ?p

answers Thus

statements the case

answer binary

both

once. ?q the

questions "? (p+q)" ) is -pq

J in

(which possible

we shall

write pq, is from

the

question provided must and of

whose only course

answers-are of these

p-q,

and -p-q, This ? (pq)

that

none

inconsistent. the questions Genetally,

be distinguished which are binary

? (p v q),

questions.

if

Ql has m possible will

answers

and Q2 has n, with (or

their

join

Ql + Q2 answers,

be a question of

up to mn possible equivalent to) a conjunction of 0-2,

each consisting answer only

of a possible and omitting 4.54

of two questions

of Ql and a possible as are

answer

such conjunctions It can easuly

contradictory. the join

be seen that
either any

Q1 and Q2 contains the join of

question Q and

separately;

and that

question

the

empty question

is

equivalent

to

Q itself.

Every

.. -m-F-.

-. -:_: ... .. ,. . _..,..... . ,.,, ,,,,,

t2S=

question

contains

the

empty

question,

and the

empty

question
4.55 to this their the

contains

no. others.
Questions may be classified and it of is clear according that

number

of

their

answers; some sort for example not

will

constitute since if.

"ranking" a question have in more this

as regards Ql can contain

content,

a question answers setting however, appropriate and natural subject lot than

Q2 only Ql.

Q2 dove

possible way is like

Banking

questions

up a "linear" that

content--measure. content particular But the special

We shall measure that it is has

find, also a simple to this

a logarithmic here, and in

interpretation. us consider

before case

we proceed of binary

que; tionil.
4.56 is twofold. In answers In the the is The importance first the second than the are place simplest place, those of binary questions with just two

a question possible binary of higher

possible question. closer

non-empty questions orders it is have with binary a the

relationship of that

"sentences" questions 4.57

sentential "unsigned At the

calculus; sentences". same time,

questions

of

any

finite
i. e. the

order
into manner

can. be broken
of

down into
questions

binary

questions,
after or parts-

sequences of counsel

yes-no

asked witness

ciloss-examining

26_

cipanta: In logical

chn a panel-game terminology is equivalent Since 2n possible

finding we can to the

the say the join

answer

to

a riddle. of of any

that join of

a question of some set

finite binary

order

questions. up to

n binary the minimum

questions number

can have of with than binary

answers, to the

questions

required is

represent smallest

a question integer not less:

N possible logt N.

answers

4.58
that should arise words oven questions

It
with in from

seems reasonablo
an infinity this the way. use of of Such special

to-postulate
possible questions answers typically

be reducible in practice

interrogative "hove many" and so

such

as "what".

"when",

"where",,

forth:

e. g.

"What. is is

the

length

of this to

room? " formulate infinite

In a

such a case it sequence question

possible questions

as a rule

of binary is

such thattthe "limit

represented

by the

of their

partial

j oina" .
4.6
possible it of first section measures becomes 3.15 of clear will

If

we turn
"content" that our

to the
in the

consideration
case; of

of

questions, criteria" by containment

"deducibility be replaced concept of

most

naturally on the

"containment defined above.

criteria" That

based iss.

_!t7

If'Ql

contains
a (Qj)

Q2 then
?C (Q2)

with

equality

when we have
number of

equivalence.
possible answers

Since
to

if

Ql

contains

Q2 the

Ql must

bQ greater
to Q2 it is

than
clear to

or equal
that number

to the

number of possible.
of content answers

answers

any measure of possible

topologically to the

equivalent

the

question But this 4.61 this

concerned is not the

would only

automatically possible case:. choice

satisfy

the

criterion.

A particular assumption, can be dotormined

Let are us,

of metric,

on

by adding
present

an'1idditivity
purpo io ,

criterion" say to that the

as follows. tho first questions is

for

indapondelit with any of

when no answer answer to the will then be

inoonsistont definition An "additivity

socond. be given framed: -

(A different below).

independence can

criterion"

If O(Ql

Q, _and + Q2)

Q2 are

independent, 0(Q1) +

then 0(Q2),

where

as before

"Ql + Q2" denotes

the Join

of the

two

questions.

Now let re spey t ive ly. sense, the their If join they will

Q1, Q2 have. m, n possible,, are independent have mit possible is a function the in the

answers

above If of the criterion

answers. f(x)

content

of any question its possible

number x of

answers,

additivity

I2$

requires4of

this f(mn)

function

that f(m)

it +

satisfy f(n)

for

any in, n.

This

immediately have answers

leads

to

the x

logarithm where x is of it minimum

function. the

Thus we shall

C(Q) - log to Q.

number of possible the logarithm. equate

The choice in that

base 2 for enables

seems appropriate approximately in the binary

us to

content required only

to the

number of though unit. 4.62 theory content

channel channel

questions is

representation; a scale

this

of course

a matter

of choosing

If of information, bears
capacity capable

we now refer we coo that rosomblanco

2.32).

back to the

technical of of
a discrete

nuah. a definition to
In

a close

the
the

definition
care n of of

(section of

transmitting

a number

symbols

of
to

oqual
logt

time-duration
n of binary unite

the

channel
symbol.

capacity

is

equal
the a channel.

per

Now consider end of such

situation

a man at

the

receiving

He is signals
wo can cyrnbol

assumed to know apriori are;

imagine will it

what the

the

possible

received

and prior
him

to

reception
the the e.

of a symbol
question cot of the our "Which

as asking If

himself

be? ".

wo regard sot (i.

symbols correct model)

an a logically operation of

exhaustive the channel

regard of

as a postulate

then

this

question
are possible

has exactly
symbols,

as many possible
and the capacity

answers
of the

as there

zq

channel 4.63 the

per

symbol

is

exactly

phe

content (section to rate the in

of

this

question. that

Now we recall capacity value of a channel of the is-equal

2.575) maximum the

possible

information

channel,

and is
symbols statistical obvious

equal
are

to the
equally redundancy

rate

achieved
in It

when the message

other words, when the

probable;. is zero. of

immediately

becomes or

that

the

concept is give

statistical also

redundancy, to questions; of

an analogous and that content to this which

concept, will in

applicable

us an alternative not to channel

meaoure cppacity

analogous

but

information

rate.

4.7 answers with to a question We regard

probabilitios,

Let

us now-assume

that

the

possible

have probabilities the probabilities

e. g. L-rango,

associated for the moment

preceding

them.

as logical chapter.

as in

the

4.71
Pl, of p29 pj". pp .. * Since

Thus lot
and let the

q have the possible

stand exhaustive for "the

answers
probability

''P(p4)II are

answers

and mutually

exclusive

we have

P(pl
s
0

Vp2v
P(pl)

...

vpn)
+ P(p2) +

0 ..

P(Pn)

1. We should reasonably expect

4.72

a measure

/3n::

of content
In order to

to

satisfy
our

our containment
ddditivity

criterion

as before.
a

apply of two

criterion it is

we need appropriate if every

new definition to say that

"independence"; are

questions

independent

answer

to of

the

first

is

independent to the

in

the

probability

sense two questions

and to be

every

answer

second.
answers

Thus given
(pl, ...

Q1 and Q2 with (q1, ... qn) if

possible respectively, and only P(Jgq)

pp) said

' Q1 and Q2 are if P(P)P()

independent

Call

j,

k).

The additivity sarge words 4.73

on the function and that give the

criterion ad before., If in

may than

be expressed

in

the

addition

to
that the

these
it

criteria

content-measure of the all

we require of

be a continuous answers it should

probabilities the probabilities as the the

possiblo are equal

when

same result determine

measure measure

already as exactly

introduced, that of

we completely Shannon, namely

A(Q)
This to may be proved Shannon's 2.44; requirement containment

areadily

P(p1) j
in

1062 P(Pj)"
closely analogous above in

a manner

"tuiiquenees where is it

theorem"

mentioned that

section third our

may be noticed the criteria.

Shannon's of

approximately

conjunction

and additivity

131 ..

'

4.74 "entropy" use the of symbol the

We shall question for it of latter the of close binary the

refer

to

this and It H(p)

measure consistently-

as the,

concerned, as above. the measure

"H"

is

clearly introduced as the

a in measure)

simple section of of the the

generalisation 3.54, entropy properties from the which of

may be. regarded question K(Q) it bears ? p. are to

The majority sufficiently the concept

measure

obvious

analogy,

of

"entropy"

as introduced

by Shannon.
notice taken the in particular the sot that of is the

We may however it has the form of an average, and that of content of

over

possible logarithmic 3.51. in fact

answers; measure

quantity as defined of this

averaged in section

The definition have of is been

content in

a question way, as the

could average

approached

content measure chapter.

the

possible

answers; one in

where the

the of

contentthe previous

an "additive" That an additive

sense of

measure

content

should

be used is consider it would

immediately nature

seen to be appropriate of an averaging process;

when we that is,

the not

be appropriate

to average

a non-additive

quantity.
4.75 anstiwor , the account probability, time, In averaging of i. e. each the answer content is of possible into It

taken factor.

a second

as a weighting

in

appropriate

here

to

say something

about

the

sort
t

132-

of

probability of Ir-range of

that (or

is

relevant,

whether sense) In

logical or

in

the

sense in the

some similar frequency. are is

empirical

sense the

relative

communication understood such to be

theory empirical

probabilities ones; it and of known designing In in devising a logical to think the in

generally that

assumed to

statistical the into

characteristics channel account equipment. of imntences appropriate When the as in are in

messages

be sent

over

advance the

and may be taken

transmitting a measure language, of of the such case, of it

and receiving "logical is content" on the other

hand ones.

probabilities contendea it, is not are at

as logical averaged, first

contents the present

however,

sight

completely

obvious

that

it

is

legitimate as the

or weighting

appropriate factors.
4.76 we might

to use logical

probabilities

To interpret refornulate the situation

the

measure

which

results,

something

as followst

A guestion division) universe"

determines of

a decomposition universes, of affairs

language is

(or

section,

or

the possible state

in to

"possible by where determined

under

we mean the

by

a "state-description" Thus an anwwer

the

consideration. equivalent

a question

a statement

to the the

disjunction

of the determined

state-descriptions by the question. is

in If

one of the the prob-

compartments definition

"L-range"

of probability

adopted,

)33
ft

ability relative

of

such

an answer of

will

be represented in the turns the

by

the

number or,

state-descriptions as we might in which say, it

appropriate number to be true. factor in

compartment; of possible

relative out

universes such

When we use an averaging as an average 4.77 There will

a probability the over Such the

as a weighting average class may thus of in possible perfectly it in

operation, taken

be regarded universes. legitimate. not*the

an averaco in for

be cases, one.

however,

which example,

appropriate

Consider,

a questionnaire

issued Let

to

each member of a population it has n possible probabilities

The actual are received will

of

some kind. p1, p2,41 "" pn answers

with however,

us suppose the
...

anzwero

and that
are which P(pl),

logical
P(pn).

of these
frenuonciec; depend,

these

answers

on the IfFIpl), logical but the

F(p, mixed

population ... F(pn)" of

concerned: . Now in

lot ordor

us write to

then

an the P(pj) of

calculate

content we want

each answor wo must average

employ the content will

if

to know the the

logical factors
content

answers
). Thus

received
for tho

weighting
logical

be the
the

avera6e

we have

expreeeion

11(pi) 4.78
a formula of. Wiener

logt

1'(p).

This
(rnf.

formula
4}3) in

has: affiliations
which ho refers

with
tar our

',

"P(pj)" as

as "apriori

probabilities"

and to (Wiener's In-Intl more irs

our

"F(p,

)" is.

"aposteriori

probabilities". over refers to the

formula probabiillty to this

actually distributions continuous affect

an integral and thus case thatr

particularly but cases, of is )"

the., not he

discrete; In both

does

general refer typo.

principles). to empirical In of this

howevr, the to

seams to froquency that for

probabilities connection the it "P(p,

relativesee formulae in of the. the

oauy in

a number

purpoooo

our

may be interproted sense of being

not

an "logical" calculated

probabilitioo in terms

probabilities

1ogi&al

form

of the

pj,

but

simply

as apriori

probabilities, in

i. e. as probabilities advance. 4.79 in accord with

regarded

an in

some way given

This Shannon's. the

interpretation In the context

is

completely of comnunioation concerned

theory,
are and

howeverzi
to

apriori

probabilities
frequencies confines for given himself him never" that

assumed since

be relative further

in to

advance;

Shannon it

ergodic and and that one

sequences aposteriori there quoted. is

becomes probabilities

axiomatic

apriori

"almost "mixed"

differ, as the

no need

for

formulae

auch

4.8 in our revis'e'd-'sense

When two of section

questions 4.72,

are the

independent entropy of

77-7-r,

das`

their
Thus if

join

iaiequal

to

the

sum of their

entropies.

QI and 112. are 11{0-1 + Q2)

independent H(Q1)

we have + H(Q2)"

In
be less than than the

general,
sum of the

the

entropy

of the
though Q1, Q2 C

join
greater

will

entropies, for any

either xQ1)

soparatgly. Q1

Thus + Q2) a

AQ + Ql)

H(Q1)

+ g(Q2)"

We have
Q`;

H(Q1)

H(Ql
case

+ Q2)
is that

if
in

and only
which

if

Q1 contains
ompty.

a particular

Q2 is

4.81_
ii(g1) As in + H(Q2) the theory over of

The quantity
H(Q.1 + Q2) communication,

which
is of

is

the

excess

of

special it

importance. "(Q1, Q2)"

N7o write

and define

R(Q1,
It Qi measures (or Q2 on

Q2)
as it vice were versa).

H(Ql)
the

+ H(Q2) * H(Q1 + Q2).

of dependence of

entropy

4.82
algebra section" however, defined noticed of the the to it the in joiin

Unlike
question-calculus parallel that-of

the

class-calculus
hays no concept

or Boolean
of "interof this, just be

"Join". to give

In the

spite.

seems appropriate name "intersection passing in of the two that it is

gaantity It might entropy of the a

entropy". related to

the

same way as the sentences is

probability to

disjunction
-- ------

related

probability

136

of their
4.83 justified all easily

conjunction.
The name "intersection by reference from join: to the following entropy" may be

properties, of the

derived of the

comparable -

proportion

information

For
(i) (ii) is empty 0< If Ql

any two questions

R(141" Q2) and Q2 are

Ql and
< or

Q2
A(Q1)" iS either

R(Q2,0-1) independent,

R(Q1, Q2) (iii) If and only R(Q19 Q2)

(This If includes relative

" Q1 "

0.
containo

if

Q2

H(Q2)"
opocial caeo are of Q2 empty). as (for

the

entropies

defined

example) wix
then we have R(Q1, Q2) H(Q1) g(Q2) The extensive in 4.72 terms of probabilities is Q2) P(P. P(P Qg) 1092 Qk) (in _ for fQ2(Ql) H4l(Q2), R(ql, of Q2)

H41

0'

H(Q1 + R2) - II(Ql)

expression the notation

section

above)

R(Q1?

j, k This Qxprossion, line that for

P(Pj)P(Qk) H(Q), has the form of an

"

average;

and the

quantity

averaged

is

the

logarithmic

measure

of

"dependence" over all

introduced possible

above in answers

section joi3ht
the as

3.81.

The average'ia
question probabilities

to the
are

the Q2, +and., -Q1 of the

weighting

factors

answers.

The same remarks

before
ac vvs.

apply

to, the use of logical

probabilities

an wr ighting

4.831

The quantity

R(Q1,

Q2)

corrspondo

with

Shannon's
If

"information
we imagine

rate"
Ql as the

in

the

case of a noisy
"What "What the is is the the

channel. transmitted received is

question question since

symbol? " and Q2 res the (-" symbol? --), or vice und allocate versa,

measure in

symmetrical with

probabilities in the channel, rate are

accordance the for quantity the

symbol

frequencies the

then symbol

R represents in which

information symbols analogies

per

ease

successive such

independent. in

(Vie shall chapter 5).

however

consider

further

4.84 chapter 3, we might is related way

Returning notice to in

briefly briefly

to

the

eubjest this

of sort by

how eloaely of might It is "correlation" be applied particularly characters,

of moasure considering practical useful for

measures it

the

which problems. of

to

statistical correlation

non-metrical

of

3$ 1

which example

we shall of

give

an example. ref. us 20a). suppose dependence eyes.

(Thin

is

a classical

Galton, Lot

we wish of

to

measure

the

degree such

of hereditary as colour of

some characteristic are grouped in

Eye-colours

n categories.
and their in the eldest

A number of observations
Sono, table of and the (matrix) results of

is
are

made on fathers
aummarisod P(E17k)* j parlance

a rectangular probability eldest is called

numbers of In

finding

a father k. table".

eye-colour otatiotical

whose this

son has

eye-colour

a "contingency

By adding
of for get values the P(Ei) fathers, for. the

up rows we should
probabilities up of columns

get

a not

eye-colours we should procedure

and by adding set. P(Plt) for the

a similar

eons.

The usual

in

calculating

a measure

of correlation
P(Ej)P(Fk) assumption the old which the

would
(an of

be to
estimate

produce of the

a new table probabilities the such

of values on the

independence) of some measures

and compare technique the This c-

new table as the

with

by means in old effect

" x2-test", between mean square

mean square is ab like of

difference the 3.631.

and new entries. of our measure

tgking section

value

4.841. "intersection entropy"

An alternative

is

to

compute

the

(QlI

Q2) _. k

1'(E

) log k2

F(EFk) --. _. .,...... PEEj)P(Pk)

I3'7

which joint

if

desired

may be "normalised"

by dividing

by the

entropy

H(Q1 "+ 92)

This gives a coefficient 0 occurs 1 occurs
that that

2 j, k
which

P(E jpk)
lies

logt

P(E jFk)
0 and 1. sense.

between in

The value The value

in the

for in
the of

independence the case of

the usual

complete
of the

dependence
is of uniquely prob-

sense by is

eye-colour the fathers

sons matrix

determined abilities by

(the or

a diagonal of rows

matrix,

can be made such

interchange

and columns).

4.842
appears fathers name is symmetry for count fathers

What we above called

as a . symmetry and loss with sons. That of the measure with

"sentence-symmetry"
respect to (--the

we call here ---)

"oign-symmetry" is represented Thus

appropriate respect athern but to to

as a

eye-colours. have blue-eyed

a tendency would blue-eyed'

blue-eyed positively; to

children for

so would

a tendency

have

green-eyed

children.

4.847
cut of of intuitive information information sometimes in

This
meaning, nay from react which

moasurc

hau a relatively
that statistical Certain of the if the

cleartheory theory advantages

and

illustrates on the derives. the For use

back it

night

accrue units.

fron

non-normalised, a succession

measure,

binary

example,

of values

wore computed

for

r , tccos , ively

finer

subdivisions

+D 'J

of

eye-colour,

their about --the

increase "fine the or

would structure" dependence

give, of

'. quite the

directly, hereditary only to.

information dependence the "broad"

whether

applied

categories

also

in

detail.

4.844 class
by

, that could
Gabor

This

is

not

the

only

measure work.

of

its

be used in
(ref... 20)

statintical
discusses

A paper

D. and A.

the

closely

similar

neasure
2j, k P(EJFk) logt of P( )` logP(EIFk) PCI; )rCPk) and connected with

called

"coefficient

depndence"j,

a measure

called

the

"diversity" VIM
j 1062 2(EJ) "generalined entropy"

Elaowhoro

the

use

of

the

Z
has been proposed,

(P(E3))12(1062
222). Our

p(Ej))n,
"entropy" and "inter-

(rer.,

section satisfy

entropy`, both

however,

are the

only

moacures

which

our containment

and additivity

criteria.

4.85

Although

our

theory

of

gquestions".

and the sphere

resulting of the

measures, "sentential

have been restricted calculus" in the sense

to that

the

we have not acteristic

introduced of the higher

quantifiers logical

or any similar languages,

char-

no special

(4 t

difficulty direction; section are likely 3.4

attaches

to

thoir the

generalioation methods of

in Kemeny

this as in

we may follow above. to Practical

applications, since in considering

however, the contents

be limited;

of Seneralisations infinities.

we continually

run up against

4.9 of expressiv-g in they our terms are

Some intoreat entropy--type of purely

attaches measures

to H(Q) of

the

problem Q2)

and (Q1, the

directly in which

logical without

terms, the

language of

defined.,

mediate

use

probability are
is the

concepts. definable
sense probability

Since in

the logical

probability terms,

concepts the problem

However, to in intro-

themselves
in a certain of the the

a straightforward concept of the entropy

one. perhaps

use

tends

obscure ducing conclude of H(Q)

rationale

concepts,

irrelevant this in

associations. chapter of with logical a brief

We shall account alone:

accordingly of a derivation B(Q1, q2)

terms

concepts

is

of course
of

derivable
the 1jo3i of Lot

from
the

it
two

so long
questions. then, content of its

as we have

the

concept 4.91 at. which c in Thus

us return, the

to of

the the

point question Answers to the

wo had simply

just of

dofinod the

terms

nunbor

possible

we have

C(Q) of binary

1002 n,

an approximation required to

smallest

number

questions

express

14z :1

in relative

an extensive content

binary measure

form.

Now let under

us introduce certain questions

a,

applicable namely

restricted which are

circumstances, such that Thus

as between the Ql; let other.

one contains let p2, qn. Q2 contain """

and let Q2 have

Ql have possible

possible answers one of n greater which pk4 to

answers ql, the q2, pk, than

p1, "..

pm and

Now each Pk In jo will

q3 implies general

some unique: have qj this

say inc. n,

we shall q's that than imply

and there lot xj is the the

be other of of the q's

izaply be xj.

pkj s Thus Pk

number number x, pk j is

possible content

answers of 0.2

Q2

iyn to

relative

j; the statement

and logt

Now we can relative over all to the cueution answerato to Q2;

define Q1 by Q2s

the

content this each in turn

of

Q2

averaging thus

quantity to

possible

answer

Q2 determines a subset of the of number

an answer anmrerw of the to

cal which

determines the for logarithm all

and we average in this subett

members

answers

to Q2.
0

Thus we define
n Q1(R2) that logt x

We easily

prove

CQl(Q2)
Now let

<0

02)

us fact

suppose the

wo have of

a finite elementary

languago,

and that

in

number

1,43

sentences speak

in

the

language

is

I.

In

this

case we can languaga,

possible language in number

of the
riU". are

"universal
This is

question"
the question

of the
whom of the

which

we denote answers

all

state-descriptions "possible C(U)

(statements 2N;

describing

universal"), a N. contains we can

accordingly

we have The univcroal

question

all deduce the

others,

Since

from

a state--description

truth
for with

or falsity
any our question above

of any sentence
Q we can definition. calculate Let

of

the

1anguate.
in

Thus

CQ(U),

accordance answers

Q have

possible

pl I p2,

" ..

pn;

and let

yi with

bo the

numbor of We find lo ('; yj 2

state-

descriptions

concictent C (U)
In terms of the

each p,. (1%N)

"
express

of

this

quantity Q; for 1062 1092

we can

H(Q),

the

entropy H(R)

qucation P(Pj)

wo have P(Pj)

j22

N-

1062YJ "
The cecond a sun over term all is equal to CQ(U) since s Thus it, is in effect

state-descriptions.

we have

H(Q)
This might have

w
been

c(U)
used

cQ(u)
definition of H(Q).

as the

Notice

that

it

has3something

of the

form

of an intersection

144

INFORMATION AND LANGUAGE

5.1 postponed the technical the question or in of

Up to of

the the

present exact theory

we have relationship of

largely between

"engineering" chapter

information theory (or

as,, presented theories) this

2 and the 3 and 4.

logical

chapters in

We must

now examine

relationship

more

debil. is of some importance technical because, theory that concepts; "semantic the engineering strongly there

Our enquiry despite very is disclaimers, often virtually seem to writers imply

on the

by their between (ref. are

terminology the 3) two that to

no difference Shannon of says

concerned. aspects problem". that in the

3.9 p. irrelevant 39 PP-

communication But Weaver of is "at (ref.

95-98)

suggests the

context

a communication least to

channel

technical

theory

a significant

degree"

also
5.11

a semantic

theory.
Relative says Weaver, to. the there broad seem to subject of be problems

communication, levels: at three

14g

Level

commu{jication LevelLB: symbols problem). convey

How accurately (The be transmitted? A: the

can the technical

symbols

of

do the How precisely (The desired meaning?

problem). transmitted semantic

Level meaning affect effectiveness

How effectively desired the in conduct C:

He says: "The semantic

does the received (The: way?

problem).
problems by are concerned

the identity, with in the interpretation compared --even when with the

or

satisfactorily meaning meaning a very deep

of intended is only

close approximation, the receiver, as of the sender". situation,

and adds: "This one degis of

and involved

speech"'. communicating However, in speaking of Shannon's "admittedly instance in the first which applies All, he goes on to say: a to problem problems "Part theory comes from of the significance levels that the fact of those signal of the Wand 0,

the with through

relatively

simpler theory, only new

can make use only out to be possible limitations arily apply

when analyzed discovered in the theory

accuracies A. at level

above, turn which Thus any

significance Level A disclloses levels

A necessat Level C. larger B But levels a part and at of the that the analysis comes from the fact at that this level overlaps the other suspect. a significant

possibly more than one could naively Thus the theory A is, to of Level at least degree, levels theory B and C". of also a

5.12
problem" (and

Weaver's
particularly of

conception
the

of the

"semantic
problem")

"effectiveness

is
..

little a
-- _ -_

vague,
_ ____--

and he does not

.., w.,,

attempt

to

go into

detail.

I"
In the at present least concepts context a model of we can be more of the "semantic 3 and 4. a definition least in the exact, theory" But of since in we terms

have of

our

chapters for at

althoughwe "logical context concept

have

provided

a basis content", language, of the

information of with a logical. those

restricted of this

the

relationship theory similarity is not of

technical

completely mathematical

obvious, form. 5.13 by

except

as regards

We shall considering it an artificially relatively is in

first

try

to

point context important

the in

analogy

restricted exact. the more More

which

becomes

for of as breaks

consideration, ordinary English. down,

however,

general language the seriously

context such

communication Here and that we would Weaver's

a natural that is

contend account

analogy

misleading.

5.2 large of set of "possible language.

Let

us

imagine

a finite specifiable

though in

very terms the

universes" For

a logical

simplicity to of be the section

we may take sentential 3.25

logical and in the they desired

language view of

concerned the theorem to

calculus;

we may take since (to any them if

possible are not degree

universes so initially of

be equally they

probable,

may be made so by "analyoing"

approximation)

further

into

equally

probable
`-

constituents.

In

such a

r47

situation

any

statement content in has senses the

of

the

logical the

language senses

has of chapter content reasons

an associated 3; in and any any

any

one of

question the for

similarly of chapter part

an associated 4. For

one of here most namely

obvious in the the

we shall measures theory, Thus

most

be interested to 3.94 p: those of

closely those for any

analogous of section

technical genoralioatione,

their and

we have,

statement

M(P)
and for any question Q with

1062 -V(P)
posoibl answers p1, p2' "0 Pnt-

xCQ) In both
probability of 5.21 radically of of distinct, possible

-G the
sense

PCpj4 lo62 "P( )"

P(Pj)

" logical
the act

these
in

cases
the

symbol
of

denotes
in

truth-frequency

universos. Now there ways in are: two which into the this different, technical picture. and concept One channel the terms physical

information these consists the

may be brought in modelling concerned,

a communication i. e. such that in

inside

universe' of

transmission

a message

can be describod

of the

language

concerned.

The other outside

consists

in

the

consideration and consisting messages=framed

of communication of messages in

the universe, i. e.

ab_ ut

the universe, of the these

a coded version consider

`..., r:..`..

language. in turn.

We shall

two cases

148 l

5.22 modelled conceived is the to inside system. it of

If a finite: Thus will

a communication universe, if. the it

system must

is

to

be

be a finitely concerned

logical

language of describing

be finite,

be incapable from

transmission consider

messages

an infinite of length, to

ensemble; a finite i. e. the number messages.

we must of from need to

messages symbols, ensemble. the -and

made up only of finite

elementary a finite be

Since passage

no bound to the

ensemble be thought of procedure

specified,

limit

might

present

no difficulty: yet assume that it

however,

as a matter made.

we cannot

has been

5.23 process,
sequence such

Let in
of

us considor, universe, which

tl, ...

than,

some "physical" a

our finite
N symbols being language

produces

at

times from

t11 respectively, set S19 to* SM

symbols

chosen will

an elementary to include

Our logical

be taken

statements

P(S describing Between hold;

P(Si2, cannot

tJ) of symbol certain tJ)

= 12

(i

- 1,
jQ1,

...
...

M;
N)

the these

occurrence statements p(il,

il

Si at time

tj.

logical inconsistent
two

relationships with
symbols

e. g. that
t occur unless at

is
(---

different such

the

same time

---);

relationships

express
There

apriori
may-also

knowledge
be constraints

of the process
of some sort

conceirned.
imposed by

what we might

call

"natural

laws",

e. g. Sl might

always

I4-q

be followed

by S21 whence P(S1,

we should

have

ti)
such the

D
apriori

P(S21 t j+1)
knaledge Quite any to

Ci ' 1 ...
be built into we shall the

N-1)

We assume the logic to

all of

language. with

generally, p in the

be able a logical of such

associate probability

statement and we take complete

language

P(p); to of the

knowledge of our

probabilities knowledge

be the prococo. us

extent

apriori 5.24 qk of

Now let statements which

consider describe

the

ensembio the process,

completely

i. e. which

specify

complete

time-sequencs

of

symbols.

Thus each qk will

p(Si, of ti distinct ), one for qk

be a conjunction
each value of j;

of N statements
the total will number be MN'. and with

(assuming

no constraints)

The q,, are each (If qk the 5.25 message of there are qk

exhaustive a logical additional

and mutually probability physical and are

exclusive; can

them are

be associated'. some of call the

constraints

inconsistent

omitted).

We. ehell

"message-descriptions". The logical description sx

content in

0(qk) terms

of of

a its

can be specified

logical

probability:
C(qk)

the with question possible

r

1062

P(qq) is the qk,

" message? ", we have

Similarly, i. e.
_.

if

Q is

"What answers

the

question

15-0

k It. of is. tempting the to-refer This of the our to. this might, messages sequence; model,

P(qk) logt P(qk)

as the suggest, cuuh that such "average however, P(qk) content that is the

messagesn... a sequence. q of. justified. in by

, we, have . frequency is-not

and in which are

an interpretation is only

there apriori of

one message ones. of the riori expectation 5.26 used the to The

and the quantity, ,

probabilities II(Q) is,

logical the to contents their

an average according it is our to

various logical with

messages, probabilities; regard Either

weiphted

contentthe question might be Q.

to

the

answer or

C(qk) per symbol

11(Q)

express of

content symbols per in

by dividing concerned of

by Ir, (or time is

number

the

message if

similarly

content

second

a measure

associated

with

the-tJ).

.503

We now asserts

these

measures

are

a generalised communication
entropy is the per content is the

version theory.
symbol per (or

of

tho measures

employed

by message

Summing-up,
per second) (or per

we might
as defined second) whose of

say:

by Shannon the question relations

symbol

"What embody

" in message?

a language

logical

the

apriori

knowledo. however, requires justification

Our assertion,

at a number

of points.

sr

5.31
of a possible ensemble All that

In objection whereas needs for more the to

the that

first our

place, definition is in for

let

us is

dispose for a

finite one. that

Shannon's be said finite than

an infinite connection is in an is

this

a definition sense since the to former the,: limit such introduce would to the

all

ensembles that for

important ensembles; case of

general latter any

infinite as a limiting in which order first the

may be derived circumstances But

under

passage to justify to

can "be justified. to the limit

in

a passage farther make it special the

we should on our

have

restrictions less general. considered to the

model;

and these necessary have been

When restrictions by Shannon presents

cases passage

applied,

limit

no difficulty.

5.32 restrictions. specified the; bare

a physical We assume,

Now let

us consider

some of these had nothing except represents

of symbols. specified still the has

Our message-ensemble about its statistical by the

produces it is not that

has so fur

characteristics fact that it

minimum implied
device however, which that laws",

a sequence completely the

by known. "natural more are than taken

i. e.

ensemble of it But

one member when known into (or with account. "stochastic") random processes This

postulates that

language

moans process. in

represents Shannon does

a "random" not deal

general;

he restricts

rsz
htmaolf to processes (a) (b) (e) (see 2.54 2.564) -

which

are

sections

stationary, orgodic, of the. Markov type,.

finally and ----(d)

5.33 are introduced the is of

doubly-infinite
The. order

in
in

time.
which these since until roetrictiona we cannot we have, at

some importance; to the limit Since

countenance least,

passage

a stationary

ensemble.

ergoicity

implicc

stationarity easily most

then by the

and since in order

the

the terms would

theory

of

orgodicity ensembles, (d),

in most tho (b) and

formulated convenient
(c), But

of infinite

(a), to be seam
of generality of finite

purpose of

would Markov in

be uervedt proceenen, the

development the

a theory to

permitting order given

restrictions

be introduced

above.

5.34
a constraint Thus for on the example

Each of theso
logical structure

restrictions
of our

represents
language.

stationarity of the probabilities; probability occurs

may be taken whence of the

as meaning the language that say, time that the this a

time-invariance must given tj represent group of to

statement at,

symbols the

commencing of at the

as equal

probability commencing

statement tj+l

same group

occurs

time

(provided

153

is falls

meaningful, within

i. e. the

that

the

time-shifted

signal Similarly

still the

allowable of

time-range). process statements than

definigg by

property

a-Markov of

can be introduced referring to number this is

specifying

independence separated vie may take

time-instants u (which

by more very

some finite than 11);

much less probabilities.

another 5.35

restriction

on the

The ergodic that of of the it provides, logical to

property in

is

of

cspdial for

importance-in the about of lach equality groups groups

a certain of

sense,

probabilities the in relative

statements

symbols

frequencies message. should feel "

as calculated of

a particular requirement

Thus remove about of

introduction any our

an ergodicity that a "relative of

apprehensions neglect in ergodic of

an engineer frequency"

might

definition one. In

probability

favour

an apriori the which of the such two

logical an, *lr

an infinite equivalent chosen for 5.36 restrictions developed ergodicity for

ensemble of

independently the of calculation zero The in theories etc. our model of (see of the

mambor-oequnc frequencies, members. of general, of these

is

except

a set

measure

of

"writing-in" can, in

various the

follow

concepts in Our

stationarity, 2) and that introduces the

references principle.

chapter assertion

no new questins

logical ication assertion accommodate in accordance theory

measures rieasures

agree is is

with thus

the

corresponding more than to

communan

scarcely

that`the within with

model it the

sufficiently of

general

models

a communication of Shannon.

system

requirements

5.37 -ac distinct from

If simply

we consider an information

ac

stern source,

of

connunication we find

again
can for Si the

nothing
express

new in principle.
two sorts of statement, the ta)"

If

our logical
'lp(Si,

lanuuaLm,

ta)'standing of symbol expressing 3i Sk and have

the at

sentence t,,

expressing and "p(S ,

transmission for the tj sots)

time

sentence (whore

recoption

of from

symbol different of

Sk at

time

may be drawn a finite question messages answers

finite

we still Let of

ensemble whose qk, are

possible are

univorses. descriptions

Q be tho

answers the

transmitted whose qi. -Then

and Q'

corresponding of received

question messages have

descriptions exactly R(a,

proceedingJ=x

as before 2 k, l

we shall P(gkqj)

2(gkq1) logt P(gq)P(ql)

for the of 5.38

the

intersection

entropy

same way as before entropy rate for the

Q and Q'; and in exactly we can use this as a definition of channel. we might notice in

"Noise", simply as a lack of

paocing,

appears

correspondence

between

"transmitted"

and "received"

messages.
,

Thus of communication the two

we. model, when a universe of but

the

technical by turn

thoory a logical out

inside

described

language, to be not

concepts analogous our

"information" identical; to embody

merely

provided our apriori

we suitably

organise

language

knowledge conclusion
of

of the

communication

system. the logical

The principal definition

than that of the

to be drawn is; that

is of greater

information

generality

tedbnical
5.39 however

theory.
This throws no light method of relating of the two theories

on the. theory

communication

with

regard
had by model

to Weaver's
to the consider

"Level
the

B";

because
of for only this the

vie have
symbols purposes

nowhere handled of of the

meanings system;

communication need

the

they events.

be considered enters

as constituents model only

physical

"Meaning"

at

a higher

level, the

i. e. at

the

level

at which

statements events)

are made about in the channel.

occurrence

of symbols

(physical

5.4 of relating ication

our torn) need

Let

us now turn

to the

second method a communof

(in coded wo

the two theories. outside

Her we imagine of discourse

system

the universe
capable the of

logical

language, about

transmitting In the this of

statements

universe. to

case the

make no explicit

reference

form

statements that it

in contains

our

language: elenentaiy

'we'shall. sentences

simply p1,

assume p21 """ Ph

and, logical

functions Now lot

ofthem. us, imagine channel, of that, there. events over the at is the transmitting

end of

our

communication, reliable them will for

an observer of the universe The informis im-

who makes and-encodes recipient ation plicit asks about in what

observations transmission. to in the is in

channel.

be-assumed the universe of

have

no independent except what

question,

the

logic

common language. there the between the

Now we can entropylogical

relationship massages content? Before

propertis'of information

channel

and -their

we can explicit

anewor about code

this details

question, of the

ho. vever, languaga,

we must the

be much more of

choice

messages,

and the

concerned.

5.41 to consider

It soma asses

will

be convenient ospecially

for

ua first rolations

in which

simplo

hold.

(i
communication observer i. e. until has takes

Firot,
place over

lot
the

ua suppose
channel of

that

no
tho

until the

made coxipieto in

obsorvgtions to transmit

universe,

he is This whether

a position will it is

a atatcfor euch obvious

description. of thepJ,

be a specification, true or is false.

The most binary

way of

coding

such

a message

as an n--digit

number whose digits

of the orclorod as squally units, pj;

ioprosent,
if and

in

order,
of the

the

truth-values
are

states it, per hae

universe

taken n binary

probable, or one bit

a lo3ical (i. e.

content-of the content

symbol

of the'iuiiversal
with Shannon's

quostion").
measure to such in any

This
sense

is
in

clearly
which

in, accord
the latter,

can, be"appliod ' 'required

an ensemble:

tho, other by the infinity

properties specificwe

are guaranteed of ,the 'ensemble to tend let If n wo our model. equivalence.

'get .

complete

5.411 -observations begins:

Thin

It should

is not

necessaxry, before

of courne, trannmiseion pari pasau.

that

be complete

we can imagine
of ,

them proceeding
that or at the least the

presupposes, a certain pj the however, any not is jth is

course,

observations that time the comes

are observation to in this impossible in the

made in respecting transmit case, to

order; complete symbol. that the

before

The basic message that

limitation code makes it

transmit

infoxnation

in '1.ncoiplete" or denial df

senso_of elemantary 5.42 inherent all of

involving sentence of (ii)

Affirmation the lanuuage. This case:

an

limitation let in us terms

is suppose of

also that

in

the

following are

observations elementary in which

describable or their

conjunctions but that the each

sentences they are

negations, arbitrary;

order

made is

and that

is, transmitbed ations .. transmit, -sentence, truth, . will units or is

as it

is

r1ade . it

If

the will

order

of

the

observto

ccmplotoly, for the each

ranciom, observation of that

be necessary

respecting aentence n e4ementary1 +'l092 plus

an oloziontary an index of r this n

number

falsehood.

-, Given

sentences, logt an against n/2 i

involve of

approximately entropy of the just'1 fact per

(n/2)

channel content allow of for

observation, unit. (We'nrrite if at

a logical of the but -here n to

ctoad occur,

that,

no repetitions any stage is not

number minus ,n that.

possible number

sontncee, already to the

n say the

the in

transmitted). information

Wo can embodied in

addition

observations
that

the

channel

messages

transmit
been to wasted.

the
made.

information
'Wo have

such-and-such thir3'a

observations information capacity of (iii) the is

have

assumed however, is 5.43 simplest of the

ditional

be irrelevant, Such wastage

and channel feature

a necessary

model. Now - let us conwidor

who

systera language. in

capable Lot us

of

communicating the form, sentonc, i. e. fora of

"oontonco say p,

suppose normal its

expressed of

disjunctive

as a disjunction

state-descriptionz;

logical by saying

can be completely each it poeaible appoars there in

and unredundantly state-description in are the expancion.

specified of the

language

whctlier

Given

n elementary and if

sentences these

2n state-descriptions,

we order

___

,,..

some way we can. code such . of that, jth the jth digit

p as a 2n-digit expresses in the the

binary

number, of absence

presence

the

statte-description

disjunctive..

normal

expansion. 5.431 , -binary digits the fora highest (and is the clearly This single content average not method of coding of involves: 2n

sentence that, is

the. language; can have than one bit).

9

whereas `ia n bits The

a sentonco loss since

rather

coding.

"optimum", to message

message-length Never-

is,, not theless is with again

adjusted it is beine of

according clear

probability. that

on any reckoning and for

channol'canacits unconnected

"wasted", the coding.

a reason

details

5.44
from is to the above or

In
examples. apprbxinate agreement for i. e. all that the

fact
If

a general
the the logical

principle
content entropy,

emerges
of thorn and rcoiver) a message,

equal be prior

massage

must

(botwoen

transmitter of the

on a Pro. pram concerned; such that

transmission

information list of coded (mertions veroiona

aGrocmont

on an ordered

need, be transmitted

are

of the
5.441 case, sent

answers.
Example where the successive answers to (i) digits a finite the message above of represents the messagg of such reprebinary can be a

successive

sequence

questions:

alternatively

as. a whole

Ito'

considered the (in 5.442 questions question. 'ing prior Join this of

aw a coded the

answer

to

a single of

question, the sequence

namely

individual universal Clearly

questions question). any finite

case

the

sequence

of

Saite

can thus Hence

be considered example (i)

as a single-finite by suppos-

can be generalised transmitter with this say the

agreement question p2, ...

(between Q. finite, PA

and receiver) n possible "mossage-question".

on a single xn: Wers p1,

We call

5.443 are arranged

If

the

answers only

to

the raecsage-question

and coded so that

answer are 1062 is transmitted:,

the number of the

approximately content the less. codinS in of which length the each equal content. message length, answers of 1062 n tho

appropriate binary question equally 5"L. 5 we could answer in This i. e. with to to digits

needed. n units but is

The 1oSical when all otherwise idoal

A is

are

probable,

By using approximate Q is of in coded the into digits) minimise of the

of

answers posoible (say a

situation a si3nal to the its

number would, the the their

binary fact

logical

expected. of

average various logical to

lengths with

messages

corresponding equAl length question Q.

answers, probabilities. the

weighting This content

factors expected of the

would

be equal

logical

/61

5,5
and. "logical" does about .. the not the provide theory theories

This. method
of

of relating

the

"technical"
again

information, of

howovor, Weaver's in

a justification of ','Level of

speculations view of

B" information, the aaoumptiono

restrictive

',nature be emphaeisld.

involved.

These-'Must 5: 51. cases between of

Firstly, ccmmunication the communicants Let

there

is of

in

ordinary'overydar port

no agroenent

this

on a "message-queStion". an everyday situation such

us imagine

fas

the

following:
that it is

I want
about of that to

to toll
rain. of In

a friend
order td

something,
dispose and coding,

e. g. for let rain

the

moment

questio4s my friend are (in

probability both

us assuno and fine

and I exhaustive

understand

that exclusive

weather and are

and mutually equally a logical

alternativeir this of means one; binary

some sense) will have

probable: content

that

my message unit.

5.52
stance involved is the in which no more circumstance been understood at simply I

Now there
could that in give one unit which between a particular saying it

is

one an& only

in

one eircumwhich that some provide

my massage of

a form

mea, age-entropy; already I in

had us that time

sense the code,

should

infornation e. g. by

and in rain

a particular and "no"

"yea"

for

for

fine

weather. but there

(The actual

code uood doos. not

of course

matter,

must be an assumption alternatives with

of two exhaustive regard in the to tho

and mutually wordo used, described). my friond

rain? ".

exclusive paralleling This would

auch proportion be possible, for

events if going
if he it no But

inotanco, ",fin it
`-3

had just
It aukod might the

askod mo tho
aluo quoction

quoation
of but that

to

bo po3uibluj explicitly

course, I hod I had

had not in and both hin

frond" done

Saco,

provided

he understood interpret of

conucquontly of thane

know how to cauco to are

n;y answer. ordinary to

atypical

conmunication. or

Aci a rule

we have

nay what

we want

Lay n ro

leas

fully

and Explicitly. questions it ib that but also,

And this au it

involves wuro,

not

merely what

nnswerinS questions 5.53 hero in is

euying

wo are anuiroring.. A second that be omphaciood

point

should

that,

at

least

in

tbo

oxnmploo

wo havo eonaidorod in order with a to mocuago-

cectionc

5.41-5.43,

there

in no room except of mccuu ; oo. of logical In

trivial nuntain entropy, definite that it

conco for the

nonuoncon

identification it in

coutont thuro in

even if

assumed that

uitwoya a to apocity unlace

"nancago-quontion",

wo should

lutvo

no Ae uonco of conformed for

ouch cuoutiona basic it

was ullowabio criteria;

to certain xanplo, e.

statistical a Markov it in

unions,

gonoratod whothor

prococa. cane that

Now

wo nay uoriously

concidor

the

d3.

practical markov 5.54 has already

sequences processes.

of

messages

are

to

be regarded

as

Shannon's been given (section short

answer-to 2.565): sequences

this

question so long

' At: least of letters

as we consider

relatively

or words,

the

! structure"
specified

of a language
statistically. about longer

such as English
But sequences; `, nothing and the may run chapters. even and

may be completely Shannon after mush mi3ht

be doubtful

all, less

when we consider than a sentence or will

1o iCRl count

Content,

as a unit, in a book or even

thread-of through

an ardent a number of

discussion

paragraphs,

pagos,

5.6 another
with ferred in the to

Here it result given

is

relevant (ref.
It This

to consider 40) in connection

been rethat to are

by Shannon
of English. 2.579). of

redundancy above the (ace.

has result it and

already

indicates is that vital there

measuring;

redundancy long which

English

consider

relatively

passages; a simple

many correlations cannot discover.

statistical

approach

An experiment ThO experimenter the and services says to of hire selects

may be conducted of EnSlish, test The

as follows: secures subject, subject if

a passage

a reasonably "Guess told the

intelligent first letter". right

guesses,

and is

whether

he is

or wrong;

re4.
wrong, guesses number second -end in of is he is asked to guess again, and so on until; down "Guess until he the the the now has letter numbers

correctly. of guesses

The experimenter required. and the Then

writes he says repeated

letter",

process'ie The

of, -the front theme, a code

passageis.

reached. of

experimenter one for this list each of

hirzi a list, of; sage. ' of Shannon

numbers, suggests:

the-message. Decoding, it is suggested, twin" with in of the would the tont be possible subject; in every

if

one could

find

an "identical subject such subject could identical a "twin" in cay the

moaning respect. to tho those

another Putting of the

first

circumstances

idontical

experiment the _as to first

as conducted, lottor" and first atop number, Provided and , the

oxporirlontor again" at the

"Guooe

"Gue8r, "twin" writing only with the

and so on, guess

exactly

before, tho

corresponding guocaod; of the

doom the that those the of

letter

and "twin" under

no on. wore identical fro.

reactions the test

identical circumtancoc, the list of

subject could

original

message

be rocovered.

figures. :1hon the pcrforiaod in practice, large greater and, the to fir: the 3t half list of of This of this o porimont obtained twos, to in hac and be highly

nwnbc ro onoo and

a :. urprising1S" few numbers

proportion than five.

ohoy3r3 it

redundant,. this

redundancy be at least

o:f English

au eutimatod

in

way appears

as hiGh arg f30^'4, and poaaibly

/6s
cis high

when 90%, as

long

passages,

are

used.

But

how

does. this give 5.61 i3 that the figures

method in

compare with the region The firnt.

the more direct

onos which

of 500%? thing statistical that we might approach notice simply wo

straightforward long
twenty length are the

does not
consider a very logical as it at

work for
passages

pacoagdo
letters ao far

of English.
long: this,

iuppoco
is. aurely

modorato content affects this

ac correlations and the the structure aubjoct

affecting of must Englich extend of

concerned, of

guefliiea> of this

tout back.

loa8t

number of

lo-Gtore length,

The number

ppoaible of of

passages

a. ieuming is 2720;

an alphabet which is

twenty-six the order there of

letters. of 10`7;

and a apace, and all could to the

pu n; agc. of

written this of passages

English number twenty bat

ever

were

scarcely collect practical the whole

add up to statistics

letters. long

Hence is one; does not it

letters

no mere slice

impossibility, field of the large of a

a theoretical language And of hero at

English "sample". "'sample"

constitute is

a oufficiontly to speak of

course all,

nonsense

unless

we are

thinking

a sample

taken
0o bold

from
as

some theorotIicRl
to lay

enoenblo;
of

but
tho

who would

be

down a definition

charactristica

of a t-heorctical
the centuries the

ensemble
statistics language

of

such a sort?
of a lanuuaCe

lAoreover,
must change

over

appreciably

as the

changes,

and the

ensemble

could

1"

not

be considered

"stationary". is defined in terms

Bhannon', s ,"redundancy" of, statistics in '.Shannon's-. been -alone 5 62 structure'which easily detailed : will have it is not language specifiable in rules; element the most in terms -such :as of_letters. experiment, any But = clearly

the'test above,

subject cannot have

as outlined' great degree

influenced-to,,

by-letter-statistics.

English

'has

aMost

statistically. of detailed many of ---

can be-specified ---gras natical

very

though

these

statistical some Perhaps

in. their

formulation. thing about

important that in

a natural of its

language, growth it has

however,

the

process the needs

become. closelyadaptwi"to

of

its

users;
from'their is

to the
mental

oztent

that

it

hardly can
The structure

be consideredof to those a

apart

processes. too it,, saying

language who have the

familiar brought difficulty Thus it

------ all up to in is

familiar though what that

--they that

been

may have structure

greatat

comprises. e have about

possible might

the

ioctationa o: in

language

be best

doscribablo

psychological that

terms,

rather of the structure'

than

statistical; would

and

a. description

of language

have to be similar. 5.63 '-Whatever the structure is, it Must

---:

y".

TV

`lr'Y"^Tfi.

"TcT

+am

fe"^^'""o'-.

mss-. -. w-i.

e. _.

-.

-_"

-_""

1Ei
betaken-into content
will.

account to be given.

if

a satisfactory It is
will

definition that
natural

of a machine
languages

is

conceivable
handle

one day, be, built

which

',

with

something

of, the

facility as coder
subject

of humans. and decoder

and his on'the well

Two ouch after the

twin".

machines ,might. beused

manner But of of Shannon's suspending sort, sceptical; giving the we it that test

"identical

while thaw .

judgnnt very such

possible at would.

achievements least to this

remain

extent without of the

a result concerned. at'first,

be imposoible,

machines which

many attributeD coem irrelovant to

a, quasi-human task of

kind

"cd, ini".

, 7 betwoe21 Laie, irifoiiai:

oioovor ion

ova Hunt

Still

d avr the by an

line

content

as indicated

oxpperiment
for artificial

of this,

sort

and logical

content;

defined

languagen. h'uppoio Shannon's rymbol letter since not, guess in in tho teat ubjoct woro making

his

; uo

ee oyribol of letter

by by

a mathematical a natural last ono;

language, and that had been

instead tho

previous

symbols, He would ae his next

"stop", have and his

"7 +5". in giving.

we feel, "12";

much hecitibion reasons with for

doing the

8o would content

be logical sentence.

ones,

connected

minimioing

of. the

But thorn

is

a hidden

fallacy

hro..

It

w p.:.,.

:. ,

16 3 8

might ofthe he that

be the

case

that of

the

mathematics

involved ad absurdum" Or it that might for

were

part

demonstration expect'

a "reductio

so that be the case

should

inc'onaistency. an, reasonably, the

he, suspec-Med,, -quito reason, ;

some

psychological

affecting

writer.

a mistake

would
"hunch" that right

be made at, this

that some other explain. hunches).,

point.
figuro -

', Or he might
would turn

simply
up, for

have a
no reason

he could in, his;

(And he might

be customarily

The, study the our other study studies of the: linGuistic of logical

of

lang-uge behaviour have

is of

no more its

than but

users: around

languages

bevolvecl

considerations. That issit is, ab it wore incidental in

here ariTin

that

,o .e, ::rental at

calculation (Quito . What

may be involved olaborate i. -0 rolelant

a mood euosi. at tiUoa)

dalculations tc is a breach so much

may be involved knowledge of the lo; icnl worse of or for

what; to

exnectr; or

and if arithmetical

one expocts rules,

grammatical tho rules. That does not

5.71 . of statistics are 5.711 definition including of a natter'of

expoctationo

are , imply or

not that

a matter they rules general writers type

automatically grammatical of

logical,

other

The Tanis contenth s been 46,

a ratheremore by terms

sketched 4). in

eevoxcal of the

Woodward

(ref'.

p)z.

: . 16q

of von at

calculations Neumann least

involved and Morgenstern to

in

the (ref. into

"theory 35). account

of This all

games"

of

reprosenta the known It wo have

an attempt of the

take in the

structure does not

message remove

computing other

redundancy.

however

difficulties

mentioned.
5.8 displays product claim results obtainable, to an attempt of the The theory. to treat but Mandelbrot (rof. 34)

-of language

as an evolutionary does although language with is not not certain are bridged.

human mind, logical

alternately Hence

define of

content.

extrememinterest the gap we are

concerning concerned

C%

LANGUAGE'AND INFORMATION

Our next d$scent from the formal

task logical

involves or

something mathonatical

of

plane

to

that

of common usage. of what it

this it is

We now ask: to bo of,

to

in

the

theory information?

1`a theory
To answer is, the in

purports
necessary sense.

namely

know what

information look at

the

ordinary of the

To do'this,; everyday

we must

usage

word

in

contexts.
.,

6.1 to disentangle It the is

Firstly, concept perhaps at all

it

is

wotth

while from that here, from to "let

trying that there since "knowing". him know". of

of. information rather of curious

"knovrlodge".. should be any is

danger quite

confusion activity is

"informing" Yet to

a'difforent of

"inform"

someone

something

(And there'-are
According grammar indifferently is to

similar
the

turns

of phrase

in most

languages).
our

simple-minded there is

metaphysics a sort or of

on which called

built,

commodity which

"information"

"knowledgoll

we can

i ., . --_-

,.

-..

. n., -.-,

.,.,.

,-

..; ff,. -. , . r ,,

t7!

transfer

from

place

to'placo

or

from

porson

to

person.

"Knowing"
a piece of

something
this

consists

in being

in

possession
someone

of
oleo

commodity,

and "informing"

is. like
6.11

giving

it

away to him.
But whatever into word is is that say that this "know" known used: it is "information" simple has is to true picture a peculiar may be, in logic

"knowledge" detail whereby or implied at

will all. truth

not For of

fit the

the

what word

always say that

presupposed somoono if it know speak turns it of is with perfect clearly

when the

knows out at

something

implies to

(and really to

false all).

we have It is

he didn't tnntradictory

practically -

"incorrect the

knowledge" term. "I

"incorroct

information" the word

appropriate ---

And one can use ... " --to

emphasis certainty.

know that

indicate

6.12
Plato that the

The notion
"commodity"

is

at
of

least

no old
will

as
not

theory

knowledgo

work.

Suppose someono "iota

--do I then

me know"

that

two and two

mako fivo 6.13 Platonic for all

know it? not of is the

We need doctrine these punts that to

course

accept

knowledge apply.

something

inborn

The Platonic

philosopher

tends

to

use

the word

"information"

for

everything

which

its not

7 f

r7z.

"knowledge". "Such mere -and-such information". that

He may even is true

use

it

rather all

slightingly: the rest is to

knowledge; no doubt theory knowledge. what theory . is

He would information not of

be concerned a theory there rather properly

emphasise of

"merely" need have bo no to

information, on this is

But

quarrel consider

score; the

wo shall is oven

whether of

describable

as a theory

information.

6.2 wotds forst a very "inform" point neat to and notice

Lot

us be guided in this

by

the

use

of

the The into

"infornation" is In the that

ordinary use does

apooch. not fit

pattern. between

particular noun

we should

notice verb

differences "inform".

"information"

and the

"Inform"
"tell". *+r else two that It represents

can be uccdhalmost
a relationship case. fact; someone There it

exactly

like

between tolls is

(typically) someone

people

and a fact is typo the of

something on the profound,

no logical false,

restriction obvious, or

may be true,

necec, iary, Some of somoti

contingent, categories to me that ... ".

contradictory are qualify ... " None, rather the or word "Ho however,

what-you-wvill. We should ---

these

atypical. a little informed are

es prefer informed that

"He blithely all ruled

me in

neriou6neea out.

completely

173 e

6.21 speak with of of any such "information". particular acts.

It

is It

not is

quite not

the

same when we connected any in set

necessarily g, nor with

act

of

informs

"Information" Thus no human type of it

can be contained subject fallt is or referred not object to

a book

on a shelf. Moreover, more of limited. also or

os implied. is as a rule to without consider (is say contained "Ise informed speak

even

the

Although incorrect usually that Yet it

contradictory the torn Thus make five proper to

information, implies two is truth.

qualification "The in information book)". two

and two quite

this

me that 6.2a information in

and two Iako

five". like the it "He gave is me the somewhere as in other its

When a phraeo that ... " is used, extent

position

between.

To a curtain with the "Ile informed

can be used ... sebno "; to but

synonymous applications more

me that

noun

"information"

impose

restrictive

connotation. The noun "information" the verb verb hau a correlate is it not is related not

"misinformation". in the same way to to phrase say

But the

"misinform" since me that me" is "an

"inform",

good rather

English the

"He misinformed

so-and-so" in itself. we

"He misinformed of

complete

When we speak refer rather to the "information" to any particular in

informed person's

portion", possession of which

thet act.

than

of

informing

74

he has been also has its

the

victim.

(Though

of use).

course There

"informed" is a similar

usual

participial

use for
different formed direct

"misinformed",
contexts,

but
as when or fact

it

tends
saying

to be used in
that someone i. e. is

rather
misin-

on some subject, reference to the

about or

something, set of facts

with

concerned.

When it
it has

is

used on its
reference

("You own
to

have been misinformed")

act of telling.

a closer

a particular

6.23
lines and we shall main point

Much more could

have to too of of certain other

be written
comments is to that

along
to

these

make. is

The first vory into above. that atised of all in

be made here compJicatdd theory

usage

complicated; the sort And, the this

be fitted we have wish to

accurately been considering

mathematical course,

no one would idioms There is of

srggest

ordinary way.

language

can be systemconvention are

now a well-founded ordinary words

"tedhnical by

language", and ways (To the words

whereby other for

taken in their traffic, use

scientists exact

specialists use within this

and redefined the is fields of

relatively

specialisms. since their these rather

some extent may later meanings,

a two-way into ordinary

come back though to them). word is to

with

technical meanings than

a large former

extent ones 6.24 to

become

additional replacing

the

completely

When an ordinary a technical vocabulary, it is

assimilated

almost

unavoidable

""

7_

that

its

meaning would concept

must like

be chi to think

in that

certain ho has

ways. simply as when to to

The taken one

specialist a vague takes

it made and "The "The case

more

precise, from f$om here here of

the

concept it in for is this the as

distance distance we might of

London" Marble of here us, London necessity the the Arch".

and defines Yet oven

speak "the

a change from enables is in

moaning, to for London"

vagueness

distance It Circus

a systomatie to say that

vagueness. Piccadilly it, or

example, of

instead for

a bus-ride that London are not

away from is a mile

and

avoids

the from use of

saying

so distant the but the

Thams. special the for point case 6.25 associations are loss

Those definition the

arguments its

against

within

own field; are is and to not of changed

clearly bettor

accepted ---

moanings Arch ---

everybody than of

Marble Castle

a better it is

reference relevant its habits. if the that amongst the word in the

Windsor

a now verbal If

candidate one does tricks is is the

study

do this meaning

-of

and special --confusion it

overlooked

common experience even oomotimos

unnecessary

created,

the
that

specialists
theta is that

themselves.
ample there opportunity already

And it
for is

would

be claimed
and certain the case

confusion, in

evidence of

confusion,

"information".

77m

; "'"-l

6.26 of of of linguistic course a word shall broad to usage acceptable to technical

We are in

not

interested like the the course unusual remaining object is

in present. of

all

the It

details is

a case in

that,

transfer and idiomatic "classified" it is

vocabulary, the and our main

uses into simply

be amputated categories. avail errors Certiin

ones clear;

due to broad in

conceptual generalisations The first "inform" that have

confusions. emerge is and one is their that there

immediately is

in

the

cave'

hand. between

an important It other is

difference not just, for of

"inform. a verb correspond. can

ation". and the ing more nouns or

course, verbs

a noun,

most

and when less replace is does not not

suitably or

bolstered

grammatically by them. in this Actually regard in

be replaced typical the act of

"information" since way the it (for act

quite

either, the denoted

denote that

informing, typically the also hadia

example) of of

"communication" --the but other in rather (but "I

communicating the act; of.

indirect atypical) communication".

object sense

of

"communication",

e. g.

Our point,
It is that the to even word.

however,

in =ore

logical.
without and We can never

"information" or suet of such "the acts

can be used of informing, or acts.

roference without

and act

presupposing speak say, of

an act

meaningfully obtain" about,

information of

we shall history.

some period

We speak

about locked same. of

it

just

as if

it

were

some kind got at it,

of but

commodity there not all ovon the speak '..

up so that Under these

we can't

circumstances since neither is is

we could of any order.

"a communication", place. the word But

no act there in

communicating act of informing;

takes yet

"information"

We begin
"information" as: uue the some of is a highly

to soo that,
Platonic of

after
word. It

all,

the word .
to doopite

begins ,

the

attributes

"kuowledgell

differences.

6.3 be considered

Lot

us ask directly: Is it

can information something and from

so,

as a commodity? from
sugar or

that person
i

can be transferred
to person like

place

to place
And if

boeawax?

how iQ it

created

and where does it In the forit

eventually place, it

go to? is certainly
might call are need speak distinction_ so insub-

the

sort

of

thing

that People

a classical but built because in it and

economist sell it. it,

a commodity. even not of whole worry

and there But this

businesses us unduly, -services"

around

economists one breathes

habitually without

"roods-and. even

and will stantial with thing the

on occasion

admit

as a commodity We are but phys ica. not rathor

a thing economics

as "goodwill". of information, as its

concerned with some-

better

described

i..

6.31
that is I its give dietinguishs reproducibility. three but to if to the

Now there information If milkman a piece as I that it is to like I

is

one

thing Groceries,

in

particular and that

from have

five two

oranges-and I have I non can give as well, being is is in by

and eat of

left; it but in

I have

information and use it

as many people only thing of

myself still There

the

can

stop

me from of .memory.

possession

a lapoo this

principle nature

no limit infinitely It

process;

information

copiablo. it not copiablo in just the way that

a book is a photostat as I have,

in which

copiable,

or a picture; he does not (though

I

for

if

I give

someone

of a book, but
he has

have the there

should

same book a sense

course the book information apart

only
the

a copy

is
of in

same book).

be reproducing when would only I reproduced be the from

any information the book, but

contained tho

reproduced original,

. -=e

information

as the

errors.

6.32
interesting the of and "physical" communication quantum is (but

These considerations
somewhat properties and in the For irrelevant) of information, fields example, or in more of

could
speculations both

lead

to
about

in

the

field

thermodynamics wci can ask appropriate whether language

mechanics. reproducible; of

"entropy" whether

reproduction

a signal

involves

an entropy

179:

change, In the

whether formor case

in case the

the it

physical does not,

sense or not

or

in

Shannon's. in

necessarily; formulated is enough to

Shannon's and perhaps indocate

question moaning. in the

os inadequately This alone

without

a difficulty

'! commodity"

concept.

For Shannon,
only . , about however, in it. is the sense that this of

information
is a useful of

is

a "commodity"
speaking, in a channel,

way of

The measure a symmetrical and from matter for all. that in

rate

information

one as between the mathematical direction whether

transmitter ppint of view

and receiver, it does not or at

which matter

the there

information is any physical

"flows", process 6.33 information (so in and far

We might is'tather as we know) the like

say having

that the

having measles,

a piece which thing of

of is

an infinitely

reproducible Measles is is a form

something its

same sense. correlate,

disease;

physical nea: le would

the

virus,

reproducible. a "commodity",

Now the since in jars it

virus

might

be considered possible to to

be perfectly it from can to

pack

measle But

virus although at

and transfer tho fron diseaso,

place

place.

neaslea, least it ,

certainly another,

be transferred the question has of

one person

whether

can bo considcre& and academic

as; a "commodi, ty" ri4g.

a faintly

comic

Igo

6.4 in. -the way of

However a commodity

there theory,

is

another that

obstacle of the somewhere, to goes to,

namely must

information

source.

A commodity But it is

come from difficult where in it

and go somewhere. say whore except (for information the

extremely from, or lout the

comes if laok it

latter) When I

is of

channel and

"noise". eay

out

window

"It's

raining",
It

havo I created
is difficult

a piece
to

of information?
ancwor of to give,

know what to the

at

any

rata which I

so long would could of to

as one adheres be in accord with

kind

interprettheory. source

ation

the

technical

Certainly, for of I the

be considered cone calculation pebeoh.

arg an information involving But in

purposes

transfer nenne me, and

information act purely touuhod sound

another

another reached by

as a transducer. ny, outotretched waves. Up to

Light hand. a point, in I

rays

water certain coding cortical

reacted the

emitting of impulses, larynx.

process nerve of the the

can be followed activity,

through

detail; motions

corresponding $o the ultimate

cource rain? there in

of

information

that 6.41 here

it

wan raining

was the However

a. conceptual

confusion

as before. me : ay "It

Let is

us

consider

a counter-examplo. may load to a listener toll with else

Hearing a certain in error

raining",

disposition that it is

and bad hearing not raining. The

someono of

same chain

events,

I81 'H j
nervous impule-es to and cortical this, as before, it could the serve activity but it nor is is and the not it the like, same version of this is can

be invoked piece of, it. it if of

describe

information (Of course playing

a coded version , but

as a coded

wo were

"bpposites" of game Is to confitoo the of

irrelevant). "of logical

The nictalke content The latter, of the former, with in

traneraiosion )hysical

the

trarsmicdon is any

signals. condition .-one.

sorge form, but not to

a necessary dcrroe a sufficient

Technically,

the

problem

of

the

' courco"

can be solved aiway within

by koeping the context

the

concept

of

"information" channel under

of the particular is all, outsictb to then

consideration. of information and the kind

But if which is

this vilid.

havo no definition 'Awe

such a context, ask about

is simply

of question
An information

we want

it

cannot

be answered. that

source i. o.

an -thing that i. e. emits almost

can be considered that

an such,

anything

anything

can bo condidered

as a signal,

anything

at all

6.5 to in use the word

It

is

abundantly in its

clear sense

that

if

we are

"information" or in to

as current equivalent

ordinary

usage,

an ppproximately have This some kind must not of

sense,

we shall in

have mind.

linguistic

communication

be interpreted

as ruling

out

the

simpler

kinds

of

("pre-linguistic")

communication drawing or pictures

such

as pointing, similar

beckoning, kinds of

smiling,

and

"sign-language", and the sane time, in like the

as ruling

out

pictorial for language.

representations At the

as a substitute case to of theso to

rudimentary make any their

languages even

no one would syst. cmatic except will

pretend

be able

approximate

statements a secondary

about

information different

content, sense which

in concemi

and rather

us in
6.51 content good

a moment.
Some of as defined approximations as applicable criteria detailed and if of the properties language of the of arc logical undoubtedly

fchr a symbolic to proportios in with metrical a natural which

"infozmation" viz. the claim

concept deducibility that at all; the

language; wo started. are

We cannot applicable be rival this is

properties there could

they the

were, sort in

scarcely But

definitions a familiar which

wo have the

described. of

phenomenon definite" of it

case

a technical one, technical to be said non-technical "pa(,,-,ca,

concept

"makofr

a non-technical must is in rest on the

and the applications. for the

justification At the

same time property;

there

something io far

add. itivity usage chapters 6.52

as in

information ...,.

is etc. "

sometimes

measured-in

lines,

In

saying

that

inforiiation

is

measurable

only

in

respect

of a language

we must be careful

to realise

153r

that the the

this technical

is

not

what

is

being

assumed "Lot iss not the Sl, in

when, S29

intuoducing ... Sn be of word of present the

theory,

one says That we are it with

fundamental in

symbols". a sentence connects etc. in is the

speaking the

information in

using

any way which words such

statistics From our

letters, standpoint 6.53 possible concepts apart fron

sentence.

a use

entirely chapter the only (in

secondary

and derived. the

When in relationships of information, a very spacial

5 we diecueiod logical kind of and case 5.45),

between the

"technical" we found, in which

case

section

there of

was any close 5.3ff

relationship the

between logical
of

them was that could be

section
to

in which
the in

theory
ctatcments

applied concerning atic their the the to

measuring the events

contents the the but

and questions no systemand

channel. entropies

We found of

relationship logical entropies corresponding emphasise of that

between contents, of messages

messages

only

rolationships logical It talking

between contents is of

and the

messap-e-c1e_; criptions. so long as we are

now important logical

about including described

content those section

a language, by means are

uhannon' of the to of

c measures, experiment our the problem. two

obtained 5.6,

in

irrelevant.

The confusion easy$ almost and in some cases that it even is this

concepts to avoid. the

is It

fatally might

difficult confusion,

be said

cyotematic

18

ambiguity much of

of the

the

word

"information", of the theory

is

the

source -

of'

interact

as a whole.

In obvious content defined 1arGp the for else lettors should that of at a distinction passages all in of terms to for natter since

the

simplest, needs to in

cases

it

is-sufficiently Thus the

be made. so far theory, and apart simply pages as it

English, of their

can be by from and

Shannon's length; we might sentences,

is

proportional

allowance for at that all,

redundancy words, there is

count or

letters

anything to treat content other hand

no absolute units).

reason

as the be in

fundamental to

Tha t lo-gical would on the

proportion

length

be accepted'. by hardly
If of the form "The it such

anyone.
other in band'we consider oentencee. y of book

on the letter

position that

x on page the logical

z is of will to

an S, ", of

becomes sentonces

obvious

content passage

a set have the

describing approximately Concerned. to the

u given in

a logical th of the that

contents passage is

proportion is of thin the

len,

And it entropy

logical passage

content as defined

related

by Shannon. 4 Thus the remarks 5, can of Weaver, into a pas is quoted porDpetive. ago of

6.55 at the beinnin, thinks (or of that of

chapter the logical soft

be put of

VIoaver English

content of

some other

message)

somothing

iss
of but the same kind with as entropy, allowance for calculable by similar process 115). of a methods; of

simply

an unspecified (ref. is 39 p.

"semantic" We have different concerned. 6.6 of the

encoding shown kind that from

and decoding "logical entropy

content" of the

something or

messago

passage

If logical concepts this cot

we ach about to the case they

the of act

applicability a natural language,

there uid

can be only o to a useful

answer: of concepts;

as a rough analogy.

a loading

No one pretends has any accurate to everyday

that

"logical

probability"

(cf,

"L-range") application

or even approximate

numerical

statements.
by tho (and doducibility

Yet such properties

conditions conditions

as those
additivity' entropy

o:.-pressed condition type noa

and the for represent concept in It that their is of

the

corroepondiug to the

uroa

applicable truths in fhe about future

"questions") ordinary usage

appro:: iriate and nay where it well

information favour

alter

does

not fact

already about way in But

correspond. information which secondly it

as it is

were a

an interacting oimplo 6.61 there is this consistent

there

can be conceivedl. and most concept concept to which importantly of for the information comparison technical

answer: the

the

logical

can be oat in order to

againnt

"technical" the extent

demonstrate

! 86

concept Technical perhaps

perverts

the

meaning

of

the

word

"information". though this

uagge: can now hardly does not mattor becauLs

bo changed; in the limited

context

of the more is
of the

major

applications thaiaithe

of the convention
one,

technical that
derived

theory

nothing

needed
word is

the meaning
from the

a "technical" but eaconelary.

correct

meaning

6.62 the logical

in

3iiailax concept
fact,

ccondary be made in
conputo

applications other fieldc;

of

could
one can

wherovQr,

probabilities.

In

fact that

ono conclusion in any field

or postulate

cugGestod whore it i

by what powoiblo

wo havo Said to measur,

is

compute be found

probability application

coofficiontothere of the concept

will of

an approxiuato

information.
almost probability of the scale.

Given

the
all

definitions,
that in

in
needed

fact,
to is

this
nonvort

is
a

a triviality; into

an information Information content

content is

a transformation

(sufficiently transfor is

nearly)

probability.

And information

correlation,
ensemble there

or very
is

like

it.

Wharever, there
and hence

is

an
in this

a'quostion",

a content

sons: 6.63

too.

But only,
sense.

we nust

re-emphasise,
tempted,, in some

in

a secondary

And when we are

context

whiabz is

close

to

th context

in which

the

187

primary getting information is in or

concept to grips which to to

is

relevant, the

to essence world

think of

that

wo are

really the

with

information, the is

makes

the

go round, ordinary are in life

corrective lived

perhaps a world forgotten.

thinly which

how much of science count is there

and logic the in words

irrelevant a book, but phrase? may

We can

how much information And even not lanuuago in

an apt

turn There

of

can be done a nod --

without.

may or there in

be one bit

but

how many are

a smile?

/36.

REFERENCES

General

reference

to: be made may on information theory and its theory,

Proceedings of conference London, 1949. Symposium, on communication London 1952.

(The

applications,

second of the above includes Reference by F. L'. Stumpers. below is indicated symposium

a bibliography to papers of by "Symp").

prepared this

1.
2.

Bar-Hillel
Bell E. T.

Y. and Carnap R.
"Development of

(Symp).
mathematics" 1940.

3. 4.

Birkhoff Birkhoff

G. D. G.

Proc, "Lattice

Nat.

Acad.

Sc.

17 (1931)

650. p.

theory"

1940.

5.
6. 7. 8. 9. 10. 11.

Brillouin
Brillouin Calder R.

L.
1.

J. App. Phys. 22 (1951) pp. 334,338.

J. App. "Science in Phys. our to 24 (1953) lives" p. 1152. 1954. 1942. 1950-

Carnap R. Carnap R. Carnap R.

"Introduction "Logical "Meaning

semantics"

foundations and necessity"

of probability" 1947-

Cawsey G. F. "Physical entropy and the entropy theory" of info rmation R. A. E. tedh. note

G.169 (1952).

12.

Cayley

A. Phil. mathematical

Trans. papers

(1858) p. 17: v. 2 no. 152.

collected

1$

13. 14. 15. 16. 17.

Eddington A. S. and electrons" Tech. Fano R. M. of electronics, Feller Fellgett W. and its

"Relativity 1936.

theory

of

protons Lab. theory Trans.

report no. 65, Research M. I. T. (19+9). to probability i, 1950vol.

"Introduc#ion applications"

Phil. P. B. and Lin foot E. H. (1955). 931 247 A vol. no. ser.

Frechet !'Methodes des fonctions M. arbitraires. des evenements dans le cas Theorie en chaine d'un nombre fini d'etats 1938possibles" Gabor Gabor D. Phil. Mag. 41 (1950) p. 1161.

18. 19.

D. "Communication theory and cybernetics", Symposium on electronics and television, Milan 1954. D. and Gabor A. ser. A, 117 (1954) S. "Information J. p. Roy. 31theory" and the Statist. 1953. weighing of Soc.

20. 21. 22. 22a. 23. 24.

25. 26.

Gabor

Goldman

Good I. J. "Probability evidence" 1950Good I. Hartley Hopf

Jelonek Kemeny

J. R. V.

Biometrika
L. Bell

40 (1953)
System

p. 237J. 7 (1928). der Vol. 5-

Tech.

E. "Er godentheorie", Mathemat ik und ihrer

Z. J. G. (Symp). J. Symb.

Ergebnisse Grenzgebiete,

Logic

13

(1948)

p.

16.

27. 28. 29. 30.

Kemeny J. Kemeny J. Keynes J.

G.

Zr. Symb.

Logic P.

18 (1953) Phil.

p. Sc.

289. 19 1921.

(1952) p . 307M. Koopman B. 0.

G. and Oppenheim "A treatise Bull. Amer.

on probability" Math. Soc. 46

(1940)

P. 763.

,1

31. 32.

Koopman B. Kolmogorov

0. A. N.

Ann.

Math.

ser.

2,41 of the

(1940) theory

p. of

269.

"Foundations

probability" 3334. 35. 36.

Eng. tr.

1950.

Loeb J.
Mandelbrot

(Symp).
B. (Symp). ' "Theory 1947. dissertation: of

0. Neumann J. von and Morgenstern behaviour" games and economic Riemann G. F. B. , Privat-docent

D. E. Smith see 1939, p. 411.

K. R. "Logik Mind

"Source book in mathematics"

Forschung" P" 193. 1935.

37. 37b.

Popper Popper

der

K. 'R.

56 (1947)

37c.
37d"
370. 37f. 38. 38a.

Popper K. R. p. 173.
Popper
Popper Popper Russell Shannon

'roc.

Roy. Dutch Acad. Sc. 51 (1948)

P. 275.
Sc. Sc. 5 (1954) 6 (1955) outlook" J. 27 P" 143. P" 51-

K. R.
K. K. R. R.

Mind 47 (1938)
Brit. Brit. W. Bell J. J. "The PhiL Phil. scientific

B. A. C. E.

1931.

System

Tech.

(1948)

Pp. 379 and 623,

39. 40. 40a.
41. 42.

Shannon C. E. and Weaver W. theory of communication" Shannon C. E. Szilard

Whittaker Wiener

"The mathematical 1949. J. 30 (1951) p.

P.

Bell Zeits. f.

System Tech. Phys.

Gazette

p.

50-

L.
E.

53 (1929)
29 (1945) extrapolation time-series"

840.
137. and 1949.

Math.

"Interpolation, N.. of stationary smoothing

43. 44.

Wiener

N.

"Cybernetics" "Unconscious 1953.

1948. origin of Berkeley's

Wisdom J. 0. philosophy"

fL

45.
46.

"Tractatus Z. Wittgenstein 1922. Eng. tr.

Woodward P. theory M. with "Probability applications

logico-philosophicus"
and information to radar" 1953.