Equivalent Test: MAP Test: January 2012 Detec9on & Es9ma9on, SP, IIT Delhi 1

Equivalent
Test: MAP Test
Compute a-posteriori probabili9es Pr(H0/R), Pr(H1/R),

Pr(H2/R), and choose the largest.
Thus, Minimum Pe criterion Maximum A Posteriori

Probability Test (MAP).
Generaliza9on to M Hypotheses case is straighLorward.
January 2012 Detec9on & Es9ma9on, SP, IIT Delhi 1

Degenerate Tests
Case 1: Combine H1 and H2:

Then C12 = C21 = 0;
For simplicity, let
C01 = C10 = C20 = C02 and Cii = 0, i= 0,1,2.
Then rst two par99oning equa9ons reduce to:
H1 or H2
P11(R) + P2 2 (R) P0
Else H0
The third equa9on becomes an iden9ty (equality).

Degenerate Case 1: Decision Regions
2(R)
H1 or H2
H0
1(R)
Since all cost eects of a decision between H1 and H2 are

eliminated, we have a Binary Problem.

Case 2: Dummy Hypotheses Techniques
Actual Problem: Binary with Hypotheses H1 and H2.

Calcula9ons some9mes simplied by introducing a dummy
hypothesis H0 with zero probability P0.
We let: P0 = 0; P1 + P2 = 1; C12 = C02, C21 = C01.
The rst 2 equa9ons imply that we always choose between H1
and H2. P (C C ) (R) P (C C ) (R)
H 2
1 12 22 2 1 21 11 1
Else H1
H2
P1 (C21 C11 )
(R)
P1 (C12 C22 ) Else H1
Where
= p(R
(R)
/H )
p(R / H )
2
as expected for binary problem
1

Dummy Hypotheses Case
May appear absurd!
Useful when p(R/H2)/p(R/H1) dicult to work with.
1(R) and 2(R) can be made simple, by a proper choice of

p(R/H0).

The General Gaussian Problem
Binary Case
Many prac9cal applica9ons: p(R|H1) and p(R|H0) Gaussian.
Important to study this special case in detail.
Deni9on: A set of r.v.s r1,r2,.rN are jointly Gaussian if all their
linear combina9ons are Gaussian r.v.s.
Deni9on: A vector r is Gaussian random vector, when its
components r1,r2,.rN are jointly Gaussian r.v.s.
Thus, if
N
z = gi ri = gT r
i=1
is a Gaussian r.v. for all nite gT, then r is a Gaussian vector.

Characteriza9on of Gaussian Vectors: A Review
Characteris.c Func.on :
Deni9on: M ( jv) = E r [exp( jv r)] T
Recall that for a scalar Gaussian r.v. x, the characteris9c

func9on is
M x (jv) = E[e jvx ] = exp(jvmx x2v 2 /2)
T T
Let
E[r]=m,Cov{r}= E[(rm)(rm) ]=, L et x = v r, m x = v T
m
jv T r
M r ( jv) =E[e ] = E[e jx ] = exp( jmx x2 /2)
= exp( jv T m v T v /2), x2 = E [v T (r m)(r
m) T
v ] = v T
v


Probability Density Func9on (pdf)
Given the characteris9c func9on, the pdf is given by

1 jv T R
p(R) = N e M r ( jv)dv
(2 )
1 jv T R
= e N exp(( jv T m v T v /2)dv
(2 )
1 jv T (R m ) v T v / 2
= N e e dv
(2 )
1 & 1 1/ 2 T ) & 1 )
=
(2 )N
(2 (
exp' j v 1/ 2
(R m))( j1/ 2
v 1/ 2
(R m))+ ( 2
*.exp' (R m) T 1
(R m) *dv
+
1 & 1 T 1 )
= N / 2 1/ 2 exp (R m) (R m) *
'
(2 ) ( 2 +
The General Gaussian Problem
When p(R|Hi) is a Gaussian density on all hypotheses.

Hi: r is a Gaussian random vector specied by its mean vector
and covariance matrix.
! E(r1 | H i ) $ !m1 $ " %
$ K11i K12i . . K1Ni '
# & # &
E(r | H
# 2 i & #m2 &) $ . . . . . '
E{r | H1} = #. & = #. & = mi Cov(r | H i ) = Ki = E{(r mi )(r mi ) | H i } = $ . . . . . '
# & # & $ '
. . . . . '
#. & #. & $
#" E(r | H )&% #"m &% $ i
K N1 i
. . . K NN '
$# '&
N N N
Dene Qi = K i1

" N/2 1/2 $1 ' 1 T *
p(R | H i ) = #(2 ) Ki % exp ( (R mi ) Qi (R mi )+,i = 0,1
) 2 ,

General Binary Gaussian Problem
LRT gives
1
(R) =
p(R | H1 )
=
1/2
2 {
exp (R m1 )T Q1 (R m1 )
K0 }
Say H1 else H0 .
p(R | H 0 ) K 1/2 exp 1 (R m )T Q (R m )
1
2 {
0 0 0 }
Taking logarithms
1 1
(R m0 )T Q0 (R m0 ) (R m1 )T Q1 (R m1 )
2 2
1 1 Say H1 else H0.
ln + lnK1 lnK 0 = *
2 2
Test: Dierence of two quadra9c forms: Basic Result: Many
applica9ons special cases of this general result.

Case A: Equal Covariance Matrices
Let K = K = K and
Q=K 1 but the means are
1 0
dierent.
Test reduces to
H1
1 T
(m1 m0 ) QR ln + (m1 Qm1 mT0 Qm0 ) = *'
T
2
Let
Else H0
m = m1 m0
Then H1
l(R) = m T QR = RT Qm *'
Else H0

Case of equal covariance matrices
Result: similar to Example 1: m1j = m = constant, m2j = 0.

Performance completely characterized by d: distance
between the means of the two hypotheses, when variance
normalized to 1.
Dene 2
d 2
=
[ E (l / H1 ) E(l / H 0 )]
var(l / H 0 )
We have

E(l / H1 ) = m T Qm1, E (l / H 0 ) = m T Qm 0
var(l / H 0 ) = m T Qm

Case of equal covariance matrices
It can be seen that

2 T
d = m Qm
Performance for this case determined completely by this
quadra9c form.

Further Insight: Independent Components with
Equal variances
Thus 1
K = 2 I,Q =
2 I

2 T 1 1 2
d = m . 2 I.m = 2 m

Or m
d=

d corresponds to distance between the two mean
vectors normalized by .
Sucient
Sta9s9c: 1
l = m T R
2
Dot product of R and the mean dierence vector

Independent components with equal variances
m1
m0

A2: Independent Components with Unequal
Variances
Thus #1 &
#12 0 . 0& % 2 . . 0(
% ( % 1 (
% 0 22 . .( % . . . .(
K=
%. . . 0 (
,Q =
% . . . .(
% (
N2 '
% 1(
$0 . . %$ . . .
22 ('
N
m i2
d =
2
2
i =1 i

dierence components weighted inversely propor9onal to
variance along that co-ordinate.

A2: Unequal Variances
Alterna9vely d2: distance in a new co-ordinate system.

d 2 = (m' )T (m' )
$ 1 '
& m1 )
1
& )
.
m' = Q1 / 2 m = & )
& . )
& 1 )
& m N )
% N (
m i ' = m i / i
R
i ' =
R
i /
i changes the scale on each axis, to
1 / 2 normalize all variances to one.
R' = Q R

A2: Unequal Variances
Sucient Sta9s9c:
N
m i Ri
l(R) = 2
i =1
i
T
l(R') = (m') R'

A3: General Case
As seen earlier
l(R) = m T QR
2 T
d = m Qm
Valuable Insight: By looking at this dierently.
Simplicity of A1 and A2: stemmed from Diagonal covariance
matrices.

Can we interpret the SS in a co-ordinate system, where the
components of R are s.i. random variables?
Possible to nd such a co-ordinate system?

Co-ordinate transforma9on for diagonaliza9on
of covariance matrix
r2
r 2 r
1
r1
r3 3

Diagonaliza9on
New set of co-ordinate axes: 1, 2, . N: orthogonal vectors

s.t. iT i = ij
Let r: observa9on in the new co-ordinate system, with
ri ' = r T i = iT r;
r ' = Wr;
Required to choose the new co-ordinate systems s.t.
T T
cov{ri 'rj } = E{ (r m)(r m) j } = iij
i
T
K j = iij
i

Diagonaliza9on
Will require that i be the solu9on of
Ki = ii,i =1,2,...,N
Eigenvectors of K: the basis func9ons; Eigenvalues: Variances
of the components ris.
Eigenvectors normalized to have unit norm:
T
=1
i i
Some well known proper9es:
Since K is symmetric, eigenvalues are real.
Since K is a covariance matrix, eigenvalues are non-nega9ve.

Diagonaliza9on
Eigenvalues are roots of N-th degree polynomial equa9on;
For dis9nct eigenvalues, corresponding eigenvectors are orthonormal.
For an eigenvalue of mul9plicity M, corresponding eigenvectors are
linearly independent.
Mean dierence vector in new co-ordinates:
$1T '
& T)
& 2 )
m' = & . )m = Wm
& )
& . )
&% NT )(
January 2012
Detec9on & Es9ma9on, SP, IIT Delhi 23
Sucient Sta9s9c in New Co-ordinates
Sucient Sta9s9c given by

l(R') = l(R) = m T QR = (m')T W T
QW 1
R'
= (m')T (WKW T ) 1 R'
%1 (
' 0 . 0 * % r1 ' (
' 1
*' . *
T ' 0 . . .. *' *
= (m')
' . . . 0 *' . *
' 1 * 'r '*
'& 0 . 0
N *) & N )
N
m i ' R'
=
i =1
i
and N
(m') 2
d =
2
i =1
i

Summary &Performance
Reduces to case A2 in a co-ordinate system which whitens K.

Always possible to nd such a co-ordinate system.

Summary: Case A
Sucient Sta9s9c: A random variable which is a linear
transforma9on of the observed vector R.
Performance for any threshold sewng: Governed by d2.
Self-Reading Exercise: Example on Page 105 from Van Trees.

Case B: Equal Mean vectors
Let m1 = m0 = m
Then LRT becomes:
H
1 T 1 K1 1
(R m) (Q0 Q1)(R m) ln + ln = *
2 2 K 0 Else H 0
Without loss of generality, can assume m =0.

H
Thus T
1
l(R) = R QR 2 * = ',where

Else H0
Q = Q1 Q0
l(R): Dot product of R and QR: Not Gaussian r.v.

Special case B1: Diagonal covariance matrix on
H0 with equal variances
Thus H 0 : K 0 = n2 I Typically when ri = ni
H1 : K1 = K s + n2 I Typically ri = si+ni
Hence 1
Q0 = I,
n2
1 # 1 & 1
Q1 = %I +
n2 $ n2
Ks(
'
Convenient to write
1
Q1 = (I H )
n2
H1
1
l(R) = 2 RT HR '
n
Else H0

B1: Equal variances in diagonal entries under H0
Where H can be variously wriyen as:
2 1 2 1 2 2
H =( I+Ks) Ks =K ( I+Ks) = (Q Q1) = Q
n s n n 0 n
Proof:
I H = n2K11;
H = I n2K11 = K11 (K1 n2 I) = K11K s
= (n2 I + K s ) 1 K s

Subcase B1a: Uncorrelated iden9cally
distributed signal components
si independent with iden9cal variances:
K s = s2 I
2 2 s21 2
H = ( I + I) I = 2
s n 2 I s
s + n
N
1 s2 1 2
2 Ri
T s 2
l(R) = 2 2 2 R R = 2 2
n s + n n s + n i=1
Alterna9vely, we have
N H1 2

l'(R) = Ri ''= '. 2 ( n + s )
2 n 2 2
i=1 Else H s 0

B1a: Performance Calcula9on
l(R): Sum of squares of N Gaussian r.v.s with variance

n2 + s2 : H1
n2 : H 0
Let N: even (for simplicity)

Start by
considering characteris9c func9on of ri2:
jvR i2 1 R i2 / 2 n2
M r 2 ( jv) =
i

e
2
e dRi
= (1 2 jv n2 ) 1/ 2

B1a: Performance Analysis
Since ri s are independent r.v.s, we can write

2 N /2
M r / H 0 = (1 2 jv ) n
Taking inverse transform

#N &
% 1(
$ 2 ' L / 2 n2
e L
pl / H 0 (L /H 0 ) = N / 2 N ,L 0
2 n (N /2)
= 0,L < 0
: 2 (chi-square) density func9on with N degrees of freedom.


For N=2, reduces to exponen9al density func9on.

Similarly #N &
% 1(
$ 2 ' L / 2 12
L e
pl / H1 (L /H1 ) = N / 2 N ,L 0
2 1 (N /2)
= 0,L < 0
12 = s2 + n2
1 ( N 1)
L / 2 12
PD = [2
''
N /2
1N (N /2)] L 2
e dL
1 ( N 1)
L / 2 n2
PF = [2
''
N /2
nN (N /2)] L 2
e dL

For N=2, same result as discussed earlier. (Slides 67 and 68).

For the general case: let
N
M = 1, ''' = '' /2 n2
2
M
''' x
PF = 1 0 e x dx
M!
Integra9ng by parts: M
( ''') k
PF = exp( ''')
k =1
k!

Similar expression for PD , with replaced with iv = /212.

B1a: ROCs
ROC plots:
Trade o
between N and
s2/n2

Subcase B1b: Independent signal components
with unequal variances
# s2 &
Here % 1
0 . 0
(
0 . . .
Ks = % (,
% . . . 0 (
% (
$ 0 . 0 s2N '
# s2 &
% 2 1 2 0 . 0 (
% n + s1 (
% 0 . . . (
H =% (
. . . 0
% (
% s21 (
0 . 0
%
$ n2 + s2 N
(
'
Characteris9c func9on easy to compute, but PF and PD dicult.

B1c: General Case: Arbitrary Signal Components
This can be always reduced to case B1b by an orthogonal

transforma9on.
Exercise: Work out the details of the orthogonal

transforma9on.

Cherno Bound: Basics
Many cases: easy to derive the test, but exact performance

calcula9on impossible.
Bounds only way out to calculate, PF, PD and Pe .
Cherno bound: convenient and useful, when it is possible to
compute/es9mate the moment genera9ng func9on of a r.v. x.
Objec9ve: To compute or bound P(x a).
Dene y = u(x a),

Hence e sx e sa y, s 0
E{e sx } e sa E{y} = e sa P{x a}

and
Cherno Bound
It follows that P{ x a} e sa E {e sx }, s 0
P{ x a} min e sa E {e sx }
s0
min exp{sa + ln E{e sx }}

s0
Can be made 9ght by choosing the best value of s.

Applica9on to ROC calcula9ons

Dene (s) = ln e sL(R )
p(R / H 0 )dR

[ln ( R )] s
= ln e p(R / H 0 )dR

& p(R / H1 ) ) s

= ln ( + p(R / H 0 )dR
' p(R / H )
0 *

s 1 s
= ln [ p(R / H )] [ p(R / H )]
1 0 dR


Tilted Density Func9ons
Tilted probability density

The 9lted family of density
func9ons obtained by
mul9plying px(X) by esX for
various values of s>0.
Normalized to have unit

area.
Tilt in this case: mean value

shied to right.

Proper9es of Tilted RV
Let x and xs and be the un9lted and the 9lted rvs.
e sX pl / H 0 (X / H 0 ) e sX pl / H 0 (X / H 0 )
px s (X ) = =
e ( s)
e sL
pl / H 0 (L / H 0 )dL

Xe sX pl / H 0 ( X / H 0 )dX
Now E{x s} = Xpx s ( X )dX =

e sL
pl / H 0 (L / H 0 )dL

d(s)
= (s),var{x s } =
= (s)
ds
Bounds on PF and PM
PF in terms of the 9lted variable xs:

PF = p l / H0 (L /H 0 )dL = e [ (s)sX ]
pxs (X)dX

=e (s)
e sX
pxs (X)dX

For s 0 and X : e-sX e-s;

Hence
PF e[ (s)s ] px s (X )dX ,s 0

e[ (s)s ],s 0

Cherno Bound on PF
To get the best bound, we minimize RHS: dieren9ate the
exponent and set to zero to yield
(s) = (A)
Because 0
(s) a solu9on will exist if
(0) ()
must be to the right of E(l/H ): always
Implies that threshold 0
the case.

(A) implies: choose s so that mean of 9lted variable xs is at the
threshold value .

Cherno Bound on PF and PM
Then bound on PF
PF exp[(s) s (s)],s0
Bound on PM
We have
X [ (s)+(1s)X ]
pl / H1 (X /H1) =e pl / H0 (X /H0 ) =e pxs (X)

PM = e ( s) e (1 s)X
px s ( X )dX


Cherno bound on PM
For s 1
e(1s)X e(1s) forX

PM e (s)+(1s) pxs (X)dX

e (s)+(1s) ,s 1
Once again bound minimized if (A) sa9sed for s 1.

(1) = E{l/H
} this requires the threshold to the le of
1
E{l/H1)

Summary
We have
PF exp[(s) s (s)]
PM exp[(s) + (1 s) (s)]
0 s 1
= (s)
With the threshold lying between the means of l on two
hypotheses

Graphical Interpreta9on of Exponents
Let s1 be the solu9on of (A) Tangent at s1 intersects the

ver9cal lines at s =0, and s =
1.
Value at s = 0: Exponent in
PF bound.
Value at s = 1: Exponent in
PM bound.

Special Case
P0 = P1 = (equally likely hypotheses);

Hence = 0
To minimize Cherno bound, choose s so that
(s) = 0:sm (say)

Hence 1
Pe = (PF + PM )
2
1 (sm ) 1 (sm ) 0
e
2
px s (X)dX + e
2
px s (X)dX
0
1
e (sm )
2
Assignment #3

Mandatory Submission: Problem Nos.
2.6.1 2.6.2 2.6.3 2.6.6 2.6.7
2.7.1 2.7.2 2.7.3 2.7.4 2.7.9
To be submiyed on 6.3.17
Try to solve as many addi9onal problems as possible.

Equivalent Test: MAP Test: January 2012 Detec9on & Es9ma9on, SP, IIT Delhi 1

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Equivalent Test: MAP Test: January 2012 Detec9on & Es9ma9on, SP, IIT Delhi 1

Uploaded by

Copyright:

Available Formats

Equivalent

Test: MAP Test

Compute a-posteriori probabili9es Pr(H0/R), Pr(H1/R),

Thus, Minimum Pe criterion Maximum A Posteriori

Generaliza9on to M Hypotheses case is straighLorward.

January 2012 Detec9on & Es9ma9on, SP, IIT Delhi 1

Case 1: Combine H1 and H2:

The third equa9on becomes an iden9ty (equality).

Since all cost eects of a decision between H1 and H2 are

January 2012 Detec9on & Es9ma9on, SP, IIT Delhi 3

Actual Problem: Binary with Hypotheses H1 and H2.

January 2012 Detec9on & Es9ma9on, SP, IIT Delhi 4

May appear absurd!

Useful when p(R/H2)/p(R/H1) dicult to work with.

1(R) and 2(R) can be made simple, by a proper choice of

January 2012 Detec9on & Es9ma9on, SP, IIT Delhi 5

is a Gaussian r.v. for all nite gT, then r is a Gaussian vector.

Recall that for a scalar Gaussian r.v. x, the characteris9c

January 2012 Detec9on & Es9ma9on, SP, IIT Delhi 7

Given the characteris9c func9on, the pdf is given by

January 2012 Detec9on & Es9ma9on, SP, IIT Delhi 8

The General Gaussian Problem

When p(R|Hi) is a Gaussian density on all hypotheses.

January 2012 Detec9on & Es9ma9on, SP, IIT Delhi 11

Result: similar to Example 1: m1j = m = constant, m2j = 0.

January 2012 Detec9on & Es9ma9on, SP, IIT Delhi 12

It can be seen that

January 2012 Detec9on & Es9ma9on, SP, IIT Delhi 13

January 2012 Detec9on & Es9ma9on, SP, IIT Delhi 15

dierence components weighted inversely propor9onal to

variance along that co-ordinate.

January 2012 Detec9on & Es9ma9on, SP, IIT Delhi 16

Alterna9vely d2: distance in a new co-ordinate system.

January 2012 Detec9on & Es9ma9on, SP, IIT Delhi 17

January 2012 Detec9on & Es9ma9on, SP, IIT Delhi 19

January 2012 Detec9on & Es9ma9on, SP, IIT Delhi 20

New set of co-ordinate axes: 1, 2, . N: orthogonal vectors

January 2012 Detec9on & Es9ma9on, SP, IIT Delhi 21

Sucient Sta9s9c given by

January 2012 Detec9on & Es9ma9on, SP, IIT Delhi 24

Reduces to case A2 in a co-ordinate system which whitens K.

Self-Reading Exercise: Example on Page 105 from Van Trees.

January 2012 Detec9on & Es9ma9on, SP, IIT Delhi 25

Without loss of generality, can assume m =0.

January 2012 Detec9on & Es9ma9on, SP, IIT Delhi 26

January 2012 Detec9on & Es9ma9on, SP, IIT Delhi 27

Where H can be variously wriyen as:

January 2012 Detec9on & Es9ma9on, SP, IIT Delhi 28

January 2012 Detec9on & Es9ma9on, SP, IIT Delhi 29

l(R): Sum of squares of N Gaussian r.v.s with variance

Let N: even (for simplicity)

January 2012 Detec9on & Es9ma9on, SP, IIT Delhi 30

Since ri s are independent r.v.s, we can write

Taking inverse transform

: 2 (chi-square) density func9on with N degrees of freedom.

January 2012 Detec9on & Es9ma9on, SP, IIT Delhi 31

For N=2, reduces to exponen9al density func9on.

January 2012 Detec9on & Es9ma9on, SP, IIT Delhi 32

For N=2, same result as discussed earlier. (Slides 67 and 68).

January 2012 Detec9on & Es9ma9on, SP, IIT Delhi 33

January 2012 Detec9on & Es9ma9on, SP, IIT Delhi 34

This can be always reduced to case B1b by an orthogonal

Exercise: Work out the details of the orthogonal