Estimation: parts of Chapters 12-13

Wiener and Kalman Filtering

Natasha Devroye
devroye@ece.uic.edu
http://www.ece.uic.edu/~devroye
Spring 2011

Summary

Wiener filtering

!"#$%&"'()*+,--$."'()*/&%0"1$"-'
!"#$%&"'()*+,--$."'()*/&%0"1$"-'
!"#$%&"'()*+,--$."'()*/&%0"1$"-'
Terminology
for
three
different
to
the Wiener
filter problem
Terminology
for three
different
ways
toways
castWiener
thecast
Wiener
filter problem
Terminology
for three
different
ways
to cast
the
filter
problem
!"#$%&"'(
!"#$%&"'(
!"#$%&"'(

+,--$."'( /&%0"1$"-'
/&%0"1$"-'
+,--$."'(
+,--$."'(
/&%0"1$"-'

Given:
![0],
![1], ,
!["]
Given:
![0],
![1],Given:
, ![#-1]
Given:
![0],
, ![#-1]
Given:
![0],![0],
![1],
, !["]
Given:
![0], ![1],
,
![#-1]
![0], ![1],
,![1],
![#-1]
Given:
![1],
, !["]
Given:
![0],
![1],
, ![#-1]
Given:
![0],
,![1],
![#-1]

[0!
],, 1$$[][#
1],!!
1] # %!]Find:
Find:
$[0], $$[[10],],
!$[,1$],[$#
!![ #
Find: Find:
Find:
!
1], $[ #Find:
[,# %#"% ]0!,[ #%#"%0] , % " 0

Find:Find:
$[" ] $[" ] Find: $[" ]

!["]

!["] !["]

2
1

$[0]
$[0]
$[1]

2
1 3

!["]

!["] !["]

3
$[0]

"1

" 3

"

13

!["]

!["]

"3

1"

2
"1
3

$[0] $[1] $[2] $[30]] $[1] $[2] $[3] ![5]

$[0] $[1] $[2] $[3]

$[1]

$[1]

$[2]

13

2
2
4 51 " 3
3 4 5

![5]

4" 5

"

![5]

All three solved

usingsolved
General
LMMSE
Est.LMMSE Est.
All three
using
General
All three solved using General LMMSE Est.

$[2]

$[2]
$[3]
$[3]

Note!! Note!! Note!!

!["]

!1
!1
! $ 3
3!22
1 3 22 2
$23!!2
! $
!23
!2 3 22 2

$[3]

http://www.ws.binghamton.edu/fowler/fowler%20personal%20page/EE522_files/EECE%20522%20Notes_28%20Ch_12B.pdf

11

11

11

Wiener filtering

!"#$%&"'()*+,--$."'()*/&%0"1$"-'
!"#$%&"'()*+,--$."'()*/&%0"1$"-'
!"#$%&"'()*+,--$."'()*/&%0"1$"-'
Terminology
for
three
different
to
the Wiener
filter problem
Terminology
for three
different
ways
toways
castWiener
thecast
Wiener
filter problem
Terminology
for three
different
ways
to cast
the
filter
problem
!"#$%&"'(
!"#$%&"'(
!"#$%&"'(

+,--$."'( /&%0"1$"-'
/&%0"1$"-'
+,--$."'(
+,--$."'(
/&%0"1$"-'

Given:
![0],
![1], ,
!["]
Given:
![0],
![1],Given:
, ![#-1]
Given:
![0],
, ![#-1]
Given:
![0],![0],
![1],
, !["]
Given:
![0], ![1],
,
![#-1]
![0], ![1],
,![1],
![#-1]
Given:
![1],
, !["]
Given:
![0],
![1],
, ![#-1]
Given:
![0],
,![1],
![#-1]

[0!
],, 1$$[][#
1],!!
1] # %!]Find:
Find:
$[0], $$[[10],],
!$[,1$],[$#
!![ #
Find: Find:
Find:
!
1], $[ #Find:
[,# %#"% ]0!,[ #%#"%0] , % " 0

Find:Find:
$[" ] $[" ] Find: $[" ]

!["]

!["] !["]

2
1

$[0]
$[0]
$[1]

2
1 3

$[1]

$[2]

$[2]
$[3]
$[3]

!["]

!["] !["]

3
$[0]

"1

"

!["]

Note!! Note!! Note!!

!["]

2
2
2
2
1"
"1
13 4 531 4" 53
"3
3
$[0] $[1] $[2] $[30]] $[1] $[2] $[3] ![5]
![5]
$[0] $[1] $[2] $[3]

" 3

!["]

13

4" 5

"

![5]

$[1]

$[2]
$[3]

All three solved

usingsolved
General
LMMSE
Est.LMMSE Est.
All three
using
General
All three solved using General LMMSE Est.

!1
!1
! $ 3
3!22
23!!2
1 3 22 2
$3
!23
! $
2
!2 22

http://www.ws.binghamton.edu/fowler/fowler%20personal%20page/EE522_files/EECE%20522%20Notes_28%20Ch_12B.pdf

11

11

11

Wiener filtering
!1
! " "!! " !!
!

Filtering

Smoothing

! " "[! ] (scalar)

! " # (vector)

# $
" % #"[! ] # $

Prediction

! " '[ ) ! 1 ( ( ] (scalar)

"'' " % '[ ) ! 1 ( ( ] ! #

" %$"" [! ]! $"" [0]&

# $
" % # #( # ( & ) $
" % # ## ( #& $

"~
$""T (vector! )

" % "" (Matrix! )

"~
$''T (vector! )

"'! " % "[! ]! #

#

#
" % ###

( && #

" '' " % (# ( & )( # ( & )#

#

x '()*s!

"!! " % #! #

#
" % ###

" %$'' [ ) ! 1 ( ( ]! $'' [( ]&

" '' " % (# ( & )( # ( & )#

( && #

# $

" '' " % !! #

" % ''

" % "" ( % &&

" % "" ( % &&

"[! ] " ~
$""# ( % "" ( % && ) !1 !

# " % "" ( % "" ( % && ) !1 !

# !1
'[ ) ! 1 ( ( ] " ~
$''
% '' !

%1) ) &% ) ) ) &% ) )1&

% ) ) ) &% ) ) ) &% ) )1&

%1)( ! (1)&%( ! (1))( ! (1)&%( ! (1))1&

12
http://www.ws.binghamton.edu/fowler/fowler%20personal%20page/EE522_files/EECE%20522%20Notes_28%20Ch_12B.pdf

Wiener filtering
!"##$%&'("%()*+&$,*%-.()/0(1*$%$,
#[$ ] ! ~
,##! ( 0 ## " 0 "" ) #1 2 ! 3! 2
$
!!
!#!!!
"
3!

4 ! & ( $ ) [0] & ( $ ) [1] ... & ( $ ) [$ ]

! $% $

% $ #1 ... % 0 %

#[$ ] !

& & ( $ ) [ ' ] ([ $ # ' ]

' !0

Wiener-Hopf Filtering Equations

( 0 ## " 0 "" )4 ! ,##
$!
!#!!
"
0 ((

,## ! $)## [0] )## [1] ... )## [$ ]% !

Wiener Filter as
Time-Varying FIR
Filter
Causal!
Length Grows!

)(( [1]
)(( [$ ] ) , & ( $ ) [0]) , )## [0])
&
, )(( [0]
*
'
* ) [1]
)(( [0] & )(( [$ # 1]'' * & ( $ ) [1] ' ** )## [1] ''
* ((
!
* %
'* % ' * % '
%
'
%
*
'
*
'
*
'
)(( [$ ] )(( [$ # 1] &
) [0] & ( $ ) [$ ]'( + )## [$ ](
+$
!!!!!!#!!!!((!!"( *+
Symmetric & Toeplitz

In Principle: Solve WHF Eqs for filter 4 at each $

In Practice: Use Levinson Recursion to Recursively Solve

13

http://www.ws.binghamton.edu/fowler/fowler%20personal%20page/EE522_files/EECE%20522%20Notes_28%20Ch_12B.pdf

Kalman filtering

Borrowed heavily from the excellent notes

http://www.ws.binghamton.edu/fowler/fowler%20personal%20page/EE522_files/EECE%20522%20Notes_29%20Ch_13A.pdf
http://www.ws.binghamton.edu/fowler/fowler%20personal%20page/EE522_files/EECE%20522%20Notes_29%20Ch_13B.pdf
http://www.ws.binghamton.edu/fowler/fowler%20personal%20page/EE522_files/EECE%20522%20Notes_29%20Ch_13C.pdf
http://www.ws.binghamton.edu/fowler/fowler%20personal%20page/EE522_files/EECE%20522%20Notes_29%20Ch_13D.pdf

Kalman filtering
Rudolf Kalman developed in 1960s
discrete-time and continuous time versions
used in Control systems, Navigation systems, Tracking systems

Background
Wiener filter: LMMSE of changing signal (varying parameter)
Sequential LMMSE: sequentially estimate fixed parameter
State-space models: dynamical models for varying parameters

Kalman filter: sequential LMMSE estimation for a time-varying parameter

vector that follows a ``state-space dynamical model (i.e. not arbitrary
dynamics)

State-space / dynamical models

System state: variables needed to predict system at future times in absence
of inputs (i.e. what you need to keep track of)
Example: constant velocity aircraft in 2-D

Vector Gauss-Markov Model

Thm. 13.1 (Vector Gauss-Markov Model)

Example - constant velocity 2-D aircraft

Ex. Set of Constant-Velocity A/C Trajectories

Red Line is Non-Random
Constant Velocity Trajectory

14000

12000

Y position (m)

10000

8000

Acceleration of
(5 m/s)/1s = 5m/s2

6000

# ! 1 sec

$ u ! 5 m/s

4000

rx [ "1] ! ry [ "1] ! 0 m

2000

v x [ "1] ! v y [ "1] ! 100 m/s

0

2000

4000

6000

8000 10000
X position (m)

12000 14000 16000

18000

12

Have state model, now observation model

Have a state-space or dynamical system model for the desired signal
Need a model for the noisy measurements and how it relates to the states
(depends on how we acquire the data)

Estimation problem

Scalar state, scalar observation Kalman

Scalar state, scalar observation Kalman

Lets derive this!!!

Some observations
dynamical model provides update from estimate to prediction
in the Kalman filter, prediction acts like the prior information about the state at
time n before we observe the data at time n
must know noise and initial state distributions and dynamical and observation
model

!"#$"%&'(#)*+,&-."#"+&-)")*&&/&&-."#"+&012*+3")(4%
-)")*&546*#:

s[n] $ as[n # 1] " u[n]

012*+3")(4%&546*#: x[n] $ s[ n] " w[n]

9%()("#(:")(4%:

2
u[n] WGN; WSS; ~ N (0, ! u )

w[n] WGN; ~ N (0, ! n2 )

Varies
with n

2
2
2
572)&!%48
572)&!%48: :%%s,s,!!s2s, ,a,a,!!u2,u,!!n2n

s[#1 | #1] $ E{s[#1]} $ % s

M [#1 | #1] $ E{( s[#1]} # s[#1 | #1]) 2 } $ ! s2

;+*6(.)(4%:

s[n | n # 1] $ as[n # 1 | n # 1]

;+*6<&5-=:

M [n | n # 1] $ a 2 M [n # 1 | n # 1] " ! u2

!"#$"%&>"(%:

K [ n] $

?@6")*:

s[n | n] $ s[n | n # 1] " K [n]& x[n] # s[n | n # 1]'

=2)<&5-=:

M [n | n] $ &1 # K [n]'M [n | n # 1]

M [n | n # 1]
! " M [n | n # 1]
2
n

13

http://www.ws.binghamton.edu/fowler/fowler%20personal%20page/EE522_files/EECE%20522%20Notes_29%20Ch_13B.pdf

!"#$"%&'(#)*+,&-*.)/+&0)")*&&1&&0."#"+&234*+5")(/%
0)")*&</=*#:

4[n ] $ ;4[n # 1] " :9[n ] 4 p ! 1; ; p ! p; : p ! r; 9 ~ N (78 6) r ! 1

T
T
234*+5")(/%&</=*#: x[n] $ > [n]4[n] " w[n]; > [n] p ! 1

B%()("#(C")(/%:

w[n] WGN;

~ N (0, % n2 )

2
<94)&!%/A:
<94)&!%/A:((s,s,??s,s,;,
;,:,
:,>,>,6,
6,%%n2n

4[#1 | #1] $ E{4[#1]} $ @ s

&

'

<[#1 | #1] $ E (4[#1]} # 4[#1 | #1])(4[#1]} # 4[#1 | #1])T $ ? s

D+*=(.)(/%:

4[n | n # 1] $ ;4[n # 1 | n # 1]

D+*=E&<0F&Gp!pH:

<[n | n # 1] $ ;<[n # 1 | n # 1];T " :6:T

!"#$"%&I"(%&Gp!JH:

![ n ] $

<[n | n # 1]>[n]
% " >$T !
[n]!
<!
[n#| n!#!
1]!
>["
n]
2
n

1!1

KL=")*:

4[n | n] $ 4[n | n # 1] " 

$!!#!!"
[ n|n #1]
$!!!
!#x!
!!!
"
~
x [ n ]: innovations

F4)E&<0F&Gp!pH: :

<[n | n] $ B # ![n]>T [n] <[n | n # 1]

14

http://www.ws.binghamton.edu/fowler/fowler%20personal%20page/EE522_files/EECE%20522%20Notes_29%20Ch_13B.pdf

!"#$"%&'(#)*+,&-*.)/+&0)")*&&1&&-*.)/+&234*+5")(/%
0)")*&</=*#:

4[n] $ ;4[n # 1] " :9[n] 4 p !1; ; p ! p; : p ! r ; 9 ~ N(78 6) r !1

234*+5")(/%:

A[n] $ @[n]4[n] " ?[n]; A M !1; @[n] M ! p; ?[n] ~ N(78 >[n]) M !1

C%()("#(D")(/%:

4[#1 | #1] $ E{4[#1]} $ B s

<94)&!%/?
<94)&!%/?: :''s,s,>>s,s,;;, ,::, ,@@, ,66, ,>>[n]}
[n]}

&

<[#1 | #1] $ E (4[#1]} # 4[#1 | #1])(4[#1]} # 4[#1 | #1])T $ > s

E+*=(.)(/%:

4[n | n # 1] $ ;4[n # 1 | n # 1]

E+*=F&<0G&Hp!pI:

<[n | n # 1] $ ;<[n # 1 | n # 1];T " :6:T

*
!"#$"%&J"(%&Hp!MI: ![ n] $ <[ n | n # 1]@ [n]+ >[ n] " @[ n]<[ n | n # 1]@ T [ n] (
$!!!#!!!" (
+
M !M
,
)

#1

4[n | n] $ 4[n | n # 1] " 

$!!#!!"
[ n|n #1]
$!!!
!#A!
!!!
"

KL=")*:

~
A [ n ]: innovations

<[n | n] $ .C # ![n]@[n]/<[n | n # 1]

G4)F&<0G&Hp!pI: :

15

http://www.ws.binghamton.edu/fowler/fowler%20personal%20page/EE522_files/EECE%20522%20Notes_29%20Ch_13B.pdf

!"#$"%#&'()'**+,',-.%/0.%(1
K--6/#''
506--%01
2(%1.34'
506--%01
804#-%01'
7%1#0$'
*(9#3
!;.%/03'
+#<='>%3.#$
(No Dynamics)

!;.%/03'
?03/01'>%3.#$
(w/ Dynamics)

5#1='**+,

>($C#'7%1#0$

Squared Cost Function

K14'LB>M'
?1(&1'N19 *(/#1.-

! ! E{! | :}
$1
": $ E{:}#
! ! E{!} % F !: F ::

"

! ! H ! % F! GT GF! GT % F w

&

$1

7**+,

": $ GH ! #

7**+,'
7%1#0$'
*(9#3

'

7%1#0$'
+#<='>%3.#$

! n ! ! n $1 % J n x[n ] $ ITn ! n $1

-[n | n ] ! -[n | n $ 1] % ? [n ]( :[n ] $ G[n ]K-[n $ 1 | n $ 1])

@A('B410/%C-D

7%1#0$'
?03/01'>%3.#$
@&E'B410/%C-D17

http://www.ws.binghamton.edu/fowler/fowler%20personal%20page/EE522_files/EECE%20522%20Notes_29%20Ch_13B.pdf

Properties of the Kalman filter

extension of sequential MMSE (fixed parameter) to time-varying parameter
with a known dynamical model
5. There is a natural
Kalman filter is a time-varying filter (Kalman gain changes
withup-down
n) progression in the error

The Prediction Stage increases the error

The Update Stage decreases the error
M[n|n 1] > M[n|n]
This is OK prediction is just a natural, intermediate step in the Optimal
processing

Kalman filter computes and uses its performance measure M[n|n]

M[5|4]
M[6|5]
M[7|6]
M[5|5]
M[8|7]
M[6|6]
M[7|7]

prediction increases error, update decreases error

Prediction is an integral part of the KF

as n! Kalman filter reaches ``steady-state and6.becomes
a linear time
And it is based entirely on the Dynamical Model!!!
invariant filter (i.e. k[n] constant, M[n|n] constant) 7. After a long time (as n ! ") the KF reaches steady-state

operation and the KF becomes a Linear Time-Invariant filter

M[n|n] and M[n|n 1] both become constant

but still have M[n|n 1] > M[n|n]

Kalman filter created uncorrelated sequence of ``innovations
Thus, the gain k[n] becomes constant, too.

Kalman filter is optimal for Gaussian, if not Gaussian, optimal Linear MMSE
M[n|n-1], M[n|n], K[n] can be computed off-line (ahead of time)

More observations
Kalman vs Wiener?

What about if dont have linear observation and dynamical models?

Much more to know about Kalman filter!!!