Least-Squares Estimation !

Robert Stengel! Optimal Control and Estimation, MAE 546, Princeton University, 2013 " ! Estimating unknown constants from redundant measurements"
! Least-squares" ! Weighted least-squares"

Perfect Measurement of a Constant Vector "

! Given "
! Measurements, y, of a constant vector, x"

! Estimate x" ! Assume that output, y, is a perfect measurement and H is invertible"

! Recursive weighted leastsquares estimator"

Copyright 2013 by Robert Stengel. All rights reserved. For educational use only.! http://www.princeton.edu/~stengel/MAE3546.html! http://www.princeton.edu/~stengel/OptConEst.html!

y=Hx
= H !1 y x

! y: (n x 1) output vector" ! H: (n x n) output matrix" ! x : (n x 1) vector to be estimated"

! Estimate is based on inverse transformation"

Imperfect Measurement of a Constant Vector "

! Given "
! Noisy measurements, z, of a constant vector, x"

Cost Function for LeastSquares Estimate "

! Measurement-error residual"

=z"y !=z"Hx

dim(!) = ( k " 1)

! Effects of error can be reduced if measurement is redundant" ! Noise-free output, y"

! Squared measurement error = cost function, J"

y=Hx

! y: (k x 1) output vector" ! H: (k x n) output matrix, k > n" ! x : (n x 1) vector to be estimated"

J=

! Measurement of output with error, z"

Quadratic norm! 1 T 1 T ) (z " H x ) ! ! = (z " H x 2 2 1 T T HT z " z T H x +x T HT H x = z z"x 2

z= y+n=Hx+n

! z: (k x 1) measurement vector" ! n : (k x 1) error vector"

! What is the control parameter?" The estimate of x!

) = ( n ! 1) dim ( x

Static Minimization Provides Least-Squares Estimate "

Error cost function "

Static Minimization Provides Least-Squares Estimate"

! Estimate is obtained using left pseudo-inverse matrix"

J=

1 T T HT z ! z T H x +x T HT H x z z!x 2
Necessary condition "

T HT H HT H x or = HT H x

)(

!1

T = z T H HT H =x

!1

(row )

T T !J 1 ) + x T HT H % = 0 = # 0 " ( HT z ) " z T H + ( HT H x $ & !x 2

!1

HT z (column )

T HT H = z T H x

Example: Average Weight of a Pail of Jelly Beans "

! Measurements are equally uncertain"

Average Weight of the Jelly Beans "

Optimal estimate "
-1 ! z1 # ' ! 1 #* % & % &, ) 1 &, ! % z2 & # # % = )! x 1 1 ... 1 1 1 ... 1 " $ ... " $% % &, ... & ) % & ) % 1 &, z ( " $+ % " k & $

zi = x + ni , i = 1 to k
! Express measurements as"

z = Hx + n
! Output matrix"

! # H=# # # "

1 1 ... 1

$ & & & & %

! Optimal estimate"

= (k)

-1

( z1 + z2 + ... + zk )
1 ! zi k i=1
k

= H H x
T

!1

Simple average "

H z

= x

[sample mean value]

Least-Squares Applications "

Error Cost Function!

Least-Squares Linear Fit to Noisy Data "

! Measurement vector" ! Find trend line in noisy data"
zi = ( a0 + a1 x ) + ni ! # # # # # " ! z1 $ # ( a0 + a1 x1 ) & z2 & # ( a0 + a1 x2 ) =# ! & # ! & zn & # ( a0 + a1 xn ) % # " $ ! & # & # &+# & # & # & " %
# & n1 $ 1 x2 & ! a0 $ &+n & z=# # ! ! &# a n2 & " 1 & % # &# 1 xn & # " % ! & & " Ha + n nn & % ! 1 x1 $

z = ( a0 + a1 x ) + n

y = a0 + a1 x

! More generally, least-squares estimation is used for"

! Higher-degree curve-tting" ! Multivariate estimation "

! Error cost function"

J= 1 T ) (z ! H a ) (z ! H a 2

! Least-squares estimate of trend line" ! Estimate ignores statistics of the error"

! a $ = # 0 & = HT H a a # " 1 & %

'1

HT z

=a 0 + a 1 x y

Least-Squares Image Processing "

Original Lenna !

Measurements of Differing Quality"

! Suppose some elements of the measurement, z, are more uncertain than others"

Degraded Image!

Restored Image (Adaptively Regularized Constrained Total Least Squares) !

z = Hx + n
! Give the more uncertain measurements less weight in arriving at the minimum-cost estimate" ! Let S = measure of uncertainty; then express error cost in terms of S1" 1
J= 2 ! T S "1!

Chen, Chen, and Zhou, IEEE Trans. Image Proc., 2000"

Error Cost and Necessary Condition for a Minimum "

Error cost function, J "

Weighted Least-Squares Estimate of a Constant Vector "

Necessary condition for a minimum "

1 1 T ) S "1 ( z " H x ) J = ! T S "1! = ( z " H x 2 2 1 T HT S "1z " z T S "1H x +x T HT S "1H x = z T S "1z " x 2

T HT S !1H ! z T S !1H $ " #x %=0 T HT S !1H = z T S !1H x

Weighted left pseudo-inverse provides the solution "

Necessary condition for a minimum "

!J =0 !x
=
T T 1# ) + x T HT S "1H % 0 " ( HT S "1z ) " z T S "1H + ( HT S "1H x $ & 2

= HT S !1H x

!1

HT S !1 z

The Return of the Jelly Beans"

! Error-weighting matrix"
" $ !1 S !A=$ $ $ $ # a11 0 ... 0 0 a22 ... 0 0 % ' ... 0 ' ... ... ' ' ... akk ' & ...

How to Chose the Error Weighting Matrix "

! Optimal estimate of average jelly bean weight"

= H S H x
T

!1

!1

H S z
0 #! &% 0 &% ... ... & % &% ... akk & % $" ... ... z1 # & z2 & ... & & zk & $

!1

a)# Normalize the cost function according to expected measurement error, SA!

J=
1 1 ... 1 ! a11 #* % &, 0 &, ! 1 1 ... 1 # % " $ % ... &, % &, , 0 $+ % "
-1

' ! a11 % ) % 0 # = )! x 1 1 ... 1 " $ % ... ) % ) ) 0 % ( "

0 a22 ... 0

0 #! & ... 0 & % % ... ... & % &% ... akk & " $ ...

0 a22 ... 0

1 T "1 1 1 T 1 1 ! S A ! = ( z " y ) S" ( z " H x )T S" A (z " y) = A (z " H x) 2 2 2

= x

!a z
i =1 k

ii i

b)# Normalize the cost function according to expected measurement residual, SB! 1 1 T 1 1 ) S" J = ! T S" (z " H x B! = B (z " H x) 2 2

!a
i =1

ii

Measurement Error Covariance, SA "

Expected value of outer product of measurement error vector "
T $ SA = E " z ! y z ! y ( ) ( ) # % T $ = E" z ! Hx z ! Hx ( ) ( ) # % T $ = E" nn # %!R

Measurement Residual Covariance, SB "

Expected value of outer product of measurement residual vector "
T SB = E " # !! $ %

T$ = E" #( z & Hx ) ( z & Hx ) %

T $ = E" #( H! + n ) ( H! + n ) %

) ! = ( z " Hx

T T T T T T = HE " # !! $ % H + HE !n + E n! H + E nn

! HPH + HM + M H + R
T T T

where T$ P= E" #( x ! x ) ( x ! x ) % T )n $ M = E" #( x ! x %

T R = E" # nn $ %

Requires iteration ( adaptation ) of the estimate to nd SB "

Recursive Least-Squares Estimation "

!! Prior unweighted and weighted least-squares estimators use batch-processing approach"
!! All information is gathered prior to processing" !! All information is processed at once"

Prior Optimal Estimate "

Initial measurement set and state estimate, with S = SA = R "

z1 = H1x + n1
T !1 T !1 1 = ( H1 x R1 H1 ) H1 R1 z1 !1
dim ( z1 ) = dim ( n1 ) = k1 ! 1 dim ( H1 ) = k1 ! n dim ( R1 ) = k1 ! k1

!! Recursive approach"
!! Optimal estimate has been made from prior measurement set" !! New measurement set is obtained" !! Optimal estimate is improved by incremental change (or correction) to the prior optimal estimate"

State estimate minimizes "

J1 =

1 T "1 1 T "1 1 ) R1 1 ) !1 R1 !1 = ( z1 " H1 x ( z1 " H1 x 2 2

New Measurement Set "

New measurement "

Cost of Estimation Based on Both Measurement Sets "

Cost function incorporates estimate made after incorporating z2 "
J 2 = " ( z1 ! H1x 2) $ #
T T

z 2 = H2 x + n2 R 2 : Second measurement error covariance

Concatenation of old and new measurements "

2) ( z 2 ! H2x

( !1 0 % * R1 ' * 0 R !1 & 2 )
T

! z1 $ z!# & # " z2 & %

dim ( z 2 ) = dim ( n 2 ) = k2 ! 1 dim ( H 2 ) = k2 ! n dim ( R 2 ) = k2 ! k2

!1 1 2 ) R1 2 ) + ( z 2 ! H2x 2 ) R! = ( z1 ! H1x ( z1 ! H1x 2 ( z 2 ! H2x2 )

+ " (z ! H x 1 1 2) -$ 2) ( z ! H2x ,$ # 2

% ' ' &

2 Both residuals refer to x

Optimal Estimate Based on Both Measurement Sets "

Simplication occurs because weighting matrix is block diagonal "
( * T 2 = ) ! H1 x " * + ! R %1 0 #& 1 $ & 0 R %1 2 " # ! H #, ( * T ' & 1 '* - )! H H2 ' * *" 1 '& $. + $"
%1 T 1 %1

Apply Matrix Inversion Lemma "

Dene "

P1!1

T !1 ! H1 R1 H1

Matrix inversion lemma "

# ! z #, ' & 1 '* z * '& $. $" 2 '

HT 2

! R %1 0 #& 1 HT 2 $ & 0 R %1 2 "

%1 1 1 %1 z + HT 2 R2 z2 )

T %1 %1 = ( H1 R1 H1 + HT 2 R 2 H2 )

(H R

T P1 ! P1HT 2 ( H2P 1H 2 + R 2 ) H 2 P 1 !1

(H R (P
T 1

!1 1

!1 H1 + HT = 2 R 2 H2 ) !1 !1 + HT = 2 R 2 H2 ) !1

!1 1

Improved Estimate Incorporating New Measurement Set "

T !1 1 = P1H1 x R1 z1

Simplify Optimal Estimate Incorporating New Measurement Set "

I = A !1A = AA !1 , with A ! H 2 P1HT 2 + R2

New estimate is a correction to the old "

T 2 = x 1 ! P1HT x 2 H2P 1H 2 + R 2 T 2

!1

1 H2x

+ P1H " I n ! H 2 P1H + R 2 #

T 2

!1

H 2 P1H $ R z %
T 2

!1 2 2

T 2 = x 1 ! P1HT x 2 ( H2P 1H 2 + R 2 )

!1

!1 T $ = " I n ! ( H 2 P1HT 2 + R 2 ) H2P 1H 2 x1 # % !1 T T $ !1 " + P1HT 2 In ! ( H2 P 1H 2 + R 2 ) H 2 P 1H 2 R 2 z 2 # %

1 ! K ( z 2 ! H2x 1 ) !x

1 ) ( z 2 ! H2x

K : Estimator gain matrix

Recursive Optimal Estimate "

!! Prior estimate may be based on prior incremental estimate, and so on" !! Generalize to a recursive form, with sequential index i!
T i = x i !1 ! Pi !1HT x i Hi P i ! 1H i + R i

Example of Recursive Optimal Estimate "

z= x+n i = x i!1 + pi!1 ( pi!1 + 1) x
!1
!1

i!1 ) ( zi ! x
H = 1; R = 1

i !1 ! K i ( z i ! H i x i !1 ) !x with Pi = P + H R Hi
T i

) (z
!1

i !1 ) ! Hi x

ki = pi!1 ( pi!1 + 1) = pi = ( pi!!11 + 1) =

!1
index p-sub-i k-sub-i 0 11 0.5 0.5 2 0.333 0.333 3 0.25 0.25 4 0.2 0.2 5 0.167 0.167

pi!1 ( pi!1 + 1) 1
!1 i !1

(p

+ 1)

!1 i !1

!1 i

dim ( x ) = n ! 1; dim ( P) = n ! n

!1

dim ( z ) = r ! 1; dim ( R ) = r ! r

dim ( H ) = r ! n; dim ( K ) = n ! r

0 = z0 x 1 = 0.5 x 0 + 0.5 z1 x 2 = 0.667 x 1 + 0.333z2 x 3 = 0.75 x 2 + 0.25 z3 x 4 = 0.8 x 3 + 0.2 z4 x !

Optimal Gain and Estimate-Error Covariance "

1 T !1 Pi = Pi! !1 + H i R i H i

T Ki = Pi !1HT i Hi P i ! 1H i + R i

!1

!1

!! With constant estimation error matrix, R,"

!! Error covariance decreases at each step" !! Estimator gain matrix, K, invariably goes to zero as number of samples increases"

Next Time:! Propagation of Uncertainty in Dynamic Systems!

!! Why?" !! Each new sample has smaller effect on the average than the sample before!

Weighted Least Squares ( Kriging ) Estimates (Interpolation)"

! Can be used with arbitrary interpolating functions"

Supplemental Material !

http://en.wikipedia.org/wiki/Kriging!

Weighted Least Squares Estimates of Particulate Concentration (PM2.5) "

Delaware Sampling Sites ! Delaware Average Concentrations ! DE-NJ-PA PM2.5 Estimates !

Delaware Dept. of Natural Resources and Environmental Control, 2008!