Parameter Estimation 2

Brian Beckman, 18 Aug 2010

Pursuant to our little exploration of least squares a few days ago, this note contains an investigation of noise statistics.

Note that the use of the words "variance" and "covariance" here arise in statistics and probability and have no obvious connection to
their uses in physics, mathematics, and programming, which latter all seem to have subtle connections amongst one another.

Statistics

Ÿ Normalization
Require the noise to have zero mean and unit covariance, meaning the expectation values E@ΕD = 0 and EAΕΕT E = Im , the m-
dimensional unit matrix.

First, recall that, in general, the variance of the uniform distribution with max x2 and min x1 is Hx2 - x1 L2 ‘ 12. Ensure E@ΕD = 0 by
setting up the distribution symmetric about the origin, that is, x2 = Ν, x1 = -Ν, and dividing by the square root of the variance

H2 ΝL ’ 12 = Ν ’ 3 . Easiest to do by choosing Ν = 1 and normalization factor 3.

In[1]:= ΕNormAm_E := 3 Table@RandomReal@8-1, 1<D, 8m<D

Now simulate the expectation value E@ΕD of Ε over a large number of trials. Expect the simulated values to be much less than 1 for a
large number of trials. To say precisely how much less, examine the statistics of an average of trials of length t, which converge to
the normal distribution by the central-limit theorem.

ã Statistics of Each Component of ΕNorm

Let X1 , X2 , ..., Xt be t copies of a random variable, each with the same mean Μ and variance Σ2 , representing the results of t
independent and identically distributed trials for each component of ΕNorm. Consider the average of these t copies

Xt = HX1 + X2 + º + Xt L  t
The central-limit theorem guarantees that

Zt = t IXt - ΜM ‘ Σ
converges to the H0, 1L standard normal distribution as the number of trials t grows

P HZt L ‘2
1 2
= ã-Zt
2Π

-J t IXt -ΜM‘ΣN “2
PI t IXt - ΜM ‘ ΣM =
2
1
ã
2Π

-tIXt -ΜM ’2 Σ2
2
1
= ã
2Π

-IXt -ΜM ’2 JΣ’ t N

2 2
1
= ã
2Π

Therefore, each component of ΕNorm, averaged over t trials, is distributed like Xt has variance JΣ ’ t N , where Σ2 = Ν2 ‘ 3, that is,
2

variance
2 Noise and Covariance.nb

IΣ ‘ t M = 1  H3 tL
2

Thus, we expect such a trial to vary by a standard deviation of 1 ’ 3 t , so a trial of length 106 should have a variation roughly

around 1 ’ 3 000 000 » 0.00058.

Here is a sample of 106 with a six-component ΕNorm, producing 6 independent trials each with standard deviation 0.00058:

In[2]:= EΕAm_, trials_E := Sum@ΕNorm@mD, 8i, trials<D  trials

In[3]:= ListLinePlot@EΕ@6, 1 000 000D, PlotMarkers ® AutomaticD

æ
0.0010

0.0005

æ
æ
Out[3]=
1 2 3 4
æ
5 6
æ
-0.0005

-0.0010
æ

The precise statistics of elements of the noise covariance matrix are much more complex. For current purposes, suffice it to say that
we expect the diagonal elements to be near 1 and the off-diagonal elements to be near 0, plus or minus a quantity proportional to
1’ t :

EΕSquaredAm_, trials_E :=
Sum@With@8o = ΕNorm@mD<, Outer@Times, o, oDD  trials, 8i, trials<D
In[4]:=

In[5]:= EΕSquared@6, 1 000 000D

1.00029 -0.000245335 0.000799381 -0.000498956 0.00161438 -0.000513165

-0.000245335 0.999602 -0.00122787 -0.00247517 -0.000665883 -0.000446644
0.000799381 -0.00122787 1.0005 -0.00116333 -0.0011422 0.00064151
Out[5]=
-0.000498956 -0.00247517 -0.00116333 0.99955 0.000538755 -0.000218657
0.00161438 -0.000665883 -0.0011422 0.000538755 0.999874 -0.000918535
-0.000513165 -0.000446644 0.00064151 -0.000218657 -0.000918535 1.00036