CGF

Moment generating function
C O L E P O L Y T E C H N I Q U E
F DR A L E D E L A U S A N N E
Definition 1 The moment generating function (MGF) of a random variable X is

defined as
!
MX (t) = E(etX ) =
"r tr /r!, t T ,
r=0
where T = {t R : MX (t) < }. The "r are called the moments of X (around
the origin).
This has the following properties:
!
!
0 T for any X (but MX is only useful if T is an open set);

(r)
if T is open, then "r = MX (0), and

"
E(X) = MX
(0),
!
!
""
"
var(X) = MX
(0) {MX
(0)}2 ;
"n
if X1 , . . . , Xn are independent, then MX1 ++Xn (t) = j=1 MXj (t);

there is an injective map between MGFs and probability distributions.
Statistics for Mathematicians
Week 5 slide 5
Cumulant generating function
Definition 2 The cumulant generating function (CGF) of a random variable X is defined

as
X
tX
KX (t) = log MX (t) = log E(e ) =
r tr /r!, t T ,
r=1
where T = {t Rp : MX (t) < }. The r are called the cumulants of X.

!
0 T for any X (but KX is only useful if T is an open set);
if T is open, the mean and variance of X may be computed as

"
E(X) = KX
(0) = 1 ,
""
var(X) = KX
(0) = 2 ;
P
if X1 , . . . , Xn are independent, then KX1 ++Xn (t) = n
j=1 KXj (t), so the cumulants
of a sum of independent variables are the sums of the cumulants;
there is an injective map between CGFs and probability distributions.
Week 5 slide 6
Reminder: Normal distribution
Definition 3 If X N (, 2 ), where R, R+ , then

$
#
1
1
2
fX (x; , 2 ) =
(x
)
,
exp
2 2
(2 2 )1/2
x R.
Properties:
! the standard normal variable Z = (X )/ has mean 0 and variance 1,
Z N (0, 1);
!
the density fZ (z) = (z) = (2)1/2 ez

z = 0, with
/2
, for z R, is symmetric around

% z
Pr(Z z) = (z) =
(x) dx.
the quantiles of Z are denoted z and satisfy z = 1 (), with
0.8
0.84
0.9
1.28
0.95
1.64
0.975
1.96
0.99
2.33
0.995
2.58
Week 5 slide 7
Normal distribution II
Lemma 4 (a) If a, b R, then Ma+bX (t) = eat MX (bt),

t {b1 T } = {s : bs T }.
(b) If Z N (0, 1), then
t2 /2
MZ (t) = e
t R,
MZ 2 (t) = (1 2t)1/2 , t < 1/2.

2
(c) If X N (, 2 ), then MX (t) = et+t
2 /2
, t R.
Lemma 5 (a) If X1 , . . . , Xn are independent random variables with MGFs MXj (t),
for t Tj , and a, b1 , . . . , bn R are constants, then
Ma+b1 X1 ++bn Xn (t) = eat
n
&
MXj (bj t),
j=1
n
'
{b1
j Tj }.
j=1
ind
(b) If Xj N (j , j2 ), for j = 1, . . . , n, then

a + b1 X1 + + bn Xn N (a + b1 1 + + bn n , b21 12 + + b2n n2 ).
Week 5 slide 8
Chi-squared distribution
Introduction
Some Distributions
Moment generating
function
Cumulant generating
function
Reminder: Normal
distribution
Normal distribution
II
Chi-squared
distribution
Student t
distribution
F distribution
Computing
Normal Random
Sample
Multivariate
Distributions
iid
Definition 6 If Z1 , . . . , Z N (0, 1), then W = Z12 + + Z2

has the chi-squared distribution on degrees of freedom, for
N. We write W 2 , the p quantile of W is c (p), and
1
/21 w/2
e
, w > 0, = 1, 2, . . . ,
w
/2
2 (/2)
( a1 u
where (a) = 0 u e du, for a > 0, is the gamma function.
fW (w) =
Lemma 7 (a) If W 2 , then MW (t) = (1 2t)/2 , t < 1/2.

(b) E(W ) = , var(W ) = 2.
(c) If W1 21 W2 22 , then W1 + W2 21 +2 .
Week 5 slide 9
Density of 21
0.0
0.1
Density
0.2
0.3
0.4
0.5
Chisquared density, nu=1
10
w
15
20
Week 5 slide 10
Density of 22
0.0
0.1
Density
0.2
0.3
0.4
0.5
10
w
15
20
Week 5 slide 11
Density of 24
0.0
0.1
Density
0.2
0.3
0.4
0.5
10
w
15
20
Week 5 slide 12
Density of 26
0.0
0.1
Density
0.2
0.3
0.4
0.5
10
w
15
20
Week 5 slide 13
Student t distribution
Introduction
Some Distributions
Moment generating
function
Cumulant generating
function
Reminder: Normal
distribution
Normal distribution
II
Chi-squared
distribution
Student t
distribution
F distribution
Computing
Definition 8 If Z N (0, 1) and W 2 are independent, then

T = Z/(W/)1/2 has a Student t distribution on degrees
of freedom; we write T t . Its density is
{( + 1)/2}
1
,
fT (t) =
(+1)/2
2
(/2) (1 + t /)
The p quantile of T is t (p).
Normal Random
Sample
Multivariate
Distributions
< t < , = 1, 2, . .
If they exist, E(T ) = 0 and var(T ) = /( 2), for 2

and 3 respectively.
As , the Student t distribution approaches the
N (0, 1) distribution.
Week 5 slide 14
Density of Student t1
Student density: black.

N (0, 1) density: red.
0.0
0.1
Density
0.2
0.3
0.4
0.5
Student t density, nu=1
0
w
Week 5 slide 15

0.0
0.1
Density
0.2
0.3
0.4
0.5
0
w
Week 5 slide 16

0.0
0.1
Density
0.2
0.3
0.4
0.5
0
w
Week 5 slide 17

0.0
0.1
Density
0.2
0.3
0.4
0.5
0
w
Week 5 slide 18
F distribution
ind
Definition 9 If W1 , W2 21 22 , then
W1 /1
F =
W2 /2
has the F distribution on 1 and 2 degrees of freedom: we write
F F1 ,2 . Its density function is
fF (u) =
1 /2 2 /2
+
2
1
2 1)
)1 *
1
2 1 2 2
)1
1
2
2
*
1
u 2 1 1
(2 + 1 u)
(1 +2 )/2
u > 0, 1 , 2 = 1, 2, . . . ,
and its p quantile is denoted F1 ,2 (p).
Week 5 slide 19
Computing
Introduction
Some Distributions
Moment generating
function
Cumulant generating
function
Reminder: Normal
distribution
Normal distribution
II
Chi-squared
distribution
Student t
distribution
F distribution
Computing
Normal Random
Sample
Multivariate
Distributions
Values of the N (, 2 ), 2 , t , and F1 ,2 quantiles are

available in tables.
Statistical packages such as R (powerful, free from

http://www.r-project.org/) have these and many other
useful functions.
Example of the use of R to compute quantiles:
R : Copyright 2005, The R Foundation for Statistical Computing

Version 2.2.1 (2005-12-20 r36812)
...
> qnorm(0.025)
# this is a comment; access normal quantiles
[1] -1.959964
# the [1] means this is the first element of a vector
> ?qnorm
# help on use of function qnorm()
> qchisq(0.025,df=3) # chi-squared quantiles
[1] 0.2157953
> qt(0.025,df=3)
# t quantiles
[1] -3.182446
> qf(0.025,df1=3,df2=4) # F quantiles
[1] 0.06622087
Week 5 slide 20
Normal random sample

Introduction
Some Distributions
Normal Random
Sample
Normal random
sample
Jackal jawbones
Consequences
iid
Theorem 10 If X1 , . . . , Xn N (, 2 ) and
X = n1
Multivariate
Distributions
n
!
Xj ,
j=1
then
X
(n 1)S 2
1 !
2
S =
(Xj X)2 ,
n1
j=1
N (, 2 /n)
2 2n1
independent;
equivalently we write
D
S2
+ n1/2 Z,
(n
1)1 2 W,
Z N (0, 1),
W
2n1 ,
Z, W independent.
A proof will be given later.
Week 5 slide 21
Covariance and correlation
Definition 1 Let Xn1 = (X1 , . . . , Xn )T be a vector random variable. Then the

covariance and correlation between Xj and Xk are
cov(Xj , Xk )
corr(Xj , Xk )
= E [{Xj E(Xj )} {Xk E(Xk )}] = E(Xj Xk ) E(Xj )E(Xk ),

!
= cov(Xj , Xk )/ var(Xj )var(Xk ),
provided these are finite. If XA , XB are p 1, q 1 subvectors of X, then their

covariance matrix has (i, j) element cov(Xi , Xj ), for i A, j B, and is
"
#
T
cov(XA , XB )pq = E {XA E(XA )} {XB E(XB )} ,
if all the needed expectations exist. In particular, the matrix cov(X, X) = nn is

called the covariance matrix (sometimes variance matrix) of X.
Lemma 2 A covariance matrix is symmetric and positive semi-definite. If it is not

positive definite, then the distribution of X is degenerate. Moreover
cov(ap1 + Bpn X, cq1 + Dqn X) = BDT .
Week 6 slide 4
Multivariate normal distribution
Definition 3 A p-dimensional multivariate normal random variable

X = (X1 , . . . , Xp )T with mean p1 and covariance matrix pp has density
f (x; , ) =
1
(2)p/2 ||1/2
exp
12 (x
)
T
(x ) ,
x, Rp ;
we write X Np (, ). We assume that the distribution is not degenerate, in which

case is positive definite, then its determinant || > 0.
Also Z Np (0p1 , Ip ) is a p-dimensional standard normal variable, with density
& 1 T '
p/2
f (z) = (2)
exp 2 z z , z Rp .
Week 6 slide 5
Joint moment-generating functions
Definition 4
Let X = (X1 , . . . , Xp )T be a random variable taking values in Rp . The joint moment
generating function of X is defined to be
tT X
MX (t) = E(e
Pp
) = E(e
j=1 tj Xj
),
t T Rp ,
where T = {t Rp : MX (t) < }.

!
0 T for any X (but MX is only useful if T is an open set);
if T is open, then
MX (t)
E(Xj ) =
,
tj t=0
2 MX (t)
MX (t)
MX (t)
cov(Xj , Xk ) =
;
tj tk t=0
tj t=0
tk t=0
the joint cumulant generating function of X may be defined as KX (t) = log MX (t);
there is an injective map between MGFs and probability distributions.
X1 , . . . , Xp are independent iff MX (t) factorises into component functions of t1 , . . . , tp .
Week 6 slide 6
MGF of Np
Introduction
Multivariate
Distributions
Covariance and
correlation
Multivariate normal
distribution
Joint
moment-generating
functions
MGF of Np
Reminder: Theorem
10, week 5
Marginal and
conditional
distributions
"
Likelihood
Lemma 5 (a) The MGF of X Np (, ) is

MX (t) = E(e
tT X
) = E(e
Pp
r=1 tr Xr
&T
'
1 T
) = exp t + 2 t t ,
t Rp .
(b) If A {1, . . . , p}, and XA is the corresponding subset of X,

then
XA XAc A,Ac = 0.
iid
(c) If X1 , . . . , Xn N (, 2 ), then
Xn1 = (X1 , . . . , Xn )T Nn (1n , 2 In ).
(d) Linear combinations of normal variables are normal:
ar1 + Brp X Nr (a + B, BB T );
this is non-degenerate iff |BB T | > 0.
Week 6 slide 7
Marginal and conditional distributions
Theorem 7 Let X Np (p1 , pp ), where || > 0, so the distribution is

not degenerate, and let A, B {1, . . . , p} with |A| = q < p, |B| = r < p and
A B = .
Let A , A , and AB respectively be the q 1 subvector of and q q, and
q r sub-matrices of conformal with A, A A, and A B. Then
(a) the marginal distribution of XA is normal,
XA Nq (A , A );
and
(b) the distribution of XA conditional on XB = xB is normal,
&
'
1
1
XA | XB = xB Nq A + AB B (xB B ), A AB B BA .
Week 6 slide 9

CGF

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

CGF

Uploaded by

Copyright:

Available Formats

Moment generating function

Definition 1 The moment generating function (MGF) of a random variable X is

0 T for any X (but MX is only useful if T is an open set);

if T is open, then "r = MX (0), and

if X1 , . . . , Xn are independent, then MX1 ++Xn (t) = j=1 MXj (t);

Statistics for Mathematicians

Cumulant generating function

Definition 2 The cumulant generating function (CGF) of a random variable X is defined

where T = {t Rp : MX (t) < }. The r are called the cumulants of X.

0 T for any X (but KX is only useful if T is an open set);

if T is open, the mean and variance of X may be computed as

there is an injective map between CGFs and probability distributions.

Statistics for Mathematicians

Reminder: Normal distribution

Definition 3 If X N (, 2 ), where R, R+ , then

the density fZ (z) = (z) = (2)1/2 ez

, for z R, is symmetric around

the quantiles of Z are denoted z and satisfy z = 1 (), with

Statistics for Mathematicians

Lemma 4 (a) If a, b R, then Ma+bX (t) = eat MX (bt),

MZ 2 (t) = (1 2t)1/2 , t < 1/2.

(c) If X N (, 2 ), then MX (t) = et+t

MXj (bj t),

(b) If Xj N (j , j2 ), for j = 1, . . . , n, then

Definition 6 If Z1 , . . . , Z N (0, 1), then W = Z12 + + Z2

Lemma 7 (a) If W 2 , then MW (t) = (1 2t)/2 , t < 1/2.

Statistics for Mathematicians

Chisquared density, nu=1

Statistics for Mathematicians

Chisquared density, nu=2

Statistics for Mathematicians

Chisquared density, nu=4

Statistics for Mathematicians

Chisquared density, nu=6

Statistics for Mathematicians

Definition 8 If Z N (0, 1) and W 2 are independent, then

The p quantile of T is t (p).

If they exist, E(T ) = 0 and var(T ) = /( 2), for 2

Statistics for Mathematicians

Student density: black.

Student t density, nu=1

Statistics for Mathematicians

Student density: black.

Student t density, nu=5

Statistics for Mathematicians

Density of Student t10

Student density: black.

Student t density, nu=10

Statistics for Mathematicians

Density of Student t20

Student density: black.

Student t density, nu=20

Statistics for Mathematicians

and its p quantile is denoted F1 ,2 (p).

Statistics for Mathematicians

Values of the N (, 2 ), 2 , t , and F1 ,2 quantiles are

Statistical packages such as R (powerful, free from

Example of the use of R to compute quantiles:

R : Copyright 2005, The R Foundation for Statistical Computing

Statistics for Mathematicians

Normal random sample

A proof will be given later.

Statistics for Mathematicians

Covariance and correlation