Distribution Theory - Autumn Quarter 2015:

Lecture Notes by Tom Hen

September 28, 2015

CONTENTS

CONTENTS

Contents
1 Distribution Functions and their inverses:
1.1

Basic Definitions: . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Generalized Inverses:

3
3

1 DISTRIBUTION FUNCTIONS AND THEIR INVERSES:

Distribution Functions and their inverses:

1.1

Basic Definitions:

Definition 1. Given a real-valued RV X : ! R the distribution function (df ) of X is a function

FX : ( 1, 1) ! [0, 1] defined by
FX (x) := P [! 2 | X (!) x] P [X x]
Remark 1. This is often called the Cumulative Distribution Function (CDF).
Theorem 1. The df F FX of a real-valued RV X has the following properties:
1. F is non-decreasing.
2. F is right-continuous.
3.

lim F (x) = 0 and lim F (x) = 1

x! 1

x!1

4. F (x ) := limu"x F (u) = P [X < x]

5. P [X = x] = F (x)

F (x ) and this is non-zero i F function has a jump-discontinuity.

Fact 1. If F is strictly increasing (which coincides with it also being continuous) then it is has an
inverse function F 1 : [0, 1] ! ( 1, 1) and this function is also continuous and strictly increasing,
we will show this soon.
Remark 2. If F is not strictly increasing it is still possible to define a similar function called the
psuedo-inverse as F 1 (u) = inf {x 2 R | F (x) u}. It can be shown that this is well-defined and
that this function has a lot of properties similar to a standard inverse function.
Claim 1. Given a RV X with df F the RV F (X) is distributed Uniform (0, 1).
Proof. By direct calculation:

P [F (X) u] = P X F

[u] = F F

Definition 2. Let X be a RV with df F and let X1 , ..., Xn

distribution function is defined by

(u) = u

i.i.d
F , the corresponding empirical

1X
Fn (x) =
1{Xi x}
n i=1

The Kolmogorov-Smirnof Statistic (KS) of F is then defined by:

KSn (F ) := sup Fn (X)

F (x)

x2R

Remark 3. If we denote Ui = F (Xi ) U (0, 1) it can be seen that:

n

1X
KSn (F ) = sup
1{Xi x}
x2R n i=1
= sup
y2R

1X
z}|{
F (x) = sup
1{F (Xi )F (x)}
x2R n i=1

F (x)

1X
1{Ui (x)y}
n i=1

This shows that if F is strictly increasing (required for ) then KSn (F ) is a "Distribution-Free"
statistics. That is the values of KSn do not depend on the distribution of X.
3

Remark 4. Given the simple one-sided hypothesis-test with statistic Z that rejects H0 when |Z| > C
we have the p-value defined by
P val = PH0 [|Z| > C | Z is distributed under H0 ]
It can be seen using an alternative formulation of this that Pvalues are distributed U (0, 1) under H0 :
P val := PH0 [|Znew | > C | |Zobs | = C]

= PH0 F|Zneq | (C) | |Zobs | = C

= PH0 F|Zneq | (Zobs ) = Uniform (0, 1)

Claim 2. Let X be a RV with df F whose inverse is F

Proof. Given x 2 R we have F

P F

(U ) x =) U = F F

. Then F
1

(U ) X for U Uniform (0, 1).

(U ) F (x) and thus

(U ) x = P (U F (x)) = F (x) = P [X x]

Since this is true for all x 2 R we have the required result.

Remark 5. This result is useful since it gives a simple way to simulate values out of a distribution F
with inverse F 1 given one is able to generate uniformly distributed random numbers.
Definition 3. The mapping X 7! F (X) is called the probability integral transformation (PIT)
whereas the mapping U 7! F 1 (U ) is called the inverse probability transformation (IPT).

Part I

Generalized Inverses:
Definition 4. Given any df F (not necessarily continuous) there may not exist a true inverse F
However, it is still possible to define a generalized inverse F : (0, 1) ! ( 1, 1) defined by

F (u) = inf {x 2 R | u F (x)}

Theorem 2. The function F is well-defined for any df F .
x!1

Proof. Given u 2 (0, 1) denote I := {x 2 R | u F (x)}, since F (x) ! 1 and u < 1 this is a
x! 1
non-empty set. Since u > 0 and F (x) ! 0 I is also bounded from below and has an infimum by
completeness of the real numbers.
Claim 3. Given u 2 (0, 1) the set I := {x 2 R | u F (x)} is an interval of the form [F (u) , 1).
x2I

z}|{

Proof. First note that if x 2 I and y > x then since F is non-decreasing F (y) F (x)
u =) y 2 I.
1
Thus for any x 2 I we have [x, 1) I. Take {xn }n=1 I s.t xn # F (u) (such a sequence exists
since is F (u) is the infimum of I). Then by right continuity of F we get:
F (F (u)) = lim F (xn )
n!1

xn 2I

z}|{

u =) F (F (u))

This shows that [F (u) , 1) is indeed and interval, as required.

4

u =) F (u) 2 I

Claim 4. Switching Formula (SF):

Let F be a df and let F be its generalized inverse. Then u F (x) () F (u) x.
Proof. =) u F (x) =) x 2 {z 2 R | u F (z)} =) F (u) = inf {z 2 R | u F (z)} x.

(= If F (u) x then by claim (3) x 2 [F (u) , 1) = {z 2 R | u F (z)} and thus u F (x).

Remark 6. This formula has an immediate counterpart of the form u > F (x) () F (u) > x.

Remark 7. One can not replace the inequalities in the switching formula with strict inequalities.
Theorem 3. Properties of generalized inverses:
Let F be the generalized inverse of the df F . Then
1. F is non-decreasing.
2. F is left-continuous.
3.

lim F (u) = inf {x 2 R | F (x) > 0}

u! 1

4. lim F (u) = sup {x 2 R | F (x) < 1}

u!1

5. F (F (u) ) u F (F (u)) for all u 2 (0, 1).

6. F (F (x)) x F (F (x) +) for all x s.t F (x) 2 (0, 1) .
Proof.
1. Given u < v 2 (0, 1) the set we have since F is non-decreasing we have {x 2 R | v F (x)}
{x 2 R | u F (x)} and thus
F (u) = inf {x 2 R | u F (x)} inf {x 2 R | v F (x)} = F (v)
1

2. Let {un }n=1 (0, 1) be non-decreasing sequence such that un " u 2 (0, 1). Since we showed F
1
is non-decreasing the sequence {F (un )}n=1 is also non-decreasing and it is also bounded from
above by 1 and thus has a limit L := lim F (un ) F (u). To show the opposite inequality let
n!1

x 2 R be s.t F (u) > x. By (SF) u > F (x) and thus un > F (x) for suciently large n (since
un " u) and for such n by (SF) F (un ) > x. This shows that L = lim F (un ) > x and taking
x ! F (u) yields that L

F (u), concluding the proof.

n!1

3. Not interesting.
4. Not interesting.
5. F (u) F (u) and thus (SF) gives the right inequality. For the left inequality let < F (u),
by (SF) we get u > F () and taking " F (u) gives u lim"F (u) F () = F (F (u) ) .
6. The left inequality is trivial by (SF). For the right inequality, let > F (x) , by (SF) we get
F () > x and taking # F (x) gives x lim#F (x) F () = F (F (x) +).
Corollary 1. If F is continous then F is an actual inverse of F .

Proof. Using continuity of F and (5) from the previous theorem we have
F (F (u)) = F (F (u) ) u F (F (u)) =) F (F (u)) = u
Similarly using (6) from the previous claim
F (F (x)) x F (F (x) +) = F (F (x)) =) F (F (x)) = x
Theorem 4. Let F be the left-continuous generalized inverse of the df F . Then
1. F (F (u)) = u 8u 2 (0, 1) i F is continuous.
2. F (F (x)) = x for all x 2 A := {x 2 R | F (x) 2 (0, 1)} i F is strictly increasing over A.
Proof. Exercise...
Theorem 5. The IPT Theorem:
Let X be a RV with df F whose generalized inverse is F . Then F (U ) X for U Uniform (0, 1).
Proof. Given x 2 R, by (SF) we have F (U ) x () U F (x). Thus
P [F (U ) x] = P [U F (x)]

U Uniform(0,1)

Since this is true for all x 2 R we have F (U ) X.

z}|{
=

F (x) = P [X x]

Theorem 6. The PIT Theorem:

Let X be a RV with df F . Then P [F (x) u] u for all u 2 (0, 1) and this inequality is an equality
for all u 2 (0, 1) i F is continuous.
Proof. Let U Uniform (0, 1) and let F be the generalized inverse of F . By the IPT F (U ) X
so F (F (U )) F (X) . By theorem (3) we have that F (F (u))
u for all u 2 (0, 1) and thus
F (F (U )) U . Thus for all u 2 (0, 1)
P [F (x) u] = P [F (F (U )) u] P [U u] = u
Now if F is continuous then by corollary (1) for all u 2 (0, 1)
P [F (F (U )) u] = P (U u) = u =) F (F (U )) Uniform (0, 1)
Thus by the IPT F (X) F (F (U )) Uniform (0, 1) .

On the other hand, if F is not continuous at x 2 R then

0 < F (x)

F (x ) = P [X = x] P [F (X) = F (x)]

But P [U = F (x)] = 0 and thus F (x) 6 U .

HW: Exercises 6,8,9,10 in first part of lecture notes.