You are on page 1of 38

Random Variables

Monotonic Transformations

Expectations

PSCI 356: STATISTICS FOR POLITICAL RESEARCH I

Class 5: Probability Theory: Random Variables and Distribution Functions II

Random Variables

Monotonic Transformations

Expectations

Outline

Random Variables Discrete Random Variables Continuous Random Variables Monotonic Transformations Expectations Properties of Expectations Measures of Dispersion

Random Variables Continuous Random Variables

Monotonic Transformations

Expectations

Random Variables

Random Variable: numerical outcome of a random experiment.


1

Denition: A discrete random variable is a random variable that can take on only a nite (or countably innite) number of values. Denition: A continuous random variable is a random variable that can take on a continuum of values (uncountable number of values).

Let F (x ) be the CDF for a continuous random variable X . Properties of F (same as for discrete random variables):
limx F (x ) = 0, limx + F (x ) = 1 F (x ) is increasing so x1 < x2 F (x1 ) F (x2 )

The CDF is a smooth increasing function over some interval of real numbers.

Random Variables Continuous Random Variables

Monotonic Transformations

Expectations

Continuous Random Variables

In dealing with continuous random variables, we talk about the distribution of the random variable over the sample space in terms of the probability density function [PDF] not probability mass functions. Denition: The CDF of a continuous random variable is x F (x ) = P (X x ) = f (x ) x . If f is continuous at x , then the PDF is f (x ) = F (x )/ y . In other words, the PDF is the derivative of the CDF. A PDF f (x ) is a function such that: f (x ) 0 for all x , < x < + f is piecewise continuous, and f (x ) x = 1. These conditions imply that: P (a X b) =
b a

f (X ) X .

By denition of F (x ), P (a X b) = F (b) F (a).

Random Variables Continuous Random Variables

Monotonic Transformations

Expectations

Continuous Random Variables EXAMPLE: The Uniform distribution on the interval [0, 1] chooses any number between 0 and 1. The PDF of the Uniform is given by: 1 if 0 x 1 f (x ) = 0 else f(x)

Random Variables Continuous Random Variables

Monotonic Transformations

Expectations

Continuous Random Variables


EXAMPLE: Uniform Distribution

A consequence is that P (X = a) =

a a

f (x ) x = 0.

The probability that a given value for a continuous random variable is realized is 0. Why is this the case? If the probability of observing the realization a was any positive probability, it would be the same for any number (by denition of the Uniform). But then the sum of any countably innity subset of [0,1] (e.g., the set of rationals) would be > , which is not possible. If P (X = x ) = p > 0, then F (x ) would have a discontinuity (jump) of size p at point x, violating the assumption of continuity. Practically speaking, what is the likelihood of seeing rainfall measurement of exactly 3.435 inches?

Random Variables Continuous Random Variables

Monotonic Transformations

Expectations

Continuous Random Variables


EXAMPLE: Uniform Distribution The CDF of the Uniform Distribution described above is (note the pdf is just the derivative of F (x ): given by: 0 if x < 0 x if 0 x 1 F (x ) = 1 if x > 1 f (x ) = 1 0 if 0 x 1 else

Pr(X <= x)

0.0 2

0.4

0.8

0 x

Random Variables Continuous Random Variables

Monotonic Transformations

Expectations

Continuous Random Variables


Normal Probability Distributions

(Standard) Normal Probability Distribution or Gaussian Distribution: bell shaped curve


For > 0 and , Y is a normal random variable if the probability distribution is dened as: 2 2 1 e(x ) /(2 ) , x p(X = x ) = p(x ) = 2 Standard Normal: = 0 and = 1

Random Variables Continuous Random Variables

Monotonic Transformations

Expectations

Continuous Random Variables


Normal Probability Distributions

How might Gauss have guessed this?

Random Variables Continuous Random Variables

Monotonic Transformations

Expectations

Continuous Random Variables


Standard Normal Probability Distributions

Random Variables Continuous Random Variables

Monotonic Transformations

Expectations

Continuous Random Variables


Other Probability Distributions

Gamma Binomial Probability Distribution Beta Probability Distribution

Random Variables

Monotonic Transformations

Expectations

Monotonic Transformation

Suppose g(x) is a strictly monotone function with inverse g 1 (). Let Y be {y |g (x ) = y for some x X )}. Suppose that fX (x ) is continuous. Then
fY (y ) = fX (g 1 (y ))
g 1 y (y )

, on Y .

WHY? FY (y ) = P (Y y ) = P (g (X ) y ) = P (X g 1 (y )) = FX (g 1 (y )). Then take derivative using the chain rule.

Random Variables

Monotonic Transformations

Expectations

Monotonic Transformation

EXAMPLE: Say pdf fX (x ) = exp(x ), x > 0 and 0 elsewhere Find the pdf for Y = 1/X
fY (y ) = e(1/y )
y

1 y2

Find the pdf for Y = ln(X)


fY (y ) = ee |ey |

Random Variables

Monotonic Transformations

Expectations

What have we learned?

Events in a sample space can be used to model experiments. Probability measure on the events enables us to evaluate the frequency of events. (Discrete and Continuous) random variables can be dened on the sample space to quantify events. Random variables can be described/characterized using pmf/pdf and cdf. However, is there a summary measure of a quantity of interest that can be used to characterize the outcome of the experiment? Of course there is. We refer to such quantities as moments.

Random Variables

Monotonic Transformations

Expectations

Measures of Central Tendency


Discrete Random Variable: Expected Value E[X] One characteristic that we might desire is the expected realization of the random variable X. For a discrete random variable X with values xi , 1 = 1...N , we calculate the expected value (or expectation or mean or the mean of the probability distribution) as follows:
N

E [X ] =
i =1

xi f (xi ) = = x

EXAMPLE 1 Let X be the number of heads in 2 fair coin tosses. Then E [X ] = 0 1/4 + 1 1/2 + 2 1/4 = 1.
.50 Pr(X=x) 0 .25

Random Variables

Monotonic Transformations

Expectations

Measures of Central Tendency


Discrete Random Variable: Median(X)

An alternative measure of central tendency is given by the median of a random variable. The median xM of random variable X is dened as the choice xM such that P (X xM ) 1/2.
Pr(X < x) 0.8 0.0 2 0.4

For the experiment described above, note that xM = 1. x Also realize that, by denition, whereas the median depends only on the rank ordering of the realizations of X, the mean depends on the actual values of X. In other words, since the realizations xi i 1...N enter into our calculations, the mean is sensitive to extreme values (sometimes called outliers").

Random Variables

Monotonic Transformations

Expectations

Measures of Central Tendency


Example: Mean vs. Median

EXAMPLE Consider the random variables X and Y with the following PMF. x f(x) y f(y) 0 .5 0 .5 1 .49 1 .5 1000000 .01

What is E[X] and E[Y] = ? What is the median of X and Y = ? E[X] = .5 E[Y] = 10000.49 Median of X and Y are both .5

Random Variables

Monotonic Transformations

Expectations

Measures of Central Tendency


Discrete Random Variable: Mode(X)

We can also dene the mode of a random variable, which is the element of X that occurs most frequently. mode(X ) = value of xi which maximizes p(xi )

Random Variables

Monotonic Transformations

Expectations

Measures of Central Tendency

Discrete Random Variable


N = E [X ] = =x i =1 xi f (xi ) median(X ) = xM such that P (X xM ) 1/2 mode(X ) = value of xi which maximizes p(xi )

Continuous Random Variable = E [X ] = xf (x ) x =x xM median(X ) = xM such that f (x ) x 1/2 mode(X ) = Value of xi which maximizes f (xi )

Random Variables

Monotonic Transformations

Expectations

Measures of Central Tendency EXAMPLE Suppose f (x ) = 2x 3 x > 1. Calculate the expected value, median and mode.

f(x) 0.0 1 1.0

2.0

3.0

1 x

Random Variables

Monotonic Transformations

Expectations

Measures of Central Tendency

EXAMPLE Suppose f (x ) = 2x 3 x > 1. Calculate the expected value, median and mode. E [X ] = = = = = = = x 2x 3 x b limb 1 x 2x 3 x 1 b limb 1 2x 2 x b limb 2 1 x 2 x 2 limb 2 2
1 x 1 1 2 2 limb b 1 1

b 1

1 2

f (x ) x limb a f (x ) x by standard improper integral results. Since kf (x ) x = k f (x ) x . 3 By Power Rule": x n x = 1/(n + 1)x n+1 for n = 1.

Random Variables

Monotonic Transformations

Expectations

Measures of Central Tendency


EXAMPLE Suppose f (x ) = 2x 3 x > 1. Calculate the expected value, median and mode. The median of X is solved for as follows.
1 2 1 2 1 2 1 m2

= =

m 1

2x 3 x
m 1

= = m=

1 x 2 2 1 1 1 m2 1 2

Inspection of the PDF reveals that the mode of X is obviously at 1. Note that as the above example indicates, there is no reason to expect that all measures of central tendency will be identical. Consequently, depending on the measure you choose, your characterization of the random variable may differ. Furthermore, there is no necessary reason for the median and mode to be unique although it is the case that the mean (E[X]) is unique.

Random Variables

Monotonic Transformations

Expectations

Measures of Central Tendency

EXAMPLE Consider the random variable X with the following PMF. x f(x) 0 .2 1 .3 2 .1 3 .3 4 .1

E [X ] = 0(.2) + 1(.3) + 2(.1) + 3(.3) + 4(.1) = 1.8.4 mode(X ) = {1, 3} 1 5 median(X ) [1, 2] since P (X < xm ) 1 2 and P (X > xm ) 2 .

4 5

Note that there is no reason why E[X] has to be an actual value of X. Sometimes expresses as 1.5.

Random Variables Properties of Expectations

Monotonic Transformations

Expectations

Properties of Expectations

For any function of the random variable g(x) we can still calculate the expected value of the transformed" random variable using the following result:
E [g (x )] =
x

g (xi )f (xi ) if X discrete

E [g (x )] =

g (x )f (x ) x if X continuous

Note that it is not the case that E [X 2 ] = (E [X ])2 .

Random Variables Properties of Expectations

Monotonic Transformations

Expectations

Properties of Expectations
EXAMPLE Consider the random variable X with the following PMF. x f (x ) 0 .5 1 .5 x2 f (x 2 ) 0 .5 1 .5

E [X ] = 0(.5) + 1(.5) = .5 implying that (E [X ])2 = .52 = .25. E [X 2 ] = 0(.5) + 1(.5) = .5. EXAMPLE Let g(x) = a + b(x). E [a + bX ] = = = = =
(a + bx )f (x ) x af (x ) x + bxf (x ) x a f (x ) x + b xf (x ) x 6 a 1 + b E [X ]

a + bE [X ]

As the above example demonstrates, for a constant c, E [c ] = c , and E [cX ] = cE [X ]

By denition of the PDF and E[X] respectively.

Random Variables Measures of Dispersion

Monotonic Transformations

Expectations

Measures of Dispersion

Just as there are several measures of central tendency, so too are there several measures of dispersion. The two most commonly used are the range and the variance. The range is dened as: range(X) = max(X) - min(X). Note that it is usually very uninformative. A more useful description of a random variables dispersion is given by the variance. Intuitively, the variance of a random variable measures how much the random variable X typically" deviates from its typical" value (or distance from the population mean). Mathematically: Var [X ] = E [(X E [X ])2 ]. 2 = E [(X E [X ])2 ] = E [X ])2 f (x ) if X is discrete (x E [X ])2 f (x ) x if X is continuous x
x (x

Random Variables Measures of Dispersion

Monotonic Transformations

Expectations

Measures of Dispersion

Why is the difference squared? Well, suppose not.


i (xi

E [X ])f (x ) = = = =

i xi f (x ) i E [X ]f (x ) E [X ] E [X ] i f (x )7 E [X ] E [X ]8 0.

In other words, the average deviation of X from its average is always 0; values above the mean cancel out values below the mean by denition of the mean. Consequently, we need to square the difference. Note that we could also take the absolute difference but that is harder" to work with mathematically.

7 8

By denition of E[X] and since E[X] is a constant respectively. Since i f (x ) = 1 by denition of probability.

Random Variables Measures of Dispersion

Monotonic Transformations

Expectations

Measures of Dispersion

EXAMPLE x f (x ) 1 .2 2 .2 3 .2 4 .2 5 .2

E [X ] = 1(.2) + 2(.2) + 3(.2) + 4(.2) + 5(.2) = .2 + .4 + .6 + .8 + 1 = 3 Var [X ] = E [(X E [X ])2 ] = (1 3)2 (.2) + (2 3)2 (.2) + (3 3)2 (.2) + (4 3)2 (.2) + (5 3)2 (.2) = 2.5

Random Variables Measures of Dispersion

Monotonic Transformations

Expectations

Measures of Dispersion

We can also derive another expression for the variance. E [(X E [X ])2 ] = = = = = E [X 2 2XE [X ] + (E [X ])2 ] E [X 2 ] 2E [XE [X ]] + E [(E [X ])2 ] E [X 2 ] 2E [X ]E [X ] + E [(E [X ])2 ] E [X 2 ] 2E [X ]2 + E [X ]2 E [X 2 ] E [X ]2

Random Variables Measures of Dispersion

Monotonic Transformations

Expectations

Measures of Dispersion
EXAMPLE Consider the continuous random variable with the PDF. Calculate Var[X]. f (x ) = 2(1 x ) 0
1 0

if 0 < x < 1 else

= E [X ] = = = = = = E [X ]2 =

x 2(1 x ) x
1 2 0 x x x 1 1 2 0 x x 0 x x 1 2 1 1 3 1 x ]0 3 x ]0 2 1 1 3 2 1 1 = 3 6 2 1 9

2 2 2 2 2
1 3

Random Variables Measures of Dispersion

Monotonic Transformations

Expectations

Measures of Dispersion

E [X 2 ] = = = = = = 2 = Var [X ]

1 0

x 2 2(1 x ) x
1 2 3 0 x x x 1 2 1 3 0 x x 0 x x 1 3 1 x ]0 1 x 4 ]1 0 3 4 1 1 3 4 1 1 = 6 12

2 2 2 2 2

= E [X 2 ] E [X ] 2 = 1 1 6 9 1 = 18

Random Variables Measures of Dispersion

Monotonic Transformations

Expectations

Measures of Dispersion

The interpretation of the variance may not be that intuitive, as it tells us the average squared deviation of a random variable X. Consequently, if X is measured in dollars, Var[X] is measures in squared dollars. To make interpretation easier, we often times refer to the standard deviation of a random variable. The standard deviation is simply the square root of the variance. More generally, we can characterize a random variable using a moment generating function. The moment generating function allows us to dene a sequence of moments which can completely characterize the probability distribution.
The kth moment around zero is dened as E [0 E [X ]]k or E [X ]k . Note that the rst moment about zero is the mean: E[X]. The kth moment around the mean is dened as: E [(X E [X ])k ]. The second moment about the mean is the variance.

Random Variables Measures of Dispersion

Monotonic Transformations

Expectations

Measures of Dispersion

Why is this terminology useful? Because it provides a common framework for talking about our measures of central tendency and dispersion. Higher moments about the mean also have special terms associated with them. The third moment around the mean E [(X E [X ])3 ] is called the skew of the distribution. The skew tells us whether the dispersion about the mean is symmetric (if skew = 0 ), or if it is negatively skewed (if skew < 0; implying that E[X] < median(X)) or positively skewed (if skew > 0; implying that E[X] > median(X)).

Random Variables Measures of Dispersion

Monotonic Transformations

Expectations

Measures of Dispersion
Skewness
Positive Skew
0.04 EX med[X] Density 0.02 0.00

50

100 x

150

Symmetric
0.015 0.010

Density

EX

0.005

med[X]

0.000

50

100 x

150

Negative Skew

0.04

EX med[X]

Density

0.00

0.02

50

100 x

150

Random Variables Measures of Dispersion

Monotonic Transformations

Expectations

Measures of Dispersion
Kurtosis

The fourth moment about the mean E [(X E [X ])4 ] is known as the kurtosis and it measures how thick the tails" of the distribution are.9

When measure using higher powers we often normalize and use

E [(X E [X ])k ] ( Var [X ])k

because E [(X E [X ])k ] increases dramatically as k increases.

Random Variables Measures of Dispersion

Monotonic Transformations

Expectations

Measures of Dispersion
Kurtosis

0.020

0.015

Density

Density 0 50 x 150

0.010

0.005

0.000

0.000 0 50 x

0.005

0.010

0.015

0.020

150

Random Variables Measures of Dispersion

Monotonic Transformations

Expectations

Measures of Dispersion
Example: Normal Distribution

EXAMPLE The Normal Distribution (also sometimes called the Gaussian Distribution) is completely characterized by two moments. The PDF is: 2 1 (x ) 1 f (x |, 2 ) = e 2 2 2 where E [X ] = and Var [X ] = 2 . Since two parameters dene a normal distribution, we denote this by: X N (, 2 ). In doing empirical work, it is always important to understand what the characteristics are of the data you are working with. A good rst step is to calculate the summary statistics of variables you work with as well as plot them out so you can visualize what you are working with. Doing so will also (hopefully) help you catch errors and identify outliers for further investigation.

Random Variables Measures of Dispersion

Monotonic Transformations

Expectations

Measures of Dispersion

If X is a random variable with nite variance, then for any constants a and b, Var [aX + b] = = = = = E [(aX + b) E [(aX + b)]2 E [aX + b aE [X ] b]2 E [aX aE [X ]]2 a2 E [X E [X ]]2 a2 Var [X ]