Statistics 512 Notes 22: Wrap up of Sufficiency, Most Powerful Tests

Rao-Blackwell Theorem: Theorem 7.3.1 (stated a little differently): Let X 1 ,K , X n be an iid sample from the pdf or pmf f ( x; ) , . Let u ( X1 ,K , X n ) be a sufficient statistic for . Let = W ( X , K , X ) be an estimator of . Because
1 n

u ( X1 ,K , X n ) is a sufficient statistic for , % = E ( | u ( X 1, K , X n ) = u; ) is a function of X 1 ,K , X n that is independent of . The theorem is that for all , % MSE ( ) MSE () % The inequality is strict unless = .
Note: If u ( X1 ,K , X n ) is not a sufficient statistic for , then E ( | u ( X , K , X ) = u; ) is not an estimator since it depends on .
1 n

Application of the Rao-Blackwell Theorem: Suppose X 1 , X 2 , X 3 are iid Bernoulli random variables with success probability p . Consider the estimator X + 2 X 2 + 3X 3 p= 1 . 6 Summing over the possible samples ( X 1 , X 2 , X 3 ) = (0, 0, 0), (0, 0,1), (0,1, 0), (0,1,1), (1, 0, 0), (1, 0,1), (1,1, 0), (1,1,1) , we have

 MSE p ( p) = (1 p )3 (0 p) 2 + p(1 p ) 2 (1/ 2 p ) 2 + p 2 (1 p)(5 / 6 p) 2 + p (1 p ) 2 (1/ 6 p ) 2 + p 2 (1 p)(2 / 3 p) 2 + p 2 (1 p )(1/ 2 p ) 2 + p 3 (1 p ) 2

p is not a function of the sufficient statistic Y = X 1 + X 2 + X 3 . We can use the Rao-Blackwell Theorem to improve p . Consider the estimator X + 2 X 2 + 3 X3 % p = E 1 | X1 + X 2 + X3 = y . 6 We have X + 2 X 2 + 3X3 E 1 | X1 + X 2 + X3 = 0 = 0 6 X + 2 X 2 + 3X3 1 E 1 | X1 + X 2 + X3 = 1 = 6 3 X + 2 X 2 + 3X3 2 E 1 | X1 + X 2 + X3 = 2 = 6 3 X + 2 X 2 + 3X3 E 1 | X1 + X 2 + X3 = 3 = 1 6

p (1 p) . Here is a 3 % comparison of the MSEs for p and p for some values of p . p MSE ( p) MSE ( p) .25 0.073 0.063 .5 0.097 0.083 .75 0.073 0.063 % % Thus, p = X . We have MSE p ( p ) =
p p

Limitation of Rao-Blackwell Theorem for Finding Best Estimator: Suppose there are two estimates 1 and 2 having the same expectation. Assuming that a sufficient % statistic Y exists, we may construct two other estimates 1 % and ,by conditioning on Y. The Rao-Blackwell
2

Theorem gives no clues as to which one of these two is better. If the probability distribution of Y has a property called completeness, then % % 1 and 2 are identical, by a theorem of Lehmann and Scheffe. This topic is pursued in the rest of Chapter 7 but we shall not cover not it.

Optimal Hypothesis Testing

Review on Hypothesis Testing Goal: Decide between two hypotheses about a parameter of interest H 0 : 0

H1 : 1 ,
where 0 U 1 = . Null vs. Alternative Hypothesis: The alternative hypothesis is the hypothesis we are trying to see if there is strong evidence for. The null hypothesis is the default hypothesis that we will retain unless there is strong evidence for the alternative hypothesis.

Critical region: A test is defined by its critical region. Let S denote the support of the random sample ( X 1 ,K , X n ) . The subset C of S for which we reject the null hypothesis is called the critical region, i.e., our decision rule is Reject H 0 if ( X 1 ,K , X n ) C C Retain (Do not reject) H 0 if ( X 1 ,K , X n ) C . Note: Here I am following the book in defining a critical region in terms of the sample space rather than a test statistic as I did in Notes 5-6. Errors in hypothesis testing: True State of Nature Decision H 0 is true H1 is true Type I error Correct decision Reject H 0 Accept (retain) H 0 Correct decision Type II error The best critical region would make the probability of a Type I error small when H 0 is true and the probability of a Type II error small when H1 is true. But in general there is a tradeoff between these two types of errors. Size of test, power of test: Power function of test = C ( ) = P (( X 1 ,K , X n ) C ) = Probability of rejecting null hypothesis when true parameter is .

Size of test = max 0 C ( ) Power at an alternative 1 = C ( ) Neyman-Pearson paradigm: Choose size of test to be reasonably small to protect against Type I error, typically 0.05 or 0.01. Among tests which have prescribed size, choose the most powerful test. What is the most powerful test? Example: Consider one random variable X that has a binomial distribution with n=5 and p = . Suppose we want to test H 0 : = 0.5 vs. H 1 : = 0.75 Let f ( x; ) denote the pmf of X . The following table gives, at points of positive probability mass, the value of f ( x;0.5), f ( x;0.75) , and the ratio f ( x;0.5) / f ( x;0.75) . x f ( x;0.5) f ( x; 0.75) f ( x;0.5) / f ( x;0.75) 0 1/32 1/1024 32/1 1 5/32 15/1024 32/3 2 10/32 90/1024 32/9 3 10/32 270/1024 32/27 4 5/32 405/1024 32/81 5 1/32 243/1024 32/243 What is the best critical region of size = 32 ? Two critical regions have size f ( x;0.5) / f ( x;0.75) : (1) C1 = { X = 0}
1

(2) C2 = { X = 0} Power of C1 = P( X = 0; = 0.75) = 1/1024 . Power of C2 = P ( X = 5; = 0.75) = 243 /1024 The test with critical region C2 = { X = 0} is the most powerful test of size = 32 . The Neyman-Pearson Lemma provides a systematic way of finding most powerful tests for testing a simple null hypothesis versus a simple alternative hypothesis. Theorem 8.1.1 (Neyman-Pearson Lemma): Let X 1 ,K , X n be an iid sample from the pdf or pmf f ( x; ) Suppose we want to test H 0 : = ' vs. H1 : = '' Then (1) any test with a critical region C of the following form (where k is some positive number) is a most powerful test of size : (a) (b)
f ( x; ') L( ';( X 1 ,K , X n )) = k for each point ( X 1 ,K , X n ) C f ( x; '') L( '';( X 1 ,K , X n )) 1

f ( x; ') L( ';( X 1 ,K , X n )) = k for each point ( X 1 ,K , X n ) C C f ( x; '') L( '';( X 1 ,K , X n ))

0

(c) = PH [( X 1, K , X n ) C ] (2) A necessary condition for a test to be a most powerful test of level is that it satisfy conditions (a), (b) and (c).

Proof: We will follow the proof in the textbook for (1). Remark 8.1.1 discusses (2).