Professional Documents
Culture Documents
3 Significance Testing
The topics in this section concern with the second course objective.
The topics in this section heavily depend on those in the last section. I
strongly recommend reading the last section at least one more time before
starting on this section.
The very hypotheses that were tested by hypothesis testing can be tested by
significance testing. Significance testing uses the same test statistics (to test
the same corresponding hypotheses) as those used in hypothesis testing.
Ironically, as you will see, significance testing has no significance level.
Here is the difference between significance testing and hypothesis testing.
The conclusion of a significance test is always reject Ho with the p-value.
However, the part reject Ho is always implicitly understood. As a result,
only the p-value is given as the conclusion of significance testing. See the
diagram of significance testing given below.
Data
Test Statistic
P-Value
Significance Testing
At this point, your question should be What the heck is p-value?
In significance testing, after the computation of the observed value from
data, a probability is found using this computed observed value. This
probability is called the p-value. The p stands for probability. It has
It would be truly helpful if you understand that p-value is probabilityvalue. Read Appendix: P-Values given below in this section.
Now, at this point, you should be asking how to find a p-value.
You compute the observed value (from data using an appropriate test
statistic). Generally, this observed value splits the entire probability of one
into two probabilities, one probability on the left and another probability on
the right of the observed value. See the diagram given below.
A probability is the area between the curve and horizontal axis when
the probability distribution is continuous. All the test statistics in
this section have continuous distributions (such as the standard
Normal, Students t, and Chi-square distributions).
The probability on the left
of the observed value (ov)
The probability on
the right of the
observed value (ov)
The probability is
the p-value for a
right-tailed case
since it is on the
right of the
observed value.
choice if you are multiplying one of them by two to obtain the p-value
because a p-value is a probability. Read Appendix: P-values again.
Of course, if the two probabilities are equal to each other, you can multiply
either probability by two to obtain the p-value in a two-tailed case.
Why?????!!!!!! Well, you could try this with each of the two equal
probabilities and compare their p-values.
As you may have noticed, there is no significance level given in significance
testing. However, the p-value is also called observed significance level,
and the term significance in significance testing has come from the term
significance in observed significance level. Do not confuse with an
observed significance level (p-value) and a significance level ().
When you hear significance level, you should think of hypothesis
testing.
When you hear significance testing, you should think of observed
significance level (p-value).
An observed significance level is a probability determined by the observed
value and the alternative hypothesis.
A p-value is an observed significance level.
Here is actually how to find numbers (values) for p-values. You need to
compute observed values from test statistics. As you remember from the last
section, the test statistics are
( x - o)/(s/ n ) for testing hypotheses on ,
^
with n 1 degree of freedom (with large data). Now, read Appendix: How
to Find P-Values given below.
Let us have a numerical example by conducting significance testing on
Ho: 20.0
vs
Ha: > 20.0
from the Orange Crate Example. The test statistic is ( x - o)/(s/ n ) which
is the Normal standardization of the sample average, x , the estimator for .
The sampling distribution of the test statistic is, of course, the standard
Normal distribution.
The observed value is 1.2086 = (20.22
20.00)(0.997/ 30 ) . This is a right-tailed case so the p-value is
P(Z > 1.2086) = 0.1134
where Z is the standard Normal variable. This probability is found using the
website given in Appendix: How to Find P-Values, and it is the p-value
for this significance test. This significance testing concludes by stating
The p-value is 0.1134 for the significance testing on the hypotheses given
above.
By the way, if you were testing
Ho: = 20.0
vs
Ha: 20.0,
then the p-value would be 0.2268 (= 0.1134*2) since it is a two-tailed case.
If this were a left-tailed case, then the p-value would be 0.8866. Can you
tell why it is 0.8866? At any rate, this means that you cannot pick a smaller
probability for the p-value for a one-tailed test all the time.
Significance testing is a system whose input is data and its output is the pvalue (see the first diagram given above in this section). That is,
significance testing ends with the p-value. However, with the p-value,
hypothesis testing can be done. That is, you can conduct hypothesis testing
after significance testing.
Namely, once a significance testing is conducted and concluded with its pvalue, a hypothesis testing can be conducted with the p-value if so desired.
If the p-value is less than or equal to the significance level (p-value ),
then the null hypothesis Ho is rejected at .
This is because, if the p-value is less than or equal to , then the observed
value must be in the rejection area. See the diagram given below.
A right-tailed case: The p-value (the area in blue) is less than (the area in the red;
that is, the area to the left of the p-value and p-value itself together) and, hence, the
observed value is in the rejection area. As a result, the Ho is rejected at .
The p-value
If the p-value is greater than the significance level (p-value > ), then the
null hypothesis Ho is not rejected .
This is because, if the p-value is greater than , then the observed value
must be outside the rejection area. See the diagram given below.
A right-tailed case: The p-value (the area in the red; that is, the area to the left of the
and itself together) is greater than the (the area in blue) and, hence, the
observed value is outside the rejection area. As a result, the Ho is not rejected at .
The p-value
vs
Ha: P({D}) 0.40.
Then, this is a two-tailed case, and the test statistic is
^
0.40, p = P({D})
= 357/850 = 0.42 and n = 850. The observed value is
(0.42 0.40)/ 0.40(1 0.40) / 850 = 1.1902. This results in the p-value of
0.234. See Appendix: How to Find P-Values given below to find the pvalue.
If hypothesis testing is to be performed on these hypotheses, then Ho cannot
be rejected at the 1%, or even 5%, significance level.
Finally, the p-value is the maximum probability for Ha to be true in light of
the obtained data. For instance, a significance test on
Ho: P(D) P(R)
vs
Ha: P(D) < P(R)
is conducted in the Political Poll Example. This test produces the p-value of
3.6%. This p-value of 3.6% means that the maximum chance for the Ho:
P(D) P(R) to be true is 3.6% given the data. If the hypothesis testing is
conducted, the null hypothesis is rejected and accepted is the alternative
hypothesis of the P(R) is greater than P(D) at 5%. However, it fails to reject
the null hypothesis at 1%.
As you have realized, if you can do significance testing, you can do
hypothesis testing with the p-value. Significance testing is performed by
finding the right test statistic and its probability distribution. The observed
value, computed from the test statistic and data, gives you the p-value (using
tables or a website). If the p-value is less than a significance level, reject
Ho. Otherwise, Ho cannot be rejected.
Appendix: P-Values
A p-value is the maximum possible probability for data falsely rejecting a
null hypothesis. Thus, a p-value is a probability value. Understanding of
this fact would prevent you from making a mistake like giving a number
greater than one or a negative number for a p-value like many students do.
Why is this so? Well, did you understand what probability is numerically?
Also, you multiply a probability by two when you find a p-value in a twosided case. You would not multiply a probability greater than 0.5 by two to
find a p-value. Why? If this is not obvious to you, please think through. I
hope you will find the reason why one should not multiply a number greater
than 0.5 by two to find a p-value.
By the way, this is where the difference between memorization and
understanding comes in play (to help you or to kill you).
Appendix: Recommendations
At this point, if you are not sure what we are talking about in this section,
please go back and read the last section, 3.2 Hypothesis Testing. You
should have learned and understood test statistics, observed values, lefttailed tests, right-tailed tests, two-tailed tests and such in the last section.
These topics (concepts) are used in the subject (significance testing) and its
related topics of this section. Lack of understanding of the topics/concepts
in the last section would impede your understanding of the subject and the
related topics in this section. On the other hand, their understanding would
enable you to understand significance testing and its related topics.
10
or
http://sites.csn.edu/mgreenwich/stat/normal.htm
For a right-tailed case, click on the circle under the second picture, put the
ov under z value and click on the button . This should give you the pvalue under probability.
For a left-tailed case, click on the circle under the first picture and repeat the
above. You should get the p-value under probability.
For a two-tailed case, click on the circle under the fourth picture and repeat
the above. You should get the p-value under probability. Do not multiply
this p-value by two. It is the p-value, given under probability for a twotailed test. This is because, see the picture, the probability given under
11
probability is P(Z < ov) + P(Z > ov) = 2* Min{ P(Z > ov), P(Z < ov)} =
the p-value.
Let us have some exercises. Conduct significance testing (that is, find the pvalue) on Ho: p 0.23 vs Ha: p > 0.23 based on data of size 800 whose
estimate for p is 0.21.
The answer: The p-value is 0.9106. Hint: The ov = -1.3442 and it is a
right-tailed case.
Conduct significance testing on Ho: = 41.85 vs Ha: 41.85 based on
data of size 36 whose sample mean and sample standard deviation are 42.57
and 2.5 respectively.
The answer: The p-value is 0.084. Hint: The ov = 1.728 and it is a twotailed case. Remember, you do not multiply the probability by two when
you use the website.
For testing hypotheses on , the p-values are
P (n-12 > ov) for a right-tailed case,
P( n-12 < ov) for a left-tailed case, and
2*Min{ P( n-12 > ov), P( n-12 < ov)} for a two-tailed case,
where n-12 is the Chi-square variable with n 1 degree of freedom. Now
go to the website,
http://surfstat.anu.edu.au/surfstat-home/tables/chi.php
or
http://sites.csn.edu/mgreenwich/stat/chi.htm
For a right-tailed case, click on the circle under the second picture, put the
degree of freedom under d.f. and put the ov under 2 value. Then, click
on the button which should give you the p-value under probability.
For a left-tailed case, click on the circle under the first picture, and the repeat
the above. You should find the p-value under probability.
12
For a two-tailed case, you repeat the above for both right-tailed and lefttailed cases. Find the smaller probability under probability and multiply it
by two, which is the p-value. This time, you have to multiply it by two.
Let us have some exercises. Conduct significance testing on Ho: > 35.0
vs Ha: 35.0 based on data of size 35 whose sample mean and sample
standard deviation are -35.7 and 32.5 respectively.
Answer: The p-value is 0.3035. Hint: The ov = 29.316 and the degree of
freedom is 34. It is a left-handed case.
Conduct significance testing on Ho: = 3.7 vs Ha: 3.7 base on data of
size 48 whose sample mean and sample standard deviation are 3.4 and 4.2.
The answer: The p-value is 0.1768. Hint: The ov = 60.561 and the degree
of freedom is 47. The two probabilities are 0.9116 and 0.0884.
13
Ho: 1 - 2 o
vs
Ha: 1 - 2 < o
Two-tailed case:
Ho: 1 - 2 = o
vs
Ha: 1 - 2 o
14
We randomly sampled 40 oranges from the second crate and took the weight
measurements of these 40 oranges. Their sample mean is 29.87 oz, and their
or
http://sites.csn.edu/mgreenwich/stat/normal.htm
and find the p-value or you realize that it must be very close to zero (since
ov is -6.054). We have the p-value = 0. The Ho is rejected at 5% or even at
1% level. There is very strong evidence that, on average, the oranges in the
second crate are heavier than those in the first crate by at least 5 oz.
Let us, now, test the claim on the variation. The hypotheses are
Ho: 2/1 0.80
vs
Ha: 2/1 < 0.80.
The test statistic to test hypotheses on standard deviations of two
independent groups is
((n1 1)s12 /12)/((n2 1)s22/ 22)
which has an F distribution. Please read Appendix: F Distributions & Pvalues given below.
15
By the way, this test statistic consists of a test statistic for testing hypotheses
on one sigma over another test statistic for testing hypotheses on another
sigma.
At any rate, we have 2 = 0.80*1, s1 = 0.997, and s2 = 0.679. As a result,
((30 1)(0.997)2/ 12)/((40 1)(0.679)2/( 0.80*1)2) = 1.02604,
which is the observed value. By going to the website,
http://stattrek.com/Tables/F.aspx,
you find the p-value of 0.46. This is a right-tailed case and, to learn how to
find a p-value, please read Appendix: F Distributions & P-values again.
The test statistic
^
^
^
^
^ ^
( p 1 - p 2 o )/ p 1(1- p 1)/n1+ p 2(1- p 2)/n2
16
Two-tailed case:
Ho: p1 - p2 = o
vs
Ha: p1 - p2 o
The test statistic has the standard Normal distribution for large n1 and n2.
Let us have a numerical example. In Indiana, a political poll of 850 was
taken, and 54% of the likely voters surveyed would vote for the Republican
candidate (recall the Political Poll Example). In Nevada, a political poll of
900 was conducted, and 51% of the likely voters surveyed would vote for
the Republican candidate. I am wondering whether P({R}) = p1 in Indiana
is greater than P({R}) = p2 by at least 2% or not. So, the hypotheses to be
tested are
Ho: p1 - p2 0.02
vs
Ha: p1 - p2 > 0.02
or
http://sites.csn.edu/mgreenwich/stat/normal.htm
you find the p-value of 0.3376. You cannot reject the null hypothesis at 5%
or 1%, and there is no evidence that the Republican candidate in Indiana has
at least a 2% point edge over the Republican candidate in Nevada in spite of
the estimates.
17
You should not compare P({R}) and P({D}) in the same state since these
18
than -10. If it were greater than -10, then the post-scores would not be
higher than the pre-scores by at least 10 points on average as speculated.
The hypotheses are tested by using the test statistic ( x - o)/(s/ n ) with the
standard Normal distribution just like a regular significance or hypothesis
testing on . By the way, x and s are the sample average and standard
deviation of the 30 differences given in the last column of the table given
below, o is -10 and n is 30 in the example.
Pre-scores
65
43
54
Post-scores
78
83
81
Differences
-13
-40
-27
.
.
.
.
Student 28
Student 29
Student 30
67
36
42
87
32
65
-20
4
-23
19
20
This is the reason why an F distribution has two degrees of freedom, d1 and
d2. The first one is always from the denominator and the second one from
the denominator. For the test statistic, it is an Fn1-1, n2-1. Go to the website,
http://stattrek.com/Tables/F.aspx,
put the first degree of freedom at Degrees of freedom (v1), the second degree of
freedom at Degrees of freedom (v2) and the observed value at f value and click
on the button Calculate. You find the p-value for a left-tailed case at
Cumulative probability. That is, Cumulative probability = P(Fn1-1, n2-1< observed
value) = p-value for a left-tailed case. Thus, the p-value for a right-tailed
case is 1 Cumulative probability and the p-value for a two-tailed case is
2*Min{ Cumulative probability,1 - Cumulative probability}.
For example, you are testing Ho: 2/1 0.80 vs Ha: 2/1 < 0.80 with n1
1 = 30 1 = 29, n2 1 = 40 1 = 39, and the observed value of 1.02604.
Then, put 29 at Degrees of freedom (v1), 39 at Degrees of freedom (v2), 1.02604 at f
value and click on the button Calculate.
You find 0.54 at Cumulative
probability. However, this 0.54 is the p-value in a left-tailed case. For the
right-tailed case, which is our case here, the p-value is 0.46 = 1 0.54. The
Ho cannot be rejected at 5% or 1% significance level.
21