WWC (PHASE I) COMPUTATION TOOLS

June 2009

This
Excel file contains
five workbooks
describedare
below.
The computations
contained
in these workbooks
based on the Technical Details of WWC-Conducted Computations docume
during the first phase of the What Works Clearinghouse (WWC), which is available online at:
http://ies.ed.gov/ncee/wwc/pdf/conducted_computations.pdf
Workbook
ES_Formulas
ES_ContinuousY
ES_DichotomousY
Clustering_Correction
BH_Correction
Technical contact:

WWC (PHASE I) COMPUTATION TOOLS

June 2009

etions
contains
five workbooks
describedare
below.
contained
in these workbooks
based on the Technical Details of WWC-Conducted Computations document created
st phase of the What Works Clearinghouse (WWC), which is available online at:
gov/ncee/wwc/pdf/conducted_computations.pdf
Description
Effect size (ES) computation formulas for both continuous and dichotomous outcomes
Workbook for computing ESs for continous outcomes under various scenarios, with two examples under each scenario
Workbook for computing ESs for dichotomous outcomes
Workbook for computing clustering-corrected statistical significance of ESs
Workbook for applying Benjamini-Hochberg correction for multiple comparisons
Dr. Mengli Song, American Institutes for Research, Email: msong@air.org.

r each scenario

ES Data

(assume

Formula

sample sizes known)

Continuous Outcomes

X1 X 2

Unadjusted posttest means and SDs

g=

T-test result

g=t

ANOVA F-test result

g=

ANCOVA adjusted posttest means and

unadjusted posttest SDs

g=

ANCOVA unadjusted pretest and posttest

means and unadjusted posttest SDs

g=

ANCOVA unajdusted pretest and posttest,

pretest/posttest correlation, and unadjusted
posttest SDs

g=

ANCOVA F-statistic and pretest/posttest

correlation

g=

HLM level-2 coefficient for intervention effect

and unadjsted posttest SDs

g=

(n1 1) S12 (n 2 1) S 22
(n1 n 2 2)

n1 n2
n1 n2
F (n1 n2 )
n1n2
X 1 ' X 2 '
(n1 1) S12 (n 2 1) S 22
(n1 n 2 2)
( X 1 X 1 pre ) ( X 2 X 2 pre )
( n1 1) S12 ( n 2 1) S 22
( n1 n 2 2)

( X 1 X 2 ) r ( X 1 pre X 2 pre )
(n1 1) S12 (n2 1) S 22
(n1 n2 2)

F (n1 n2 )(1 r 2 )
n1 n 2

(n1 1) S12 (n2 1) S 22

(n1 n2 2)

Dichotomous Outcomes
proportion/probabiblity for each group

OR =

p1 (1 p 2 )
p 2 (1 p1 )

p1 (1 p 2 )
p 2 (1 p1 )

LOR = In(OR)
LORcox = LOR/1.65

Notation
X1: unadjusted postest mean for the intervention group
X2: unadjusted postest mean for the comparison group
n1: sample size for the intervention group
n2: sample size for the comparison group
S1: standard deviation of the intervention group
S2: standard deviation of the comparison group
t: t-statistic for group mean difference

F: ANOVA F-statistic for group mean difference

X1': ANCOVA adjusted postest mean for the intervention group

X2': ANCOVA adjusted postest mean for the comparison group

X1-pre: pretest mean for the intervention group

X2-pre: pretest mean for the comparison group

r: correlation between prestest and posttest

F: ANCOVA F-statistic for adjusted group mean difference

: HLM level-2 coefficient for the intervention's effect, representing

group mean difference adjusted for level-1 covariates

OR: odds ratio

p1: propability of the occurrence of the invent for the intervention group

p2: propability of the occurrence of the invent for the comparison group
LOR: logged odds ratio
LORcox: Cox index

ES Computation for Continuous Outcomes

Note: Data entry in black; programmed computations in blue.

Type of ES Data

Unadjusted posttest means and SDs

T-statistic

ANOVA F-statistic

ANCOVA adjusted posttest means

and SDs
ANCOVA unadjusted pretest &
posttest means and SDs
ANCOVA unajdusted pretest and
posttest, pretest/posttest correlation,
and unadjusted posttest SDs

Outcome
Measures
Outcome A
Outcome B
Outcome C
Outcome A
Outcome B
Outcome C
Outcome A
Outcome B
Outcome C
Outcome A
Outcome B
Outcome C
Outcome A
Outcome B
Outcome C
Outcome A
Outcome B
Outcome C

pretest

Data Entry: ES Data Reported

Treatment group
adjusted
posttest
sample
posttest SD (S1)
(X1)
size (n1)
(X1)
3.06
2.74

0.99
1.11

173
173
212
26
22
65

56.03
66.23

62.45
78.98

56.03
66.23

62.45
78.98

67.50
32.25

6.40
4.33

34
48

6.42
12.75

9.63
11.02

80
80

9.63
11.02

80
80

ta Entry: ES Data Reported by the Study Authors

Control group
adjusted
posttest
pretest
posttest
SD (S2)
(X2)
(X2')
2.76
2.38

1.16
1.29

ES Com
sample
size (n2)
169
169
134
26
25
65

58.64
65.33

62.03
77.65

58.64
65.33

62.03
77.65

pretestT-statistic F-statistic posttest

correlation

68.6
30.25

8.71
5.46

36
48

3.39
12.32

8.75
10.67

78
77

8.75
10.67

78
77

Pooled
SD
1.08
1.20
0.00

1.56
2.31
3.14
1.02

0.75
0.82

7.68
4.93
0.00
9.21
10.85
0.00
9.21
10.85
0.00

ES Computation
ES (g)

Bias corrected
ES (g)

U3

Improvement
Index

0.28
0.30
#DIV/0!
0.17
0.64
#DIV/0!
0.52
0.18

0.28
0.30
#DIV/0!
0.17
0.63
#DIV/0!
0.51
0.18

0.61
0.62
#DIV/0!
0.57
0.74
#DIV/0!
0.69
0.57

0.11
0.12
#DIV/0!
0.07
0.24
#DIV/0!
0.19
0.07

-0.14
0.41
#DIV/0!
0.33
0.04
#DIV/0!
0.26
0.05
#DIV/0!

-0.14
0.40
#DIV/0!
0.33
0.04
#DIV/0!
0.26
0.05
#DIV/0!

0.44
0.66
#DIV/0!
0.63
0.52
#DIV/0!
0.60
0.52
#DIV/0!

-0.06
0.16
#DIV/0!
0.13
0.02
#DIV/0!
0.10
0.02
#DIV/0!

ES Computation for Dichotomous Outcomes

Note: Data entry in black; programmed computations in blue.
Outcome
Measures
Outcome A
Outcome B
Outcome C

Treatment group
Comparison Group
Proportion Sample
Odds
Proportion Sample
Odds
(p1)
size (n1) (Odds1)
(p2)
size (n2) (Odds2)
0.28
30.00
0.39
0.22
32
0.28
0.78
56.00
3.55
0.82
53
4.56
0.21
185.00
0.27
0.322
160
0.47
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00

ES Co
Odds
Ratio
1.38
0.78
0.57
#DIV/0!
#DIV/0!
#DIV/0!
#DIV/0!
#DIV/0!
#DIV/0!
#DIV/0!

Logged Odds ratio

(LOR)
0.32
-0.25
-0.56
#DIV/0!
#DIV/0!
#DIV/0!
#DIV/0!
#DIV/0!
#DIV/0!
#DIV/0!

ES Computation
Cox index
Bias corrected
(LORcox)
ES (g)
0.19
0.19
-0.15
-0.15
-0.34
-0.34
#DIV/0!
#DIV/0!
#DIV/0!
#DIV/0!
#DIV/0!
#DIV/0!
#DIV/0!
#DIV/0!
#DIV/0!
#DIV/0!
#DIV/0!
#DIV/0!
#DIV/0!
#DIV/0!

U3
0.58
0.44
0.37
#DIV/0!
#DIV/0!
#DIV/0!
#DIV/0!
#DIV/0!
#DIV/0!
#DIV/0!

Improvement
Index
0.08
-0.06
-0.13
#DIV/0!
#DIV/0!
#DIV/0!
#DIV/0!
#DIV/0!
#DIV/0!
#DIV/0!

Computing the Statistical Significance of ESs Corrected for Clustering

Note: Data entry in black; programmed computations in blue.
DATA ENTRY
Number of students in treatment group (Nt)
Number of students in control group (Nc)
Total number of students (N = Nt + Nc)

Outcome 1
166
147
313

Outcome 2
173
169
342

Number of clusters in treatment group (Mt)

Number of clusters in control group (Mc)
Total number of clusters (M = Mt + Mc)

10
9
19

10
9
19

ES based on student-level analysis ignoring clustering

0.39

0.42

Intraclass correlation (ICC)

0.13

0.10

Average cluster size (N/M)

16.47

18.00

t-statistic derived from ES that ignores clustering

t-statistic corrected for clustering

3.44
1.97

3.88
2.35

Adjusted degrees of freedome (h) -- nominator

Adjusted degrees of freedome (h) -- denominator
Adjusted degrees of freedome (h)

94234.78
376.71
250.15

113299.56
385.56
293.86

Significance (p-value) for corrected t with df=h

0.0498

0.0194

COMPUTATION

Outcome 3
173
169
342
10
9
19
0.37
0.13
18.00
3.42
1.90
112613.94
419.65
268.35
0.0589

Benjamini-Hochberg Correction for Multiple Comparisons for Domains with Multiple Outcomes
Note: Data entry in black; programmed computations in blue.

Procedures:
Step 1: enter only outcome measures with significant (clustering-corrected) p values in the accending order of the p values in C
Step 2: enter the number of comparisons within the domain, the clustering-corrected p-values, and the corresponding p-value r
Step 3: identify and highligt the outcome with the largest i (p-value rank) that has a "YES" in Column 6--THIS IS THE CUTOFF
Step 4: in Column 7, enter "significant" for this outcome and all the outcomes before this outcome (i.e., outcomes with a smalle
Outcome
Measures
Column 1
Outcome A
Outcome B
Outcome C
Outcome D

Number of comparisons
within the domain (M)

Clusetering-corrected p- value
(Pi)

P-value rank (i)

Column 2
7
7
7
7

Column 3
0.0047
0.0082
0.0230
0.0460

Column 4
1
2
3
4

ultiple Outcomes

in the accending order of the p values in Column 1.

p-values, and the corresponding p-value ranks (i) Columns 2, 3&4. Values in Columns 5&6 will be generated automatically.
YES" in Column 6--THIS IS THE CUTOFF POINT (Outcome B in the example).
this outcome (i.e., outcomes with a smaller i); enter "ns" for all outcomes below this outcome (i.e., outcomes with a larger i).

Pi' = i*0.05/M

Pi <= Pi' ?

Statistical significant after

BH correction ( = .05)

Column 5
0.0071
0.0143
0.0214
0.0286

Column 6
YES
YES
NO
NO

Column 7
significant
significant
ns
ns

rated automatically.

mes with a larger i).