COMPARISON OF BAYESIAN AND CLASSICAL META-ANALYSIS

Joe P King Educational Psychology Measurement, Statistics, and Research Design

Meta Analysis
Overview of Meta-Analysis Traditional Meta Analysis Bayesian Meta Analysis Why are they different? Why is it important? Comparing methods using experimental data

Implications
Conclusions

Introduction
Meta analysis is used in every field of scientific research

Meta analysis allows us to compile many studies to test a

theory across many studies. The goal is to take the many studies which have looked at an outcome variable and try to inform on a theory. This satisfies one of the tenants of science that research must be reproducible and not one study can confirm or deny a theory

Introduction
Methods of Meta Analysis
Problem formulation
Data collection and selection of relevant studies Evaluation of data collected Interpretation and analysis Presentation of results

Example
Study by Rakes, Valentine, McGatha, and Ronau (2010)

analyzed different methods of algebra instruction they found 82 relevant studies, 109 independent effect sizes, and a total sample size of 22424 students. Searched electronic journals for relevant articles, calculated effect sizes and in some cases weighted effect sizes due to different sampling techniques or small sample size In 5 strategies of teaching algebra they found statistical significance for each in at least one model, which should inform on how math instructors teach algebra.

Treatment
Communities that Care Seeks to prevent anti-social behaviors among the youth in a community and strengthen pro-social behaviors. Provides community leaders with training, materials and technical assistance in the advancement of the program (Coie, Watt, & West, 1993; Mrazek, Haggerty, & Committee on Prevention of Mental Disorders, 1994).

Data
This replication summarizes 12 independent studies of

the effects of communities that care 12 pairs of two communities were formed based on comparable demographic data. After pairing, one city was randomly assigned to receive the treatment, the other served as the control. The treatment community in each pair was trained in and delivered the communities that care program for 5 years. Data was collected on specific adolescent measures, here binge drinking and delinquent behavior.

Outcomes
Two Measures were Used for this analysis
Delinquent behavior

Fifth grade, delinquent behaviors were categorized as

stealing, property damage, shoplifting, and attacking someone with intention of hurting them. Eighth grade, delinquent behaviors were carrying a gun to school, beating up someone, stealing a vehicle, selling drugs, and being arrested.
Binge Drinking -

5 or more drinks at once occasion, and the measurement was how many instances of binge drinking per month

Traditional Meta Analysis

Effect size was collected for each pair of cities

Used multilevel modeling with two levels Level 1 was the pairs of cities and accounting for the

within experiment variance. Level 2 sought to account for the between experiment variance.

Why Bayes?
Specify Priors (this analysis uses non-informative priors)

Can see the effect of adding each experiment on effect

size and precision associated with the estimate. Less Uncertainty in the experiment Easier to implement

Bayesian Analysis
Replication of Howard, Maxwell, & Fleming (2000)

They used Bayesian Calculations to calculate a posterior

distribution of 3 studies. Basics of Bayesian Approach Prior Distribution Initial estimate of effect size Likelihood Principle Effect size estimate of data Posterior Distribution Final Effect Size Combining Prior and Likelihood.

Calculations
What we know
Effect size for each Comparison
Uncertainty within Effect Size

What we will calculate

Precision Posterior Effect Size for Each City then All

Cities Uncertainty Around the Effect Size

Precision

Effect Size

Results Delinquent Behavior

Effect Size Traditional Meta Analysis Bayesian Meta Analysis -0.31 -0.17 SE 0.1400 0.0025

Results Binge Drinking

Effect Size Bayesian Meta Analysis -0.43 SE 0.0011

Results for Each Comparison

Posterior Distributions for Deliquent Behavior
0.8 0.6

0.4

0.2

Effect Size

0.0

-0.2

-0.4

-0.6

-0.8

-1.0 1 2 3 4 5 6 7 8 9 10 11 12

Comparison Number

Cumulative Results After Each Iteration

Posterior Distributions for Deliquent Behavior
0.6

0.5

0.4

0.3

Effect Size

0.2

0.1

0.0

-0.1

-0.2

-0.3 1 2 3 4 5 6 7 8 9 10 11 12

Posterior Iteration

Results for Each Comparison

Posterior Distributions for Binge Drinking
2.5 2.0

1.5

1.0

Effect Size

0.5

0.0

-0.5

-1.0

-1.5

-2.0 1 2 3 4 5 6 7 8 9 10 11 12

Comparison Number

Cumulative Results After Each Iteration

Posterior Distributions for Binge Drinking
2.5

2.0

1.5

Effect Size

1.0

0.5

0.0

-0.5

10

11

12

Posterior Iteration

Summary
Meta Analysis is important to research

Many methods exist yet pose limitations

Bayesian approach is an additional method that shows

promise.

Thank you
Dr. Charles Peck

Dr. Robert Abbott

Dr. Joe Lott Kelly Jewell Mom and Dad

References
Coie, J., Watt, N., & West, S. (1993). The science of prevention: A conceptual framework and some directions for a national research program. American Psychologist, 48(10), 1013-1022. Hedges, L. V., & Hedberg, E. C. (2007). Intraclass Correlation Values for Planning Group-Randomized Trials in Education. Educational Evaluation and Policy Analysis, 29(1), 60-87. Howard, G. S., Maxwell, S. E., & Fleming, K. J. (2000). The proof of the pudding: an illustration of the relative strengths of null hypothesis, meta-analysis, and Bayesian analysis. Psychological Methods, 5(3), 315-332. Mrazek, P. J., Haggerty, R. J., & Committee on Prevention of Mental Disorders, I. on M. (1994). Reducing risks for mental disorders: Frontiers for preventive intervention research. Washington, D.C. National Academy Press. Noble, J. H. (2006). Meta-analysis: Methods, strengths, weaknesses, and political uses. Journal of Laboratory and Clinical Medicine, 147(1), 7-20. R Development Core Team. (2010). R: A Language and Environment for Statistical Computing. Vienna Austria: R Foundation for Statistical Computing.