Evaluu~ion and Program Pianning, Vol. 5, pp. 371-375, 1982
~/0
Printed in the U.S.A. All rights reserved.
BOOK ~VI~WS
Meta-Analysis in Social Research, Gene V. Glass, Barry S. McGaw and Mary Lee Smith. Beverly Hills: Sage
Publicatiork, Inc., 1981, 273 pp., $25.00 (hardcover).
Reviewers: David A. Kenny and Jeffrey S. Berman
Looking back over the 1970s there were a number of
major developments in methodology. These
developments include work in log linear analysis,
structural equation modeling, and time-series analysis.
However, it seems to us that the most important work
in the past decade was in the area of summarizing
results across studies by quantitative methods. This
area has been called meta-analysis. This effort is the
most promising not because of its complexity in
statistical analysis, but rather because it takes standard
methods and applies them to important questions in
social science. No longer are we limited in thinking of
persons within a study as the unit of analysis but now
the study itself becomes the unit of analysis. Meta-
analysis represents a bold new step in the analysis of
data.
The area of meta-analysis is developed enough
where a book is in order and Glass, McGaw, and
Smith have given us one. As they mention in their
preface they have given us a unified perspective on
meta-analysis but not the final word. They cover the
basic topics of research integration: how to locate
studies, how to classify studies, how to measure ef-
fects, and how to analyze studies. Moreover, they pro-
vide a rationale for meta-analysis as well as respond to
the critics of the enterprise.
For the novice the book provides a superb introduc-
tion to the topic. Real examples are continually used
and although there are a few obscure passages here
and there, the book is very readable to anyone familiar
with multiple regression and most of the book is in-
telligible to those familiar with only analysis of
variance. To someone planning to do a meta-analysis
the book is particularly lucid since so many concrete
examples are used. The authors even provide a sample
form for coding studies. To those who already have
some knowledge of the literature, the book is not as in-
structive since very little new ground is broken. A good
deal of the material has been published elsewhere. It is,
however, useful to have all of this material in one
volume and there is some new material (e.g., work on
nonindependence).
371
0149-7~89/82/~371-~03.
Copyright @ 1983 Pergamon Press Ltd
As mentioned earlier, the Glass, McGaw, and Smith
book is not the final word. We wish to raise three
issues that we hope that they would address in their
second edition.
Because quantitative reviews are relatively new,
there has been considerable variability in how such
analyses are handled and reported. Researchers seem
to be adopting the “kitchen sink” approach and they
overwhelm readers with ail sorts of numbers in their
meta-analyses. Instead of doing all possible types of
analyses (e.g., combining p values, computing effect
sizes, and multiple regression analyses), researchers
need some very clear guidelines about which pro-
cedures are appropriate. While somewhat helpful
Glass, McGaw, and Smith do not provide these
guidelines.
The purpose of an effect size measure is to provide a
standardized estimate of treatment effects that can be
compared across studies. For example, the Glass,
McGaw, and Smith’s effect size measure is the dif-
ference between treatment groups relative to the con-
trol group standard deviation. (We find it quite
curious that the authors do not acknowledge that the
measure of effect size preferred by most experts in the
area has as its denominator the pooled within groups
standard deviation and not the control group standard
deviation.) The difficulty with this approach,
however, lies in the possibility that within-group
variability may not be equivalent across the studies in
the domain. This nonequivalence problem is especially
im~rtant since most of the effort in a meta-analysis is
devoted to a comparison of effect sizes. If these effect
sizes differ because of design or other extraneous
features of the study, these comparisons can be
misleading. This nonequivalence of within-group
variance may result from differences in the subject
populations (e.g., some studies may have more
homogeneous samples than others), differences in the
reliability of the dependent measure, or in the way er-
ror variance is computed (e.g., effect sizes in some
studies may be based on ANCOVA or repeated-
measures ANOVA, resulting in much smaller
372 BOOK REVIEWS

estimates of error variance). The authors’ recommen- authors note this nonindependence can artificially
dations for extracting effect sizes (see chapter 5) do underestimate the variance of the effect sizes. A less
recognize this problem. However, the proposed solu- problematic approach would be to compute a single
tions often require information that is not typically measure of effect size from each study. If multiple
available from published reports and must therefore dependent variables are available and they are all in
be guessed. It is not clear to us that guessing is the best the same domain, an average effect size could be com-
way to make effect sizes comparable. The noncom- puted for each study. If the different dependent
parability of effect sizes from different studies may measures are assessing different constructs, separate
end up being one of the more difficult problems facing analyses could be carried out for each type of out-
those who conduct quantitative reviews. come.
One aspect of the Glass et al. procedures that has Meta-Analysis in Social Research is a book to be
raised controversy is their use of data from the same read and studied by social scientists. It provides the
study to generate multiple effect sizes and their treat- novice with a good introduction and the practitioner
ment of these estimates as if they were independent. with enough examples. Let us hope it acts like John the
With this method it is not uncommon for there to be Baptist in preparing for the gospel of meta-analysis to
twice as many effect size measures as studies. As the be written sometime during the next five years.

A Zero-Based Look at Zero-Base Budgeting, Thomas H. Hammond and Jack H. Knott. New Brunswick: Trans-
action Books, 1980, 148 pp., $14.95 (hardcover).

Reviewer: William R. Meyers

This important book, a masterful analysis of zero-base can make the competition for scarce resources more
budgeting, provides much insight into the political and intense and acrimonious, and they view achieving con-
administrative realities of the budgeting process. The sensus on goals as unnecessary if there is already agree-
Hammond and Knott book will be valuable to practic- ment on policy. They are even more skeptical that ac-
ing evaluators, to students of evaluation, and also to tivities and functions can be sharply defined and they
managers and administrators who have budgetary reject entirely the idea that they can or should be ag-
responsibilities. And there is little doubt that many of gregated into decision packages, that is, program
the issues they raise for the public sector are relevant categories that can logically be “prioritized.” The
also to the private sector because of the complex in- authors also note a problem with alternatives: it is
termediate programmatic, planning and budgetary hard to specify alternative means to the same goal,
considerations - for example, those concerning per- since alternative means reach different goals.
sonnel selection and training-that must intervene Moreover, owing to lack of data and lack of causal
before one can anticipate what the “bottom line” will theory, the question of which specific alternatives lead
turn out to be. to which particular goals has not been fully resolved.
Chapter 1, Introduction, describes the authors’ Hammond and Knott believe that in single-mindedly
research procedures: a careful analysis of all available focusing on the economic costs and benefits of any
case studies on the use of Zero-Base Budgeting (ZBB) given expenditure, ZBB advocates blind themselves to
on the federal, state, and local levels, and a systematic the political costs and benefits of categorizing any
effort to relate these findings to what is already known given activity in different ways.
about budgeting, organizations, and politics from ex- Chapter 3, Planning Plus Scarce Resources Equals
tensive past experience and from the literature. Thus Budgeting, considers allocation of dollars among the
the authors are able to link their conclusions about packages. Hammond and Knott point out that instead
ZBB with broader issues in politics and public ad- of integrating planning with budgeting, as it claims to,
ministration. The Introduction also provides an over- ZBB actually separates planning from budgeting by re-
view of the book, and gives some genera1 conclusions. quiring that we first choose our preferred way of ac-
Chapter 2, A Political Analysis of ZBB’s Pro- complishing a task, and only then concern ourselves
cedures, considers the specification of objectives, the with how much money to spend carrying out the task
grouping of “activities” and “functions” into “decision in this preferred way. Traditional budgeting considers
packages,” and the use of performance measures in preferences and amount of money available simulta-
governmental organizations. As political scientists, neously, which has the effect of uniting planning and
Hammond and Knott are skeptical about the value of budgeting, a better procedure.
clarifying goals, for often bringing out divergent views The authors believe that decision packages are in-