This article was downloaded by: University College London

On: 28 Jan 2017

Access details: subscription number 11237
Publisher:Routledge
Informa Ltd Registered in England and Wales Registered Number: 1072954 Registered office: 5 Howick Place, London SW1P 1WG, UK

Handbook of Advanced Multilevel Analysis

Joop J. Hox, J. Kyle Roberts

Explained Variance in Multilevel Models

Publication details
https://www.routledgehandbooks.com/doi/10.4324/9780203848852.ch12
J. Kyle Roberts, James P. Monaco, Holly Stovall, Virginia Foster
Published online on: 19 Jul 2010

How to cite :- J. Kyle Roberts, James P. Monaco, Holly Stovall, Virginia Foster. 19 Jul
2010 ,Explained Variance in Multilevel Models from: Handbook of Advanced Multilevel
Analysis Routledge.
Accessed on: 28 Jan 2017
https://www.routledgehandbooks.com/doi/10.4324/9780203848852.ch12

PLEASE SCROLL DOWN FOR DOCUMENT

Full terms and conditions of use: https://www.routledgehandbooks.com/legal-notices/terms.

This Document PDF may be used for research, teaching and private study purposes. Any substantial or systematic reproductions,
re-distribution, re-selling, loan or sub-licensing, systematic supply or distribution in any form to anyone is expressly forbidden.

The publisher does not give any warranty express or implied or make any representation that the contents will be complete or
accurate or up to date. The publisher shall not be liable for an loss, actions, claims, proceedings, demand or costs or damages
whatsoever or howsoever caused arising directly or indirectly in connection with or arising out of the use of this material.
Downloaded By: University College London At: 14:31 28 Jan 2017; For: 9780203848852, chapter12, 10.4324/9780203848852.ch

12
Explained Variance in Multilevel Models

J. Kyle Roberts
Annette Caldwell Simmons School of Education and Human
Development, Southern Methodist University, Texas

James P. Monaco
Laboratory for Computational Imaging and
Bioinformatics, Rutgers University, New Jersey

Holly Stovall and Virginia Foster

Dedman College, Southern Methodist University, Texas

With the rise of the use and utility of multilevel modeling (MLM), one ques-
tion has consistently been posed to authors and on listserves: “How much
variance does my model explain?” Answering this question within the MLM
framework is not an easy task where it is actually possible to explain “nega-
tive variance” when the addition of explanatory variables increases the corre-
sponding variance components (Snijders & Bosker, 1999). Because effect size
measures previously proposed consider variance at each level, a single mea-
sure is needed that helps researchers interpret the strength of the model as a
whole. The purpose of this paper is to provide a history of past MLM effect
sizes and present three new measures that consider “whole model” effects.
The utility of effect sizes in research interpretation has generated con-
siderable discussion, much of which centers on the role and function of
effect sizes, especially concerning the relationship to statistical signifi-
cance tests (cf. Harlow, Mulaik, & Steiger, 1997). Many authors agree that
effect sizes can serve a valuable function to help evaluate the magnitude
of a difference or relationship (cf. Cohen, 1994; Kirk, 1996; Schmidt, 1996;
Shaver, 1985; Thompson, 1996; Wilkinson & APA Task Force on Statistical
Inference, 1999). Their articles, along with current publications (cf., Knapp
& Sawilowsky, 2001a; Roberts & Henson, 2002) continue to debate both the
use and utility of measures of effect size when considered both in conjunc-
tion with and peripheral from statistical significance testing.
One positive thing that has occurred while researchers began to debate the
issue of effect size reporting (e.g., Knapp & Sawilowsky, 2001b; Thompson,

219
Downloaded By: University College London At: 14:31 28 Jan 2017; For: 9780203848852, chapter12, 10.4324/9780203848852.ch
220  •  J. Kyle Roberts, James P. Monaco, Holly Stovall, and Virginia Foster
2001) is the encouragement of researchers
to consider more than just p-values before
making interpretations as to the notewor-
thiness (or lack thereof) of a given study.
And although it may seem that the field of
research follows changes and adopts a new
course with the speed and acuteness of a
glacier, the fact that this glacier is moving is
predicated by the adoption of such language
in the APA Publication Manual (2001):
The general principle to be followed, how-
ever, is to provide the reader not only
with information about statistical signifi-
cance but also with enough information
to assess the magnitude of the observed
effect or relationship. (p. 26)
12.1 Catalog of Effect
Sizes in MLM
A history of effect sizes has been dealt with
exhaustively in Huberty (2002), and does
not bear repeating here. One thing absent
from Huberty’s catalog was the use of effect
size indices in multilevel analysis. It was
wisely absent from Huberty’s manuscript,
since there is much misconception, and
even disagreement, as to the interpretation
of these effects. We will quickly list some of
the proposed effect size indices for use in
MLM and give brief explanations as to their
utility.
12.1.1  Intraclass Correlation
Intraclass correlation (ICC) is generally
thought of as the degree of dependence of
individuals upon a higher structure to which
they belong; or, the proportion of total vari-
ance that is between the groups of the regres-
sion equation. Put more succinctly, it “is the
degree to which individuals share common
experiences due to closeness in space and/
or time” (Kreft & de Leeuw, 1998, p. 9). Hox
(1995) explains the ICC as a “population
estimate of the variance explained by the
grouping structure” (p. 14). The ICC for a
two-level model can be represented as:
τ02 σ u2 0
ρI = = , (12.1)
τ 02 + σ 2 σ u2 0 + σ e20
where the numerator is represented by the
variance at the second level of the hierarchy
( τ02 ) , and the denominator represents the
total variation in the model at both level-2
and level-1 (σ2).
Although the ICC actually is not a mea-
sure of the effect size of an MLM model, it
bears mentioning here because it sometimes
is wrongly thought of as a measure of the
“power” or strength of MLM over ordinary
least squares (OLS) regression. However,
this type of thinking is commonly illus-
trated in passages like the following:
Determining the proportion of the total
variance that lies systematically between
schools, called the intraclass correla-
tion (ICC), constitutes the first step in an
HLM analysis. We conduct this analysis
with a fully unconditional model, which
means that no student or school charac-
teristics are considered. This first step can
also indicate whether HLM is needed or
whether a single level analytic method is
appropriate. Only when the ICC is more
than trivial (i.e., greater than 10% of the
total variance in the outcome) would the
analyst need to consider multilevel meth-
ods [emphasis mine]. Ignoring this step
(i.e., assuming an ICC of either 0 or 1)
would be inappropriate if the research
question were multilevel. Investigation of
contextual effects, I argue, is by nature a
multilevel question. (Lee, 2000, p. 128)
Downloaded By: University College London At: 14:31 28 Jan 2017; For: 9780203848852, chapter12, 10.4324/9780203848852.ch
Explained Variance in Multilevel Models  •  221
Roberts (2002) has rightly pointed out
that it would be incorrect to interpret
this statistic as a measure of the magni-
tude of difference between OLS and MLM
estimates.
12.1.2 Proportion Reduction
in Variance
The process of building a multilevel model
often begins with a null model (also called
the baseline model by Hox, 2002). In this
baseline model, just the grand mean is fit in
the model such that:
yij = γ 00 + u0 j + eij , (12.2)
where, γ00 is the model grand mean (or inter-
cept), u0 j is the random group-level effect
with variance σ u02
, and eij is the person level
effect with variance σ e2 . This baseline mod-
el’s variance estimates serve as a bench-
mark for determining the R2 at each level of
the hierarchy. By using variance estimates
from the null model (e.g., σ u2 0|b ) and vari-
ance estimates from the model where all
predictors are entered (e.g., σ u2 0|m ), the per-
centage reduction in variance between the
null model and the complete model can be
estimated by:
σ u2 0|b − σ u2 0|m
R22 = , (12.3)
σ u2 0|b
for the percentage reduction in level-2 vari-
ance and by:
σ e2|b − σ e2|m
R12 = , (12.4)
σ e2|b
for the reduction percentage in level-1 vari-
ance, where |b and |m represent the baseline
and full models, respectively. This formula
is reflected in many forms by Hox (2002,
p. 64), or conversely as:
τ 00 (null ) − τ00 ( full )
R22 = (12.5)
τ 00 (null )
by Raudenbush and Bryk (2002, p. 74) and
Kreft and de Leeuw (1998, p. 118).
Although perceived as a tool for noting
the reduction in variance at each level of the
model, Hox (2002) and Snijders and Bosker
(1999) are quick to caution researchers
against directly interpreting this statistic,
since it is possible to obtain negative values
for R2 with these formulas when either σ e2
is a biased estimator or when level-2 pre-
dictors are included in the model. This is a
difficult concept to grasp, because in nor-
mal OLS models, the addition of variables
to the model can only help prediction of
the dependent variable (raise R2), not hurt
prediction. A negative R2 value in MLM
might be wrongly interpreted to mean that
the predictor variables are performing at
worse levels than just the grand mean as a
predictor.
Negative variance can occur in an exam-
ple where we use a variable that has almost
no variation at one of the levels. Consider
the case of a model where we have one single
level-1 predictor. It would be safe to assume
that the addition of this variable would
reduce both the between-and within-groups
variance. If we add a group-level predictor,
then we could expect that it would reduce
only the between-groups variance, not the
within-groups variance, ultimately increas-
ing the estimate for the population variance
σ̂ u0
2
. For example, consider the output in
Table 12.1 from a two-level model.
The data in this example were adapted from
a hypothetical data set written to illustrate
multilevel models (Roberts, 2004). In model
Downloaded By: University College London At: 14:31 28 Jan 2017; For: 9780203848852, chapter12, 10.4324/9780203848852.ch
222  •  J. Kyle Roberts, James P. Monaco, Holly Stovall, and Virginia Foster

TABLE 12.1 as one minus the combined variance at both

Illustration of Negative Variance With the Addition levels for the full model divided by the com-
of a Level-1 Predictor bined variance for the null model, or:
Estimate
Model Formula σ e2 σ u0
2

R12 = 1 −
( ∑γX )
var Yij −
h
h hij
M0: scienceij = γ 00 + u0 j + eij 1.979 23.923
var (Yij )
M1: scienceij = γ 00 + γ 10 (ses ) + u0 j + eij 0.651 80.890 (12.6)
σˆ ( full ) + τˆ ( full )
2 2
0
= 1− ,
M1, just a single level-1 predictor is included σˆ 2 (null ) + τˆ (null )
2
0
in the model with variance estimates of
σ̂ e2  = 0.651 and σ̂ u0
2
 = 80.923. If we are to where Yij is the outcome variable, γh repre-
consider this in terms of Equation 12.4, we sents the coefficient for outcome variable
could see that we actually would be decreas- Xhij for all h variables, σ̂ 2 is an estimate of
ing the variance explained (or increasing σ̂ u0
2
) at the variance at the first level, and τ̂02 is an
the second level with the introduction of this estimate of the variance at the second level.
predictor (hence negative variance explained The level-2 R2 is then found by dividing
between the two models). the σ̂ 2 by the group cluster size (B), or by
Although the amount of variance explai­ the average cluster size for unbalanced data,
ned is noteworthy at level-1 (R12 = (1.979  – such that:
.0651)/1.979 = 0.671), the amount of variance
explained at the second level is actually
R22 = 1 −
( ∑γX )
var Y. j −
h
h h. j
–2.381(R22 = (23.923 – 80.890)/23.923 = –2.381). var (Y. j )
Not only is this number troubling, but it is (12.7)
counter-intuitive to the way most researchers σˆ 2 ( full ) / B + τˆ 02 ( full )
= 1− .
think about the effectiveness of a model. If we σˆ 2 (null )/ B + τˆ 02 (null )
were to interpret this model without previous
knowledge of multilevel models, we might In this formula, it is easy to see that the
be inclined to say that the addition of the R2 estimate at level-2 is similar to the R2
predictor “ses” is a worse predictor of “math for level-1, having just reduced the level-1
achievement” at the second level than if we variance to represent an average variance
had no predictor at all, being that it explains for each group. Although this estimation
–238% variance! differs from the previous definition of R2
(Equations 12.3 and 12.4), it is still possible
to obtain “negative” values for R2.
12.1.3 Explained Variance as Using Table 12.1, R2 at level-1 is:
a Reduction in Mean
Square Prediction Error 0.651 + 80.890
R12 = 1 − = −2.148,
1.979 + 23.923
Snijders and Bosker (1999) argue for a
slightly different approach to computing R2 and for level-2 is:
values in multilevel models by computing
the model’s associated mean square predic- 0.651 / 10 + 80.890
R22 = 1 − = −2.356.
tion error. The R2 for level-1 is then computed 1.979 / 10 + 23.923
Downloaded By: University College London At: 14:31 28 Jan 2017; For: 9780203848852, chapter12, 10.4324/9780203848852.ch
Once again this solution is extremely prob-
lematic, as we have again obtained negative
values for R2.
12.1.4 Pooling as a Measure of
Explained Variance
Like Snijders and Bosker (1999), Gelman
and Pardoe (2006) propose a method that
defines a measure of explained variance (R2)
at each level of the model. (Here the term
level “corresponds to the separate variance
components rather than the more usual
measurement scales.”) However, their mea-
sure does not require fitting multiple models
and can be interpreted similarly to the clas-
sical R2 in that it is the proportion of vari-
ability explained by the linear predictors.
However, it is important to note that since
the proposed R2 is computed at each level,
it does not directly measure the overall pre-
dictive accuracy of the model.
They define the explained variance at the
data level as:
σˆ 2y / n
R2 = 1 −
σˆ α2 / (n + 1) + σˆ 2y / n
where σ̂ 2y is an estimate of the variance of yj
given y j ~ N ( α j , σˆ 2y ) and σ̂ α2 is an estimate
of the variance of αj given α j ~ N ( µ α , σˆ α2 ) .
Gelman and Pardoe also define a sum-
mary measure for the average amount of
pooling (λ) at each level of the model. A low
measure of pooling (< 0.5) suggests a higher
degree of within-group information than
population-level information. The opposite
is true for a high measure of pooling. Note
that this pooling factor is clearly related to
the number of effective parameters. This
pooling factor may be defined at the data
level as:
Explained Variance in Multilevel Models  •  223
n / (n + 1)σˆ α2 + σˆ 2y / n
λ = 1− ,
σˆ α2 + σˆ 2y / n
and at the group level as:
σˆ α2
λ = 1− .
σˆ + σˆ 2y / n
2
α
Together these two measures clarify the
role of predictors at the different levels
and are very useful for understanding the
behavior of complex multilevel models.
These measures shed insight into model fit
at each level as well as the extent to which
the errors are pooled toward their common
prior distribution.
12.1.5 Whole Model
Explained Variance
Xu (2003) proposes three measures that
could be used to estimate the amount of vari-
ation being explained by predictors in a lin-
ear mixed effects model. To obtain all three
of these measures both a null and full model
, must be fit to the data. The first of these three
measures, r2, uses maximum likelihood
to directly estimate the variance compo-
nents under these two models. An empiri-
cal Bayes approach is taken to estimate the
random effects, and a second measure, R2, is
obtained by comparing the residuals of the
full and null models. The third measure, ρ2,
is based on the idea of explained random-
ness and relies on the Kullback-Leibler gain.
Xu defines the following formulae where:
σˆ 2
r2 = 1− ,
σˆ 0
RSS
R2 = 1 − ,
RSS0
Downloaded By: University College London At: 14:31 28 Jan 2017; For: 9780203848852, chapter12, 10.4324/9780203848852.ch
224  •  J. Kyle Roberts, James P. Monaco, Holly Stovall, and Virginia Foster

and predictor variables and the dependent vari-

able. Another way to think about this value
σˆ 2  RSS RSS0  is that it is the correlation between the
ρ2 = 1 − exp − ,
ˆ
σ0  Nσˆ 2 Nσˆ 02  dependent variable, y, and ŷ , the values of
the dependent variable predicted from the
independent variables. This association can
where σ̂ 2 is an estimate of the residual
be seen in Figure 12.1 where x1, x2, and x3
variance at the lowest level (given by Xu as
are all predictors of y.
εij) with all predictors present, and σ̂ 02 is
Therefore we could write a formula to
an estimate of the residual variance given
represent Figure 12.1 as:
only the clustering. RSS is then defined
as the residual sum of squares such that
R y2( x1,x 2 ,x 3) = R yy
2
ˆ. (12.8)
σˆ 2 ≈ RSS / ( N − df ).
The performance of the three measures, r2,
If we are to think theoretically about this
R2, and ρ2, was assessed by Xu through sim-
formula, we can describe ŷ as simply any
ulation, and the results suggest that r2 gives
given person’s predicted y score based on
accurate estimates of the true amount of vari-
the weights derived for each independent
ation explained. R2 and ρ2 are good estimates
variable. The distance between an individ-
when the cluster size is large, but they overes-
ual’s predicted score, ŷ , and their observed
timate if the cluster sizes are too small with R2
score, y, would simply be thought of as an
over estimating the truth slightly more than
error, or variance unaccounted for.
ρ2. Thus, all three of these measures seem well
Although this is a relatively simple for-
suited for summarizing the predictive ability
mula, it can be applied easily to MLM, if
of a linear mixed effects model.
we are to think of MLM as belonging to
the General Linear Model of statistics. If we
consider that the point of any analysis is to
try to produce a series of coefficients that
12.2 Distance Measures
closely approximates an individual’s origi-
for Calculating R2 in
nal score, then we can see that in MLM, the
Multilevel Models
ŷ is simply any predicted value based on
In turning our thoughts to multiple regres- a set of regression coefficients derived from
sion, the multiple R2 can be thought of as the MLM model, and the error term is sim-
the correlation between a function of the ply the distance between ŷ and y. In the

y x1

x2

Figure 12.1
Graphical representation of correlation between y x3
ŷ
and predictor variables.
Downloaded By: University College London At: 14:31 28 Jan 2017; For: 9780203848852, chapter12, 10.4324/9780203848852.ch
Explained Variance in Multilevel Models  •  225

case of hierarchical linear modeling, these
weights are derived through maximum like-
lihood estimates of the fixed effects, with
the individual estimates being the product
of empirical Bayesian estimates.
Although it would seem that a researcher
could simply correlate these ŷ and y val-
ues to obtain an estimate of R2, we must
remember and maintain in MLM that we
wish to honor the procedure by which the
estimates were obtained. In OLS regression,
we can typically compute the total variance
in the model as:
n
∑ (Y − Y ) .
i =1
i
2
(12.9)
In a multilevel model, we must remember
that we typically define the null model as hav-
ing both fixed effects (the grand mean for the
dependent variable) and random effects (the
where y j is the random estimate for group
j and yij is the original outcome score for
person i in group j. This number is the
same value as the sum of the square of all
of the residual values eij, but is distinctly
different from the variance estimate from
the unbiased estimator of σ2 that contains
a correction factor for the Q + 1 regression
parameters such that:
σˆ 2 = ∑ e / (n − Q − 1).
2
ij (12.12)
As noted in the OLS model, the error
variance in a model can be viewed as the
distance between ŷ and y. Likewise, in
MLM after the ŷ are calculated using the
full model, the error variance for the total
model can be explained as:
J nj

variation of each group’s mean around that

grand mean). In defining our model in this
σ′ 2
error = ∑ ∑( y
j =1 i =1
ij − yˆij ) , (12.13)
2

manner, we cannot simply compute the total

variance in the same manner as it is computed where ŷij is the predicted value for person i
in Equation 12.9. Instead, the total variance in group j based on the full model:
must be the predicted values of y when only
the grand mean is used as a predictor, or: yˆij = γ 00 + γ q 0 Xij + γ 0qWj + u0 j + eij , (12.14)

yij = γ 00 (cons ) + u0 j . (12.10) and Wj is a level-2 predictor. Once these val-

ues are computed, an R2 value may be com-
Notice that the random estimate for indi- puted as:
viduals has been left out of Equation 12.10.
By doing this, the grand mean value of the σ error
′2
1− . (12.15)
dependent variable is being estimated for the σ total
′2
entire model and the random level-2 deviate
u0j (also known as β0j for each group). We
then can compute the total amount of vari- 12.2.1  R2 Measure that
ance in this model as: Incorporates a Gaussian
Probability Density Function
J nj
So far, these formulas are not very dissimilar
σ′ 2
total = ∑ ∑( y
j =1 i =1
ij − y j ) = eij2 , (12.11)
2
from the previously proposed estimators of
variance explained, with the difference being
Downloaded By: University College London At: 14:31 28 Jan 2017; For: 9780203848852, chapter12, 10.4324/9780203848852.ch
226  •  J. Kyle Roberts, James P. Monaco, Holly Stovall, and Virginia Foster

that they do not use the unbiased estimator
for variance. However, consider if we were to
aggregate this formula to the level-2 grouping
structure such that we gain an R2 value for
each level-2 group and then average across
all groups. Doing so would further enhance
the above formulas such that the estimate of
variance explained would be defined by:
R 2j k , (12.16)
where k is the number of level-2 groups
and,
where eij′ is an estimate of the residual for
each i person in the jth group in the null
model, and:
p( yij ) = p(dij | s j ) p(s j ) (12.19)
where p(dij • sj) is the probability of the per-
son i, given that they belong to the jth group,
and p(sj) is the probability of group j, given
the entire sample of level-2 units. In extrap-
olating Equation 12.19 further and apply-
ing the probability density function from a
Gaussian distribution, it can be shown that:

ei2 ( full ) 1  (eij′ )2 

R 2j = 1 − , (12.17) p( yij ) = • exp  − 2σ 2 
ei2 (null ) 2 πσ ij2 ⋅ σ 2j  ij 
(12.20)
 (u ′j )2 
with ei2 representing the measure of each • e xp  − 2 
,
residual for the ith person in each distinct  2σ j 
group j for both the full model and the null
model. Although this would represent an with σ ij2 representing the variance of the
“average” R2 for the entire model by produc- i individuals around their jth group mean
ing a mean R2 based on each group’s R2, it for the null model, σ 2j is the variance of β0j
does not take into account that the original around γ00, and eij′ and u ′j are the residual
estimation method used to produce these scores for the level-1 and level-2 estimates,
values was based on a probability of inclu- respectively.
sion in the model from maximum likelihood As would follow from Equation 12.13, the
estimates. Unless each group has exactly the model error could be thought of as:
same sample size and the same probability of
selection, it would not follow to use Equations ∑ p( y )(e ′′)
ij ij
2

12.16 and 12.17 to solve for a model R2. σ error

2 =
ij
, (12.21)
By inserting the Gaussian probability ∑ p( y ) ij
density function into the above equations, ij

we could gain a measure of R2 as a function

where eij′′ is an estimate of the residual for
of the probability of inclusion of the given
each i person in the jth group in the full
value assuming the model. Doing so would
model, and:
modify Equation 12.11 such that:
1  (eij′′)2 
∑ p( y )(e ′ )
ij
ij ij
2 p( yij ) =
2 πσ ij2 ⋅ σ 2j
• exp  − 2σ 2 
 ij 
(12.22)
σ 2
= , (12.18)
∑  (u ′′j )2 
total
p( y ) ij • exp  − ,
2 
ij  2σ j 
Downloaded By: University College London At: 14:31 28 Jan 2017; For: 9780203848852, chapter12, 10.4324/9780203848852.ch
Explained Variance in Multilevel Models  •  227

with σ ij2 representing the variance of the the slope coefficient is simply the weighted
i individuals around their jth group mean least squares estimator (or maximum likeli-
for the full model, σ 2j is the variance of β0j hood estimate) γ00 where:
around γ00, and eij′′ and u ′′j are the residual
scores for the level-1 and level-2 estimates,
respectively. The final estimate of variance
γˆ 00 = ∑∆ −1
j •jY ∑∆ −1
j , (12.24)

explained could then be derived from com-

bining Equations 12.18 and 12.21:
σ error
2
1− . (12.23)
σ total
2
The strength of a measure such as this is
twofold. First, it puts what would normally
be a complicated interpretation of a model
into a palatable form for the less-informed
researcher who might be reading an MLM
analysis. Although it often seems that the
goal of many statistical concepts is to con-
fuse the graduate student (e.g., the multi-
plicity of effect sizes currently available for
use), in doing so, we only confuse the future
researcher, and likewise, future research.
Second, it allows the researcher, outside
of the Akaike Information Criterion AIC
and ∆j is the sum of the two variance compo-
nents Var(u0j) and Var( e• j ; see Raudenbush
& Bryk, 2002, p. 40 for further discussion).
Put simply, the grand estimate for the mean
of all of the groups (γ00) is the sum of all of
the group means ( Y• j ) after applying the
precision parameter ( ∆ −1 j ) and then divid-
ing by the sum of the precision parameters.
The effect of these precision parameters on
the grand estimate is to apply more weight to
the groups that are measured with more pre-
cision (cf., more level-1 units). This formula
for grand estimates also could be applied to
the idea of an R2 effect size measure.
In typical OLS regression, the multiple R2
can be expressed as:
n

(Akaike, 1987) or the Bayesian Information

Criterion BIC (Schwarz, 1978), with a single
∑ ( yˆ − y )i i
2

R2 = 1 − i =1
n
, (12.25)

∑( y − y )
statistic to interpret just how well a model is 2
i
performing. Since the goal of most research is
i =1
to find variables that fully describe the varia-
tion in the dependent variable, a measure
like this could potentially prove very useful where yi is any given individual’s score
in helping the researcher make judgments on the dependent variable, ŷi is that indi-
about the effectiveness of a MLM model. vidual’s predicted score from the linear
regression equation, and y is the mean of
all individuals on the dependent variable.
12.2.2 Group Initiated R2 based on For any given group within a set of level-2
Weighted Least Squares units, Equation 12.25 could be considered
In addition to alternative ways of comput- the mathematical equivalent of:
ing an effect size mentioned above, another
type of effect size can be conceived through nj
( yˆij − yij )2
∑
1
maximum likelihood methods. In a typical R = 1−
2
j , (12.26)
nj σ ij2
multilevel ANOVA, the grand estimate for i =1
Downloaded By: University College London At: 14:31 28 Jan 2017; For: 9780203848852, chapter12, 10.4324/9780203848852.ch
228  •  J. Kyle Roberts, James P. Monaco, Holly Stovall, and Virginia Foster

where nj is the number of people in group
j and σ ij2 is the variance of the individuals
in group j. For simplification purposes, we
will define the latter part of Equation 12.26
as being an error term corresponding to a
normalized error for a given group. In rep-
resenting this with the term Ei, Equation
12.26 can be thought of as:
nj
∑E .
1
R = 1−
2
j i (12.27)
nj i =1
we are producing an equation for each
­second-level group based on weighted least
squares estimators. It seems appropriate,
then, to produce an entire model R 2 that
is also weighted for the probability of the
group from which the estimate was drawn.
In expanding Equation 12.27 to include
all groups and also reflect the need to use
a weighted estimator, RT2 could be thought
of as:
1
RT2 = 1 − J
E j , (12.30)

This would mean that the total weighted ∑ p ( s )n

j =1
j j

least squares normalized error for all groups

could be thought of as:
where Ej is the total weighted least squares
J
normalized error for all groups from
Ej = ∑ p(s ) E ,
j =1
j i (12.28) Equation 12.28. We can further deduce
that RT2 :

where p(sj) is the probability of group j exist- J

∑ p(s ) E ,
ing given the sample of all j groups such 1
= 1− J j j
that:
∑ p ( s )n
j =1
j j
j =1

1  (u ′j )2 
p(s j ) = • exp  −  , (12.29) J  nj

∑ ∑
1
2 πσ 2j  2σ j   p(s j )
2
= 1− J
Ei  ,

∑ p ( s j )n j j =1  i =1 

with u ′j being the residual score for group j =1

j around the mean of all groups and σ 2j nj

J J  
the variance of j groups’ means around the
grand mean of the dependent variable.

∑
j =1
p ( s j )n j − ∑ j =1
 p(s j )

∑
i =1
Ei 

= ,
In performing analyses that are not J

­multilevel in nature, we could simply com- ∑ p ( s )n j j

pute the R 2 for each group, and then aver- j =1

age these values across all groups to obtain nj

J  
an average group R 2 based on the sample
of groups that we drew from the greater
∑
j =1
p ( s j )n j  1 − 1/ n j

∑ i =1
Ei 

= .
population of all level-2 units. As was J

previously stated, this makes little sense, ∑ p ( s )n j j

however, since in multilevel modeling j =1

Downloaded By: University College London At: 14:31 28 Jan 2017; For: 9780203848852, chapter12, 10.4324/9780203848852.ch
And since we already have defined R 2j in
Equation 12.26, we can then interpret:
J There is a caution, however, in making
∑ n ⋅ p(s )⋅ R
j =1
j j
2
j
RT2 =
J
∑ n ⋅ p(s )
j =1
j j
which, theoretically, is simply the weighted
least squares average of all of the R2 val-
ues from each group. What is present
in Equation 12.31 is a solution that will
produce estimates similar in interpreta-
tion to OLS R2 measures. Equation 12.31
seems considerably more appropriate than
Equations 12.15 and 12.23, since it honors
both the nesting structure of the data and
the fact that the model was derived through
weighted least squares estimates.
12.3 Conclusions
Snijders and Bosker (1999) presented good
arguments for instances when Equations
12.6 and 12.7 produce results that yield
negative values for R2 in multilevel mod-
els. While it is sometimes helpful to be
able to know the variance accounted for
at each level of the MLM model, the lan-
guage with which researchers must refer to
these estimates is, at best, confusing to the
non-MLM minded researcher. With the
further encouragement from editors to
begin reporting effect sizes in all research,
it is becoming more necessary for research-
ers using MLM to be able to explain their
results in a way that is common with other
statistical methods. Although results from a
multilevel model probably will need further
Explained Variance in Multilevel Models  •  229
explanation, it is hoped that the continued
development of these models will help in
their proliferation.
these models more accessible. Just because
a researcher has the software and program-
, (12.31)
ming skills to utilize complicated techniques
does not mean that that technique is war-
ranted. With the growth of a likewise com-
plicated field of statistics, MLM, Goldstein
(1995) voiced similar concerns:
There is a danger, and this paper reminds
us of it, that multilevel modeling will
become so fashionable that its use will be
a requirement of journal editors, or even
worse, that the mere fact of having fitted a
multilevel model will become a certificate
of statistical probity. That would be a great
pity. These models are as good as the data
they fit; they are powerful tools, not uni-
versal panaceas. (p. 202)
It is our sincere hope that develop-
ing MLM as a more user-friendly field of
statistics will improve its utilization and
interpretation, but along with this must
come responsibility in evaluating such
models.
References
Akaike, H. (1987). Factor analysis and the AIC.
Psychometrica, 52, 317–332.
American Psychological Association. (2001). Publi-
cation manual of the American Psychological
Association (5th ed.). Washington, DC: Author.
Cohen, J. (1994). The earth is round (p < .05).
American Psychologist, 49, 997–1003.
Gelman, A., & Pardoe, I. (2006). Bayesian measure of
explained variance and pooling in multilevel
(hierarchical) models. Technometrics, 48(2),
241–251.
Goldstein, H. (1995). Hierarchical data modeling in
the social sciences. Journal of Educational and
Behavioral Statistics, 20, 201–204.
Downloaded By: University College London At: 14:31 28 Jan 2017; For: 9780203848852, chapter12, 10.4324/9780203848852.ch
230  •  J. Kyle Roberts, James P. Monaco, Holly Stovall, and Virginia Foster
Harlow, L. L., Muliak, S. A., & Steiger, J. H. (Eds.).
(1997). What if there were no significance tests?
Mahwah, NJ: Erlbaum.
Hox, J. (1995). Applied multilevel analysis. Amsterdam:
TT-Publikaties.
Hox, J. (2002). Multilevel analysis: Techniques and
applications. Mahwah, NJ: Erlbaum.
Huberty, C. J. (2002). A history of effect size indices.
Educational and Psychological Measurement,
62(2), 227–240.
Kirk, R. E. (1996). Practical significance: A con-
cept whose time has come. Educational and
Psychological Measurement, 56, 746–759.
Knapp, T. R., & Sawilowsky, S. S. (2001a). Constructive
criticisms of methodological and editorial prac-
tices. The Journal of Experimental Education, 70,
65–79.
Knapp, T. R., & Sawilowsky, S. S. (2001b). Strong argu-
ments: Rejoinder to Thompson. The Journal of
Experimental Education, 70, 94–95.
Kreft, I., & de Leeuw, J. (1998). Introducing multilevel
modeling. Thousand Oaks, CA: Sage.
Lee. V. E. (2000). Using Hierarchical linear modeling to
study social contexts: The case of school effects.
Educational Psychologist, 35(2), 125–141.
Raudenbush, S. W., & Bryk, A. S. (2002). Hierarchical
linear models: Applications and data analy-
sis methods (2nd ed.). Thousand Oaks, CA:
Sage.
Roberts, J. K. (2002). The importance of intraclass
correlation in multilevel and hierarchical lin-
ear modeling designs. Multiple linear regression
viewpoints, 28(2), 19–31.
Roberts, J. K. (2004). An introductory primer on multi-
level and hierarchical linear modeling. Learning
disabilities: A contemporary journal, 2(1), 30–38.
Roberts, J. K., & Henson, R. K. (2002). Correction for
bias in estimating effect sizes. Educational and
Psychological Measurement, 62(2), 241–253.
Schmidt, F. (1996). Statistical significance testing
and cumulative knowledge in psychology:
Implications for the training of researchers.
Psychological Methods, 1, 115–129.
Schwarz, G. (1978). Estimating the dimension of a
model. Annals of Statistics, 6, 461–464.
Shaver, J. (1985). Chance and nonsense. Phi Delta
Kappan, 67, 57–60.
Snijders, T., & Bosker, R. (1999). Multilevel analysis.
Thousand Oaks, CA: Sage.
Thompson, B. (1996). AERA editorial policies regarding
statistical significance testing: Three suggested
reforms. Educational Researcher, 25(2), 26–30.
Thompson, B. (2001). Significance, effect sizes, step-
wise methods, and other issues: Strong argu-
ments move the field. Journal of Experimental
Education, 70, 80–93.
Wilkinson, L., & American Psychological Association
Task Force on Statistical Inference. (1999).
Statistical methods in psychology jour-
nals: Guidelines and explanation. American
Psychologist, 54, 594–604. [reprint available
through the APA Home Page: http://www.apa.
org/journals/amp/amp548594.html]
Xu, R. (2003). Measuring explained variation in linear
mixed effects models. Statistics in Medicine, 22,
3527–3541.