Professional Documents
Culture Documents
Sociology 761
John Fox
Lecture Notes
c
Copyright 2006
by John Fox
Sociology 761
The standard linear model has one random effect, the error term %l,
and one variance component, 2 = Var(%l).
When the assumptions of the standard linear model hold, ordinaryleast-squares (OLS) regression provides maximum-likelihood estimates of the regression coefficients,
b = (X0X)1X0y
Sociology 761
c
Copyright 2006
by John Fox
0
b
b
y X
y X
b2 =
q
b2 is a biased estimator of 2; usually, the unbiased estimator
b
b
y X
y X
v2 =
qs
is preferred.
I The standard linear model and OLS regression are generally inappropriate for dependent observations.
Dependent (or clustered) data arise in many contexts, the two most
common of which are hierarchical data and longitudinal data.
Sociology 761
c
Copyright 2006
by John Fox
I In all of these cases, it is not generally reasonable to assume that observations within the same higher-level unit, or longitudinal observations
within the same individual, are independent of one-another.
c
Copyright 2006
by John Fox
Sociology 761
Sociology 761
c
Copyright 2006
by John Fox
Sociology 761
c
Copyright 2006
by John Fox
Sociology 761
c
Copyright 2006
by John Fox
I Topics:
The linear mixed-effects model.
Modeling hierarchical data.
Modeling longitudinal data.
Generalized linear mixed models (time permitting).
1> 2> = = = > s are the fixed-effect coefficients, which are identical for all
groups.
Sociology 761
c
Copyright 2006
by John Fox
10
{2lm > = = = > {slm are the fixed-effect regressors for observation m in group
l; there is also implicitly a constant regressor, {1lm = 1.
e1l> = = = > etl are the random-effect coefficients for group l, assumed
to be multivariately normally distributed, independent of the random
effects of other groups. The random effects, therefore, vary by group.
The eln are thought of as random variables, not as parameters, and
are similar in this respect to the errors %lm .
}1lm > = = = > }tlm are the random-effect regressors.
The } s are almost always a subset of the {s (and may include all of
the { s).
When there is a random intercept term, }1lm = 1.
# 2n are the variances and # nn0 the covariances among the random
effects, assumed to be constant across groups.
In some applications, the # s are parametrized in terms of a smaller
number of fundamental parameters.
Sociology 761
c
Copyright 2006
by John Fox
Sociology 761
c
Copyright 2006
by John Fox
11
Sociology 761
c
Copyright 2006
by John Fox
12
13
c
Copyright 2006
by John Fox
14
Sociology 761
c
Copyright 2006
by John Fox
Sociology 761
c
Copyright 2006
by John Fox
15
Sociology 761
c
Copyright 2006
by John Fox
16
Catholic
-2
5819
-1
-2
1433
17
Public
-1
2336
-2
4642
7345
-1
-2
8628
25
25
20
20
15
15
10
10
-1
-2
-1
-2
-1
2336
4642
5619
6469
7345
8627
7919
7635
2458
5640
1477
1461
0
3716
6484
9359
6089
2768
3039
25
Math Achievement
Math Achievement
25
20
15
10
5
20
15
10
5
0
8854
8009
9292
3499
8983
5937
4523
25
25
20
20
15
15
10
10
5783
3377
1296
9508
1288
0
-2
-1
-2
-1
1
Centered SES
-2
-1
-2
c
Copyright 2006
by John Fox
18
In each scatterplot, the broken line is the linear least-squares fit to the
data, while the solid line gives a nonparametric-regression fit. The
number at the top of each panel is the ID number of the school.
Particularly given the relatively small numbers of students in individual
schools, the linear regressions seem to do a reasonable job of
summarizing the relationship between math achievement and SES
within schools.
Although there is substantial variation in the regression lines among
schools, there also seems to be a systematic difference between
Catholic and public schools: The lines for the public schools appear to
have steeper slopes on average.
I SES in these scatterplots is expressed as deviations from the school
mean SES.
That is, the average SES for students in a particular school is
subtracted from each individual students SES.
Sociology 761
c
Copyright 2006
by John Fox
-1
-2
-1
1
Centered SES
-2
-1
c
Copyright 2006
by John Fox
19
Sociology 761
c
Copyright 2006
by John Fox
20
21
Catholic
(Intercept)
school
ses
|
9586
9198
8628
8193
7688
7332
7011
6469
6366
5619
5404
3427
3020
2990
2755
2526
1436
1433
1308
3039
1477
2208
9359
3838
3610
2458
7635
5761
4173
3992
1906
6074
4042
8150
9104
8165
3688
3498
5720
5667
4931
1462
4223
2629
1317
8857
4292
9508
2658
5650
7364
3499
7342
4253
3533
3705
9347
6578
4511
9021
4530
8009
2277
6816
4523
5192
8800
2305
4868
7172
|
|
|
|
|
|
|
|
|
|
|
|
|
15
20
-5
|
|
|
|
|
|
|
|
|
10
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Sociology 761
22
c
Copyright 2006
by John Fox
Sociology 761
23
Public
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
20
-5
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
15
|
|
|
|
|
|
|
|
|
|
|
|
|
10
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Catholic
|
|
Public
Catholic
Public
10
Slopes
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Intercepts
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
ses
|
-2
|
|
|
|
20
15
10
school
(Intercept)
8627
8202
7697
7345
7276
4642
3657
3351
3332
3152
2771
2626
2336
1946
1942
1637
1461
6397
5819
2768
6089
5640
9225
8874
8175
7734
6897
6484
4325
3999
4420
8531
8357
2030
6170
2917
7232
8946
9158
5838
2995
3881
6600
1374
2467
2651
1909
3716
7919
5937
6464
9550
6415
3967
1224
6291
1288
8477
1499
8707
4410
8188
8983
9292
2655
9397
4350
1296
3377
2639
7101
3013
5783
9340
2818
6808
6443
6144
7890
8775
3088
4383
1358
7341
5815
6990
5762
4458
8854
8367
c
Copyright 2006
by John Fox
Figure 5. Boxplots of within-school coefficients for the least-squares regression of math achievement on school-centered SES, for 70 Catholic
and 90 public schools.
Sociology 761
c
Copyright 2006
by John Fox
25
Slope
-2
10
20
15
24
Intercept
-1.0
-0.5
0.0
0.5
Mean SES
-1.0
-0.5
0.0
0.5
Mean SES
c
Copyright 2006
by John Fox
26
Sociology 761
c
Copyright 2006
by John Fox
Sociology 761
c
Copyright 2006
by John Fox
27
Sociology 761
c
Copyright 2006
by John Fox
28
Note that all explanatory variables in the Laird-Ware form of the model
carry subscripts l for schools and m individuals within schools, even
when the explanatory variable in question is constant within schools.
Thus, for example, {2lm = sesl (and so all individuals in the same
school share a common value of school-mean SES).
There is both a data-management issue here and a conceptual
point:
With respect to data management, software that fits the Laird-Ware
form of the model (such as the lme or lmer functions in R) requires
that level-2 explanatory variables (here sector and school-mean
SES, which are characteristics of schools) appear in the level-1 (i.e.,
student) data set much as the person time-period data set
that we employed in survival analysis with time-varying covariates
required that time-constant covariates appear on the data record for
each time period.
Sociology 761
c
Copyright 2006
by John Fox
30
29
Sociology 761
c
Copyright 2006
by John Fox
31
Two observations |lm and |lm 0 in school l are not independent because
they share the random effect, e1l.
I There are also two variance components for this model:
# 21 = Var(e1l) is the variance among school means.
Sociology 761
c
Copyright 2006
by John Fox
Sociology 761
c
Copyright 2006
by John Fox
32
33
Sociology 761
c
Copyright 2006
by John Fox
34
c
Copyright 2006
by John Fox
Sociology 761
c
Copyright 2006
by John Fox
35
Sociology 761
c
Copyright 2006
by John Fox
36
Sociology 761
c
Copyright 2006
by John Fox
38
Sociology 761
c
Copyright 2006
by John Fox
37
Sociology 761
c
Copyright 2006
by John Fox
39
Sociology 761
c
Copyright 2006
by John Fox
40
I Cautionary Remarks:
Because REML estimates are calculated integrating out the fixed
effects, one cannot legitimately perform likelihood-ratio tests across
models with different fixed effects when the models are estimated by
REML.
Likelihood-ratio for variance-covariance components across nested
models with identical fixed effects are perfectly fine, however.
A common source of estimation difficulties in mixed models is the
specification of overly complex random effects.
Interest usually centers in the fixed effects, and it often pays to try to
simplify the random-effect part of the model.
Sociology 761
c
Copyright 2006
by John Fox
42
41
Sociology 761
c
Copyright 2006
by John Fox
43
I After fitting this model to the data by REML, I tested to check whether
random intercepts and slopes are still required:
Model Omitting logh O
1
23> 247=70
2
# 21> # 12 23> 357=86
3
# 22> # 12 23> 247=93
Sociology 761
c
Copyright 2006
by John Fox
Sociology 761
c
Copyright 2006
by John Fox
44
45
c
Copyright 2006
by John Fox
46
c
Copyright 2006
by John Fox
Sociology 761
Public
Catholic
-1
b4,
b5,
b6, and
b7 combine to
Figure 7 shows how the coefficients
produce the level-1 (i.e., within-school) coefficient for SES.
At fixed levels of school SES, individual SES is more positively
related to math achievement in public than in Catholic schools.
The maximum positive effect of individual SES is in schools with a
slightly higher than average SES level; the effect declines at low
and high levels of school SES, and becomes negative at the lowest
levels of school SES.
An alternative, and more intuitive representation of the fitted model
is shown in Figure 8, which graphs the fitted within-school regression
of math achievement on centered SES for Catholic and public
schools and for three levels of school SES: 0=7 (the approximate
10th percentile of school SES), 0 (the median), and 0=7 (the 90th
percentile).
47
Sociology 761
-1.5
-1.0
-0.5
0.0
0.5
1.0
c
Copyright 2006
by John Fox
Sociology 761
c
Copyright 2006
by John Fox
48
49
15
20
10
Math Achievement
15
10
5
Math Achievement
20
Public Schools
-3
-2
-1
SES
-3
-2
-1
SES
Figure 8. Fitted within-school regressions of math achievement on centered SES for public and Catholic schools at three levels of mean school
SES.
Sociology 761
c
Copyright 2006
by John Fox
50
Sociology 761
c
Copyright 2006
by John Fox
51
Sociology 761
c
Copyright 2006
by John Fox
Sociology 761
c
Copyright 2006
by John Fox
Sociology 761
54
Patients
8 10 12 14 16
log2Exercise
309
317
127
161
8 10 12 14 16
196
338
141
162
8 10 12 14 16
152
337
325
175
8 10 12 14 16
121
189
104
329
8 10 12 14 16
150
149
163
307
4
2
0
-2
-4
4
2
0
-2
-4
8 10 12 14 16
8 10 12 14 16
8 10 12 14 16
Age
8 10 12 14 16
8 10 12 14 16
Control Subjects
8 10 12 14 16
log2Exercise
210
217
231
263
8 10 12 14 16
283
234
213
201
8 10 12 14 16
227
229a
250
258
8 10 12 14 16
281
273a
236
205
8 10 12 14 16
204
235
215
229b
4
2
0
-2
-4
4
2
0
-2
-4
8 10 12 14 16
8 10 12 14 16
8 10 12 14 16
Age
8 10 12 14 16
8 10 12 14 16
53
2
0
-2
log2exercise
10 15 20 25 30
52
Exercise (hours/week)
control
patient
control
patient
c
Copyright 2006
by John Fox
55
With so few observations per subject, and without clear evidence that
it is inappropriate, we would be loath to fit a model more complicated
than a linear trend.
c
Copyright 2006
by John Fox
Sociology 761
c
Copyright 2006
by John Fox
56
I Substituting the level-2 model into the level-1 model produces the
combined model
log -exerciselm = ( 00 + 01groupl + x0l)
+( 10 + 11groupl + x1l)(agelm 8) + %lm
= 00 + 01groupl + 10(agelm 8)
+ 11groupl (agelm 8) + x0l + x1l(agelm 8) + %lm
or, in Laird-Ware form,
|lm = 1 + 2{2lm + 3{3lm + 4{4lm + e1 + e2}2lm + %lm
Sociology 761
c
Copyright 2006
by John Fox
58
1807=07
2
#21> #12 (random intercepts) 1911=04
3
#22> #12 (random slopes)
1816=13
Both likelihood-ratio tests are highly statistically significant (particularly the one for random intercepts), suggesting that both random
intercepts and random slopes are required.
I The model that I have fit to the Blackmoor et al. data assumes
independent errors, %lm .
The composite errors, lm = e1 + e2}2lm + %lm , are correlated within
individuals, however, as we previously established for mixed models
applied to hierarchical data.
Sociology 761
57
I Fitting this model to the data produces the following estimates of the
fixed effects and variance-covariance components:
REML Estimate Std. Error
Parameter Term
1
intercept
0=2760
0=1824
groupl
0=3540
0=2353
2
agelm 8
0=0640
0=0314
3
groupl (agelm 8)
0=2399
0=0394
4
#1
(intercept)
1=4435
(agelm 8)
0=1648
#2
#12
(intercept, agelm 8)
0=0668
(%lm )
1=2441
c
Copyright 2006
by John Fox
Sociology 761
c
Copyright 2006
by John Fox
59
In the current context }2lm is the time of observation (i.e., age minus
eight years), and the variance and covariances of the composite
residuals are (as we previously established)
2
Var( lm ) = # 21 + }2lm
# 22 + 2}2lm # 12 + 2
Cov( lm > lm 0 ) = # 21 + }2lm }2lm 0 #22 + (}2lm + }2lm 0 )#12
The actual observations are not taken at entirely regular intervals, but
assume that we have observations for the same individual l taken at
}2l1 = 0> }2l2 = 2> }2l3 = 4> and }2l4 = 6 (i.e., at 8, 10, 12, and 14 years
of age).
Then the estimated covariance
5 matrix for the composite6 errors is
3=631 1=950 1=816 1=683
9 1=950 3=473 1=900 1=875 :
d ( > > > ) = 9
:
Cov
l1 l2 l3 l4
7 1=816 1=900 3=532 2=068 8
1=683 1=875 2=068 3=808
Sociology 761
c
Copyright 2006
by John Fox
60
I The linear mixed model allows for correlated level-1 errors within
individuals,
%l Nql (0> 2\l)
For a model with correlated errors to be identified, however, the matrix
\l cannot consist of independent parameters; instead, the elements
of this matrix are expressed in terms of a much smaller number of
fundamental parameters.
c
Copyright 2006
by John Fox
Sociology 761
0.7
0.7
1.0
0.5
s
-0.5
0.0
I
Us
0.0
6
lag s
10
10
lag s
Sociology 761
c
Copyright 2006
by John Fox
63
The occasions for the Blackmoor et al. data are not equally spaced,
however.
For data such as these, lme provides a continuous first-order
autoregressive process, with the property that
corr(%lw> %l>w+v) = (v) = !|v|
where the time-interval between observations, v, need not be an
integer.
0.8
0.6
0.2
0.4
I
U s
61
For example, for equally spaced occasions, a very common model for
the intra-individual errors is the first-order autoregressive [or AR(1)]
process:
%lm = !%l>m1 + m
where
2
m QLG(0> )
and
|!| ? 1
1.0
62
c
Copyright 2006
by John Fox
I I tried to fit the same mixed-effects model to the data as before, except
allowing for first-order autoregressive level-1 errors.
The estimation process did not converge; a closer inspectation
suggests that the model has redundant parameters.
I then fit two additional models, retaining autocorrelated within-subject
errors, but omitting in turn random slopes and random intercepts.
Sociology 761
c
Copyright 2006
by John Fox
64
c
Copyright 2006
by John Fox
66
65
Sociology 761
67
Patients
Controls
Exercise (hours/week)
A graph showing the fit of the model, translating back from log-exercise
to the exercise scale, appears in Figure 12.
10
12
14
16
18
Age (years)
Figure 12. Fitted exercise as a function of age and group: Average trajectories based on fixed effects.
Sociology 761
c
Copyright 2006
by John Fox
Sociology 761
c
Copyright 2006
by John Fox