My Favorite

M
F
it (Mixed)
(Mi d) Models:
M d l
An Overview
Rod Sturdivant
Center for Data Analysis and Statistics (CDAS)
Feb 2007

AGENDA

The Problem
The Traditional Approach
The Mixed Model
An Example
l
References

Center for Data Analysis and Statistics (CDAS)

The Problem

Center for Data Analysis and Statistics (CDAS)

Initial Model Specification

Reasonable? A straight line:

returni = 0 + 1 usei + i
Where:
i is the subject index (I = 1,,40)
ij is an error term (assumed independent with
y
mean 0 and constant variance usually
assume a normal distribution)

Center for Data Analysis and Statistics (CDAS)

Initial Model Specification

Parameters estimated using least squares (or
Maximum Likelihood) results in:

returni = 72.9 1.8 usei

Center for Data Analysis and Statistics (CDAS)

A Good Model?

Regression is statistically significant

(p<0.0001)
Standard regression diagnostics reveal
nothing
g suspicious.
p

BUT WHAT IF
BUT,
IF

Center for Data Analysis and Statistics (CDAS)

A NEW PICTURE
The data came from 4 treatment centers:

Center for Data Analysis and Statistics (CDAS)

Traditional Solution

Use 0-1 design variables to represent the group

differences
Results in a different line for each group
4 Groups
3 Design
variables

Center
1
2
3
4

Center for Data Analysis and Statistics (CDAS)

D1
0
1
0
0

D2
0
0
1
0

D3
0
0
0
1
8

Revised Model Specification

returni = 0 + 1 usei
+ 2 D1i + 3 D 2i + 4 D3i + i
USING LEAST SQUARES THE ESTIMATED
MODEL IS:

return
t i = 80.8
80 8 2 usei 22.9
9 D1i 88.8
8 D 2i 15
15.8
8 D3i

Center for Data Analysis and Statistics (CDAS)

Revised Model Specification

Note each center has a different estimated line:
returni = 80.8
80 8 2 usei 22.9
9 D1i 88.8
8 D 2i 15
15.8
8 D3i
Center

Design Variables

Estimated Line

0, 0, 0

returni = 80.8 2 usei

1, 0, 0

returni = 77.9 2 usei

0, 1, 0

returni = 72.0
2 0 2 usei

0, 0, 1

returni = 65.0 2 usei

Center for Data Analysis and Statistics (CDAS)

10

Model Results

Excellent fit (R2 increase from 0.69 to 0.99)

All variables highly
g y significant
g
(p < 0.0001))

Center for Data Analysis and Statistics (CDAS)

11

ONE MORE TWIST

Slopes are not all the same:

Center for Data Analysis and Statistics (CDAS)

12

Revised Model Specification

ADD use and D3 interaction:
returni = 0 + 1 usei + 2 D1i + 3 D 2i + 4 D3i
+ 4 D3i usei + i

THE ESTIMATED MODEL IS:

returni = 80.7 2 usei 2.9 D1i 8.9 D 2i 44.1 D3i + 3.8 D3i usei
The Center 4 Line Changes

Center for Data Analysis and Statistics (CDAS)

returni = 36.6 + 1.8 usei

13

Model Results

Excellent fit (R2 of 0.97)

All variables highly significant (p < 0.0001 and d1 0.0015)

Center for Data Analysis and Statistics (CDAS)

14

The Mixed Model

Q: What is wrong with the traditional model it looked pretty good

Increase in the number of groups leads to large number
of parameters
Data management and parameter estimation issues
Decrease in estimate precision (note: in our example, standard
errors more than doubled when interaction added)

May have a large number of predictors unique at the

hi h level
higher
l
l only
l
Observed groups are a random sample
Independence assumption data CORRELATED
Center for Data Analysis and Statistics (CDAS)

16

Introduction

Issue of clustered data

Common in many studies and fields

Standard models fail to adequately address
Inference and estimates affected

Mixed models as a solution

IIncreasing
i use
Estimation algorithms and software
available

Center for Data Analysis and Statistics (CDAS)

17

Mixed Model Specification

Random intercept only

Emphasis on hierarchy

Emphasis on fixed/random

yij = 0 j + 1 xij + ij
0 j = 0 + 0 j
where
h

We assume:

ij

y ij = ( 00 + 1 xij ) + ( 0 j + ij )

j = 1,..., J

Level 2 (group) index

i = 1,..., n j

Level 1 (subject) index

N ((0, 2 )

Independent
p
of

Center for Data Analysis and Statistics (CDAS)

0 j

N ((0, 02 )
18

Random Slope Model

Emphasis on hierarchy

yij = 0 j + 1 j xij + ij
where

Emphasis on fixed/random

y ij = ( 0 + 1 xij ) + ( 0 j + 1 j xij + ij )

0 j = 0 + 0 j

1 j = 1 + 1 j
Additional assumptions:

1 j

N (0,
(0 12 )

cov( 0 j , 1 j ) = 01
Center for Data Analysis and Statistics (CDAS)

19

Matrix Form of the Model

y = X + Z +
As an example, consider the matrices for the 2 level random slope
model:
y ij = ( 0 + 1 x ij ) + ( 0 j + 1 j x ij + ij )
y11

M
yn 1
1
y12
M

y =
yn2 2

M
y
1J
M
yn J
J

1 x11

M M
1 xn 1
1
1 x12
M M

X=
1 x n2 2

M M
1 x
1J
M M
1 x
nJ J

= 0
1

M
1

Z=

x11
M
xn11

Center for Data Analysis and Statistics (CDAS)

1
M

x12
M

x n2 2
O
1
M
1

x1 J

M
xnJ J

01

11
M
=

0 J

1J

11

M
n 1
1
12
M

=
n2 2

M

1J
M

nJ J

20

Covariance Structure of the

Linear Hierarchical Model
var( y ) = V = ZZ + W
Where:

= var()

and

W = Var ( )

In the 2 level random slope example:

02 01

2
01 0

=
O

0
01

01 02

Center for Data Analysis and Statistics (CDAS)

W = diag( 2 )

21

Model Estimation

Recent (10 years) computing

advances allow estimation
Maximum Likelihood (ML) Estimates

Iterative algorithms
Bayesian MCMC methods

Software including MLwiN,

MLwiN HLM,
HLM
Winbugs, SAS, VARCL

Center for Data Analysis and Statistics (CDAS)

22

For Our Example

Parameter Estimates
Parameter
Fixed

Estimate Standard Error

Intercept

66.9

10.2

Slope

-1.0

0.96

40
4.0

0 97
0.97

412.8

293.6

3.6

2.6

-38.1

27.3

Level 1
Random

Intercept
Slope
Covariance

NOTE: only estimate 6 parametersregardless

parameters regardless of number of
groups!
Center for Data Analysis and Statistics (CDAS)

23

Predictions

Residual estimates (posterior means) for level h of model

substitute parameter estimates in:

rh = R h V 1 (y X)

where

R h = Z h h

cov(rh ) = R h V 1R h

(Design and covariance

matrices for level h)

Referred to as shrunken residuals. Example for level two intercept

only case:

n j 02
( y. j y.. )
0 j =
2
2
n j 0 +
Center for Data Analysis and Statistics (CDAS)

24

Model Results
Center (j)

Intercept

Slope

66.9+12.9
= 79.8

-1 - 0.86
= -1.86

80.7
-2

66.9+11.7
= 78.6
78 6

-1 - 1.03
= -2.03
2 03

77.8
-2
2

66.9+ 5.7
= 72.6

-1 - 0.98
= -1.98

71.8
-2

66.9 30.2
66.9-30.2
= 36.7

-1
1 + 2.87
= 1.87

36.6
1.8

Estimate
intercept and
slope from:
Center for Data Analysis and Statistics (CDAS)

Std.
Model

0 j = 00 + 0 j
1 j = 10 + 1 j
25

Model Extensions
There are numerous possible extensions at this
point. In addition to additional predictors (with
either fixed or random coefficients) at level 1
returnij = 0 j + 1 j useij + ij
Additional
l
level
l

0 j = 00 k + 01 z0j + 0 j
00 k = 000 + 00 k
1 j = 10 + 11z1j + 1 j

Center for Data Analysis and Statistics (CDAS)

Center specific
predictor variable

26

Example

Batting Average in
DAY/NIGHT games

MIXED MODEL (SAS PROC MIXED)

Effect
DAY
Intercept
DAY
N

Solution for Fixed Effects

Standard
Estimate
Error
DF
0.2850 0.005203
19
-0
0.0045
0045 0.001027
0 001027
379

t Value
54.77
-4
4.38
38

Pr > |t|
<.0001
<.0001
< 0001

In data: Day average = 0.28499 Night average = 0.28048

No random effect for Player
Parameter Estimate
Intercept
0.2850
DAY
N -0.0045

Error
0.00174983
0.00247463

Center for Data Analysis and Statistics (CDAS)

t Value
162.87
-1.82

Pr > |t|
<.0001
0.0696
28

Random effects
Covariance Parameter Estimates
Cov Parm
Subject
Estimate
Intercept
PLAYER
0.000531
0 000531
Residual
0.000105

Player

Day

Night

Model Day

Model Night

0.24943

0.24545

0.25004

0.24554

0 26905
0.26905

0 26712
0.26712

0 27048
0.27048

0 26598
0.26598

0.29949

0.29169

0.29772

0.29321

0.26852

0.26626

0.26979

0.26529

Center for Data Analysis and Statistics (CDAS)

29

Acute Myocardial Infarction (AMI)

Article by Austin, Tu, Alter in the American Heart

Journal ((JAN 2001))
Analyzed patients admitted to Ontario hospitals
between 1994 and 1999 (>100,000)
Compared traditional to hierarchical logistic models
3 levels: patients, physicians and hospitals
Separate analysis for 9 outcomes classified as
fatal, non-fatal and processes of care

Center for Data Analysis and Statistics (CDAS)

30

Study Findings

Found patient level variables agreed

Hi
Hierarchical
hi l method
th d led
l d to
t different
diff
t conclusions
l i
44% of the time for hospital level factors

Traditional methods overestimated the statistical

significance of these factors

Traditional models tended to underestimate the

magnitude of physician level factors

Center for Data Analysis and Statistics (CDAS)

31

References

BOOKS

Bryk and Raudenbush (1992) good diagnostic sections

Longford (1993) can be tough to read; Fisher scoring
Goldstein ((1995)) details sometimes omitted,, but manyy of
technical issues are there (download for free)
Hox (1995) appears very introductory (download for free)
Kreft and DeLeeuw (1998) Nice introduction with many
examples using MLn
Snijders and Bosker (1999) Good introduction; some math
left out. Best sections on model checking/diagnostics
Goldstein and Leyland
y
Eds ((2001)) Great for intuitive
understanding; recent and good source of further references

Center for Data Analysis and Statistics (CDAS)

33

PAPERS

Topic dependent; I have lists

Useful Websites:

Multilevel models project (UK) - Goldstein

http://multilevel.ioe.ac.uk/index.html
Multilevel Analysis Page (Netherlands) - Snijders
http://stat.gamma.rug.nl/multilevel.htm
Multilevel Modeling Page (Leipzig) Mayerhofer
http://www lrz muenchen de/~wlm/wlmmule htm#Author
http://www.lrz-muenchen.de/~wlm/wlmmule.htm#Author
LAMMP (Michigan) Raudenbush
p
p
g
g
g
http://www-personal.engin.umich.edu/~gibsong/
Center for Data Analysis and Statistics (CDAS)

34

Marginal Models

Example is Generalized Estimating

Equations (GEE)
Primary interest in fixed parameters
Random structure specified nuisance
parameters
Hierarchical
h l models
d l consider
d random
d
parameters of interest themselves

Estimate fixed and random simultaneously

Center for DataAnalysis
and Statistics
Focus
of our(CDAS)
proposed research

35

Linear Hierarchical Model

Using
the Deviance
Comparison

IGLS ((ML estimation only)

y)
Comparing models with different random parts must have
the same fixed part
Given the maximum of the likelihood function

Di = 2 log(
l ( Li )

Difference in Deviance to compare two models

D1 D2 ~ 2 ( p2 p1 )

Where the number of parameters in the models

p1 < p2

Wald Tests of Fixed Parameters

Concern about appropriateness of assymptotic standard

normal distribution
h
Test of hypothesis

H0 : h = 0

Center for Data Analysis and Statistics (CDAS)

T (h ) =

SE( h )

36

Linear Hierarchical Model

Comparison
Wald
test in small samples compare to a tdistribution
Random parameters (Bryk & Raudenbush)

Obtain LS estimates of parameter within each

group and use chi-square test of equality
across groups

Multivariate Wald test available for a group

of parameters
Other tests all appear to have problems
and/or are not available in most software
packages
Center for Data Analysis and Statistics (CDAS)

37