1

"#$%&' ()*%+, &$* +#$%&' (#-%* %..%/0, ()*%+, #$ 1

2#03 +#$45#,0#/ &66+#/&0#)$,

Bouo Wintei

0niveisity of Califoinia, Neiceu, Cognitive anu Infoimation Sciences




750)'#&+ 8&'0 9: "#$%&' ()*%+#$4

Lineai mouels anu lineai mixeu mouels aie an impiessively poweiful anu flexible
tool foi unueistanuing the woilu. This tutoiial is a ueciueuly conceptual
intiouuction to this class of mouels. The focus is on !"#$%&'("#)"* what these
mouels aie uoing . anu then we'll spenu most of the time (++,-)"* this
unueistanuing, using the R statistical piogiamming enviionment. The iuea is to
bootstiap youi knowleuge as quickly as possible so that you can stait with youi
own analyses anu then tuin to moie technical texts if neeueu. The examples that I
uiaw fiom come fiom linguistics anu phonetics, but you neeu not be a linguist to
be able to follow this tutoiial.


You'll neeu about 2 houis to complete the full tutoiial (maybe a bit moie). Each
pait takes about 1 houi.




So, what uoes the lineai mouel uo. Assume you knew nothing about males anu
females, anu you weie inteiesteu in whethei the voice pitch of males anu females
uiffeis, anu if so, by how much.

So you take a bunch of males anu a bunch of females, anu ask them to say a single
woiu, say "mama", anu you measuie the iespective voice pitches. Youi uata might
look something like this:

2

;5<=%/0 ;%- >)#/%?8#0/3
1 female 2SS Bz
2 female 2u4 Bz
S female 242 Bz
4 male 1Su Bz
S male 112 Bz
6 male 142 Bz

"Bz" (Beitz) is a measuie of pitch wheie highei values mean highei pitch.

You might look at this table anu say that it's quite obvious that females have
highei voice pitch than females. Aftei all, the female values seem to be about 1uu
Bz above the male ones.

But, in fact, it coulu be the case that females anu males have the same pitch, anu
you weie just unlucky anu happeneu to choose some exceptionally high-pitcheu
females anu some exceptionally low-pitcheu males. Intuitively, the pattein in the
table seems pietty stiaightfoiwaiu, but we might want a moie piecise estimate
of the uiffeience between males anu females, anu we might also want an estimate
about how likely (oi unlikely) that uiffeience in voice pitch coulu have aiisen just
because of uiawing an unlucky sample.

This is wheie the lineai mouel comes in. In this case, its task is to give you some
values about voice pitch foi males anu females. as well as some piobability
value as to how likely those values aie.

The basic iuea is to expiess youi ielationship of inteiest (in this case, the one
between sex anu voice pitch) as a simple foimula. such as this one:

pitch ~ sex

This ieaus "pitch pieuicteu by sex" oi "pitch as a function of sex". Some people
call the thing on the left the "uepenuent vaiiable" (the thing you measuie) anu
the thing on the iight the "inuepenuent vaiiable". 0theis call the thing on the
iight the "explanatoiy vaiiable" (this sounus too causal to me) oi the "pieuictoi".
I'll call it "fixeu effect", anu this teiminology will make sense latei on in tutoiial 2.

Now, the woilu isn't peifect. Things aien't quite as ueteiministic as the above
foimula suggests. Pitch is not "#$%&'('&) ueteimineu by sex, but also by a bunch
of uiffeient factois such as language, uialect, peisonality, age anu what not. Even
if we measuieu all of these factois, theie woulu still be othei factois influencing
pitch that we cannot contiol foi. Peihaps, a subject in youi uata hau a hangovei
on the moining of the iecoiuing (causing the voice to be lowei than usual), oi the
subject was just moie neivous on that paiticulai uay (causing the voice to be
highei). We can nevei measuie anu contiol all of these things. The woilu is full of
S
stuff that is outsiue the puiview of oui little expeiiment. Bence, let's upuate oui
foimula to captuie the existence of these "ianuom" factois.

pitch ~ sex + c

This "c" (ieau "epsilon") is an eiioi teim. It stanus foi all of the things that affect
pitch that aie not sex, all of the stuff that - fiom the peispective of oui
expeiiment - is ianuom oi uncontiollable.

The foimula above is a schematic uepiction of the lineai mouel that we'ie going
to builu. Note that the pait of the foimula on the iight-hanu siue conceptually
uiviues the woilu into stuff that you can unueistanu (the "fixeu effect" sex) anu
stuff that you can't unueistanu (the ianuom pait "c"). You coulu call the foimei
the "stiuctuial" oi "systematic" pait of youi mouel anu the lattei the "ianuom" oi
"piobabilistic" pait of the mouel.

!"#$%&'# )*)+,-%). /)01% %0"+02
0.k., let's move to R, the statistical piogiamming enviionment that we'll use foi
the iest of this tutoiial
1
. Let's cieate the uataset that we'll use foi oui analysis.
Type in:

pitch = c(233,204,242,130,112,142)
sex = c(rep("female",3),rep("male",3))

The fiist line concatenates oui 6 uata points fiom above anu saves it in an object
that we nameu pitch. The seconu line iepeats the woiu "female" S times anu
then the woiu "male" S times . anu concatenates these 6 woius into an object
that we nameu sex.

Foi a bettei oveiview, let's combine these two objects into a uata fiame:

my.df = data.frame(sex,pitch)

Now we have a uata fiame object that we nameu my.df, anu if you type that,
you'll see this:


1
You uon't have R. Bon't woiiy, it's fiee anu woiks on all platfoims. You can get it heie:
http:www.i-pioject.oig You might want to ieau a quick intio to R befoie you pioceeu - but
even if you uon't, you'll be able to follow eveiything. }ust type in eveiything you see in uaik blue.
4


0.k., now we'll pioceeu with the lineai mouel. We take oui foimula above anu
feeu it into the lm() function . except that we omit the "c" teim, because the
lineai mouel function uoesn't neeu you to specify this.

xmdl = lm(pitch ~ sex, my.df)

We moueleu pitch as a function of sex, taken fiom the uata fiame my.df . anu
we saveu this mouel into an object that we nameu xmdl. To see what the lineai
mouel uiu, we have to "summaiize" this object using the function summary():

summary(xmdl)

If you uo this, you shoulu see this:



Lots of stuff heie. Fiist, you'ie being ieminueu of the mouel foimula that you
enteieu. Then, the mouel gives you the iesiuuals (what this is will be uiscusseu
latei), anu the coefficients of the fixeu effects (again, explanations follow. beai
with me foi a moment). Then, the output piints some oveiall iesults of the mouel
that you constiucteu.

S
We have to woik thiough this output. Let's stait with "Nultiple R-Squaieu". This
iefeis to the statistic *
+
which is a measuie of "vaiiance explaineu" oi if you
piefei less causal language, it is a measuie of "vaiiance accounteu foi". *
+
values
iange fiom u to 1. 0ui *
+
is u.921, which is quite high . you can inteipiet this as
showing that 92.1% of the stuff that's happening in oui uataset is "explaineu" by
oui mouel. In this case, because we have only one thing in oui mouel uoing the
explaining (the fixeu effect "sex"), the *
+
ieflects how much of oui uata is
accounteu foi by uiffeiences between males anu females.

In geneial, you want *
+
values to be high, but what is consiueieu a high *
+
value
uepenus on youi fielu anu on youi phenomenon of stuuy. If the system you stuuy
is veiy ueteiministic, *
+
values can be appioach 1. But in most of biology anu the
social sciences, wheie we stuuy complex oi messy systems that aie affecteu by a
whole bunch of uiffeient phenomena, we fiequently ueal with much lowei *
+

values.

The "Aujusteu R-squaieu" value is a slightly uiffeient *
+
value that not only looks
at how much vaiiance is "explaineu", but also at how many fixeu effects you useu
to uo the explaining. So, in the case of oui mouel above, the two values aie quite
similai to each othei, but in some cases the aujusteu *
+
,-. can be much lowei if
you have a lot of fixeu effects (say, you also useu age, psychological tiaits, uialect
etc. to pieuict pitch).

So much foi *
+
. Next line uown you see the thing that eveiybouy is ciazy foi: Youi
statistical test of "significance". If you've alieauy uone ieseaich, youi eyes will
piobably immeuiately jump to the p-value, which in many fielus is youi ticket foi
publishing youi woik. Theie's a little bit of an obsession with p-values . anu
even though they aie iegaiueu as so impoitant, they aie quite often
misunueistoou! So what exactly uoes the p-value mean heie.

0ne way to phiase it is to say that ,//0$123 )#04 $#-'& 1/ -#123 2#(51236 (5'
%4#7,71&1() #8 )#04 -,(, 1/ 4'&,(19'&) &#: (because the p-value is small in this case).
Technically speaking, the p-value is a /0"#)')0"(, +%01(1),)'-, it is a piobability
02-'4 (5' "#2-1(1#2 (5,( (5' 20&& 5)%#(5'/1/ 1/ (40'. In this case, the null hypothesis
is "sex has no effect on pitch". Anu, the lineai mouel shows that if this hypothesis
is tiue, then the uata woulu be quite unlikely. This is then inteipieteu as showing
that the alteinative hypothesis "sex affects pitch" is moie likely anu hence that
youi iesult is "statistically significant".

0sually, howevei, you have to uistinguish between the significance of the oveiall
mouel (the p-value at the veiy bottom of the output), which consiueis all effects
togethei, fiom the p-value of inuiviuual coefficients (which you finu in the
coefficients table above the oveiall significance). We'll talk moie about this in a
bit.

6
Then comes the F-value anu the uegiees of fieeuom. Foi an explanation of this,
see my tutoiial on AN0vAs anu the logic behinu the F-test
(http:bouowintei.comtutoiialbw_anova_geneial.puf). Foi a geneial lineai
mouel analysis, you piobably neeu this value to iepoit youi iesults. If you wanteu
to say that youi iesult is "significant", you woulu have to wiite something like
this:

"We constiucteu a lineai mouel of pitch as a function of sex. This mouel
was significant (F(1,4)=46.61, p<u.u1). (.)"

Now, let's look at the coefficient table. Beie it is again:



Note that the p-value foi the oveiall mouel was p=u.uu24u7, which is the same as
the p-value on the iight-hanu siue of the coefficients table in the iow that staits
with "sexmale". This is because youi mouel hau only one fixeu effect (namely,
"sex") anu so the significance of the oveiall mouel is the same as the significance
foi this coefficient. If you hau multiple fixeu effects, then the significance of the
oveiall mouel anu the significance of this coefficient will be uiffeient. That is
because the significance of the oveiall mouel takes all fixeu effects (all
explanatoiy vaiiables) into account wheieas the coefficients table looks at each
fixeu effect inuiviuually.

But why uoes it say "sexmale" iathei than just "sex", which is how we nameu oui
fixeu effect. Anu wheie uiu the females go. If you look at the estimate in the iow
that staits with "(Inteicept)", you'll see that the value is 226.SS Bz. This looks like
it coulu be the estimateu mean of the female voice pitches. If you type the
following.

mean(my.df[my.df$sex=="female",]$pitch)

. you'll get the mean of female voice pitch values, anu you'll see that this value is
veiy similai to the estimate value in the "(Inteicept)" column.

Next, note that the estimate foi "sexmale" is negative. If you subtiact the estimate
in the fiist iow fiom the seconu, you'll get 128, which is the mean of the male
voice pitches (you can veiify that by iepeating the above commanu anu
exchanging "male" foi "female").

To sum up, the estimate foi "(Inteicept)" is the estimate foi the female categoiy,
anu the estimate foi "sexmale" is the estimate foi the -188'4'2"' between the
7
females anu the male categoiy. This may seem like a veiy iounuabout way of
showing a uiffeience between two categoiies, so let's unpack this fuithei.

Inteinally, lineai mouels like to think in lines. So heie's a pictuie of the way the
lineai mouel sees youi uata:



The lineai mouel imagines the uiffeience between males anu females as a slope.
So, to go "fiom females to males", you have to go uown -98.SS . which is exactly
the coefficient that we've seen above. The inteinal cooiuinate system looks like
this:


Females aie sitting at the x-cooiuinate zeio at the y-inteicept (the point wheie
the line ciosses the y-axis), anu males aie sitting at the x-cooiuinate 1. So now,
the output makes a hella moie sense to us:

0
5
0
1
0
0
1
5
0
2
0
0
2
5
0
3
0
0
female male
V
o
i
c
e

p
i
t
c
h

(
H
z
)
0
5
0
1
0
0
1
5
0
2
0
0
2
5
0
3
0
0
female male
V
o
i
c
e

p
i
t
c
h

(
H
z
)
8


The females aie hiuuen behinu this mysteiious "(Inteicept)" anu the estimate foi
that inteicept is the estimate foi female voice pitch! Then, the uiffeience between
females anu males is expiesseu as a slope. "going uown" by 98.SS. The p-values
to the iight of this table coiiesponu to tests whethei each coefficient is "non-
zeio". 0bviously, 226.SS Bz is uiffeient fiom zeio, so the inteicept is "significant"
with a veiy low p-value. The slope -98.SS is also uiffeient fiom zeio (but in the
negative uiiection), anu so this is significant as well.

You might ask youiself: Why uiu the mouel choose females to be the inteicept
iathei than males. Anu what is the basis foi choosing one iefeience level ovei
the othei. The lm() function simply takes whatevei comes fiist in the alphabet!
"f" comes befoie "m", making "females" the inteicept at x=u anu "males" the slope
of going fiom u to 1.

It might not appeai stiaightfoiwaiu to you why we can expiess categoiical
uiffeiences (heie, between men anu women) as a slope. The ieason why this
woiks is because the uiffeience between two categoiies is exactly coiielateu with
the slope between two categoiies. The following figuies will help you iealize this
fact. In those pictuies, I incieaseu the uistance between two categoiies . anu
exactly piopoitional to this inciease in uistance, the slope incieaseu as well.



What's the big auvantage of thinking of the uiffeience between two categoiies as
a line ciossing those two categoiies. Well, the big auvantage is that you can use
0
5
1
0
1
5
2
0
2
5
c(rep(0, 4), rep(1, 4))
c
(
m
y
.
y
s
,

m
y
.
y
s

+

4
)
Slope: 4
Difference = 4
0
5
1
0
1
5
2
0
2
5
c(rep(0, 4), rep(1, 4))
c
(
m
y
.
y
s
,

m
y
.
y
s

+

6
)
Slope: 6
Difference = 6
0
5
1
0
1
5
2
0
2
5
c(rep(0, 4), rep(1, 4))
c
(
m
y
.
y
s
,

m
y
.
y
s

+

8
)
Slope: 8
Difference = 8
0
5
1
0
1
5
2
0
2
5
c(rep(0, 4), rep(1, 4))
c
(
m
y
.
y
s
,

m
y
.
y
s

+

1
0
)
Slope: 10
Difference = 10
0
5
1
0
1
5
2
0
2
5
c(rep(0, 4), rep(1, 4))
c
(
m
y
.
y
s
,

m
y
.
y
s

+

1
2
)
Slope: 12
Difference = 12
0
5
1
0
1
5
2
0
2
5
c(rep(0, 4), rep(1, 4))
c
(
m
y
.
y
s
,

m
y
.
y
s

+

1
4
)
Slope: 14
Difference = 14
9
the same piinciple foi something that is not categoiical. So, if you hau a
continuous factoi, say age, you coulu also fit a line. Eveiything woulu woik
exactly the same. Let's tiy this out. Say you weie now inteiesteu in whethei age
pieuicts voice pitch. The uata might look something like this:

;5<=%/0 @4% >)#/%?8#0/3
1 14 2S2 Bz
2 2S 244 Bz
S SS 24u Bz
4 48 2SS Bz
S S2 212 Bz
6 67 2u4 Bz

Anu heie's a scatteiplot of this uata:


0.k., same thing as befoie: We expiess this as a function, wheie oui "fixeu effect"
is now "age".

pitch ~ age + c

Let's constiuct the uata in R anu iun the mouel:

age = c(14,23,35,48,52,67)
pitch = c(252,244,240,233,212,204)
my.df = data.frame(age,pitch)
xmdl = lm(pitch ~ age, my.df)
summary(xmdl)

In the output, let's focus on the coefficients:

0 20 40 60 80
1
8
0
2
0
0
2
2
0
2
4
0
2
6
0
2
8
0
V
o
i
c
e

p
i
t
c
h

(
H
z
)
Age (years)
1u


Again, the significance of the inteicept is not veiy inteiesting. Remembei that the
p-value in each iow is simply a test of whethei the coefficient to the left is
significantly uiffeient fiom zeio. The inteicept (267.u76S) heie is the pieuicteu
pitch value foi people with age u. This inteicept uoesn't make much sense
because people who aie not boin yet uon't ieally have voice pitch.

What ieally inteiests us is "age", which emeiges as a significant "pieuictoi" of
voice pitch. The way to ieau the output foi age ("-u.9u99") is that foi eveiy
inciease of age by 1 you ueciease voice pitch by u.9u99 Beitz. Easy-peasy: just go
one step to the iight in youi giaph (in youi unit of measuiement, heie: age in
yeais) anu one step uown (in youi unit of measuiement, heie: voice pitch in Bz).

The scatteiplot below neatly summaiizes the mouel: The line iepiesents the
mean that the mouel pieuicts foi people at age u, 1, 2, S etc. This is the line that
iepiesents the coefficients of the mouel. It's woith looking at this pictuie anu
compaiing it to the coefficients table above. See that the line at x=u is 267.u76S
(oui inteicept), anu the slope is -u.9u99.





0 20 40 60 80
1
8
0
2
0
0
2
2
0
2
4
0
2
6
0
2
8
0
V
o
i
c
e

p
i
t
c
h

(
H
z
)
Age (years)
11
3)"#-#4567 "#$ 8)"#-#47)%% -#0)+,)90%
You might want to iemeuy the above-uiscusseu situation that the inteicept is
meaningless. 0ne way of uoing this woulu be to simply subtiact the mean age
fiom each age value, as is uone below:

my.df$age.c = my.df$age - mean(my.df$age)
xmdl = lm(pitch ~ age.c, my.df)
summary(xmdl)

Beie, we just cieateu a new column "age.c" that is the age vaiiable with the mean
subtiacteu fiom it. This is the iesulting coefficient table fiom iunning a lineai
mouel analysis of this "centeieu" uata:



Note that while the estimate has changeu fiom 267.u76S (pieuicteu voice pitch at
age u) to 2Su.8SSS (pieuicteu voice pitch at aveiage age), the slope hasn't
changeu anu neithei uiu the significance associateu with the slope oi the
significance associateu with the full mouel. That is, you haven't messeu at all with
the natuie of youi mouel, you just changeu the metiic so that the inteicept is now
the mean voice pitch. So, via centeiing oui vaiiable we maue the inteicept moie
meaningful.

:'-#4 '#
Both of these examples have been aumitteuly simple. Bowevei, things easily
"scale up" to moie complicateu stuff. Say, you measuieu two factois ("age" ,2-
"sex") . you coulu put them in the same mouel. Youi foimula woulu then be:

pitch ~ sex + age + c

0i, you coulu auu uialect as an auuitional factoi:

pitch ~ uialect + sex + age + c

Anu so on anu so on. The only thing that changes is the following. The p-value at
the bottom of the output will be the p-value foi the #9'4,&& $#-'&. This means that
the p-value consiueis how well all of youi fixeu effects togethei help in
accounting foi vaiiation in pitch. The coefficient output will then have p-values
foi the inuiviuual fixeu effects.

12
This is what people sometimes call "multiple iegiession", wheie you mouel one
iesponse vaiiable as a function of multiple pieuictoi vaiiables. The lineai mouel
is just anothei woiu foi multiple iegiession.


;%%6890-'#%


Theie's a ieason why we call the lineai mouel a $#-'&. Like any othei mouel, the
lineai mouel has (&&!2+')0"& . anu it's impoitant to talk about these
assumptions. So heie's a whiilwinu toui thiough the conuitions that have to be
satisfieu in oiuei foi the lineai mouel to be meaningful:

A9B "#$%&'#0C
It's calleu "lineai mouel" foi a ieason! The thing to the left of oui simple foimula
above has to be the iesult of a lineai combination of the things on the iight. If it
uoesn't, the %$&)#!(, +,0' will inuicate some kinu of cuive, oi it will inuicate some
othei pattein (e.g., two lines if you have categoiical binaiy uata).

We haven't talkeu about iesiuual plots yet, let alone iesiuuals. So, let's uo that!
Bave a look at the pictuie below, which is a uepiction of the agepitch
ielationship again:



The ieu lines inuicate the iesiuuals, which aie the ueviations of the obseiveu uata
points fiom the pieuicteu values (the so-calleu "fitteu values"). In this case, the
iesiuuals aie all faiily small . which is because the line that iepiesents the lineai
mouel pieuicts oui uata veiy well, i.e., all points aie veiy close to the line.

To get a bettei view of the iesiuuals, you can take a snapshot of this giaph like
this.
0 20 40 60 80
1
8
0
2
0
0
2
2
0
2
4
0
2
6
0
2
8
0
V
o
i
c
e

p
i
t
c
h

(
H
z
)
Age (years)
1S



. anu iotate it ovei. So, you make a new plot wheie the line that the mouel
pieuicts is now the centei line. Like heie:


This is a iesiuual plot. The fitteu values (the pieuicteu means) aie on the
hoiizontal line (at y=u). The iesiuuals aie the veitical ueviations fiom this line.
This view is just a iotation of the actual uata (compaie the iesiuual plot with the
scatteiplot to see this). To constiuct the iesiuual plot foi youiself, simply type:

plot(fitted(xmdl),residuals(xmdl))
2


2
Youi plot will have no cential line anu it will have uiffeient scales. It's woith spenuing some
time on tweaking youi iesiuual plot anu making it pietty. in paiticulai, you shoulu make the plot
200 210 220 230 240 250 260
-
1
5
-
1
0
-
5
0
5
1
0
1
5
Fitted values
R
e
s
i
d
u
a
l
s
14

In this case. theie isn't any obvious pattein in the iesiuuals. If theie :'4' a
nonlineai oi cuivy pattein, then this woulu inuicate a violation of the lineaiity
assumption. Beie's an example of a iesiuual plot that cleaily shows a violation of
lineaiity:


What to uo if youi iesiuual plot inuicates nonlineaiity. Theie's seveial options:

You might miss an impoitant fixeu effect that inteiacts with whatevei
fixeu effects you alieauy have in youi mouel. Potentially the pattein in the
iesiuual plot goes away if this fixeu effect is auueu.
Anothei (commonly chosen) option is to peifoim a nonlineai
tiansfoimation of youi iesponse, e.g., by taking the log-tiansfoim.
You can also peifoim a nonlineai tiansfoimation of youi fixeu effects. So,
if age weie somehow ielateu to pitch in a 0-shapeu way (peihaps, if veiy
young people hau high pitch anu veiy olu people hau high pitch, too, with
inteimeuiate ages having a "uip" in pitch), then you coulu auu age anu age
2

(age-squaieu) as pieuictois.
Finally, if you'ie seeing stiipes in youi iesiuual plot, then you'ie most
likely uealing with some kinu of categoiical uata - anu you woulu neeu to
tuin to a somewhat uiffeient class of mouels, such as logistic mouels.




so that theie's moie space aiounu the maigins. This will make any patteins easiei to see. Bave a
look at some R giaphic tutoiials foi this.
200 210 220 230 240 250 260
-
1
5
-
1
0
-
5
0
5
1
0
1
5
Fitted values
R
e
s
i
d
u
a
l
s
1S
ADB @<,%$/% ). /)++#$%&'#0C
When two fixeu effects (two pieuictois) aie coiielateu with each othei, they aie
saiu to be /0,,)"$(%. Say, you weie inteiesteu in how aveiage talking speeu
affects intelligence iatings (i.e., people who talk moie quickly aie iateu to be
moie intelligent).

intelligence iatings ~ talking speeu

. anu you measuieu seveial uiffeient inuicatois of talking speeu, foi example,
you syllables pei seconus, woius pei seconus anu sentences pei seconus. These
uiffeient measuies aie going to be coiielateu with each othei because if you
speak moie quickly, then you say moie syllables, woius anu sentences in a given
amount of time. If you'u use all of these coiielateu pieuictois to pieuict
intelligence iatings within the same mouel, you aie likely going to iun into a
collineaiity pioblem.

If theie's collineaiity, the inteipietation of the mouel becomes unstable:
Bepenuing on which one of coiielateu pieuictois is in the mouel, the fixeu effects
become significant oi cease to be significant. Anu, the significance of these
coiielateu oi collineai fixeu effects is not easily inteipietable, because they might
steal each othei's "explanatoiy powei" (that's a veiy coaise way of saying what's
actually going on, but you get the iuea).

Intuitively, this makes a lot of sense: If multiple pieuictois aie veiy similai to
each othei, then it becomes veiy uifficult to ueciue what, in fact, is playing a big
iole.

Bow to get iiu of collineaiity. Well fiist of all, you might pie-empt the pioblem in
the uesign stage of youi stuuy anu focus on a few fixeu effects that you know aie
not coiielateu with each othei. If you uiun't uo this anu you have seveial multiple
measuies to choose fiom at the analysis stage of youi stuuy (e.g., thiee uiffeient
ways of measuiing "talking speeu"), think about which one is the most
meaningful anu uiop the otheis (be caieful heie: uon't base this uiopping
uecision on the "significance"). Finally, you might want to consiuei uimension-
ieuuction techniques such as Piincipal Component Analysis. These can tiansfoim
seveial coiielateu vaiiables into a smallei set of vaiiables which you can then use
as new fixeu effects.


16
AEB F)(),G%*&,0#/#0C H )' I&<,%$/% ). 3%0%'),G%*&,0#/#0CJ
Being able to pionounce "heteioskeuasticity" seveial times in a iow in quick
succession will make you a stai at youi next cocktail paity, so go aheau anu
ieheaise pionouncing them now!

}okes asiue, homoskeuasticity is an extiemely impoitant assumption. It says that
the vaiiability of youi uata shoulu be appioximately equal acioss the iange of
youi pieuicteu values. If homosceuasticity is violateu, you enu up with
heteioskeuasticity, oi, in othei woius, a pioblem with unequal vaiiances.

Foi the homosceuasticity assumption to be met, the %$&)#!(,& of youi mouel
neeu to ioughly have a similai amount of ueviation fiom youi pieuicteu values.
Again, we can check this by looking at a iesiuual plot. Beie's the one foi the
agepitch uata again:



Theie's not ieally that many uata points to tell whethei this is ieally
homosceuastic. In this case, I woulu concluue that theie's not enough uata to
safely ueteimine whethei theie is oi isn't heteioskeuasticity. 0sually, I woulu
constiuct mouels foi much laigei uata sets anyway.

So, heie's a plot that gives you an iuea of how a "goou" iesiuual plot looks with
moie uata:

200 210 220 230 240 250 260
-
1
5
-
1
0
-
5
0
5
1
0
1
5
Fitted values
R
e
s
i
d
u
a
l
s
17

Anu anothei one:



A goou iesiuual plot essentially looks blob-like. It's a goou iuea to geneiate some
ianuom uata to see how a plot with ioughly equal vaiiances looks like. You can
uo so using the following commanu line:

plot(rnorm(100),rnorm(100))

This cieates two sets of 1uu noimally uistiibuteu ianuom numbeis with a mean
of u anu a stanuaiu ueviation of 1. If you type this in multiple times to cieate
multiple plots, you can get a feel of how a "noimal" iesiuual plot shoulu look like.

The next iesiuual plot shows obvious heteioskeuasticity:

-3 -2 -1 0 1 2 3
-
3
-
2
-
1
0
1
2
3
Fitted values
R
e
s
i
d
u
a
l
s
-3 -2 -1 0 1 2 3
-
3
-
2
-
1
0
1
2
3
Fitted values
R
e
s
i
d
u
a
l
s
18


In this plot, highei fitteu values have laigei iesiuuals . inuicating that the mouel
is moie "off" with laigei pieuicteu means. So, the vaiiability is not
homosceuastic: it's smallei in the lowei iange anu laigei in the highei iange.

What to uo. Again, tiansfoiming youi uata often helps. Consiuei a log-tiansfoim
heie as well.

AKB L)'(&+#0C ). '%,#*5&+,
The noimality of iesiuuals assumption is the one that is least impoitant.
Inteiestingly, many people seem to think it is the most impoitant one, but it tuins
out that lineai mouels aie ielatively iobust against violations of the assumptions
of noimality. Reseaicheis uiffei with iespect to how much weight they put onto
checking this assumption. Foi example, uellman anu Bill (2uu7), a famous book
on lineai mouels anu mixeu mouels, uo not even iecommenu uiagnostics of the
noimality assumption (ibiu. 46).

If you wanteu to test the assumption, how woulu you uo this. Eithei you make a
histogiam of the iesiuuals of youi mouel, using.

hist(residuals(xmdl))

. oi a Q-Q plot .

qqnorm(residuals(xmdl)))

Beie's a iesiuual plot anu a Q-Q plot of the same uata next to each othei.

-3 -2 -1 0 1 2 3
-
3
-
2
-
1
0
1
2
3
Fitted values
R
e
s
i
d
u
a
l
s
19


These look goou. The histogiam is ielatively bell-shapeu anu the Q-Q plot
inuicates that the uata falls on a stiaight line (which means that it's similai to a
noimal uistiibution). Beie, we woulu concluue that theie aie no obvious
violations of the noimality assumption.

AMB @<,%$/% ). #$.+5%$0#&+ *&0& 6)#$0,
Some people woulun't call "the absence of influential uata points" an ,//0$%(1#2
of the mouel. Bowevei, influential uata points can uiastically change the
inteipietation of youi iesults, anu similai to collineaiity, it can leau to instable
iesults.

Bow to check. Beie's a useful R function, dfbeta(), that you can use on a mouel
object like oui xmdl fiom above.



Foi each coefficient of youi mouel (incluuing the inteicept), the function gives
you the so-calleu BFbeta values. These aie the values with which the coefficients
have to be aujusteu if a paiticulai uata point is excluueu (sometimes calleu
"leave-one-out uiagnostics"). Noie concietely, let's look at the age column in the
uata fiame above. The fiist iow means that the coefficient foi age (which, if you
iemembei, was -u.9u99) has to be aujusteu by u.u64S7S7S if uata point 1 is
excluueu. That means that the coefficient of the mouel without the uata point
x
F
r
e
q
u
e
n
c
y
-3 -2 -1 0 1 2
0
5
1
0
1
5
2
0
-2 -1 0 1 2
-
2
-
1
0
1
2
Normal Q-Q Plot
Theoretical Quantiles
S
a
m
p
l
e

Q
u
a
n
t
i
l
e
s
2u
woulu be -u.97424S1 (which is -u.9u99 minus u.u64S7S7S. if the slope is
negative, BFbeta values aie subtiacteu, if it's positive, they aie auueu).

Theie's a little bit of ioom foi inteipietation in what constitutes a laige oi a small
BFbeta value. 0ne thing you can say foi suie: Any value that changes the sign of
the slope is -'8121('&) an influential point that waiiants special attention.
because excluuing that point woulu change the inteipietation of youi iesults.
What I then uo is to eyeball the BFbetas anu look foi values that aie uiffeient by
half of the absolute value of the slope. Say, my slope woulu be 2 . then a BFbeta
value of 1 oi -1 woulu be alaiming to me. If it's a slope of -4, a BFbeta value of 2
oi -2 woulu be alaiming to me.

Bow to pioceeu if you have influential uata points. Well, it's uefinitely not legit to
simply excluue those points anu iepoit only the iesults on the ieuuceu uata set. A
bettei appioach woulu be to iun the analysis :1(5 the influential points anu then
again :1(5#0( the influential points . then you can iepoit both analyses anu state
whethei the inteipietation of the iesults uoes oi uoesn't change. The only case
when it is o.k. to excluue influential points is when theie's an obvious eiioi with
them, so foi example, a value that uoesn't make sense (e.g., negative age, negative
height) oi a value that obviously is the iesult uue to a technical eiioi in the uata
acquisition stage (e.g., voice pitch values of u). Influence uiagnostics allow you to
spot those points anu you can then go back to the oiiginal uata anu see what went
wiong
S
.

ANB O$*%6%$*%$/% PPPPPPP
The inuepenuence assumption is by fai the most impoitant assumption of all
statistical tests. In the lineai mouel analyses that we uiu so fai, each uata point
came fiom a uiffeient subject. To ieminu you, heie's oui two uata sets that we
woikeu on:

;05*C 9 ;05*C D
;5<=%/0 ;%- >)#/%?8#0/3 ;5<=%/0 @4% >)#/%?8#0/3
1 female 2SS Bz 1 14 2S2 Bz
2 female 2u4 Bz 2 2S 244 Bz
S female 242 Bz S SS 24u Bz
4 male 1Su Bz 4 48 2SS Bz
S male 112 Bz S S2 212 Bz
6 male 142 Bz 6 67 2u4 Bz

We weie able to iun the lineai mouel on this uata the way we uiu only because
each iow in this uataset comes fiom a uiffeient subject. If you elicit multiple

S
Foi an inteiesting back-anu-foith on a paiticulai example of how much influence uiagnostics
anu extieme values can change the inteipietation of a stuuy, have a look at the uelightful episoue
of acauemic bantei between 0lliich anu Schltei (2u11) anu Bianut (2u11).
21
iesponses fiom each subject, then those iesponses that come fiom the same
subject cannot be iegaiueu as inuepenuent fiom each othei.

So, what exactly is inuepenuence. The iueal case is a coin flip oi the ioll of a uie:
Each coin flip anu each ioll of a uie is absolutely inuepenuent fiom the outcome
of the pieceuing coin flips oi uie iolls. The same shoulu holu foi youi uata points
when you iun a lineai mouel analysis. So, the uata points shoulu come fiom
uiffeient subjects. Anu each subject shoulu only contiibute one uata point.

When you violate the inuepenuence assumption, all hell bieaks loose. The othei
assumptions that we mentioneu above aie impoitant as well, but the
inuepenuence assumption is by fai the most impoitant one. violating
inuepenuence may gieatly inflate youi chance of finuing a spuiious iesult anu it
iesults in a p-value that is completely meaningless. 0nfoitunately, violations of
the inuepenuence assumption aie quite fiequent in many bianches of science -
so much in fact, that theie's a whole liteiatuie associateu with this violation,
staiting fiom Builbeit (1984) foi ecology, Fieebeig anu Lucas (2uu9) foi
psychology, Lazic (2u1u) foi neuioscience anu my own small papei foi
phoneticsspeech science (Wintei, 2u11).

Bow can you guaiantee inuepenuence. Well, inuepenuence is a question of the
expeiimental uesign in ielation to the statistical test that you use. Besign anu
statistical analyses aie closely inteitwineu anu you can make suie that you meet
the inuepenuence assumption by only collecting one uata point pei subject.

Now, a lot of the times, we want to collect moie uata pei subject, such as in
iepeateu measuies uesigns. If you enu up a uata set that has non-inuepenuencies
in it, you neeu to %$&0,3$ '4$&$ "0"5)"#$+$"#$"/)$& at the analysis stage. This is
wheie 2)6$# 20#$,& come in hanuy. anu this is wheie we'll switch to the
seconu tutoiial.



22
Part 2: A very basic tutorial for performing
linear mixed effects analyses

This pait seives as a quick boot camp to jump-stait youi own analyses with
lineai mixeu effects mouels. This text is uiffeient fiom othei intiouuctions by
being ueciueuly "#2"'%(0,&; I will focus on :5) you want to use mixeu mouels
anu 5#: you shoulu use them. While many intiouuctions to this topic can be veiy
uaunting to ieaueis who lake the appiopiiate statistical backgiounu, this text is
going to be a softei kinu of intiouuction. so, uon't panic!

Pait 2 will take you about 1 houi (possibly a bit moie).


Introduction: Fixed and random effects
In tutoiial 1, we talkeu about how we coulu use the lineai mouel to expiess the
ielationships in oui uata in teims of a function. In one example, we moueleu pitch
as a function of age.

pitch ~ age + c

We calleu "age" a fixeu effect, anu c was oui "eiioi teim" to iepiesent the
ueviations fiom oui pieuictions uue to "ianuom" factois that we cannot contiol
expeiimentally. You coulu call this pait the "piobabilistic" oi "stochastic" pait of
the mouel. Now, we'll unpack this "c" anu auu complexity to it. That is, we change
the ianuom aspect of oui mouel, essentially leaving the systematic pait
unchangeu. In mixeu mouels, eveiything in the "systematic" pait of youi mouel
woiks just like with lineai mouels in tutoiial 1.

In one of my stuuies, we have been inteiesteu in the ielationship between pitch
anu politeness (Wintei & uiawunuei, 2u12). So, essentially we'ie aiming foi a
ielationship that looks like something like this:

pitch ~ politeness + c

In oui stuuy, politeness was tieateu as a categoiical factoi with two levels. a
foimal iegistei anu an infoimal iegistei. 0n top of that, we also hau an auuitional
fixeu effect, sex, anu so oui foimula looks moie like this:

pitch ~ politeness + sex + c

So fai so goou. Now things get a little moie complicateu. 0ui uesign was so that
we took multiple measuies pei subject. That is, each subject gave multiple polite
iesponses anu multiple infoimal iesponses. If we go back to the uiscussion of the
assumptions of the lineai mouel in tutoiial 1, we can immeuiately see that this
2S
woulu violate the inuepenuence assumption: Nultiple iesponses fiom the same
subject cannot be iegaiueu as inuepenuent fiom each othei. Eveiy peison has a
slightly uiffeient voice pitch, anu this is going to be an iuiosynciatic factoi that
affects all iesponses fiom the same subject, thus ienueiing these uiffeient
iesponses intei-uepenuent iathei than inuepenuent.

The way we'ie going to ueal with this situation is to auu a %("#02 $77$/' foi
subject. This allows us to iesolve this non-inuepenuence by assuming a uiffeient
"baseline" pitch value foi each subject. So, subject 1 may have a mean voice pitch
of 2SS Bz acioss uiffeient utteiances, anu subject 2 may have a mean voice pitch
of 21u Bz pei subject. Beie's a visual uepiction of how this looks like:


Subjects F1 to F9 aie female subjects. Subjects N1 to N7 aie male subjects. You
immeuiately see that males have lowei voices than females (as is to be expecteu).
But on top of that, within the male anu the female gioups, you see lots of
inuiviuual vaiiation, with some people having ielatively highei values foi theii
sex anu otheis having ielatively lowei values.

We can mouel these inuiviuual uiffeiences by assuming uiffeient %("#02
)"'$%/$+'& foi each subject. That is, each subject is assigneu a uiffeient inteicept
value, anu the mixeu mouel estimates these inteicepts foi you.

Now you begin to see why the mixeu mouel is calleu a "mixeu" mouel. The lineai
mouels that we consiueieu so fai have been "fixeu-effects-only" mouels that hau
one oi moie fixeu effects anu a geneial eiioi teim "c". With the lineai mouel, we
essentially uiviueu the woilu into things that we somehow unueistanu oi that aie
somehow systematic (the fixeu effects, oi the explanatoiy vaiiables); anu things
that we cannot contiol foi oi that we uon't unueistanu (c). But ciucially, this
F1 F2 F3 F4 F5 F6 F7 F8 F9 M1 M2 M3 M4 M5 M6 M7
1
0
0
1
5
0
2
0
0
2
5
0
3
0
0
3
5
0
4
0
0
M
e
a
n

P
i
t
c
h

(
H
e
r
t
z
)
Subjects
24
lattei pait, the unsystematic pait of the mouel, uiu not have any inteiesting
stiuctuie. We simply hau a geneial acioss-the-boaiu eiioi teim.

In the mixeu mouel, we auu one oi moie ianuom effects to oui fixeu effects.
These ianuom effects essentially give stiuctuie to the eiioi teim "c". In the case
of oui mouel heie, we auu a ianuom effect foi "subject", anu this chaiacteiizes
iuiosynciatic vaiiation that is uue to inuiviuual uiffeiences.

The mixtuie of fixeu anu ianuom effects is what makes the mixeu mouel a mixeu
mouel.

0ui upuateu foimula looks like this:

pitch ~ politeness + sex + (1|subject) + c

"(1|subject)" looks a little enigmatic. I'm alieauy using the R-typical notation
foimat heie. What this is saying is "assume an inteicept that's uiffeient foi each
subject" . anu "1" stanus foi the inteicept heie. You can think of this foimula as
telling youi mouel that it shoulu expect that theie's going to be multiple
iesponses pei subject, anu these iesponses will uepenu on each subject's
baseline level. This effectively iesolves the non-inuepenuence that stems fiom
having multiple iesponses by the same subject.

Note that the foimula still contains a geneial eiioi teim "c". This is necessaiy
because even if we accounteu foi inuiviuual by-subject vaiiation, theie's still
going to be "ianuom" uiffeiences between uiffeient utteiances fiom the same
subject.

0.k., so fai so goou. But we'ie not uone yet. In the uesign that we useu in Wintei
anu uiawunuei (2u12), theie's an auuitional souice of non-inuepenuence that
neeus to be accounteu foi: We hau uiffeient items. 0ne item, foi example, was an
"asking foi a favoi" scenaiio. Beie, subjects hau to imagine asking a piofessoi foi
a favoi (polite conuition), oi asking a peei foi a favoi (infoimal conuition).
Anothei item was an "excusing foi coming too late" scenaiio, which was similaily
uiviueu between polite anu infoimal. In total, theie weie 7 such uiffeient items.

Similai to the case of by-subject vaiiation, we also expect by-item vaiiation. Foi
example, theie might be something special about "excusing foi coming too late"
which leaus to oveiall highei pitch (maybe because it's moie embaiiassing than
asking foi a favoi), iegaiuless of the influence of politeness. Anu whatevei it is
that makes one item uiffeient fiom anothei, the iesponses of the uiffeient
subjects in oui expeiiment might similaily be affecteu by this ianuom factoi that
is uue to item-specific iuiosynciasies. That is, if "excusing foi coming to late"
leaus to high pitch (foi whatevei ieason), it's going to uo so foi subject 1, subject
2, subject S anu so on. Thus, the uiffeient iesponses to one item cannot be
iegaiueu as inuepenuent, oi, in othei woius, theie's something similai to
2S
multiple iesponses to the same item - even if they come fiom uiffeient people.
Again, if we uiu not account foi these inteiuepenuencies, we woulu violate the
inuepenuence assumption.

Beie's a visual iepiesentation of the by-item vaiiability:



The vaiiation between items isn't as big as the vaiiation between subjects - but
theie aie still noticeable uiffeiences, anu we bettei account foi them in oui
mouel!

We uo this by auuing an auuitional ianuom effect:

pitch ~ politeness + sex + (1|subject) + (1|item) + c

So, on top of uiffeient inteicepts foi uiffeient subjects, we now also have uiffeient
inteicepts foi uiffeient items. We now "iesolveu" those non-inuepenuencies (oui
mouel knows that theie aie multiple iesponses pei subject anu pei item), anu we
accounteu foi by-subject anu by-item vaiiation in oveiall pitch levels.

Note the efficiency anu elegance of this mouel. Befoie, people useu to uo a lot of
aveiaging. Foi example, in psycholinguistics, people woulu aveiage ovei items foi
a subjects-analysis (each uata point comes fiom one subject, assuiing
inuepenuence), anu then they woulu also aveiage ovei subjects foi an items-
analysis (each uata point comes fiom one item). Theie's a whole liteiatuie on the
auvantages anu uisauvantages of this appioach (Claik, 197S; Foistei &
Bickinson, 1976; Wike & Chuich, 1976; Raaijmakeis, Schiijnemakeis, &
uiemmen, 1999; Raaijmakeis, 2uuS; Lockei, Boffman, & Bovaiiu, 2uu7; Baayen,
Baviuson, & Bates, 2uu8; Baii, Levy, Scheepeis, & Tilly, 2u1S).
1 2 3 4 5 6 7
1
0
0
1
5
0
2
0
0
2
5
0
3
0
0
3
5
0
4
0
0
M
e
a
n

P
i
t
c
h

(
H
e
r
t
z
)
Items
26

The upshot is: while tiauitional analyses that uo aveiaging aie in piinciple legit,
mixeu mouels give you much moie flexibility . anu they take the full uata into
account. If you uo a subjects-analysis (aveiaging ovei items), you'ie essentially
-1/4'3,4-123 by-item vaiiation. Conveisely, in the items-analysis, you'ie
uisiegaiuing by-subject vaiiation. Nixeu mouels account foi both souices of
vaiiation 12 , /123&' $#-'&. Neat, init.

Let's move on to R anu apply oui cuiient unueistanuing of the lineai mixeu
effects mouel!!

Mixed models in R
Foi a stait, we neeu to install the R package &$'; (Bates, Naechlei & Bolkei,
2u12). While being connecteu to the inteinet, open R anu type in:

install.packages(lme4)

Select a seivei close to you. Aftei installation, loau the &$'; package into R with
the following commanu:

library(lme4)

Now, you have the function lmer() available to you, which is the mixeu mouel
equivalent of the function lm() in tutoiial 1. This function is going to constiuct
mixeu mouels foi us.

But fiist, we neeu some uata! I put a shoiteneu veision of the uataset that we
useu foi Wintei anu uiawunuei (2u12) onto my seivei. You can loau it into R the
following way:

politeness= read.csv("http://www.bodowinter.com/tutorial/politeness_data.csv")

0i you can uownloau it by hanu.

http:www.bouowintei.comtutoiialpoliteness_uata.csv

.anu loau it into R the following way:

politeness = read.csv(file.choose( ))

Now, you have a uata fiame calleu politeness in youi R enviionment. You can
familiaiize youiself with the uata by using head(), tail(), summary(),
str(), colnames(). oi whatevei commanus you commonly use to get an
oveiview of a uataset. Also, it is always goou to check foi missing values:

27
which(is.na(politeness)==T)

Appaiently, theie is a missing value in iow 26S. This is impoitant to know but
foitunately, a few missing values pioviue no pioblems foi oui mixeu mouel
analyses.

The uiffeience in politeness level is iepiesenteu in the column calleu "attituue".
In that column, "pol" stanus foi polite anu "inf" foi infoimal. Sex is iepiesenteu as
"F" anu "N" in the column "genuei". The uepenuent measuie is "fiequency",
which is the voice pitch measuieu in Beitz (Bz). To ieminu you, highei values
mean highei pitch.

The inteiesting ianuom effects foi us aie in the column "subject" anu "scenaiio",
the lattei being the name of the item column (iemembei the uiffeient scenaiios
like "asking foi a favoi".).

Let's look at the ielationship between politeness anu pitch by means of a boxplot:

boxplot(frequency ~ attitude*gender,
col=c("white","lightgray"),politeness)

What uo we see. In both cases, the meuian line (the thick line in the miuule of the
boxplot) is lowei foi the polite than foi the infoimal conuition. Bowevei, theie
may be a bit moie oveilap between the two politeness categoiies foi males than
foi females.

inf.F pol.F inf.M pol.M
1
0
0
1
5
0
2
0
0
2
5
0
3
0
0
28
Let's stait with constiucting oui mouel!

Type in the commanu below .

lmer(frequency ~ attitude, data=politeness)

. anu you will ietiieve an eiioi that shoulu look like this:



This is because the mouel 2''-/ a ianuom effect (aftei all, "mixing" fixeu anu
ianuom effects is the point of mixeu mouels). We just specifieu a single fixeu
effect, attituue, anu that was not enough.

So, let's auu ianuom inteicepts foi subjects anu items (iemembei that items aie
calleu "scenaiios" heie):

politeness.model = lmer(frequency ~ attitude +
(1|subject) + (1|scenario), data=politeness)

The last commanu cieateu a mouel that useu the fixeu effect "attituue" (polite vs.
infoimal) to pieuict voice pitch, contiolling foi by-subject anu by-item vaiiability.
We saveu this mouel in the object politeness.model. To see what the iesult
is, simply type in politeness.model to piint the output (in contiast to lm()
you uon't neeu to use summary() to get this output).

This is the full output:



29
Again, let's woik thiough this: Fiist, the output ieminus you of the mouel that you
fit. Then, theie's some geneial summaiy statistics such as Akaike's Infoimation
Ciiteiion, the log-Likelihoou etc. We won't go into the meaning of these uiffeient
values in this tutoiial because these aie conceptually a little bit moie involveu.
Let's focus on the output foi the ianuom effects fiist:



Bave a look at the column stanuaiu ueviation. This is a measuie of the vaiiability
foi each ianuom effect that you auueu to the mouel. You can see that scenaiio
("item") has much less vaiiability than subject. Baseu on oui boxplots fiom
above, wheie we saw moie iuiosynciatic uiffeiences between subjects than
between items, this is to be expecteu. Then, you see "Resiuual" which stanus foi
the vaiiability that's not uue to eithei scenaiio oi subject. This is oui "c" again,
the "ianuom" ueviations fiom the pieuicteu values that aie not uue to subjects
anu items. Beie, this ieflects the fact that each anu eveiy utteiance has some
factois that affect pitch that aie outsiue of the puiview of oui expeiiment.

The fixeu effects output miiiois the coefficient table that we consiueieu in
tutoiial 1 when we talkeu about the iesults of oui lineai mouel analysis.



The coefficient "attituuepol" is the slope foi the categoiical effect of politeness.
Ninus 19.69S means that to go fiom "infoimal" to "polite", you have to go uown -
19.69S Bz. In othei woius: pitch is lowei in polite speech than in infoimal speech,
by about 2u Bz. Then, theie's a stanuaiu eiioi associateu with this slope, anu a t-
value, which is simply the estimate (2u Bz) uiviueu by the stanuaiu eiioi (check
this by peifoiming the calculation by hanu).

Note that the lmer() function (just like the lm() function in tutoiial 1) took
whatevei comes fiist in the alphabet to be the iefeience level. "inf" comes befoie
"pol", so the slope iepiesents the change fiom "inf" to "pol". If the iefeience
categoiy woulu be "pol" iathei than "inf", the only thing that woulu change woulu
be that the sign of the coefficient 19.69S woulu be positive. Stanuaiu eiiois,
significance etc. woulu iemain the same.

Su
Now, let's consiuei the inteicept. In tutoiial 1, we alieauy talkeu about the fact
that oftentimes, mouel inteicepts aie not paiticulaily meaningful. But this
inteicept is especially weiiu. It's 2u2.S88 Bz . wheie uoes that value come fiom.

If you look back at the boxplot that we constiucteu eailiei, you can see that the
value 2u2.S88 Bz seems to fall halfway between males anu females - anu this is
inueeu what this inteicept iepiesents. It's the aveiage of oui uata foi the infoimal
conuition.

As we uiun't infoim oui mouel that theie's two sexes in oui uataset, the inteicept
is paiticulaily off, in between the voice pitch of males anu females. This is just
like the classic example of a faim with a uozen hens anu a uozen cows . wheie
the mean legs of all faim animals consiueieu togethei is thiee, not a paiticulaily
infoimative iepiesentation of what's going on at the faim.

Let's auu genuei as an auuitional fixeu effect:

politeness.model = lmer(frequency ~ attitude +
gender + (1|subject) +
(1|scenario), data=politeness)

We oveiwiote oui oiiginal mouel object politeness.model with this new
mouel. Note that we auueu "genuei" as a fixeu effect because the ielationship
between sex anu pitch is systematic anu pieuictable (i.e., we expect females to
have highei pitch). This is uiffeient fiom the ianuom effects subject anu item,
wheie the ielationship between these anu pitch is much moie unpieuictable anu
"ianuom". We'll talk moie about the uistinction between fixeu anu ianuom effects
latei.

Let's piint the mouel output again. Let's have a look at the iesiuuals fiist:



Note that compaieu to oui eailiei mouel without the fixeu effect genuei, the
vaiiation that's associateu with the ianuom effect "subject" uioppeu
consiueiably. This is because the vaiiation that's uue to genuei was confounueu
with the vaiiation that's uue to subject. The mouel uiun't know about males anu
females, anu so it's pieuictions weie ielatively moie off, cieating ielatively laigei
iesiuuals.

Let's look at the coefficient table now:

S1


We see that males anu females uiffei by about 1u9 Bz. Anu the inteicept is now
much highei (2S6.846 Bz), as it now iepiesents the female categoiy (foi the
infoimal conuition). The coefficient foi the effect of attituue uiun't change much.

Statistical significance
So fai, we haven't talkeu about significance yet. But, if you want to publish this,
you'll most likely neeu to iepoit some kinu of p-value. 0nfoitunately, p-values foi
mixeu mouels aien't as stiaightfoiwaiu as they aie foi the lineai mouel. Theie
aie multiple appioaches, anu theie's a uiscussion suiiounuing these, with
sometimes wiluly uiffeiing opinions about which appioach is the best. Beie, I
focus on the Likelihoou Ratio Test as a means to attain p-values.

Likelihoou is the piobability of seeing the uata you collecteu given youi mouel.
The logic of the likelihoou iatio test is to compaie the likelihoou of two mouels
with each othei. Fiist, the mouel :1(5#0( the factoi that you'ie inteiesteu in (the
null mouel), then the mouel :1(5 the factoi that you'ie inteiesteu in. Naybe an
analogy helps you to wiap youi heau aiounu this: Say, you'ie a hikei, anu you
caiiy a bunch of uiffeient things with you (e.g., a gallon of watei, a flashlight). To
know whethei each item affects youi hiking speeu, you neeu to get iiu of it. So,
you get iiu of the flashlight anu iun without it. Youi hiking speeu is not affecteu
much. Then, you get iiu of the gallon of watei, anu you iealize that youi hiking
speeu is affecteu a lot. You woulu concluue that caiiying a gallon of watei with
you significantly affects youi hiking speeu wheieas caiiying a flashlight uoes not.
Expiesseu in foimula, you woulu want to compaie the following two "mouels"
(think "hikes") with each othei:

mul1 = hiking speeu ~ gallon of watei + flashlight
mul2 = hiking speeu ~ flashlight

If theie is a significant uiffeience between "mul2" anu "mul1", then you know that
the gallon of watei matteis. To assess the effect of the flashlight, you woulu have
to uo a similai compaiison:

mul1 = hiking speeu ~ gallon of watei + flashlight
mul2 = hiking speeu ~ gallon of watei

In both cases, we compaieu a full mouel (with the fixeu effects in question)
against a ieuuceu mouel without the effects in question. In each case, we
S2
concluue that a fixeu effect is significant if the uiffeience between the likelihoou
of these two mouels is significant.

Beie's how you woulu uo this in R. Fiist, you neeu to constiuct the null mouel:

politeness.null = lmer(frequency ~ gender +
(1|subject) + (1|scenario), data=politeness,
REML=FALSE)

Note one auuitional technical uetail. I just auueu the aigument REML=FALSE.
Bon't woiiy about it too much - but in case you'ie inteiesteu, this changes some
inteinal stuff (in paiticulai, the likelihoou estimatoi), anu it is necessaiy to uo
this when you compaie mouels using the likelihoou iatio test (Pinheiio & Bates,
2uuu; Bolkei et al., 2uu9).

Then, we ie-uo the full mouel above, this time also with REML=FALSE:

politeness.model = lmer(frequency ~ attitude +
gender +
(1|subject) + (1|scenario), data=politeness,
REML=FALSE)

Now you have two mouels to compaie with each othei - one with the effect in
question, one without the effect in question. We peifoim the likelihoou iatio test
using the anova() function:

anova(politeness.null,politeness.model)

This is the iesulting output:



You'ie being ieminueu of the foimula of the two mouels that you'ie compaiing.
Then, you finu a Chi-Squaie value, the associateu uegiees of fieeuom anu the p-
value
4
.

4
You might wonuei why we'ie uoing a Chi-Squaie test heie. Theie's a lot of technical uetail heie, but the main thing is
that theie's a theoiem, calleu Wilk's Theoiem, which states that negative two times the log likelihoou iatio of two mouels
appioaches a Chi-Squaie uistiibution with uegiees of fieeuom of the numbei of paiameteis that uiffei between the
mouels (in this case, only "attituue"). So, somebouy has uone a pioof of this anu you'ie goou to go! Bo note, also, that some
people uon't like "stiaight-jacketing" likelihoou into the classical null-hypothesis significance testing fiamewoik that we'ie
following heie, anu so they woulu uisagiee with the inteipietation of likelihoou the way we useu it in the likelihoou iatio
test.
SS

You woulu iepoit this iesult the following way:

". politeness affecteu pitch (!
2
(1)=11.62, p=u.uuu6S), loweiing it by
about 19.7 Bz S.6 (stanuaiu eiiois) ."

If you'ie useu to t-tests, AN0vAs anu lineai mouel stuff, then this likelihoou-
baseu appioach might seem weiiu to you. Rathei than getting a p-value
stiaightfoiwaiuly fiom youi mouel, you get a p-value fiom a compaiison of two
mouels. To help you get useu to the logic, iemembei the hikei anu the analogy of
putting one piece of luggage away to estimate that piece's effect on hiking speeu.

Note that we kept the pieuictoi "genuei" in the mouel. The only change between
the full mouel anu the null mouel that we compaieu in the likelihoou iatio test
was the factoi of inteiest, politeness. In this paiticulai test, you can think of
"genuei" as a contiol vaiiable anu of "attituue" as youi test vaiiable.

We coulu have also compaieu the following two mouels:

full mouel: fiequency ~ attituue + genuei
ieuuceu mouel: fiequency ~ 1

"mul.null" in this case is an inteicept only mouel, wheie you just estimate the
mean of the uata. You coulu compaie this to "mul.full", which has two moie
effects, "attituue" anu "genuei". If this uiffeience became significant, you woulu
know that "mul.full" anu "mul.null" aie significantly uiffeient fiom each othei -
but you woulu not know whethei this uiffeience is uue to "attituue" oi uue to
"genuei". Coming back to the hikei analogy, it is as if you uioppeu both the gallon
of watei anu the flashlight anu then you iealizeu that youi hiking speeu changeu,
but you woulun't be able to ueteimine conclusively which one of the two pieces
of luggage was the ciucial one.

A final thing iegaiuing likelihoou iatio tests: What happens if you have an
inteiaction. We uiun't talk much about inteiactions yet, but say, you pieuicteu
"attituue" to have an effect on pitch that is somehow mouulateu thiough
"genuei". Foi example, it coulu be that speaking politely veisus infoimally has the
opposite effect foi men anu women. 0i it coulu be that women show a uiffeience
anu men uon't (oi vice veisa). If you have such an intei-uepenuence between two
factois (calleu an inteiaction), you can test it the following way:

full mouel: fiequency ~ attituue*genuei
ieuuceu mouel: fiequency ~ attituue + genuei

In R, inteiactions between two factois aie specifieu with a "*" iathei than a "+".

S4
If you compaie the above mouels in a likelihoou iatio test using the anova()
function, then you woulu get a p-value that gives you the significance of the
inteiaction. If this compaiison is significant, you know that attituue anu genuei
aie significantly intei-uepenuent on each othei. If this is compaiison is not
significant, theie is no significant intei-uepenuence.

It might be a goou iuea to tiy out uiffeient likelihoou compaiisons with the uata
pioviueu above, say "attituue*genuei" veisus "attituue + genuei" veisus simply
"1" (the inteicept only mouel). Remembei to always put REML=FALSE when
cieating youi mouel.

Super-crucial: Random slopes versus random intercepts
We'ie not uone yet. 0ne of the coolest things about mixeu mouels is coming up
now, so hang on!!

Let's have a look at the coefficients of the mouel by subject anu by item:

coef(politeness.model)

Beie is the output:



You see that each scenaiio anu each subject is assigneu a uiffeient inteicept.
That's what we woulu expect, given that we've tolu the mouel with "(1|subject)"
anu "(1|scenaiio)" to take by-subject anu by-item vaiiability into account.

SS
But not also that the fixeu effects (attituue anu genuei) aie all the same foi all
subjects anu items. 0ui mouel is what is calleu a %("#02 )"'$%/$+' 20#$,. In this
mouel, we account foi baseline-uiffeiences in pitch, but we assume that whatevei
the effect of politeness is, it's going to be the same foi all subjects anu items.

But is that a valiu assumption. In fact, often times it's not - it is quite expecteu
that some items woulu elicit moie oi less politeness. That is, the effect of
politeness might be uiffeient foi uiffeient items. Likewise, the effect of politeness
might be uiffeient foi uiffeient subjects. Foi example, it might be expecteu that
some people aie moie polite, otheis less. So, what we neeu is a %("#02 &,0+$
mouel, wheie subjects anu items aie not only alloweu to have uiffeiing
inteicepts, but wheie they aie also alloweu to have uiffeient slopes foi the effect
of politeness. This is how we woulu uo this in R:

politeness.model = lmer(frequency ~ attitude +
gender + (1+attitude|subject) +
(1+attitude|scenario),
data=politeness,
REML=FALSE)

Note that the only thing that we changeu is the ianuom effects, which now look a
little moie complicateu. The notation "(1+attituue|subject)" means that you tell
the mouel to expect uiffeiing baseline-levels of fiequency (the inteicept,
iepiesenteu by 1) as well as uiffeiing iesponses to the main factoi in question,
which is "attituue" in this case. You then uo the same foi items.

Bave a look at the coefficients of this upuateu mouel by typing in the following:

coef(politeness.model)

Beie's a iepiint of the output that I got:

S6


Now, the column with the by-subject anu by-item coefficients foi the effect of
politeness ("attituuepol") is uiffeient foi each subject anu item. Note, howevei,
that it's always negative anu that many of the values aie quite similai to each
othei. This means that uespite inuiviuual vaiiation, theie is also consistency in
how politeness affects the voice: foi all of oui speakeis, the voice tenus to go
uown when speaking politely, but foi some people it goes uown slightly moie so
than foi otheis.

Bave a look at the column foi genuei. Beie, the coefficients uo no change. That is
because we uiun't specify ianuom slopes foi the by-subject oi by-item effect of
genuei.

0.k., let's tiy to obtain a p-value. We keep oui mouel fiom above
(politeness.model) anu compaie it to a new null mouel in a likelihoou iatio
test. Let's constiuct the null mouel fiist:

politeness.null = lmer(frequency ~ gender +
(1+attitude|subject) + (1+attitude|scenario),
data=politeness, REML=FALSE)

Note that the null mouel neeus to have the same ianuom effects stiuctuie. So, if
youi full mouel is a ianuom slope mouel, youi null mouel also neeus to be a
ianuom slope mouel.

Let's now uo the likelihoou iatio test:

anova(politeness.null,politeness.model)
S7

This is, again, significant.

Theie aie a few impoitant things to say heie: You might ask youiself "Which
ianuom slopes shoulu I specify." . oi even "Aie ianuom slopes necessaiy at all."

A lot of people constiuct ianuom inteicept-only mouels but conceptually, it
makes a hella sense to incluue ianuom slopes most of the time. Aftei all, you can
almost always expect that people uiffei with how they ieact to an expeiimental
manipulation! Anu likewise, you can almost always expect that the effect of an
expeiimental manipulation is not going to be the same foi all items.

Noieovei, ieseaicheis in ecology (Schielzeth & Foistmeiei, 2uu9),
psycholinguistics (Baii, Levy, Scheepeis, & Tilly, 2u1S) anu othei fielus have
shown via simulations that mixeu mouels without ianuom slopes aie anti-
conseivative oi, in othei woius, they have a ielatively high Type I eiioi iate (they
tenu to finu a lot of significant iesults which aie actually uue to chance).

Baii et al. (2u1S) iecommenu that you shoulu "keep it maximal" with iespect to
youi ianuom effects stiuctuie, at least foi contiolleu expeiiments. This means
that you incluue all ianuom slopes that aie justifieu by youi expeiimental uesign
. anu you uo this foi all fixeu effects that aie impoitant foi the oveiall
inteipietation of youi stuuy.

In the mouel above, oui whole stuuy ciucially iesteu on stating something about
politeness. We weie not inteiesteu in genuei uiffeiences, but they aie well woith
contiolling foi. This is why we hau ianuom slopes foi the effect of attituue (by
subjects anu item) but not genuei. In othei woius, we only moueleu by-subject
anu by-item vaiiability in how politeness affects pitch.

Assumptions
In tutoiial 1, we talkeu a lot about the many uiffeient assumptions of the lineai
mouel. The goou news is: Eveiything that we uiscusseu in the context of the
lineai mouel applies stiaightfoiwaiuly to mixeu mouels. So, you also have to
woiiy about collineaiity anu influential uata points. Anu you have to woiiy about
homosceuasticity (anu potentially about lack of noimality). But you uon't have to
leain much new stuff. The way you check these assumptions in R is exactly the
same as in the case of the lineai mouel, say, by cieating a iesiuual plot, a
histogiam of the iesiuuals oi a Q-Q plot.

Inuepenuence, being the most impoitant assumption, iequiies a special woiu:
0ne of the main ieasons we moveu to mixeu mouels iathei than just woiking
with lineai mouels was to iesolve non-inuepenuencies in oui uata. Bowevei,
mixeu mouels can still violate inuepenuence . if you'ie missing impoitant fixeu
S8
oi ianuom effects. So, foi example, if we analyzeu oui uata with a mouel that
uiun't incluue the ianuom effect "subject", then oui mouel woulu not "know" that
theie aie multiple iesponses pei subject. This amounts to a violation of the
inuepenuence assumption. So choose youi fixeu effects anu ianuom effects
caiefully, anu always tiy to iesolve non-inuepenuencies.

Then, a woiu on influential uata points. You will finu that the function dfbeta()
that we useu in the context of lineai mouels uoesn't woik foi mixeu mouels. If
you woiiy about influential points, you can check out the package 128&0'2"'<=>
(Nieuwenhuis, te uiotenhuis, & Pelzei, 2u12), oi you can piogiam a foi loop that
uoes the leave-one-out uiagnostics by hanu. The following coue gives you an
outline of the geneial stiuctuie of how you might want to uo this (you can check
my "uoouling" tutoiials on loops anu piogiamming stiuctuies in R to get a bettei
giasp of this):

all.res=numeric(nrow(mydataframe))
for(i in 1:nrow(mydataframe)){
myfullmodel=lmer(response~predictor+
(1+predictor|randomeffect),POP[-i,])
all.res[i]=fixef(myfullmodel)[some number]
}
5




uo aheau anu play with checking the assumptions. You can go back to tutoiial 1
anu apply the coue in theie to the new mixeu mouel objects in this tutoiial.


A final note on random versus fixed effects
I have evaueu a piecise uefinition of the uiffeience between fixeu anu ianuom
effects. I uelibeiately uiu this because I wanteu you to get some expeiience with
lineai mixeu effects mouels in R befoie we finally take a step back anu shaipen
oui concepts.

S
The basic iuea of this coue snippet is this: Pie-uefine a vectoi that has as many
elements as you have iows in youi uataset. Then, cycle thiough each iow. Foi
each iteiation, make a new mixeu mouel :1(5#0( (5,( 4#: (this is achieveu by
POP[-i,]). Then, the function fixef() extiacts whatevei coefficient inteiests
you.
You will neeu to auapt this coue to youi analysis. Besiues the names of youi uata
fiame anu youi vaiiables, you neeu to iun fixef() on youi mouel once so you
know which position the ielevant coefficient is. In oui case, I woulu put a "2" in
theie because the effect of "attituuepol" appeais seconu in the list of coefficients.
"1" woulu give me the inteicept, always the fiist coefficient mentioneu in the
coefficient table.
S9

So, a ianuom effect is geneially something that can be expecteu to have a non-
systematic, iuiosynciatic, unpieuictable, oi "ianuom" influence on youi uata. In
expeiiments, that's often "subject" anu "item", anu you geneially want to
geneialize ovei the iuiosynciasies of inuiviuual subjects anu items.

Fixeu effects on the othei hanu aie expecteu to have a systematic anu pieuictable
influence on youi uata.

But theie's moie to it. 0ne uefinition of fixeu effects says that fixeu effects
"exhaust the population of inteiest", oi they exhaust "the levels of a factoi". Think
back of sex. Theie's only "male" oi "female" foi the vaiiable "genuei" in oui stuuy,
so these aie the only two levels of this factoi. 0ui expeiiment incluues both
categoiies anu thus exhausts the categoiy sex. With oui factoi "politeness" it's a
bit tiickiei. You coulu imagine that theie aie moie politeness levels than just the
two that we testeu. But in the context of oui expeiiment, we #%'4,(1#2,&&) -'812'-
politeness as the uiffeience between these two categoiies - anu because we
testeu both, we fully "exhaust" the factoi politeness (as uefineu by us).

In contiast, ianuom effects geneially sample fiom the population of inteiest. That
means that they aie fai away fiom "exhausting the population" . because theie's
usually many many moie subjects oi items that you coulu have testeu. The levels
of the factoi in youi expeiiment is a tiny subset of the levels "out theie" in the
woilu.

The write-up
A lot of tutoiials uon't covei how to wiite up youi iesults. Anu that's a pity,
because this is a ciucial pait of youi stuuy!!!

The most impoitant thing: You neeu to uesciibe the mouel to such an extent that
people can %$+%0#!/$ '4$ ("(,-&)&< So, a useful heuiistic foi wiiting up youi
iesults is to ask youiself the question "Woulu I be able to ie-cieate the analysis
given the infoimation that I pioviueu." If the answei is "yes" youi wiite-up is
goou.

In paiticulai, this means that you specify all fixeu effects anu all ianuom effects,
anu you shoulu also mention whethei you have ianuom inteicepts oi ianuom
slopes.

Foi iepoiting inuiviuual iesults, you can stick to my example with the likelihoou
iatio test above. Remembei that it's always impoitant to iepoit the actual
coefficientsestimates anu not just whethei an effect is significant. You shoulu
also mention stanuaiu eiiois.

4u
Anothei impoitant thing is to give enough cieuit to the people who put so much
of theii fiee time into making &$'; anu R woik so efficiently. So let's cite them!
It's also a goou iuea to cite exactly the veision that you useu foi youi analysis.
You can finu out youi veision anu who to cite by typing in.

citation()

. foi youi R-veision . anu .

citation(lme4)

. foi the lme4 package.

Finally, it's impoitant that you mention that you checkeu assumptions, anu that
the assumptions aie satisfieu.

So heie's what I woulu have wiitten foi the analysis that we peifoimeu in this
tutoiial:

"We useu R (R Coie Team, 2u12) anu &$'; (Bates, Naechlei & Bolkei,
2u12) to peifoim a lineai mixeu effects analysis of the ielationship
between pitch anu politeness. As fixeu effects, we enteieu politeness anu
genuei (without inteiaction teim) into the mouel. As ianuom effects, we
hau inteicepts foi subjects anu items, as well as by-subject anu by-item
ianuom slopes foi the effect of politeness. visual inspection of iesiuual
plots uiu not ieveal any obvious ueviations fiom homosceuasticity oi
noimality. P-values weie obtaineu by likelihoou iatio tests of the full
mouel with the effect in question against the mouel without the effect in
question."



Yay, we'ie uone!! I hope this tutoiial was of help to you.



References
Bates, B.N., Naechlei, N., & Bolkei, B. (2u12). lme4: Lineai mixeu-effects mouels using
S4 classes. R package veision u.999999-u.
Baayen, R.B. (2uu8). ?2,&)@123 A12301/(1" B,(,C ? D4,"(1",& E2(4#-0"(1#2 (# F(,(1/(1"/ G/123
*. Cambiiuge: Cambiiuge 0niveisity Piess.
Baayen, R.B., Baviuson, B.}., Bates, B.N. (2uu8). Nixeu-effects moueling with ciosseu
ianuom effects foi subjects anu items. H#042,& #8 ='$#4) ,2- A,230,3', S9, S9u-
412.
41
Baii, B.}., Levy, R., Scheepeis, C., & Tilly, B. }. (2u1S). Ranuom effects stiuctuie foi
confiimatoiy hypothesis testing: Keep it maximal. H#042,& #8 ='$#4) ,2-
A,230,3', 68, 2SS-278.
Bolkei, B. N., Biooks, N. E., Claik, C. }., ueange, S. W., Poulsen, }. R., Stevens, N. B. B., &
White, }. S. S. (2uu9). ueneializeu lineai mixeu mouels: a piactical guiue foi
ecology anu evolution. I4'2-/ 12 >"#&#3) J >9#&0(1#2, 24(S), 127-1SS.
Bianut, N. }. (2u11). Nasty uata can still be ieal: A ieply to 0lliich anu Schltei.
D/)"5#&#31",& F"1'2"', 2S, 826.
Claik, B. B. (197S). The language-as-fixeu-effect fallacy: A ciitique of language statistics
in psychological ieseaich. H#042,& #8 K'47,& A',42123 ,2- K'47,& L'5,91#4, 12,
SSS-SS9.
Foistei, K.I., & Bickinson, R.u. (1976). Noie on the language-as-fixeu-effect fallacy:
Nonte Cailo estimates of eiioi iates foi F1, F2, F' anu minF'. H#042,& #8 K'47,&
A',42123 J K'47,& L'5,91#46 1S, 1SS-142.
Fieebeig, T.N., & Lucas, }.R. (2uu9). Pseuuoieplication is (still) a pioblem. H#042,& #8
M#$%,4,(19' D/)"5#&#3), 12S, 4Su-4S1.
uellman, A., & Bill, }. (2uu7). B,(, ,2,&)/1/ 0/123 4'34'//1#2 ,2- $0&(1&'9'&N51'4,4"51",&
$#-'&/. Cambiiuge: Cambiiuge 0niveisity Piess.
Builbeit, S.B. (1984). Pseuuoieplication anu the uesign of ecological fielu expeiiments.
>"#&#31",& =#2#34,%5/, S4, 187-211.
Lazic, S.E. (2u1u). The pioblem of pseuuoieplication in neuioscientific stuuies: is it
affecting youi analysis. L=M O'04#/"1'2"', 11, 1-17.
Lockei, L., Boffman, L., & Bovaiiu, }.A. (2uu7). 0n the use of multilevel moueling as an
alteinative to items analysis in psycholinguistic ieseaich. L'5,91#4 *'/',4"5
='(5#-/, S9, 72S-7Su.
Nieuwenhuis, R., te uiotenhuis, N., & Pelzei, B. (2u12). Influence.NE: Tools foi Betecting
Influential Bata in Nixeu Effects Nouels. * H#042,&, 4(2): pp. S8-47.
Pinheiio, }.C., & Bates, B.N. (2uuu). =1P'-Q>88'"(/ =#-'&/ 12 F ,2- FDAGF. New Yoik:
Spiingei.
Raaijmakeis, }.u. (2uuS). A fuithei look at the "language-as-fixeu-effect fallacy". M,2,-1,2
H#042,& #8 >P%'41$'2(,& D/)"5#&#3), S7, 141-1S1.
Raaijmakeis, }.u., Schiijnemakeis, }.N.C., & uiemmen, F. (1999). Bow to Beal with "The
Language-as-Fixeu-Effect Fallacy": Common Nisconceptions anu Alteinative
Solutions. H#042,& #8 ='$#4) ,2- A,230,3', 41, 416-426.
R Coie Team (2u12). R: A language anu enviionment foi statistical computing. R
Founuation foi Statistical Computing, vienna, Austiia.
Schielzeth, B., & Foistmeiei, W. (2uu9). Conclusions beyonu suppoit: oveiconfiuent
estimates in mixeu mouels. L'5,91#4,& >"#&#3)6 2u, 416-42u.
0lliich, }., & Schltei, E. (2u11). Betecting nasty uata with simple plots of complex
mouels: Comment on Bianut (2u11). D/)"5#&#31",& F"1'2"', 2S, 824-82S.
Wike, E.L., & Chuich, }.B. (1976). Comments on Claik's "The language-as-fixeu-effect
fallacy". H#042,& #8 K'47,& A',42123 J K'47,& L'5,91#4, 1S, 249-2SS.
42
Wintei, B. (2u11). Pseuuoieplication in phonetic ieseaich. D4#"''-123/ #8 (5'
E2('42,(1#2,& M#234'// #8 D5#2'(1" F"1'2"' (pp. 21S7-214u). Bong Kong, August
2u11.
Wintei, B., & uiawunuei, S. (2u12). The Phonetic Piofile of Koiean Foimality. H#042,& #8
D5#2'(1"/, 4u, 8u8-81S.