Jorge Mndez Seijas

Individual Differences in Reactivity Research:

Working Memory and Extraversion/Introversion

Abstract
Verbalizations, also known as verbal protocols or think-alouds, have become a common
methodological tool in second language acquisition (SLA) research studies. This procedure is
used to tap into participants mental processes, and therefore provides researchers with
information about the thought processes participants engage in as they complete a task. A
challenge for researchers using verbalizations is that having participants think aloud while they
complete a task may be reactive, that is, it may enhance or hinder participants performance.
Although having a control group that does not think aloud as a point of comparison was a
solution proposed in the very first study about reactivity in SLA (Leow & Morgan, 2004), many
subsequent studies have attempted to determine whether verbalizations are reactive, and if so,
with which tasks, under what conditions, and so forth.
Relevant to the question of reactivity in SLA research is that verbalizations have been
examined in absolute terms, with the conditions being with or without verbal protocols. A deeper
look into this method of data elicitation may reveal a much more complicated picture, as not all
think-alouds are the same (Ericsson & Simon, 1993). For instance, does the amount of
verbalizations (measured in words, syllables, etc.) result in differences within the participants in
the think-aloud conditions? Do individual differences play a role in the reported reactivity, or
lack thereof, in some of the tasks? Does a personality trait such as introversion/extroversion have

an impact on how reactive or non-reactive a task may be? This paper will attempt to answer this
question by adding to independent variables, working memory and introversion/extroversion,
and slightly modifying the research design of Bowles & Leow (2005). It will therefore be both a
partial replication and an extension

Introduction
Individual differences play an important role in second language acquisition (Dornyei,
2009) and second language acquisition research. In terms of think-aloud protocols, however,
Goo (2010) is in the only SLA study that has investigated if an individual difference
(specifically, working memory capacity) should be taken into consideration when employing
think-aloud protocols. Besides working memory, given the characteristics of verbalizations, the
introversion/extraversion dimension of personality may be an interesting addition to this strand
of research.
Working memory capacity (WMC) and introversion/extroversion may be relevant
because they show two dimensions that may define how participants interact with the concurrent
measures: examining WMC could allow researchers to determine to what extent the innate and
varying capacity that participants have to hold information in memory while completing a task
may induce positive or negative reactivity. Some of these questions regarding WMC have been
dealt with in Goo (2010), but limitations will be pointed out that may cast doubt on the reliability
of his results.
To have an even better picture of what reactivity is and what tasks it impacts, given the
conditions in which the data is collected, an individual difference like introversion/extroversion
would introduce a personality trait that may determine, to some extent, whether or not a task is

reactive. It is not unlikely that verbalizations may impose additional pressure on introverted
participants, as compared to extroverted ones. Furthermore, introverted participants may simply
verbalize less than extroverted ones, which could, in turn, affect whether or not the think-alouds
are reactive. By introducing these two individual difference variables, we could further explore
reactivity with regards to task type, and also the role that innate and social individual differences
play.

Verbal Protocols and Reactivity

Verbal protocols can be broadly divided into two types: concurrent and retrospective.
Concurrent verbal protocols, which will be the subject of my discussion in this paper, occur
when participants verbalize their thoughts out loud while they are completing a task.
Retrospective verbal protocols, on the other hand, occur when participants verbalize their
thoughts a posteriori, that is, after they have completed a given task. Verbal protocols can be
further divided into two different types: metacognitive (also known as metalinguistic) and nonmetacognitive (non-metalinguistic). In the first type, participants are asked for explanations and
justifications for their verbalizations, in the second, they are simply asked to voice their thoughts
without any explanations or justifications.
Concurrent protocols pose a potential threat to studies internal validity, though, as it is
possible that the imposition of a secondary task (i.e., verbalizing ones thoughts) may somehow
affect the primary process(es) required for the task(s) under investigation. Such a threat is known
as reactivity. Reactivity, when present, can affect performance on the primary task in two
different ways: it can potentially enhance performance (positive reactivity) or hinder it (negative
reactivity). The way researchers generally test for reactivity is by comparing the performance of

an experimental group (i.e., those who think aloud) to that of a silent control group. If the groups
perform equally well (or poorly), then the researcher assumes that the verbal protocols are nonreactive. Currently, the results on reactivity have been mixed. That is, in the SLA literature, it is
possible to find studies that report positive reactivity in recognition and production tasks
(Rossomondo, 2007); negative reactivity in a written reformulation task (Sachs & Polio, 2007
(first experiment)), text completion (Sachs & Suh, 2007), reading comprehension (Bowles &
Leow, 2005; Goo, 2010; Morgan-Short, Heil, Botero-Moriarty & Ebert, 2012), L2 problemsolving (Bowles, 2008), aural posttest accuracy (Sanz et al., 2009), and latency (Bowles & Leow,
2005; Sanz, Lin, Lado, Bowden & Stafford; 2009: Yoshida, 2008). It is also possible to find
studies that report no reactivity in reading comprehension tasks (Bowles & Leow, 2005; Leow &
Morgan-Short, 2004; Rossomondo, 2007; Yoshida, 2008), posttest production (Bowles & Leow,
2005; Leow & Morgan-Short, 2004; Sachs & Suh, 2007), system learning (Bowles, 2008),
writing (Sachs & Polio, 2007 (second experiment)), posttest accuracy (Sanz et al., 2009), written
reformulation tasks (Sachs & Polio, 2007), and latency (Sachs & Suh, 2007; Sanz et al., 2009).
The previously cited studies differ in many crucial aspects, such as the proficiency level
of the participant, the primary task under investigation (reading, writing, etc.), the type of
secondary task (metacognitive vs. non-metacognitive think-alouds), the dependent variables
(comprehension tasks, form recognition, etc.), and so forth. Having so many differences among
studies makes it difficult to make comparisons and generalizations. I will illustrate this by
organizing some of the studies according to two tasks: reading comprehension and writing
production, as only these tasks will be part of my experimental design.
Reactivity in Reading Comprehension and Writing Production
Let us first take a look at studies whose primary task was reading comprehension. Leow

& Morgan-Short (2004) compared a non-metalinguistic think-aloud group and a silent control
group. Analyses comparing the think-aloud and control group showed think-alouds were nonreactive. That is, no significant differences between the groups were found. Similar results were
found in other studies that also used reading as a primary task, such as Rossomodo (2007), and
Yoshida (2008). Goo (2010), conversely, found that think-alouds were indeed reactive for
reading comprehension.
As for writing as a primary task, to my knowledge, two studies deal with the issue of
reactivity in writing production: Sachs and Polio (2007), and Yanguas & Lado (2012). Sachs &
Polio (2007)s found reactivity for one experiment and non-reactivity for the other experiment.
In this study, they investigated the effects of different types of written feedback (error correction
and reformulations) and the role of reactivity in L2 development of writing skills. Their results
for the first experiment (within-subjects) indicated that think-alouds were negatively reactive in
the reformulation group. In their second experiment (between-subjects), no reactivity was
reported. A look into this design and the comparisons it offers complicates matters further, as
they compare results from a within-subject design with results from a between-subject design.
Generalizations in writing can hardly be drawn by comparing such disparate designs. If we
include Yanguas & Lado (2012) to try to disentangle the problem, more differences are found:
the participants in their study are heritage speakers, while those in Sachs and Polio (2007) are
intermediate L2 English learners. Also, while the think-alouds in Yanguas & Lado could be
performed in either language, in Sachs & Polio (2007) participants were asked to think aloud in
the target language. Once again, as it was the case with reading as a primary task, generalizations
about reactivity cannot be easily made and further studies addressing the writing process are
clearly warranted.

Besides the internal and external validity issues, in all these studies, with the exception of
Goo (2010), verbalizations have been treated as if they were a uniform construct. That is, as
though it is enough to report that they took place, without further exploring their characteristics
or the individual differences in the participants that produce them. Some of these individual
differences may prove to have an impact in the quality and quantity of the verbalization, which
may, in turn, influence whether or not the verbalizations are reactive. Understanding these
variables will allow for a better understanding of verbal protocols and the type of reactivity they
may or may not generate. For the purposes of this study, I propose that two relevant individual
differences be taken into consideration: working memory and introversion/extroversion.

Working Memory
In the field of SLA, the interaction between working memory and learning has been
widely studied (Kormos & Sfr, 2008; Linck, Osthus, Koeth & Bunting, 2013, for a metaanalysis; Mackey, Adams, Stafford, & Winke, 2008). This interaction is expected and of great
interest in the field given the cognitive load involved in speaking a second language (L2),
especially at low levels of proficiency. The concept of working memory is relatively recent, and
was introduced by Baddeley & Hitch (1974). WM has been gaining track in psychology and
SLA studies. WM is fundamentally a form of memory, but it is more than memory, for it is
memory at work (Conway, Jarrold, Kane, Miyake, & Towse, 2008, p. 3). Therefore, WM
involves cognitive multitasking, as information must be kept active while performing an
additional task. Such multitasking is especially relevant in think-aloud protocols, as participants
must perform a task while at the same time speaking aloud. The impact WM may have in terms
of reactivity (positive and negative) may very well depend of the type of task.

Goo (2010), for instance, examined working memory capacity (WMC) as an independent
variable. In his study, the think-aloud and non-think-aloud groups were further subdivided into
two groups, according to the working memory capacity levels of participants. His results
indicated that there was reactivity for participants with high memory capacity in the think-aloud
condition, and no reactivity for low memory capacity participants. Although having this variable
could be an important development in the reactivity strand, some internal validity issues hamper
this studys overall reliability. One major limitation is the sample size, which only included 5-8
participants per condition. The tasks in this study were intended to investigate the role of WMC
in reading comprehension and rule learning. One significant problem with this design is that how
rule learning is operationalized is not easily comprehensible, as it is not clear how a fill-in-theblank test serves this purpose. Furthermore, the structure under investigation (vamos + infinitive)
requires no rule learning whatsoever, for participants are given an infinitive form and they only
need to reproduce it. Another internal validity issue with this study is that it only presented
results for the relationship between WMC and performance in the think-aloud groups. The
aforementioned shortcomings make it necessary to further investigate the role WM plays in
reactivity, so as to make sure that the results in this study in fact hold in a more robustly designed
investigation.

Introversion-extroversion
Another important factor that should be taken into consideration in reactivity research is
the introversion-extroversion dichotomy. This personality trait may positively or negatively
impact the act of thinking aloud, thus potentially generating positive or negative reactivity. In the
previous section, I discussed that an innate trait such as WM may differentiate participants to

such degree that a concurrent measure might be more or less taxing. The same may prove to be
true about introversion and extroversion. The introvert/extrovert dimension of personality could
be of great interest in reactivity research, because performing better or poorer on a test may have
to do, to a certain extent, with how personality affects such performance. Even more so when the
activities in the experiment include speaking out loud, which could impose different levels of
pressure depending on whether or not a participant is an introvert or an extrovert. Revelle,
Amaral, and Turriff (1976), for instance, compared verbal intelligence scores of introverts and
extroverts in three different conditions: a) untimed, b) timed + placebo caffeine, and c) timed +
caffeine. The results indicated that extroverts performed well in all three conditions, while
introverts performance dramatically dropped from the untimed condition to those of timed +
placebo caffeine and timed + caffeine. Bates and Rock (2004) compared introverts and
extroverts performance in different conditions according to different levels of noise, ranging
from silent environments to very noisy ones, and his result indicated that introverts performance
peaked in the silence condition, and decreased performance was reported in all other conditions.
Extrovert outperformed both introverts and ambiverts (the neither-nor group) in all other
conditions. Introversion/extroversion with regards to SLA has been studied in terms of learning
(for a review, see Zafar & Meenakshi, 2013). To my knowledge, however, this dimension of
personality has not been studied in the second language acquisition field in terms of test taking
conditions or verbalization protocols.
As Goo (2010) suggested, an individual difference such as working memory capacity
might play a role in the reported reactivity, or lack thereof, in studies that involve verbalizations.
Given the nature of verbalizations, the dichotomy introvert/extrovert may also have an effect. In
order to address the role these individual differences play in the issue of reactivity, this study will

introduce to the robust design of Bowles & Leow (2005) two individual differences: working
memory capacity and the introvert/extrovert dimension. This study will be guided by the
following research questions:
1. Are verbalizations reactive during:
a.

a reading comprehension task?

b. a writing production task in which students produce old exemplars of the

Spanish pluperfect subjective?
c. a writing production task in which students produce new exemplars of the
Spanish pluperfect subjective?
2. Does working memory capacity mediate the possible effects of reactivity?
3. Does introversion and extroversion mediate the possible effects of reactivity?

Method
The current study will be a partial replication and expansion of Bowles and Leow (2005).
Most of the materials and procedures of Bowles and Leow (2005) will be maintained, and small
methodological changes will be implemented. Two tests will be added to the design of Bowles
and Leow (2005): a working memory test and an introversion/extroversion test.

Participants
A total of 80 Spanish advanced students will participate in the current study: 40 will be
assigned to a control group, and 40 will be assigned to a think-aloud group. As in Bowles and
Leow (2005), participants will be given a pretest to make sure they only have minimal
knowledge of the Spanish pluperfect subjunctive. Any participant that scores over 2 out of 10 on

the pretest will be eliminated from the analyses.

Reading Materials and Tests

The target structure for the current study will be the Spanish pluperfect subjective. As in
Bowles and Leow (2005), the participants will read an 861-word article titled Si Bush hubiera
sido porteo. This article contains 28 instances of the Spanish pluperfect subjective. After the
students read the article, they will take one comprehension test and two written production tests.
The comprehension test and the written production tests will be the same tests used in Bowles
and Leow (2005). The comprehension test will be done in English, and it will consist of 10
multiple-choice items. There will be two written production tests: one will test the participants
ability to produce old exemplars of the Spanish pluperfect subjective and one will test their
ability to produce new exemplars. These tests are both fill-in-the-blank tests. The first test
consists of 20 sentences directly from the text that the participants read. Ten instances will test
their knowledge of the Spanish pluperfect subjective, and ten instances will be distractor items.
The second test will be similar, but it will consist of sentences that were not presented in the
original test so as to determine if the participants are able to create new exemplars of the Spanish
pluperfect subjective. As in the previous test, 10 items will test the Spanish pluperfect subjective
and 10 will be distractors.

Working Memory and Introversion/Extroversion Test

Working Memory Span Tests, as described in Conway, Kane, Bunting, Hambrick, Wilhelm, &
Engle (2005), will be used in order to determine the participants levels of WMC. Following Goo
(2010), two tasks will be used to test working memory capacity: a listening span task, as

developed by Mackey et at (2008), and a operation span task that is a variant of Turners and
Eagles, developed specifically for the said study by Goo (2010). The introversion/extroversion
dimension will be measured by means of the Myers-Briggs Type Indicator, developed in the
1940s and 1950s by Isabel Briggs-Myers.

Procedure
Participants will first take the pretest on the Spanish pluperfect subjective in class. Only
those who do not answer more than 2 out of 10 correctly will be asked to participate in the
primary data collection. Unlike Bowles and Leow (2005), the data for the primary data collection
will be collected on an individual basis, and not in a language laboratory. The reason why it will
be done this way is the following: if reactivity depends on the introversion/extroversion level of
the participants, then it is likely that this method of data collection would have an effect when
participants are seated in a room with the researcher. That is, having the participants complete
the tasks in this way might have a greater effect on a personality trait such as
introversion/extroversion, making it possible for the researcher to better study the differences
between participants.
The introversion/extroversion test and the working memory test will be carried out at the
beginning of the first session. Depending on the results, participants will be assigned to different
groups: think-aloud, low WMC; think-aloud, high WMC; think-aloud, introvert; think-aloud,
extrovert; non-think-aloud, low WMC; non-think-aloud, high WMC; non-think-aloud, introvert;
non-think-aloud, extrovert.
Participants assigned to the think-aloud groups will be given the same non-metalinguistic
instruction as that in Bowles and Leow (2005). The data for the control groups will still be

collected individually with the researcher, but they will not be asked to think aloud while reading
or completing the production task.
Once participants read the text they will then complete the comprehension test and the
written production tests (old and then new exemplars).

Scoring Procedure
The same procedures will be used as in Bowles and Leow (2005) for scoring the
comprehension test and the two written production tests: participants will receive 1 point for
each correct answer for a total of 10 possible points per test. Participants will receive a numerical
score for both the introversion/extroversion and working memory tests.

Analyses
To determine whether think-alouds are reactive (research question 1) a series of three
independent t-tests will compare the silent control groups performance on each of the three
dependent variables.
To determine if working memory (research question 2) mediates the effects of reactivity,
separate tests will be run for each of the three dependent variables (comprehension, production
of old exemplars, production of new exemplars). The possible mediating effects of working
memory will be investigated in two ways: (1) by dividing the participants into high and low
working memory groups 1and, (2) by entering working memory as a covariate. For (1),
participants working memory scores will first be ranked. The bottom one-third of the group will
be considered to have low working memory capacity and the top one-third high working

1 Unlike Goo, I think its better to eliminate participants with MID WMC. That way, comparisons
will be clearer.

memory capacity.
To determine if introversion/extroversion (research question 3) affects reactivity, similar
tests will be conducted as with the working memory scores. Separate tests will be conducted for
each of the three dependent variables, and the tests will be run in 2 ways: (1) by dividing the
participants into introversion and extroversion groups and, (2) by entering
introversion/extroversion as a covariate. For (1), participants introversion/extroversion scores
will first be ranked. Then, the bottom one-third of the introversion/extroversion class will be
considered introverted and the top one-third will be considered extroverted. Finally, 2 ANOVAS
will be conducted for each of the three dependent variables.

References
Bates, T. C., & Rock, A. (2004). Personality and information processing speed: Independent
influences on intelligent performance. Intelligence, 32, 3346.
Bowles, M. A. (2008). Task type and reactivity of verbal reports in SLA: A first look at a L2 task

other than reading. Studies in Second Language Acquisition, 30, 359-387.

Bowles, Melissa A. The Think-aloud Controversy in Second Language Research. New York:
Routledge, 2010. Print.
Bowles, M. A., & Leow, R. P. (2005). Reactivity and type of verbal report in SLA research
methodology: Expanding the scope of investigation. Studies in Second Language
Acquisition, 27, 415-440.
Baddeley, A. D., & Hitch, G. J. (1974). Working memory. In S. Dornic (Ed.), Recent advances
in learning and motivation Vol. VIII (pp. 47-89). New York: Academic Press.
Conway, A. R. A., Kane, M. J., Bunting, M. F., Hambrick, D. Z., Wilhelm, O., & Engle, R. W.
(2005). Working memory span tasks: A methodological review and users guide.
Psychonomic Bulletin & Review, 12, 769-786.
Conway, A. R. A., Jarrold, C., Kane, M. J., Miyake, A., & Towse, J. N. (Eds.). (2007).
Variation in working memory. Oxford, UK: Oxford University Press.
Horwitz, E. K. (2001). Language Anxiety and Achievment. Annual Review of Applied
Linguistics, 21, 112-126.
Ericsson, K.A. and Simon, H.A. (1993), Protocol Analysis; Verbal Reports As Data (revised
edition; original edition published 1984) (Cambridge, MA: Bradford Books/MIT Press).
Goo, J. (2010). Working memory and reactivity. Language Learning, 43, 245254.
Horwitz, E. K., Horwitz, M. B., & Cope, J. (1986). Foreign language classroom anxiety. Modern
Language Journal, 70 , 125-132.
Kormos, J. and A. Sfr (2008) Phonological short term-memory, working memory and
foreign language performance in intensive language learning. Bilingualism: Language
and Cognition 11.2: 26171.

Linck, J.A., Osthus, P., Koeth, J.T., & Bunting, M.F. (2013). Working memory and second
language comprehension and production: A meta-analysis Psychonomic Bulletin &
Review
Leow, R. P., & Morgan-Short, K. (2004). To think aloud or not to think aloud: The issue of
reactivity in SLA research methodology. Studies in Second Language Acquisition, 26,
3557.
Mackey, A. R. Adams, C. Stafford and P. Winke (2010) Exploring the relationship between
modified output and working memory capacity. Language Learning 60.3: 50133.
Morgan-Short, K., Heil, J., Botero-Moriarty, A., Ebert, S. (2012). Issues of Think-Alouds and
Depth of Processing. Studies in Second Language Acquision, 34, 659-685.
Revelle, W., Amaral, P., & Turriff, S. (1976). Introversion-extraversion, time stress, and
caffeine: effect on verbal performance. Science, 192, 149-150.
Rossomondo, A. E. (2007). The role of lexical temporal indicators and text interaction format in
the incidental acquisition of the Spanish future tense. Studies in Second Language
Acquisition, 29, 39-66.
Sachs, R., & Polio, C. (2007). Learners uses of two types of written feedback on a L2 writing
revision task. Studies in Second Language Acquisition, 29, 67100.
Sachs, R., & Suh, B. R. (2007). Textually enhanced recasts, learner awareness, and L2 outcomes
in synchronous computer-mediated interaction. In A. Mackey (Ed.), Conversational
interaction in second language acquisition: A collection of empirical studies (pp.197
227). Oxford: Oxford University Press.

Sanz, C., Lin, H.-J., Lado, B., Bowden, H. W., & Stafford, C. A. (2009). Concurrent
verbalizations, pedagogical conditions, and reactivity: Two CALL studies. Language
Learning, 59(1), 3371.
VanPatten, B. & J. Rothman (2014). Against rules. In A. Benati, C. Laval, & M. J.
Arche (Eds.), The grammar dimension in instructed second language acquisition:
theory, research, and practice (pp. 15-35). London: Bloomsbury.AILA Applied
linguistics series, 9 (9). John Benjamins, Amsterdam, pp. 243-256.
Yanguas, I., & Lado, B. (2012). Is Thinking Aloud Reactive When Writing in the Heritage
Language. Foreign Language Annals 45, 380-399.
Yoshida, M. (2008). Think-aloud protocols and type of reading task: The issue of reactivity in L2
reading research. In M. Bowles, R. Foote, S. Perpin, & R. Bhatt (Eds.), Selected
proceedings of the 2007 Second Language Research Forum (pp. 199-209). Somerville,
MA: Cascadilla Proceedings Project.