Audiovisual Incongruence in Multimedia: An Exploratory EEG Study

AUDIOVISUAL INCONGRUENCE IN MULTIMEDIA: AN EXPLORATORY EEG STUDY
BY
MICHAEL ZWICK
A Thesis
Submitted to the Department of Psychology, Neuroscience, and Behaviour
in Partial Fulfillment of the Requirements
for the Degree Honours Bachelor of Science
McMaster University
April 2013
HONOURS BACHELOR OF SCIENCE (2013)

MCMASTER UNIVERSITY
Hamilton, Ontario
TITLE: Audiovisual incongruence in multimedia: an exploratory EEG study

AUTHOR: Michael Zwick
SUPERVISORS: Dr. Joe Kim & Dr. Jen Heisz
NUMBER OF PAGES: 28
Abstract:
Despite the prominent use of multimedia in education, very little is known about the
cognitive processes that underlie learning from multimedia. In order to explore this topic, we
used electroencephalography (EEG) to measure subjects' brain wave activity as they viewed
a simulated online lecture consisting of images and narration, and then tested subjects on
their knowledge of the lecture material. Since alpha power has been shown to increase with
task difficulty (Cooper et al. 2002; Galin, Johnstone, Herron, 1978; Handel, Haarmeier,
Jensen, 2011; Jensen, Gelfand, Kounios, Lisman, 2002; Osaka, 1984; Tuladhar et al., 2007),
we examined how it would be altered by changes in lecture design and whether this affects
learning of lecture material. We designed the lecture so that one half was higher in cognitive
load than the other. For one half of the lecture, the images did not directly correspond to the
narration (incongruent half), whereas the other half did not contain this audiovisual
incongruence (congruent half). Subjects showed greater alpha power while watching the
incongruent half of the lecture than the congruent half. Subjects also performed worse on quiz
questions derived from material presented in the incongruent half. This suggests that
audiovisual incongruence imposes great demands on working memory and is detrimental to
learning. These results provide evidence that EEG allows insight into how students learn from
multimedia and highlight the value of using EEG in pedagogical research.
Acknowledgements:
I would like to thank everyone who supported me and helped me with completing my
thesis. In particular, I would like to thank my supervisors Dr. Joe Kim and Dr. Jen Heisz for all
of their valuable advice and guidance, as well as Barbara Fenesi for mentoring me throughout
the year. Special thanks also goes to Ali Hashemi for teaching me how run EEG, Katherine
Lajkosz for her help in conducting the experiment, and Dr. Patrick Bennett and Dr. Allison
Sekuler for use of their lab.
Table of Contents
Abstract
Acknowledgements
Introduction
6-11
Methods
12-15
Results
16
Discussion
17-21
Figures
Figure 1. Topoplot comparing differences in alpha power
22
Figure 2. Mean alpha power exhibited during each half of the lecture
23
Figure 3. Mean comprehension quiz scores by question type and origin
24
References
25-27
Appendix
28
Introduction:
Over the past couple decades there has been a clear influx of technology into
education. Especially in higher education, instructors are making increased use of multimedia,
such as PowerPoint presentations, interactive animations, and online lectures, as vehicles to
present course content to students (Apperson, Laws, Scepansky, 2008; Bartsch & Cobern,
2003). Multimedia instruction has been touted as an antidote for student disengagement and
as a way to enrich learning (Beeland, 2002; Jacques, 1995; Cutrim Schmid, 2008). These
claims are supported by research in cognitive psychology and the popularity of instructional
multimedia among students (Najjar, 1996; Shuell & Farber, 2001). Given this, it is likely that
multimedia will continue to have a prominent role in education in the years to come. In
response, the focus of pedagogical research has shifted towards understanding the effects of
incorporating multimedia into teaching practice and how to design instructional media to best
facilitate learning. However, there is comparatively less known in regards to the cognitive
processes that underlie multimedia learning, which was the focus of the present study. We
sought to determine whether a variation in multimedia presentation design results in the
evocation of different cognitive processes and how this affects learning of instructional
material.
When we take into account Baddeley's (1992) model of working memory, the
advantages of multimedia learning become apparent. At the core of this model is the
assumption that visual and auditory information are processed in independent channels within
working memory, referred to as the visuospatial sketchpad and phonological loop,
respectively. These channels are thought to have a finite capacity for the amount of
information they can hold or process at any given time, and while they are distinct, they
interact with one another. Multimedia presentations typically include visual components, such
as images or written text, as well as some sort of narration or auditory information, thereby
stimulating both of these channels. Therefore, learners must manipulate these different forms
of sensory stimuli within their working memory to create a cohesive whole and integrate this
with previous knowledge stored in long-term memory. Not only does this promote more
elaborate processing of instructional material, but it also allows for the distribution of cognitive
load among these two channels, so no one channel becomes overburdened.
However, not all educational multimedia is created equal. One way to judge the
effectiveness of a multimedia presentation is to examine the cognitive processes it evokes
and determine to what extent these processes tax limited working memory resources. In other
words, we can measure the cognitive load imposed by instructional media. Although cognitive
load is important to consider when evaluating the effectiveness of a multimedia presentation,
it is difficult to assess, as cognitive processes are hidden from observation. It is for this reason
that cognitive load is often assessed using subjective measures. Studies often make use of
self-reports to measure the difficulty or engagement of multimedia materials. However, this
approach is flawed as perceived understanding does not always accurately reflect actual
understanding (Benjamin & Bjork, 1996; Kornell & Bjork, 2008). Even when objective
measures are used, they typically provide only an indirect assessment of cognitive load. A
common research approach is to have subjects watch one of two multimedia presentations
that have the same content but differ in some aspect of their design, and then quiz subjects
on the content of the presentation (Bruenken, Plass, Leutner, 2003). The logic is that poorer
quiz performance should be an indicator of the amount of cognitive load induced by
instructional design, however learning outcomes can be influenced by quiz design and
individual differences in working memory capacity (Mayer, 2001). Behavioural measures also
have another significant disadvantage in that they cannot be used to gauge cognitive load
online during the actual learning process. Therefore, it is difficult to determine how a particular
manipulation in presentation design directly affects cognitive activity and learning, which limits
the types of hypotheses that can be tested. Supplementing behavioural data with a
physiological measure of cognitive load may remove some of these limitations and provide
greater flexibility in the types of questions that can be asked in pedagogical research. This
would allow for a deeper understanding of the processes involved in multimedia learning and
improved testing for the effectiveness of multimedia design.
One promising approach is to borrow physiological measures that are commonly used
in cognitive neuroscience research. By using neuroimaging techniques, such as positron
emission topography (PET), functional magnetic resonance imaging (fMRI), and EEG, we can
determine what exactly is going on in learners' brains as they view multimedia. Although this
may seem intuitive, neuroimaging is very rarely used in pedagogical research. To the author's
knowledge, EEG has been used in only two prior pedagogical investigations (Antonenko &
Niederhauser, 2010; Gerlio & Jauovec, 1999). We decided to implement EEG methodology
into the present study not only because of the advantages it offers, but also to provide further
evidence that EEG is indeed a viable tool in educational research.
EEG is a non-invasive technique used to measure electrical activity in the brain via
electrodes placed on the scalp and can provide a continuous and objective measure of
cognitive activity (Picton, 2012). An advantage of EEG is its high temporal resolution, which
allows for the tracking of changes in brain activity on the millisecond scale (Picton, 2012).
Researchers can then determine fluctuations in cognitive activity at specific points of time,
such as when a stimulus is presented. This allows for a more accurate understanding of the
direct effect that some manipulation in multimedia design has on learning and cognition. In
this way, the use of EEG can compensate for the shortcomings of behavioural data.
While EEG is rarely used to measure cognitive load in a multimedia learning
environment, it is frequently used to measure cognitive load during the execution of cognitive
tasks not pertinent to pedagogy. One common approach is to look at brain wave activity in the
alpha band (8-13 Hz), as prior investigations have found that it varies with task difficulty, but
this variation is not predictable and consistent. Therefore, there are several hypotheses
regarding the functional significance of alpha and how exactly it is affected by task demands.
There are reports of alpha activity decreasing as cognitive load increases, which suggests
that alpha is inversely related to mental effort (Butler & Glass, 1976; Gerlio & Jauovec,
1999). Confusingly, a substantial amount of studies have found just the opposite: that alpha
activity increases with task demands (Cooper et al. 2002; Galin et al., 1978; Handel et al.,
2011; Jensen et al., 2002; Osaka, 1984; Tuladhar et al., 2007). Researchers are divided in
their interpretation of these findings. For a number of years, the prevailing belief was that
alpha activity reflects cortical idling (see Pfurtscheller, Stancak, Neuper, 1996 for a review),
meaning that when a task is more difficult, subjects simply pay very little attention, and
cognitive activity is relatively diminished. In the past decade, however, support has increased
for the alpha inhibition hypothesis the idea that alpha activity is an attentional suppression
mechanism. There is a growing body of evidence demonstrating increased alpha activity in
cortical regions that are involved in the processing of task-irrelevant stimuli, and that this is
correlated with better task performance (see Foxe & Snyder, 2011 for a review; Handel et al.,
10
2011; Jensen et al., 2002; Tuladhar et al., 2007). All in all, there is no consensus on what
exactly is reflected by alpha oscillations or how alpha is involved in learning.
The present study was primarily exploratory, with the overall goal of simply gaining a
better understanding of the cognitive processes involved in learning from multimedia more
specifically, in an online lecture setting. We wanted to show that EEG methodology is useful
for evaluating the effects of multimedia design on cognitive processing and learning.
Furthermore, we sought to highlight how EEG can be used to measure brain activity in reallife settings, and not just in laboratory-type cognitive tasks. Lastly, we strove to further
elucidate how alpha activity varies in response to changes in multimedia design and cognitive
load.
In order to minimize between-subject variability, we used a within-subjects design. We
ran EEG on subjects as they watched a simulated online lecture that included images and
narration. In order to increase the applicability of our findings, the lecture we presented to
subjects was an adaptation of a real online lecture used in McMaster's introductory
psychology course. In order to create two different types of lecture environments, half of the
lecture was manipulated to have greater cognitive load than the other. We did this by
scrambling the order in which the images appeared, thereby causing incongruence in audio
and visual stimuli, as irrelevant images increase the cognitive load of instructional media
(Mayer, Heiser, Lonn, 2001). We were primarily concerned with subjects' alpha power, which
is simply a measure of how much of the EEG signal contains waves oscillating within the
alpha band of frequency. We compared subjects' alpha power from each half of the lecture,
and hypothesized that it would be significantly greater in the half in which the images and
audio were incongruent. This is because attention towards the images needs to be
11
suppressed in order for subjects to gain a clear understanding of the aurally presented lecture
material. This hypothesis is in line with the view that alpha represents cortical inhibition (see
Foxe & Synder, 2011 for a review; Handel et al. 2011; Jensen et al., 2002; Tuladhar et al.,
2007).
Not only were we interested in seeing how audiovisual incongruence changes
cognitive activity, but we also wanted to determine how it affected subjects' learning.
Following the lecture, subjects completed a quiz measuring their knowledge of the lecture
material. The quiz contained questions that measured basic retention, while others required
application of the lecture's general principles. We hypothesized that subjects would perform
more poorly on application questions compared to recall questions overall, as they require a
deeper level of processing, which makes them naturally more challenging. In addition, we
predicted that scores would be lower on questions dealing with material from the incongruent
half of the lecture, regardless of question type, as the required inhibition of images should
leave fewer cognitive resources for attending to the audio track, which was the mode in which
the question material was presented.
12
Methods:
Participants:
Forty-six Psych 1XX3 students at McMaster University volunteered to be part of
the study in return for course credit. Subjects were all between 18 and 25 years of age,
in order to prevent age-related differences from confounding the EEG data.
Approximately twice as many females as males participated in the study. All subjects
completed the quiz (mean age = 19.09, SD = 1.49), but EEG was run on just eighteen
subjects, and only eight EEGs yielded analyzable data. Five EEGs were from female
subjects, and the other three were from males (mean age = 19.25, SD = 1.49). All
participants provided informed consent, and all procedures were reviewed by and
received clearance from the McMaster Research Ethics Board.
Stimuli & Procedure:

Lecture:
Subjects watched a nine-minute condensed adaptation of an actual
online lecture from Psych 1XX3, the second course in the introductory
psychology sequence at McMaster. The topic of the lecture was the
physiological and evolutionary bases of hunger and satiety. The lecture
consisted of 18 slides: each slide contained a single image with no text.
Participants viewed these images while simultaneously listening to a narration
that delivered the actual lecture content. For one half of the lecture, the images
appeared in the proper order and were congruent with the audio track, meaning
that they visually depicted a key aspect of what was being discussed at that
13
point of the lecture. However, for the other half of the lecture, the images were
scrambled. This meant that instead of the images matching the audio track in
terms of content, they corresponded to content that was discussed earlier or
later in the lecture. We counterbalanced for which half of the lecture had the
audiovisual incongruence. Half of the participants watched a version of the
lecture where the first half was incongruent, while the other half of the subjects
watched a version where the second half was incongruent. Prior to viewing,
subjects were informed that they would have to complete a quiz based on the
content of the lecture and instructed to pay close attention to the lecture.
EEG Recording & Analysis:

Participants were fitted with a 256-channel Hydrocel Geodesic Sensor
Net from EGI and seated in a soundproof booth 100 cm away from a computer
monitor. EEG was recorded by Net Station version 4.3.1 while subjects watched
the lecture. The signal was filtered with a 200 Hz low-pass filter and a 0.1 Hz
high-pass filter. The sampling rate was 500 Hz. Epochs were 14 seconds in
length they begun 2 seconds prior and ended 12 seconds after the
presentation of a new image/slide. Welchs Method was used to reduce noise
and estimate power within the alpha band. We first determined subjects' alpha
power at each of these slide changes. Then for each subject, we calculated an
average of their alpha power associated with the presentation of slides 1-9 and
10-18. This gave us the mean alpha power exhibited by each subject during
each half of the lecture. We then calculated the mean alpha power in the
14
congruent and incongruent halves of the lecture across all 8 subjects. Through
use of a topoplot, we determined that it was in the parietal region where
subjects showed the greatest alpha power in the incongruent half of the lecture
relative to the congruent half (see Figure 1). This disparity was most prominent
around E140, so we narrowed our scope of analysis to this single electrode, for
sake of simplicity.
Comprehension Quiz:
Immediately after viewing the lecture, subjects had to complete a
multiple-choice quiz in order to measure their comprehension of the lecture
material. There were 20 questions in total, with each question having four
possible options. In order to mimic a more realistic university-level evaluation
and to see how different types of cognitive skills were affected by the
incongruence, two different styles of questions were used. Half of the questions
measured basic retention of presented facts (recall), while the other half
required application of general principles to solve novel problems (application).
The quiz contained equal focus on material from each half of the lecture.
Examples of quiz questions are located in the Appendix. All participants were
briefed after completing of the quiz.
Statistical Analysis:
In order to compare the overall mean alpha power across all subjects during
each half of the lecture, we conducted a paired t-test. A mixed model repeated-
15
measures analysis of variance (ANOVA) was used to analyze comprehension quiz

scores. There were two within-subjects factors, each with two levels. One factor was
style of question, comparing application and recall questions. The other was origin of
question content; in other words, whether the question was derived from material
presented in the congruent or incongruent half of the lecture. In order to determine
whether subjects for whom the first half of the lecture was incongruent differed in quiz
performance from those who received the incongruent stimuli in the second half, we
included the between-subjects factor of condition.
16
Results:
Figure 2 shows the mean alpha power exhibited by subjects during each half of the
lecture at electrode E140. Overall, subjects showed greater alpha power when viewing the
incongruent half of the lecture compared to when they viewed the congruent half. This was
supported by a significant paired t-test contrasting mean alpha power within each lecture
halve with a moderate effect size [t(7) = 2.46, p = .043, d = .41].
Figure 3 illustrates performance on the comprehension quiz. Subjects performed
significantly better on recall-style questions than those that required application of knowledge
[F(1,44) = 75.97, p < .001]. Questions derived from the incongruent half of the lecture proved
more challenging for subjects than those derived from the congruent half, as the ANOVA
found a significant effect of question content origin [F(1,44) = 6.18, p = .017]. We did not find
an effect of condition [F(1,44) = .655, n.s.], meaning that there was no difference in quiz
performance between subjects for whom the first half of the lecture was incongruent and
those for whom the second half was incongruent. There was no significant interaction
between any of the factors [all Fs < 2.5].
17
Discussion:
As predicted, audiovisual incongruence in a multimedia presentation is detrimental to
learning. Subjects performed worse on quiz questions derived from the half of the lecture that
contained incongruent audio and visual stimuli. Collapsing across style of question, subjects
scored, on average, almost 7% lower on questions from the incongruent half, compared to
those taken from the congruent half. This difference in performance is significant as it
amounts to a difference of two grade points on most university grading scales. A possible
interpretation of these results is that subjects expended a large amount of cognitive resources
trying to resolve the incongruence between audio and visual stimuli, leaving them with an
insufficient amount of processing power to devote to the encoding of the lecture material.
It is also worth noting that the ANOVA found no effect of condition on quiz scores. This
means that subjects who received the incongruent stimuli first did not perform differently than
those for whom the second half of the lecture was incongruent. Taken together with effect of
question origin, this suggests that the initial audiovisual incongruence did not influence
subjects to merely ignore or not attend to the entirety of the presentation. It is likely that
subjects showed renewed interest when the images began to correspond with the audio track,
as reflected by higher quiz scores.
This is line with previous research demonstrating that inclusion of irrelevant stimuli in
instructional materials is detrimental to learning (Mayer, 2001; Mayer et al., 2001; Moreno &
Mayer, 2000). Learners seem especially sensitive to audiovisual incongruence, however. To
create incongruence, we merely scrambled the order of images, which means that although
the image did not directly correspond to the audio track at that time, it did coincide with what
was narrated a few minutes before or after. Thus, the images still related to the audio track in
18
overall theme they all were related to some aspect of hunger, the topic of the lecture. This
shows that learning is harmed by even subtle disparities in audio and visual content in a
presentation, further stressing how important it is for instructors to select images very
carefully when preparing instructional materials.
We also found that subjects performed much worse on application questions than on
recall questions, regardless of which half of the lecture the questions were derived from. This
was expected because application is inherently more difficult than recall. Recall questions
measure basic retention and therefore, subjects needed only a surface knowledge of the
question content to correctly answer these types of questions. In contrast, application
questions evaluated subjects' ability to learn general principles and apply them to novel
contexts. In order to do well on questions of this style, subjects must have engaged in a
comparatively deeper level of processing and had a much more thorough understanding of
the material.
Not only did the quiz questions vary in their inherent difficulty, but so did the lecture
itself. Our key experimental manipulation was inducing cognitive load during one half of the
lecture by scrambling the order in which the images appeared. Our results indicate that this
manipulation was successful. Indirect support comes from subjects' lower quiz scores on
questions derived from this part of the lecture in combination with direct evidence from their
EEG data. Subjects showed significantly greater alpha power when the images were
incongruent with the accompanying narration. The disparity in alpha power between the two
halves of the lecture was most pronounced in the parietal region.
These results are in contrast to those from a prior multimedia study that used EEG
methodology (Gerlio & Jauovec, 1999). Gerlio & Jauovec (1999) found no difference in
19
alpha power at parietal electrodes when subjects were learning from a text-only presentation
or from multimedia, despite the difference in cognitive load between the two formats. The
disparity in results between the two studies is likely due to the use of different stimuli. The
parietal lobe is involved in the integration of audio and visual stimuli (Molholm et al., 2006),
and therefore, is more sensitive to the experimental manipulation used in our study.
Although the functional significance of alpha is highly debated in the literature, one
interpretation of our results lends support to the theory that alpha represents cortical
inhibition. When viewing educational multimedia, one must process both visual and auditory
stimuli and integrate them into a cohesive whole in order to gain a complete understanding of
the presented concepts. By presenting subjects with incongruent audiovisual stimuli, this not
only made it more difficult for subjects to integrate information from both sensory modalities,
but it made integration detrimental to learning. Because the images did not complement the
narration, the best learning would occur from ignoring the images and directing full attentional
capacity towards the audio track. In other words, during the incongruent half of the lecture,
the parietal lobe is not needed to carry out audiovisual integration. Thus, it is possible that the
increase of alpha power during this half of the lecture was a mechanism to inhibit audiovisual
integration in the parietal lobe, thereby providing subjects with more cognitive resources to
use towards auditory processing and making sense of the lecture content. This interpretation
is in line with previous findings that alpha activity increases in task-irrelevant regions (Handel
et al., 2011; Pfurtscheller & Klimesch, 1990). Following this logic, we would expect that alpha
activity would decrease in task-relevant regions, and therefore, subjects would show much
less alpha power in temporal regions than in parietal regions during the incongruent half of
the lecture. However, this was beyond our scope of analysis. Furthermore, we would expect
20
higher alpha power during the incongruent half of the lecture would lead to better
performance on questions pertaining to that portion of the lecture. However, due to our small
sample size, we could not compute a correlation between quiz scores and alpha power.
Therefore, we cannot state for certain that our results support the alpha inhibition hypothesis.
We cannot rule out the possibility that higher alpha power during the incongruent half of the
lecture actually reflects cortical idling. If subjects are not integrating what they see with what
they hear, the parietal lobe could merely be in a state of relative cognitive inactivity.
The overarching goal of this study was to explore a real-life application of EEG by
exploring how it can inform pedagogical research. Our results demonstrate that EEG can
indeed provide insight into the cognitive processes involved in multimedia learning.
Specifically, EEG shows promise as a reliable measure of cognitive load during authentic
learning situations, as we observed significantly higher alpha power in the incongruent half of
the lecture. As cognitive load is an important consideration in multimedia design, this
suggests that EEG may be a useful tool when interpreting the effectiveness of different
formats and designs of multimedia presentations. By complementing behavioural measures
with EEG data, we can obtain a greater breadth of understanding in regards to multimedia
learning.
Although our study highlights the value of using EEG in educational research, our
results should be interpreted with caution. Despite running EEG on many more subjects, only
eight EEGs yielded analyzable data, which limits the generalizability of our findings. In order
to get a more reliable measure of how alpha power changes in response to audiovisual
incongruence, this study should be repeated with a greater number of subjects. In addition,
we were only able to analyze alpha power at a single electrode in the parietal area. In order to
21
more accurately assess how parietal alpha power varies with cognitive load, follow-up studies
should calculate the mean alpha power across all electrodes in the parietal region. This
should also be done for all other brain regions, so that we can gain insight into the topography
of alpha activity, as this would be useful in elucidating alpha's functional significance in
multimedia learning, and its role in general cognitive performance.
Despite these limitations, our study achieved its overall purpose of exemplifying that
pedagogical research can benefit from the implementation of methodology from cognitive
neuroscience. EEG and other types of neuroimaging techniques can provide insight into the
neural correlates of multimedia learning and enable researchers to investigate a wider variety
of research questions. However, further work is needed to determine how exactly data
obtained using neuroscience techniques can be used to inform multimedia design and
instruction. While this study provides a glimpse into the cognitive processes that underlie
learning, there is much left to be discovered. By combining findings and perspectives from
cognitive neuroscience and pedagogy, we can gain a more thorough understanding of this
complex topic. This study provides great incentive of a merger between these two fields of
psychology and it will be exciting to see how they flourish and interact with each other in
years to come.
22
Figures:
Figure 1.Topoplot comparing differences in alpha power. Regions in red represent the
brain areas in which alpha power was much greater in the incongruent half of the lecture than
in the congruent half; the circled area represents the area where this difference was greatest.
This is the area corresponding to electrode E140. Regions in blue represent regions in which
the alpha power was much lower in the incongruent half of the lecture compared to the
congruent half, but this was outside our scope of analysis.
23
Figure 2. Mean alpha power exhibited during each half of the lecture ( SE).
24
Figure 3. Mean comprehension quiz scores by question type and origin ( SE).
25
References:
Apperson, J. M., Laws, E. L., & Scepansky, J. A. (2008). An assessment of student
preferences for PowerPoint presentation structure in undergraduate courses.
Computers & Education, 50(1), 148-153.
Baddeley, A. (1992). Working memory. Science, 255(5044), 556-559.
Bartsch, R.A., Cobern, K.M. (2003). Effectiveness of PowerPoint presentations in lectures.
Computers & Education, 41, 7786.
Beeland, W. D. (2002). Student engagement, visual learning and technology: Can
interactive whiteboards help. In Annual Conference of the Association of Information
Technology for Teaching Education.
Bruenken, R., Plass, J. L., & Leutner, D. (2003). Direct measurement of cognitive load in
multimedia learning. Educational Psychologist, 38(1), 53-61.
Butler, S. R., & Glass, A. (1976). EEG correlates of cerebral dominance. Advances in
Psychobiology, 3, 219.
Cooper, N. R., Croft, R. J., Dominey, S. J., Burgess, A. P., & Gruzelier, J. H. (2003). Paradox
lost? Exploring the role of alpha oscillations during externally vs. internally directed
attention and the implications for idling and inhibition hypotheses. International Journal
of Psychophysiology, 47(1), 65-74.
Cutrim Schmid, E. (2008). Potential pedagogical benefits and drawbacks of multimedia use in
the English language classroom equipped with interactive whiteboard technology.
Computers & Education, 51(4), 1553-1568.
26
Foxe, J. J., & Snyder, A. C. (2011). The role of alpha-band brain oscillations as a sensory
suppression mechanism during selective attention. Frontiers in Psychology, 2.
Galin, D., Johnstone, J., & Herron, J. (1978). Effects of task difficulty on EEG measures of
cerebral engagement. Neuropsychologia, 16(4), 461-472.
Gerlio, I., & Jauovec, N. (1999). Multimedia: Differences in cognitive processes observed
with EEG. Educational Technology Research and Development, 47(3), 5-14.
Jacques, R. (1995). Engagement as a Design Concept for Multimedia. Canadian Journal of
Educational Communication, 24(1), 49-59.
Jensen, O., Gelfand, J., Kounios, J., & Lisman, J. E. (2002). Oscillations in the alpha band (9
12 Hz) increase with memory load during retention in a short-term memory task.
Cerebral Cortex, 12(8), 877-882.
Kornell, N., Bjork, R.A. (2008). Learning concepts and categories: is spacing the enemy
of induction? Psychological Science, 19(6), 585592.
Mayer, R. E. (2001). Multimedia learning. New York: Cambridge University Press.
Mayer, R. E., Heiser, J., & Lonn, S. (2001). Cognitive constraints on multimedia learning:
When presenting more material results in less understanding. Journal of Educational
Psychology, 93(1), 187-198.
Molholm, S., Sehatpour, P., Mehta, A. D., Shpaner, M., Gomez-Ramirez, M., Ortigue, S., ... &
Foxe, J. J. (2006). Audio-visual multisensory integration in superior parietal lobule
revealed by human intracranial recordings. Journal of neurophysiology, 96(2), 721-729.
Moreno, R., & Mayer, R. E. (2000). A coherence effect in multimedia learning: The case for
minimizing irrelevant sounds in the design of multimedia instructional messages.
Journal of Educational Psychology, 92(1), 117-125.
27
Najjar, L. J. (1996). Multimedia information and learning. In Journal of Educational Multimedia

and Hypermedia.
Osaka, M. (1984). Peak alpha frequency of EEG during a mental task: Task difficulty and
hemispheric differences. Psychophysiology, 21(1), 101-105.
Pfurtscheller, G., & Klimesch, W. (1990). Topographical display and interpretation of eventrelated desynchronization during a visual-verbal task. Brain Topography, 3(1), 85-93.
Pfurtscheller, G., Stancak Jr, A., & Neuper, C. (1996). Event-related synchronization (ERS) in
the alpha bandan electrophysiological correlate of cortical idling: a review.
International Journal of Psychophysiology, 24(1), 39-46.
Picton, T. (2012). What is an EEG? Retrieved from http://research.baycrest.org/eeg.
Shuell, T. J., & Farber, S. L. (2001). Students' perceptions of technology use in college
courses. Journal of Educational Computing Research, 24(2), 119-138.
Tuladhar, A. M., Huurne, N. T., Schoffelen, J. M., Maris, E., Oostenveld, R., & Jensen, O.
(2007). Parieto-occipital sources account for the increase in alpha activity with working
memory load. Human brain mapping, 28(8), 785-792.
28
Appendix:
Sample application-style question:
20. Alex, Sam, and Amanda all eat the same amount of food for dinner. Alex eats a triple
cheeseburger with fries. Sam eats a large turkey sandwich with a plate of steamed
vegetables. Amanda eats a large plate of chicken thighs and wings. Who will feel full the
longest?
a.
b.
c.
d.
Alex
Amanda
Sam
They will all feel full for the same amount of time
Sample recall-style question:

10. Which nutrient signals the need to replenish ones food intake?
a.
b.
c.
d.
glucose
fructose
adipose tissue
glycogen

Audiovisual Incongruence in Multimedia: An Exploratory EEG Study

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Audiovisual Incongruence in Multimedia: An Exploratory EEG Study

Uploaded by

Copyright:

Available Formats

AUDIOVISUAL INCONGRUENCE IN MULTIMEDIA: AN EXPLORATORY EEG STUDY