2012 IEEE International Conference on Systems, Man, and Cybernetics

October 14-17, 2012, COEX, Seoul, Korea

Post-Saccadic Event Related Potential

toward a New Technique for Information Retrieval
Hideaki Touyama
Toyama Prefectural University, Japan
touyama@pu-toyama.ac.jp

Abstract- In this paper, toward a brain-computer interface for

information retrieval, we performed preliminary experiments on
post-saccadic event related potential (ERP) during web browsing.
Two

channel

electroencephalogram

was

analyzed

with

five

subjects during simple string retrieval tasks using web browser

with active eye movements, i.e., saccadic eye movements. After
the saccadic eye movements the ERP signals were observed in
target string retrievals, whereas not observed in standard string
retrievals.

The classification algorithm was applied and the

average decoding performance of single-shot post-saccadic ERP

could be 78.2% and 77.6% in recall and precision, respectively.
The eye-tracking was also studied and the eye gaze points were
confirmed to be on and around target strings when the ERP was
elicited. This result suggests that the post-saccadic ERP with eye
tracking leads to a possibility of the use in the brain-computer
interface for information retrieval.

Keywords; B rain-Computer Interface, Saccadic Eye Movement,

ElectroEncephawGram (EEG), Event Rehlted Potentml (ERP),
Information

Retrieval,

Web

Browsing,

NeuroMarketing,

Chlssification.

I.

with no active eye movements during retrieval tasks. However,

recent work has been reported the brain-computer interface
based on ERP with the eye movements according to the
moving target stimuli which the user wants to select.
We consider that the study on information retrieval with
active eye movements, that includes saccadic eye movements
[6], is important to realize a new application of brain-computer
interface. In the preliminary work, we studied a feasibility of
decoding single-shot ERP after the saccadic eye movements,
that is, post-saccadic ERP, in the information retrieval
paradigm [7]. However, in that study the eye-tracking data has
not been analyzed at all. In addition to those results of previous
work, in this paper, the data of eye-tracking will be analyzed
and shown in order to confirm the feasibility of post-saccadic
ERP.
This paper is organized as follows. Next section explains
experiments performed in this study. The section III describes
analyses to decode single-shot post-saccadic ERP as well as the
analyses of eye-tracking data. Results and summary will be
mentioned in section IV and V, respectively.

INTRODUCTION

The brain-computer interface [ 1] enables people to operate

computer or machine only by means of brain activities, which
has been more and more investigated today. It requires the
recording of functional brain signal, the signal processing such
as machine learning and classification, and the feedback loop to
the user.
Several types of brain activities have been proposed for the
brain-computer interface. Based on ElectroEncephaloGram
(EEG), the technique of extracting desired events to which
people pay attention has been developed and applied in recent
studies. The Event Related Potential (ERP) [2] is a good
measure to know the degree of attention to the desired events.
By using this brain signal, the novel speller application has
been developed [3].
By using ERP the new research direction of information
retrieval has been emerging. The author investigated the photo
pickup [4] by making use of P300 signals which is one of the
ERP observed at approximately 300 ms after the relevant and
rare stimulus. A.D. Gerson et al. proposed cortically-coupled
computer vision [5] and they could show the possibility of
extracting the large number of desired photo images based on
P300 signals by combining with the technology of image
processing. These works on information retrieval studied P300

978-1-4673-1714-6/12/$31.00 2012 IEEE

II.
A.

EXPERIMENTAL

Subjects

Five healthy males (sl-s5, aged from 22 to 40 years old)

with normal or corrected-to-normal vision participated in this
experiment. They were comfortably sitting on a chair facing a
web browser displayed by LCD.
B.

EEG Recording and Eye-Gaze Tracking

Two-channel EEG signal was recorded from Cz and pz in

the international 10/20 system [8]. The EEG signals were
amplified at a biosignal amplifier (Polymate II (AP2 16), TEAC
Corporation). The amplified EEG signals were band-pass
filtered between 0.16 and 30 Hz and sampled at 200 Hz.
In order to monitor the subject's eye blinks and movements
the Vertical and Horizontal ElectroOculoGram(EOG: VEOG
and HEOG, respectively) were simultaneously recorded.
Furthermore, an eye-tracking device (EMR-9, NAC Image
Technology Corporation) was applied to measure the subject's
eye gaze directions on the web browser during the string
retrieval tasks. The experimental system configuration is shown
in Figure 1.

2939

LCD
(27 inch)

Eye Tracking
Device
I-(EMR-9)
Figure 1.

C.

PC
(laptop)

I--

Subject
(51-55)

bio5ignal
Amplifier
I-(Polymate II)

Experimental System Configuration.

Tasks

Strings in Japanese language with two Chinese characters

were used in this information retrieval tasks. One string ('the
nuclear power plant') was defined as target string. Residual
ones (,work', 'special', 'Japan' etc.) were defined as standard
strings. These strings have been selected before this experiment.
In one experimental session 80 strings were displayed in an 8 x
10 matrix array. There were four matrix arrays and each of
arrays was presented one by one in each experimental session
(There were totally four experimental sessions). In one matrix
array 20 target strings were included in random order. An
example of the visual stimulation of matrix array is shown in
Figure 2.

a!!

.31:'

::II:

.!I[

#1.

'

.'

'T'

If

iN.

l1Iffl

JIl

!fiIl

!I[

:0:.

.;r;

:!i!

ttl

"F-li

i!*

tlii;

litH

i*.

1t1:J

mj]

)I;

Ul:Wi

iEll

!;

811)

tt

}.,;1;i

-.

*'

jj.

:t:::t

:!Il'

Pi

:Jli

AT

I.

an:

. ,I!

IBM

II

[{i

_'!:r

Figure 2.

.f)!

collect the standard samples after the saccadic eye movements

the simple eye movement task using four eye fixation points
was applied. There were about 40 samples of standard retrieval
for each subject. In this study, all the strings were visible in one
web page, and the browser scrolling operation had not been
imposed to the subjects.
III.
A.

Average EEG

To confirm the ERP occurrence in target tasks the saccadic

time-locked averaging was applied to collected EEG data. The
onset of saccadic eye movement was extracted from HEOG. If
the HEOG just after the saccadic eye movement accompanied
by the button press exceeded the preset threshold the subjects
were considered to discover the target strings. And thus the
EEG signals were labeled as target samples. As for standard
samples, the similar averaging was performed to confirm no
occurrence of ERP. Furthermore, the timing of the discovery
of the target strings had been analyzed by using eye-tracking
data, which will be discussed later.
Figure 3 shows the previous result of the average EEG
waveforms at pz on which the ERP signals are expected to
appear [7]. The corresponding EOGs and EMG, which are
appropriately normalized, are also shown in order to confirm
the task achievement.

3:

-10

@
w

EOO & EMG

EOO

An example of the visual stimulation.

--

5 -L
m
0

--

The task of the subjects was to retrieve the 20 target strings

('nuclear power plant') by moving their eye gaze directions
actively, i.e., by performing saccadic eye movements. The
subjects were instructed to select the target strings (p=.25) by
pressing a button with their dominant hand when they
discovered them just after the saccade. They moved their eyes
from left to right direction and immediately after the target
discovery, the subjects kept their eyes for about 1 sec at the
fixation point. One session consisted of about 120 sec and four
sessions were performed. For each subject totally 80 target
samples were collected. To monitor whether the button press
was done when the subject discovered the target string, we
measured ElectroMyoGram (EMG) from subject's dominant
hand. Furthermore, the subjects were also instructed to ignore
completely the standard strings (p=.75). Here, in order to

AN AL YSES

Figure 3.

B.

--

-L

--

--

(target)

EOO (standard)
EMG (target)

==

--

time [sec]
Average EEG, EOGs and EMG [7].

EEG Classification

Toward the future application such as brain-computer

interface the decoding performance of single-shot post-saccadic
ERP was studied for five subjects.
At the electrode Cz and Pz, 160 time points of EEG data for
0.4 sec after the saccadic eye movement were taken into
account as one EEG epoch. After the preprocessing (baseline
correction, band-pass filtering between 1 to 8 Hz) Principal
Component Analysis (PCA) was applied for the significant

2940

feature extraction. Finally the PCA reduced the number of

feature components into 15 at most.
In order to estimate the decoding performance whether the
classifier can correctly detect single-shot post-saccadic ERP or
not, a leave-one-out cross validation was applied where only
one EEG epoch was used for testing and all the others
contributed to training or machine learning. The Linear
Discriminant Analysis (LDA) was used as a classifier in order
to classify two brain states during target and standard string
retrievals. The result of the decoding performance is shown in
TABLE I.
TABLE l.

DECODING PERFORMANCE
classification
standard

target
__
sI

s2

s3

s4

s5

C.

task

task

task

task

task

target
standard
target

precision

10

0.84

0.85

31

0.78

0.76

12

0.82

0.88

36

0.84

0.75

58

0.89

0.92

target
standard

recall

..

7
5

38

0.88

0.84

target

41

21

0.66

0.73

standard

15

28

0.65

0.57

target

44

20

0.69

0.83

34

0.77

0.63

standard

standard

Eye-tracking Data Analyses

In this subsection the analyses on eye-tracking data will be

shown. To confirm the feasibility of future applications the
points of cursors (the eye gaze points) when the post-saccadic
ERP occurred will be analyzed.
The results of the extracted camera shots from recorded
movie of eye-tracking device are shown in Figure 4. The
square points denote the eye gaze points at the time when the
EEG was classified as 'saccadic ERP' or 'target' class and the
thick red lines in the shot images denote the corresponding
target strings ('nuclear power plant' in Japanese language).

Figure 4.

2941

Examples of camera shots by eye-tracking device.

IV.

RESULTS AND DISCUSSIONS

As shown in Figure 3 the obtained ERP had rather broad

signals than usual ones, which is considered to be caused by the
jitter of the occurrence of the ERP, that is, the interval between
the saccadic-time-Iocking and the target discovery. However,
the positive EEG component was obtained only after the target
string discovery, while not observed after the standard string
discoveries. The ERP amplitude could be about 10 micro volts
at Pz. And the corresponding latency was between about 0.2
and 0.8 seconds after the EOG onset. Although being broad,
the EMG signal was observed after the EOG onset. This means
the subjects performed the task correctly.
TABLE I shows the decoding results of single-shot post
saccadic ERP. The average decoding performance values were
found to be rather high, that is 78.2% and 77.6% in recall and
precision, respectively. The performances might tend to depend
on the waveforms of the ERP. Clear ERP waveforms for
subject s l-s3 resulted in the high decoding performances.
Figure 4 describes the points of cursors (the eye gaze
points) at the time when the post-saccadic ERP occurred. In
this figure the eye-tracking was to some extent precise. We
confirmed that the cursor could be on and around the target
string. In our future work the image processing technique will
be applied in order to specify in online the desired strings,
images and possible stimulations.
In the future application such as brain-computer interface or
neuromarketing the post-saccadic ERP application will be
available in online by combining the EEG and eye-tracking
data, which enables the user to make indexing or labelling to
the desired information and events in the user's environments.
Note that the group of the author investigated the detection of
ambulatory P300 by using auditory oddball stimuli in outdoor
environments [9]. The current study suggests the decoding
performance of single-shot ambulatory P300 can be about 70%
in outdoor environment. In the future work the post-saccadic
ERP would be used in outdoor environments.
Our decoding algorithm in this study is very simple based
on PCA and LDA, and thus the optimized signal processing
method will result in the improved values of the decoding
performance. And we used only two electrodes in this study.
More number of electrodes with modular electrode cap would
contribute to the improvement.
Finally, in this study the subjects were instructed to keep
their eye gaze points for about 1 sec after the discovery of
target and standard strings. This was required to avoid the
artifacts from eye movements. However, this kind of task will
be a limitation in the future application such as web browsing
and neuromarketing. To improve this point the removal of
artifacts from free eye movements will be studied in the future
work based on Independent Component Analysis (ICA).
Furthermore, in this preliminary study the target retrieval tasks
was performed only by moving eyes from left to right direction
and the opposite direction was not investigated. On the other

hand, the opposite direction was taken into account only for
standard retrieval task. This point will be investigated in online
application of web browsing with the algorithm of complete
removal of artifacts from eye movements.
V.

SUMMARY

In this paper, toward a brain-computer interface for

information retrieval, we performed preliminary experiments
on post-saccadic ERP during web browsing. Two channel EEG
was analyzed with five subjects during simple string retrieval
tasks using web browser with saccadic eye movements. After
the saccadic eye movements the ERP signals were observed in
target string retrievals, whereas not observed in standard string
retrievals. The classification algorithm was applied and the
average decoding performance of single-shot post-saccadic
ERP could be 78.2% and 77.6% in recall and precision,
respectively. By applying the eye-tracking the eye gaze points
were confirmed to be on and around target strings when the
ERP was elicited. This result suggests that the post-saccadic
ERP with eye-tracking leads to a possibility of the use in the
brain-computer interface for information retrieval.
ACKNOWLEDGMENT

The author thanks to Shohei Yoshihara and Takahide Saito

for useful discussions on the experiments and analyses.
REFERENCES
[I]

Wolpaw JR, Birbaumer N, McFarland OJ, Pfurtscheller G, Vaughan TM.

Brain-computer interfaces for communication and control. Clin
Neurophysiol. 2002 Jun;113(6):767-91.

[2]

Steven J. Luck: An Introduction to the Event-Related Potential

Technique. Cambridge, Mass.: The MIT Press, 2005. ISBN 0-26262196-7.

[3]

L. A. Farwell and E. Donchin, "Talking off the top of your head: A

mental
prosthesis
utilizing
event-related
brain
potentials,"
Electroencephalogr. Clin. Neurophysiol., vol. 70, pp. 510-523, 1988.

[4]

H. Touyama and M. Hirose. EEG-based photo pickup. Proceedings of

18th International Conference on Artificial Reality and Telexsistance
2008 (ICAT2008), pp. 277-280.

[5]

A.D. Gerson, L.c. Parra and P. Sajda (2006) Cortically-coupled

computer vision for rapid image search, Neural Systems and
RehabiLitation Engineering, IEEE Transactions on. 14(2) 174-179.

[6]

Cassin, B. and Solomon, S. Dictionary of Eye Terminology. Gainsville,

Florida: Triad Publishing Company, 1990.

[7]

T. Saito and H. Touyama. Detection of Event Related Potential after

Saccade. Transactions of Virtual Reality Society of Japan, Vol. 17 , No.1,
pp.55-56, 2012. (in Japanese)

[8]

H. H. Jasper. The ten-twenty electrode system of the International

Federation. Electroencephalography and Clinical Neurophysiology
(\958) Volume: 10, Issue: 2, Pages: 371-375.

[9]

H. Touyama, K. Maeda: EEG Measurements towards Brain Life-Log

System in Outdoor Environment. HCI International 2011 - Posters'
Extended Abstracts, Communications in Computer and Information
Science, 2011, Volume 174, Part V, 308-311.

2942