Adaptive Brain Interfaces for Physically-Disabled People

J. del R. Millána, J. Mouriñoa, M.G. Marcianib, F. Babilonib, F. Topanic, I. Canalec, J. Heikkonend, K. Kaskid
a
Joint Research Centre of the EC, 21020 Ispra (VA), Italy. E-mail: jose.millan@jrc.it
b
Ospedale di Riabilitazione S. Lucia, Via Ardeatina 306, 00179 Roma, Italy
c
Fase Sistemi Srl, Via Ildebrando Vivanti 12, 00144 Roma, Italy
d
Helsinki University of Technology, Miestentie 3, 9400 Espoo, Finland

Abstract–This paper presents first results of an Adaptive
Brain Interface suitable for deployment outside controlled
laboratory settings. It robustly recognizes three purely mental
states from on-line spontaneous EEG signals and has them
associated to simple commands. Three commands allow to
interact intelligently with a computer-based system through
task decomposition. Our approach seeks to develop indi-
vidual interfaces since not two people are the same either
physiologically or psychologically. Thus the interface adapts
to its owner as its neural classifier learns user-specific filters.
I. INTRODUCTION
Physiological studies indicate that EEG signals are a reliable
mirror of mental activity. In addition, the combination of
EEG, MRI and PET are providing gradually better maps of
brain functions (i.e., which cortical areas are responsible for
specific mental activities). Thus, it is quite appealing to try to
use EEG signals as an alternative means of interaction with
computers. This paper describes a recent European research
effort whose objective is to build Adaptive Brain Interfaces
(ABI) suitable for deployment outside controlled laboratory
settings. The immediate application is to extend the
capabilities of physically-disabled people (e.g., select items
from a computer screen or guide a motorized wheelchair).
We aim to recognize from three to five mental states (e.g.,
relaxation, visualization, music composition, arithmetic,
verbal) from on-line spontaneous EEG signals1 by means of
artificial neural networks and to associate them with simple
commands such as “move wheelchair straight”, “stop” and so
on. Thus, users will be able to operate computer-based
systems by composing sequences of these patterns.
An ABI requires users to be conscious of their thoughts and
to concentrate sufficiently on those few mental tasks
associated to the commands. Any other EEG pattern different
from those corresponding to these mental tasks will be
associated with the command “nothing”, which has no effect
on the computer-based system.
The current ABI prototype is built upon the following
experience of the different partners in the whole spectrum of
areas covering the multidisciplinary nature of the project:
1 controlled laboratory conditions and learn a classification
We will also refer to them as “EEG patterns”, or just “patterns.”
• expertise in the neurological basis of EEG signals [7] for
optimal positioning of electrodes and feature extraction,
• development of Laplacian operators for deblurring and
spatial enhancement [2, 3],
• studies with off-line EEG signals concerning feature
extraction and classification [12],
• development of neural classifiers for the robust recognition
of mental states from on-line EEG signals [10], and
• design of a portable and easy-to-use EEG system as well as
other biomedical devices for rehabilitation [4].
An obstacle to the achievement of the ABI project is the
robust recognition of EEG patterns outside laboratory
settings. This presumes the existence of an appropriate EEG
equipment that should be compact, easy-to-use, and suitable
for deployment in real-world environments. No commercial
product fulfilling these requirements exists. We have set up a
first prototype for the acquisition of high-quality EEG
signals. We are also able to robustly recognize three purely
mental states from on-line spontaneous EEG signals.
II. RELATED WORK
In the last years several other research groups have begun to
develop EEG-based brain interfaces (BI). Several companies
are also commercializing basic mind-controlled devices. By
basic devices we mean that they can only recognize two
patterns or use muscular activity.
Two groups are developing BIs based on the recognition of
mental states associated to motor activities. McFarland and
Wolpaw’s approach relies completely on user training (users
must control their mu rhythm on each brain hemisphere) and
looks for a fixed EEG pattern that should be present in a
large majority of individuals (e.g., [8]). The ABI project
adopts an opposite approach: rather than putting all the
training requirements on the user, who has to learn to
generate a fixed EEG pattern, it makes the brain interface
adapt to the user. This approach is partially followed by
Pfurtscheller's group who seeks to recognize the motor
readiness potential generated while people are planning
movements. They are using artificial neural networks with the
aim of developing universal BIs (e.g., [5]). That is, they
gather EEG signals from a given number of users in well-
function that should be valid for everybody. They have
obtained good results with a few healthy subjects, but there is
no definite evidence that motor readiness also happen in
motor-impaired people. Instead of using brain activities
related to motor functions, the ABI project focuses on the
recognition of purely mental tasks outside controlled
laboratory settings. Anderson’s group is also using artificial
neural networks to build universal BIs (e.g., [1]). They are
using pre-recorded EEG signals and try to derive invariant
information from purely mental tasks. Their results, however,
are not so promising.
This universal BI approach suffers, in our opinion, from a
major limitation. We cannot expect a neural classifier built
with EEG data from a few persons to generalize across
individuals since not two people are the same, both
physiologically and psychologically. Our approach seeks to Fig. 1. High-quality EEG mapping after the computation of the
develop individual BIs rather than universal ones valid for Laplacian derivation for deblurring.
everybody. This means that the interface adapts to its owner
as the artificial neural network learns user-specific filters that Fig. 2 shows the ABI current prototype at work. In this
classify the incoming EEG signals into different categories. picture, the user holds the cap with integrated electrodes
Furthermore, since users can choose their more natural located according to the International 10-20 system. These
strategy to undertake a given mental task, they can regularly electrodes are directly plugged into amplifiers before sending
generate those individuals EEG patterns that are better the signals to the dedicated hardware (left). In the computer
distinguished by their personal interface. screen one can see two of the bipolar EEG signals being
processed (left windows), their corresponding power spectra
III. APPROACH AND RESULTS (top right window), and a circle (bottom right) indicating that
one of the mental tasks has been recognized.
One of our concerns is the acquisition of high-quality EEG
signals by means of robust and easy-to-use equipment,
suitable for deployment outside controlled laboratory
environments. To this end, we have built a first prototype
(see Fig. 2). The EEG system is made of a standard PC
running LabVIEW and C++, a commercial signal acquisition
board, a cap with integrated electrodes, and a dedicated
hardware for the acquisition of EEG signals. This hardware is
a stand-alone, fully isolated, portable system that gathers
analog brain-wave voltages from up to eight scalp electrodes,
amplifies and filters them, converts to digital values, and
transmits them via the acquisition board to the PC for
analysis. This prototype is very easy to operate (healthy users
can run it without external assistance), what greatly reduces
the preparation time for the acquisition of good signals.
A major step in the direction of improving the quality of the
recorded EEG signals is the measurement of reference-free Fig. 2. ABI current prototype at work.
potentials instead of conventional reference-dependent ones.
The latter are not only blurred by the skull, but show also EEG potentials are measured on the 8 channels F3, F4, C3, C4,
different spectral properties depending on the reference P3, P4, O1, and O2, with a reference electrode located in
electrode used for recording. To avoid these undesired between FZ, Fp1, and Fp2. Ground is applied to one of the ear
effects, we have developed Laplacian derivation methods to lobes. The sampling rate is 128 Hz and data is preprocessed
transform the recorded EEG signals into reference-free in temporal windows of half a second. This preprocessing
potentials [2, 3] (see Fig. 1 for an example). McFarland and consists of a Hanning windowing, a Butterworth bandpass
co-workers have recently found this computational method to filtering (4-30 Hz), on-line removal of temporal windows
increase the performance of their brain interface [8]. corrupted by ocular artifacts, and computation of either the
Interestingly, Pfurtscheller’s group is currently using the energy of the 5 differential channels (F3C3, C3P3, F4C4, C4P4,
Laplacian derivation too. O1O2) or the coherence between 10 pairs of channels (6 intra-
and 4 inter-hemispheres). The energy and/or coherence
features are fed to the neural classifier.
The experimental protocol is as follows. The subject is seated
and spontaneously concentrates on a mental task. The mental
tasks used in this study are “relaxation”, “cube rotation”,
“subtraction”, and “nothing” (whatever task other than the
previous ones). Relaxation is done with closed eyes, and all
other tasks with opened eyes. The subject performs the
selected task during 10 to 15 seconds, and he/she chooses
when to stop doing it and the next to be undertaken. Each
recording session lasts no less than 3 minutes. We have
recorded 4 sessions for each of two users. For the training
and testing of the neural classifier, the subject informs an
operator of the task he/she will perform; then 1 second before
and 1 second after are removed from the recording.
Recognizing mental states from on-line spontaneous EEG
signals is a complex task where we cannot expect to reach
recognition rates near to 100%. But a practical ABI doesn’t
require such a high performance; it should be enough a
recognition rate in between 70% and 80% provided that it has
an insignificant proportion (less than 2%) of false positives.
This is what we mean by robust recognition. In other words,
the neural classifier (almost) never takes a relevant pattern
for another (what would make the wheelchair move in the
wrong direction), but doesn’t eventually recognize EEG
patterns corresponding to the desired mental tasks (which
would be associated with the command “nothing”).
To illustrate the hard task faced by the neural classifier, Fig.
3 shows the PCA projection on the two first eigen directions
of the energy features of the five bipolar channels recording
EEG signals while a user carried out five mental tasks
according to the experimental protocol above (see [10] for
details). Every sampled EEG pattern is indicated with the
number of its associated mental task (from 0 to 4). The figure
shows a high degree of overlap among the classes. Thus
compact networks, such as classical multi-layer perceptrons,
will fail since they cannot compute different outputs for very
similar inputs. Their performance will not improve even if
the data is pre-processed with self-organizing maps [6] since
every unit of the map will codify EEG patterns of different
categories. We have confirmed experimentally this suspicion
[10]. On the other hand, one could use a single local network,
such as RBF (e.g., [11]) or LVQ [6]. Our results, however,
show that local networks achieve good results during training
but generalize poorly. For example, Table 1 reports the
results2 we have obtained with Platt’s RAN algorithm [11]
for the classification of the task “cube rotation” using one of
the recorded sessions for training and the remaining three for
testing. Similar results are obtained for the task “subtraction”,
whereas the task “relaxation” is better classified.
2
The results of this section are referred to the personal brain
interface of one of the users. Similar levels of performance are got
for the second user.
Table 1. Performance of the RAN algorithm for the mental task
“cube rotation”.
Good False Positive N Units
Training Set 92.0 % 2.4 % 444
Testing Set 68.3 % 12.4 %
Fig. 3. PCA projection of the power spectral energy features on the
two first eigen directions.
The alternative we adopt is a hierarchical committee of
networks. At the top level, there is a committee where each
network tries to classify a given task against all the others.
Then, for each task, there exists a network per EEG channel.
The output of any of the classifiers should be 1.0 if the input
pattern belongs to the mental task it has to recognize and 0.0
otherwise. Each of these networks is built incrementally3
using a variation of the RAN algorithm (see [10] for details).
Two important features of this kind of neural classifier are: i)
units are only added if an EEG pattern corresponding to the
mental task to be recognized is incorrectly classified, and ii)
their centers are moved so as to find clusters of those EEG
patterns. After training, it comes out that some of the units
have learned quite robust user-specific filters whereas some
other units are tuned to EEG patterns that are too similar to
patterns of different mental tasks. Our approach is, then, to
label as unknown the output of the classifier if one of the
latter units is the closest to the observed EEG pattern. In this
way, the neural classifier doesn’t make risky decisions for
uncertain EEG patterns, which are thus associated to the
command “nothing”. Furthermore, users can take this “no
answer” of the brain interface as a “warning” that they should
either concentrate more intensively on the desired mental task
or choose another strategy to undertake it. Indeed, initial
observations seem to indicate that, with practice, users learn
to generate those individual EEG patterns that are better
distinguished by their personal brain interface. But more
extensive testing of the approach is needed before confirming
this hypothesis.
3
That is, the number of units is not fixed. Rather, units are created
dynamically as they are required to better cover the input space.
Fig. 4 shows two user-specific filters discovered from the item above or obstacle avoidance). The implementation of
differential channel F4C4 that classify quite robustly the the elementary commands associated to the EEG patterns of
mental task “cube rotation”. Table 2 reports the performance interest will depend on the application. For example, we can
of our approach for the classification of the task “cube use these three patterns to guide a motorized wheelchair. The
rotation” using one of the recorded sessions for training and first pattern makes the wheelchair either start moving forward
testing over the three other sessions. Similar results are (if it is stopped) or stop (if it is moving). The remaining two
obtained for the task “subtraction”, whereas the task other patterns are used to make the wheelchair turn right or
“relaxation” is considerably better classified. It is worth left, respectively. These elementary commands (i.e., move
noting that the channel O1O2 is irrelevant for the forward, turn right, and turn left) are sent to a second learning
classification of the three mental tasks of interest. system (e.g., [9]) that uses the on-board sensors to bring the
wheelchair in the desired direction in a safe (avoiding
collisions) and smooth way .
These preliminary results have been obtained with a couple
of users. Before going on to recognize a larger set of patterns,
we are now experimenting the ABI with more users and
trying to improve its robustness. In this latter respect we are
exploring alternative feature extraction methods (e.g.,
Laplacian operators, autoregressive models, wavelets, etc.).
This on-going work is partially built upon previous studies
with off-line EEG signals. One of them confirms that an
artificial neural network distinguishes EEG patterns better if
it uses the temporal dynamics of brain activity [12]. This is
not surprising since EEG signals carry temporal information.
We deal with the time dimension (or history) by means of a
novel recurrent self-organizing map.
Fig. 4. Two user-specific filters for the robust classification of the
mental task “cube rotation” using energy features. V. REFERENCES
[1] C.W. Anderson and Z. Sijercic. Classification of EEG signals from
Table 2. Performance of the hierarchical committee of networks for four subjects during five mental tasks. Intl. Conf. on Engineering
the mental task “cube rotation”. Applications of Neural Networks, 407–414, 1996.
[2] F. Babiloni et al. Performances of surface Laplacian estimators: A
Good Unknown False Pos N Units study on simulated and real scalp potential distributions. Brain
Network Train Set 39.8 % 55.7 % 0.0 % 24 Topography, 8: 35–45, 1995.
[3] F. Babiloni et al. Improved realistic Laplacian estimate of highly-
F4C4 Test Set 28.3 % 62.1 % 0.4 % sampled EEG potentials with regularization techniques.
Electroencephalography and Clinical Neurophysiology, in press.
Network Train Set 25.3 % 65.1 % 0.0 % 12 [4] M Bernardi, I. Canale, et al. Ergonomy of paraplegic patients working
C4P4 Test Set 26.5 % 66.3 % 0.0 % with a reciprocating gait orthosis. Paraplegia, 33: 458-463, 1995.
[5] J. Kalcher et al. Graz brain-computer interface II. Medical &
Network Train Set 36.1 % 50.4 % 0.0 % 17 Biological Engineering & Computing , 36: 382–388, 1996.
[6] T. Kohonen. Self-Organizing Maps. Berlin: Springer-Verlag, 1995.
F3C3 Test Set 27.3 % 63.6 % 0.0 %
[7] M.G. Marciani et al. Quantitative EEG evaluation in normal elderly
Network Train Set 41.6 % 49.4 % 0.0 % 13 subjects during mental processes. Intl. Journal of Neurosciences, 76:
131–140, 1994.
C3P3 Test Set 46.2 % 40.9 % 0.3 % [8] D.J. McFarland et al. Spatial filter selection for EEG-based
communication. Electroencephalography and Clinical
Top Train Set 86.1 % 10.0 % 0.2 % 23
Neurophysiology, 103: 386–394, 1997.
Network Test Set 75.1 % 14.3 % 1.6 % [9] J. del R. Millán. Rapid, safe, and incremental learning of navigation
strategies. IEEE Trans. on SMC–Part B, 26: 408–420, 1996.
[10] J. del R. Millán, J. Mouriño, et al. Incremental networks for the robust
IV. DISCUSSION recognition of mental states from EEG. Technical Report. Joint
Three elementary commands is the minimum set to interact Research Centre of the EC, Ispra, Italy, 1998.
[11] J. Platt. A resource allocating network for function interpolation.
with a computer-based system in an intelligent way provided Neural Computation, 3: 213–225, 1991.
that we adopt the principle of task decomposition, namely the [12] M. Varsta, J. del R. Millán, and J. Heikkonen. A recurrent self-
user addresses the task at high level and all the low level organizing map for temporal sequence processing. 7th Intl. Conf. on
details are handled separately. In other words, the user just Artificial Neural Networks, 421–426, 1997.
tells the ABI to execute an elementary action (e.g., move the
pointer up or the wheelchair forward), but does not worry
about its implementation (e.g., how far so as to reach the next