BCICONTROLLED TONE MATRIX

By Laura Ileana Stoilescu, Tristan Didier

ABSTRACT
Usually, the devices needed for detecting neural signals are expensive, difficult to work with and fragile just to name a few. With the Emotiv EPOC headset we dont have to worry about these aspects at the price of lower quality signals. Still, the electroencephalography (EEG) data recorded with this headset makes possible a large range of applications. We demonstrate a brain-controlled tune matrix which works on similar principles to the P300 speller: the interface flashes the rows associated to one column in the matrix and a P300 brain potential is elicited when the flashed row matches the position on which we want to set the tone. Each position represents a different sound frequency mapped increasing from bottom up.

Algorithm, Design Implementation, Measurement, Performance

KEYWORDS
EEG, P300, Neural Signal Processing, Brain -Computer Interface, Emotiv, Event Related Potentials

INTRODUCTION
In the present paper we present the concept, design, implementation and evaluation of a wave synthesizer inspired by Andre Michelles Tone Matrix [1] which is controlled through a braincomputer interface (BCI).The application itself is represented by a graphical interface (GI) through which the user sets the tonality of each sound. In Andre Michelles version, the interface is represented by a 16- by - 16 interface whose axes are time (horizontal direction) and frequency (vertical direction) respectively. We followed the same idea with an 8-by-8 matrix in which the positions are set by using a P300 detection algorithm on the electrical signals acquired from an Emotiv EPOC EEG headset. We used two methods of detection: linear discriminant analysis (LDA) applied over data transformed using the principal component analysis (PCA) and multi-class support vector machine implemented (MC- SVM) in libsvm 3.0.1.

EEG signals are transmitted wirelessly from the headset to the processing module in Matlab which selects, adjusts and classifies the data received in order to discriminate the P300 signal. When a certain position triggers a P300, the matrix marks the position as set and generates the associated tone. We discuss the challenges in making our initial application more practical and We demonstrate a brain-controlled input method reliable. which works on the same principles to a P300 speller: for each time slot the rows are flashed and a P300 brain potential is elicited when a flashed row

GENERAL TERMS

matches the position the users wants to set. After For each column: setting a position, the user has the possibility to add - The target square is colored in another tone to that time slot by choosing another red. position. The column is changed when the interface has received the classifiers decision so just as we - Delay : 3 seconds have the pointed position, we switch automatically to the next column. - The target is colored with its initial color.

BRAIN-COMPUTER INTERFACE
For the actual realization of the BCI there are many challenges. Since the Emotiv headset is not a research-grade EEG headset and, as a result, there is a significant amount of noise in the recorded brain waves which require more complex signal processing and machine learning techniques to classify neural events such as the P300. However, the headset provides an encrypted wireless interface between the headset and the computer.
TONE MATRIX

Delay: 1 second The blinking is launched. During a blinking session, each line is flashed three times, in a random order. Each line-blinking is composed of an on-slot of 100 ms and of an off-slot of 75 ms. During the on-slot, the line is covered with a white rectangle. During the off-slot, the white rectangle is removed.

DESIGN

The Interface has been programmed - When the blinking session is using C++. The SDL library [2] has been finished, the next column is used to display the matrix, and the fmod automatically selected and the library [3] has been used to play the same operations are repeated. sounds. To communicate with the other This is done until all the columns parts of the system, we used virtual have been swept. serial ports [4]. An additional library has been used for these serial The programs send different markers through the serial port linked to Emotiv: communications [5]. y Start interface The offline protocol: The aim of the z All columns have been swept offline part was to log data in order to a New column compare the effectiveness of different b End of column classifiers. To do that, the subject was x The flashed line x is a target asked to focus on a particular square of 1 the matrix, before starting the flashing. x The flashed line x is nonWe have arbitrarily chosen to flash the 0 target squares of the diagonal which is between the top left hand corner and the bottom The online protocol: aims at using the whole system to compose and play a right hand corner. musical-loop. The modus operandi is briefly described The main differences with the offline below. part are:

For each column, the user can Matrix application and motivation for the design select the line he wants. So the choices we made. markers that the interface sends Event-Related Potentials (ERPs) are those EEGs are now different for each line. that directly measure the electrical response of the - The Matlab program sends a cortex to sensory, affective or cognitive events. feedback after each blinking They are voltage fluctuations in the EEG induced session, and these feedbacks are within the brain, as a sum of a large number of used to upload the matrix. This action potentials (APs) that are time locked to feedback is just the line number sensory, motor or cognitive events. ERPs are quite small- amplitude signals relatively to the resulting from the classifier. background EEG activity getting to 1-30 V - The flashing is now totally therefore they often need a synchronous signalrandom, and we switch to the next averaging procedure to enhance the evoked column only when the classifier potential and suppress the background noise. [6] has sent a feedback. The elicited ERP is comprised of two main So the new modus operandi is: components: the mismatch negativity (MMN) and the novelty P300 which refers to the component We sequentially highlight each column elicited by events about which the subject has not as the current one. A blinking session been instructed prior to the experiment. The P300 is launched. When the Matlab answer is wave represents cognitive functions involved in received, the blinking is stopped, the orientation of attention, contextual updating, corresponding square is colored in green, response modulation and response resolution and and the corresponding sound is played. consists of two overlapping subcomponents: P3a The next column is automatically and P3b.The two sub- components are most visible selected and we repeat the same in different areas of the scalp: operations. This is repeated until all the P3a- maximum amplitude over frontal area columns have been swept. P3b- maximum amplitude over parietal areas Once all the columns have been swept, the musical-loop corresponding to the When somebody concentrates on a task-specific selected squares is played, until the user stimulus (e.g., a highlighted row in Tune Matrix presses the escape key. among a pool of stimuli (e.g., non highlighted rows), the task-related stimulus will elicit a positive The program sends mainly the same peak with a latency of about 300ms from the markers as the offline version excluding, stimulus onset in subjects EEG signal. of course, the target/ non-target markers which are to be decided by the classifier. A classic example experiment driven by P300

P300
Using the P300 neural signal, Tune Matrix recognizes the positions which will be set for each time slot. In the next sections we will present an overview of the P300 signal and the wireless Emotiv EPOC EEG headset used by our Tune

signals is the P300 speller: a grid of 66 alphanumeric characters is presented to a subject. The subject focuses on a specific character, while the rows and columns are randomly flashed. Whenever a row or column containing those specific character flashes, a P300 signal is elicited in the subjects EEG. The speller then predicts the

specific character that the subject intends to select by determining the row and column that correspond to P300 signals in the subjects EEG and takes the letter at the intersection point.

Such a signal is presented in Figure 1 after averaging over all trials in a dataset. Initially we considered averaging the target trials and all nontarget trials, independently of the flashed row. This method had very good results in the offline part (87%-93% accuracy) of the project but in the online The software that comes with the Emotiv headset part the classifier couldnt discriminate between the provides the following detection functionality: target and non-target. This is because in the online - various facial expressions (Expressive) part we couldnt know beforehand which were nontarget trials so we had to average them separately, - levels of engagement, frustration, for each line. Doing so, we now had the meditation, and excitement (Affective) neighboring lines of the target one which also presented a P300 and now this effect wasnt - subject-specific training and detection of dissipated anymore by averaging with all noncertain cognitive neuro-activities such as push neighboring lines. pull, rotate and lift (Cognitive) By averaging the recording, the brain activity which is time-locked to the stimulus onset will be extracted as the ERP while the activity that is not time-locked will be averaged out. The drawback of averaging is that it leads to a dramatic increase of the time needed to communicate a position, hence the continued effort to develop new features and classifiers to reduce the required number of trials. [7] Also built in the headset is a gyroscope that detects the change of orientation of subjects head. The headset is not meant to be an extremely reliable device, thus it is challenging to extract finer P300 signals from the EEGs this headset produces. But this headset can be easily deployed at large scale because of its low price, and can be extremely handy if we can extract useful signals (e.g., P300) from it through smart signal processing and classification algorithms.

We used the Emotiv EPOC headset which has 14 data collecting electrodes and two reference electrodes. The sampling rate is 128 Hz and the electrodes are placed in roughly the international 10-20 system and are labeled as such. The headset transmits encrypted data wirelessly to a Windows based machine; the wireless chip is proprietary and operates in the same frequency range as 802.11 (2.4 GHz).

Figure 1: Trial and sensor-averaged data presenting a P300 ERP (blue) versus brain activity when non-target stimuli has been presented (red) on a voltage scale (10-6)

WIRELESS EEG HEADSET

Figure 2: Emotiv EPOC electrodes

DESIGN
In the design of the application we had to research many aspects in order to have a reliable and robust Tune Matrix system. Since the Emotive headset isnt intended for more accurate signal detection, each EEG electrode will have more noise than other headsets (e.g., Biosemi). In an always noisy scenario, the data which we use for obtaining the P300 ERP can be completely invalidated. For time-locked events, such as P300, the only reals solution is averaging. It is also used for BAEP (Brainstem Auditory Evoked Potentials) where the averaging is made over 1000-2000 trials.

Figure 3: Magnitude response of a ZPG filter design (green) versus standard IIR filter design (blue)

We studied different solutions for increasing the Signal to Noise Ratio (SNR) such as band pass filtering, detrending, re-referencing, electrode and trials averaging. The averaging introduces a delay in the acquisition of a P300 signal because we need several As classification algorithms we used a trials before actually detecting the P300. Bayesian statistical classifier based on The band pass filter is used to get rid of PCA and LDA and a Support Vector any noise that is not in the P300 Machine (SVM) classifier provided by the Matlab library. For both frequency range which we estimated LibSVM between 1.75 and 5 Hz. We used a Zero- classifiers we obtained almost the same Pole- Gain (ZPG) design 4th order results but the difference is the Butterworth filter to keep the 0.1- 9 Hz processing time which is roughly 2.5 segment after the example set in the seconds for the SVM classifier and 6.5 Neurophone application.[8] The ZPG seconds for the Optimal Bayes. version of the filter proved to be more LINEAR DISCRIMINANT ANALYSIS stable as seen in Figure 3. LDA is a method used in machine learning for finding a linear combination of features which separate two or more classes of events. The resulting combination may be used as a linear classifier or for dimensionality reduction

Given the strong tendency of rear sensors to display the P300 signal, the default reference locations at approximately P3/P4 will have a significant P300 component which will be subtracted from all the other channels in other words the P300 signal will be bigger and more detectable on more channels if you use a reference away from the parietal area. Therefore we chose as reference electrodes the AF3 and AF4.

before classification. LDA projects the data in a space of maximum variance between classes (between-class) and minimum variance inside a class (withinclass). The within-class covariance matrix is inversed and if there isnt enough variance on all dimensions, we will have a singular matrix. PCA is used for eliminating the redundant dimensions and obtaining a non-singular covariance matrix within-class.

finding maximum margin hyper-planes which optimally separate the two classes. The maximum margin hyperplane is the one that is furthest away from the nearest data point of each class. Whereas the original problem may be stated in a finite dimensional space, it often happens that the sets to discriminate are not linearly separable in that space.

For this reason, it was proposed that the original finite-dimensional space be PRINCIPAL COMPONENT ANALYSIS Because of the low spatial resolution, mapped into a much higher-dimensional presumably making the EEG channel readings are strongly space, correlated. This implies that any separation easier in that space. topographical difference between the To keep the computational load two classes would be unobserved. It is reasonable, the mapping used by SVM therefore important to de-correlate the schemes are designed to ensure that dot channels prior to classification. products may be computed easily in PCA is a mathematical procedure that terms of the variables in the original uses an orthogonal transformation to space by defining them in terms of a convert a set of observations of possibly kernel function selected to suit the correlated variables into a set of values problem. The hyper-planes in the higher of uncorrelated variables called principal dimensional space are defined as the set of points whose inner product with a components. vector in that space is constant. The number of principal components is less than or equal to the number of original variables meaning that the data The classification method used for is projected into a lower- dimensionality training is automatically set the same in space which maximizes the variance. the online testing for the same user. The principal components account for as much of the variability in the data as possible. Each succeeding component in MPLEMENTATION turn has the highest variance possible under the constraint that it be In the offline implementation we focus orthogonal to (uncorrelated with) the on data pre-processing and training the classifier. We used one dataset for each preceding components. of the seven subjects where we collected SUPPORT VECTOR MACHINE CLASSIFICATION data from all channels using the offline Support Vector Machines (SVMs) are two- interface described in section .

class robust classifiers that work by

Since synchronization is essential in a them. As a general idea of the results in P300 detection our greatest challenge this first step, we presented in Table was to correctly assign each trial to its 1.Error: Reference source not found corresponding marker. This was due to Dataset LDA+PCA SVM the fact that the data was collecting by accuracy accuracy reading the header from the Fieldtrip [%] [%] interface and the markers were set Dataset 1 90.265 92.920 through a serial port. Dataset 2 87.500 89.930 Dataset 3 91.667 88.095 Therefore the operation order in the data Dataset 6 84.210 87.134 -collecting and decision algorithms was Dataset 7 88.452 90.909 of major significance. In order to have an Average 88.418 89.797 Accuracy efficient processing of the data, Matlab waits for all the rows to be flashed at Table 1: Averaging without discriminating non-target trials each column and only after that begins to pre-process the data and add it to the In our second approach we apply the training set. The training set is saved same procedures to the data collected and used for the same user in the online both offline (from an existing dataset taken either through Fieldtrip, either with performance. Testbench) and online (from current data The features fed to the classifier were stream). The obtained dataset was the time-domain amplitude variances of named training. the trial- averaged data. For the offline part we applied a mask to In the classifier training we used the divide training into training-set and same interface as for the online test-set. We varied the training-set/ total performance to keep the training dataset ratio as described in Table 2 and conditions as similar as the testing ones. Table 3. The intended positions were marked before the online testing so we could In the online evaluation we kept the whole training dataset and used it, as evaluate the application. the name suggests, as a training set in We chose to play the resulted tune at evaluating the online data. Although the the end of the matrix evaluation to avoid classifications got high accuracy scores, having audio P300 artifacts in the data the rows chosen as target were not the collected.[6] We concluded the visual real choices. feedback of the set point is sufficient. Because all the processing steps were In our first approach, described in the same for both scenarios we tried section , we averaged the target and varying different parameters to see non-target trials separately without which has a bigger impact on the taking into account the identifier of non- classification. target line. The results were very optimist but they didnt match the online Dataset 1 LDA classification data-collecting method so we adjusted

accuracy 0.5 98.96 0.6 99.12 0.7 99.25 0. 100 8 0.9 Not enough variance SVM classification accuracy 0.5 87.50 0.6 87.72 0. 87.97 7 0.8 87.58 0.9 87.79
Table 2: Different training-set/ ratios for the first dataset

that the data for the rows pointed out as target are far from the P300 shape. The performance in the online evaluation isnt the desired one due to the fact that, unlike a P300 speller application, we use a lot less non-target stimuli having a target/ non-target ratio of 8/56 instead of 16/128 by flashing the rows only. Therefore, the oddball is more difficult to detect since the random character of the presented stimuli is less obvious. A solution is flashing the columns along with the rows and introducing a dataset distracter stimulus such as flashing diagonals, random points on the matrix.
Average of all electrodes. Column 4, target = 4 30 1 2 3 4 5 6 7 8

Dataset 2 LDA classification accuracy 0.5 98.96 0.6 97.37 0. 99.25 7 0.8 98.04 0.9 Not enough variance SVM classification accuracy 0.5 87.50 0.6 87.72 0. 87.97 7 0.8 87.58 0.9 87.79
Table 3: Different training-set/ ratios for the second dataset dataset

20

10

-10

-20

-30

-40

100

200

300

400

500 600 Time [ms]

700

800

900

1000

Table 4: Dataset 1, TDR = 0.7, channel 4

The selection of the proper training and testing set sizes slightly improves the performance. This is not a permanent character since the training set and the test set are chosen randomly in these limits by the crossvalind Matlab function. Although these are promising results, the visual inspection of each column show

Another explanation for this peculiar behavior might be the serial port used for markers transmission since it is known for introducing a delay from sending to reception. If the effects of the serial communications are consistent enough for data to be assigned to the wrong marker, then the misbehavior of

the

classifier

is

understandable.

Average of F3 and F4 6 5 Target Non-target

Channel 4. Column 4, target = 4 15 1 2 3 4 5 6 7 8 4 3 2 1 0 -1 -5 -2 -10 0 100 200 300 400 500 600 Time [ms] 700 800 900 1000

10

-15

100

200

300

400

500 600 Time [ms]

700

800

900

1000

Table 5: Dataset 2, TDR = 0.7, channel 4

Table 6: Electrode-averaged data over six trials: 3 target (blue), three non-target (red) . We can see in the first 300ms the rising trend of the target trials

As suggested in the literature, we tried an average both over time and electrodes. The accuracy results were situating at the same levels but the only channel now, plotted over different trials gives us extra information. Provided the serial communication introduces a delay in the markers transmissions, which is a known fact, the P300 would be detected a lot faster since we lose the data between the real marker transmission and the marker reception. Indeed, if we consider the image below, where we have the target trials (blue) and the non-target trials (red), we can see that before 300 ms all target trials are increasing and all non-target trials arent. We didnt consider an improvement to down-sample from 128 Hz as it already is a low sampling frequency (e.g., in comparison with Biosemi) but we still tried it. Down-sampling from 128 Hz to 32 Hz didnt have any influence on the results.

CONCLUSIONS
One major drawback of this application is the serial port communication which introduces a delay large enough to complicate a lot the decision process. Applying LDA and PCA for each trial is a time equivalent of getting a common spatial pattern feature. This explains the slightly better results obtained in the case of using a Bayesian statistical classifier based on PCA and LDA. The time needed for reading the FieldTrip header is 0.008-0.009s .The classification of each trial is realized only after all the rows are flashed for the current column, so we have a quasioffline decision-making algorithm which doesnt add time to the online data acquisition. If the assumption that the markers are delayed is correct, we have the certainty that the processing part doesnt have any role, only the serial interface.

Amplitude [V]

Hanover, : MobiHeld, 2010. ACM 978-14503-0197-8/10/08.

BIBLIOGRAPHY
1. Michelle, Andre. http://lab.andremichelle.com/tonematrix. http://lab.andre-michelle.com/. [Online] April 11, 2009. 2. http://www.libsdl.org/. Directmedia Layer. [Online] Simple

3. middleware, fmod interactive audio. http://www.fmod.org/. [Online] 4. http://com0com.sourceforge.net/. Nullmodem emulator . [Online] 5. http://www.codeproject.com/KB/system/s erial.aspx. Serial library for C++. [Online] 6. Saeid Sanei, J.A. Chambers. EEG Signal Processing. London : John Willey& Sons Ltd., 2007. ISBN 978-0-470-025819. 7. Feature Extraction and Classification of EEG Signals for Rapid P300 Mind Spelling. Adrien Combaz, Nikolay V. Manyakov, Nikolay Chumerin, Johan A. K. Suykens, Marc M. Van Hulle. Leuven : International Conference on Machine Learning and Applications, 2009. 978-0-7695-3926-3. 8. NeuroPhone: Brain-Mobile Phone Interface using a Wireless EEG Headset. Andrew T. Campbell, Tanzeem Choudhury, Shaohan Hu, Hong Lu,Matthew K. Mukerjee!, Mashfiqui Rabbi, and Rajeev D. S. Raizada.