SPSS Modeler Project

EEG Eye State Dataset

Syam Jamaly 6812354

Phillip Zaphiropoulos 6803131

ADM3308 Section M
Bijan Raahemi
Tuesday, April 14, 2015

Table of Contents
Abstract

EEG Eye State Dataset

Data Pre-Processing Tasks

Data Mining Techniques

Bayesian Network Model

C5.0 Model

Neural Network Model

10

Results Analysis and Conclusion

14

Abstract
Eye state identification is a kind of common time-series classification problem which is also a
popular topic in recent research. Electroencephalography (EEG) is widely used in eye state
classification to detect human's cognition state. Previous research has validated the feasibility of
machine learning and statistical approaches for EEG eye state classification (Wang, Guan, Man,
& Ting, 2014). This paper aims to propose a novel approach for EEG eye state identification
using three different modeling techniques learned in ADM3308; Predictive Decision Trees,
Neural Networks, and Bayesian Network Models.
In this paper, we investigate how the eye state (open or closed) can be predicted by measuring
brain waves with an EEG. To this end, we recorded a dataset containing the activation strength
of the fourteen electrodes of a commercial EEG headset as well as the manually annotated eye
state corresponding to the recorded data. We set the dataset training partition size to 70%, while
30% was set to the testing partition. We then tested different machine learning algorithms on
their performance to predict the eye state after training with the dataset.
The best performing classifier, the C5.0 Predictive Decision Tree, produced a classification error
rate of only 8.36% which meant a correct classification rate of 91.64%, resulting in a relatively
strong model and predictor. Additionally, we performed a test using Neural Networks which
resulted in a classification error rate of only 14.7%, and a Gini of 0.852 under evaluation metrics.
Finally, the third and worst test performed on the EEG Eye State dataset was the Bayesian
Network Model, which yielded a rather high error rate of 44.87%, concluding that the model is
only slightly more accurate than a random guess, with a Gini output of 0.001.

EEG Eye State Dataset

In this study, the EEG eye state dataset from UCI machine learning repository is employed for
the experiments. This EEG eye state dataset was donated by Rsler and Suendermann from
Baden-Wuerttemberg Cooperative State University (DHBW), Stuttgart, Germany. All data were
derived from one continuous EEG measurement with the Emotiv EEG neuroheadset, which is
shown in Figure 1. There are 14,980 patterns and 14 features (attributes) in the dataset, where the
14 features (attributes) are the data obtained by 14 sensors shown in Figure 2. The duration of the
measurement was 117 seconds. The eye state was detected via a camera during the EEG
measurement and added later manually to the file after analyzing the video frames. An overview
of each attributes minimum, maximum, and mean values can be seen in Table 1.
Table 1: Overview of Sensor Values

Figure 1: Emotiv EEG neuroheadset

Figure 2: Emotiv EEG neuroheadset sensor position and corresponding behavior groups. AF3,
AF4, FC5, F8, T7, P7, and O1 correspond to a maximum increase while F7, F3, F4, FC6, T8, P8,
and O2 correspond to a minimum decrease when the tester opens eyes.

Data Pre-Processing Tasks

There were several pre-processing tasks done to the data as result of the nature of the dataset
provided by the authors; however, nothing had to be done in regards to missing values, as there
were none. And although no pre-processing tasks were performed within the dataset itself
(internal), file conversion, an external pre-processing task, had to be completed. Firstly, the
original arff file, which was taken from the UCI Machine Learning Repository, had to be
converted into a txt file in order to be properly read and opened in Microsoft Excel. Once in
Excel, each attribute was separated into its respective column using delimited comma spacing
3

(text-to-column function), as all attributes were merged into one cell. Upon successfully splitting
each attribute into its respective column, the Excel file was ready to be saved and loaded into
IBM SPSS Modeler, using the Sources: Excel function within the application.
Additionally, once the Excel file was loaded into SPSS Modeler, the Type function was added
into the stream. All inputs (14 attributes/sensors) were loaded as Continuous measurements,
while the output (Eye Detection), was changed to a Flag measurement, as the values for the
output column were binary; 0 corresponding to eye-open, and 1 corresponding to eye-closed.
Next, the Partition function was added into the stream in order to partition 70% of the dataset
into Training, and 30% into Testing.

Data Mining Techniques

Bayesian Network Model
The Bayesian Model was chosen due to its function as a classifier model. It takes the attributes of
a data entry and determines the probability of what eye state will result in the row.
We can see in Figure 3 that the modeling our data using the Bayesian Network Model produces a
very low accuracy value of about 55.13%. It shows that the model itself is a very bad predictor
for the data we have been given. One of the main factors in the large amount of errors is due to
attribute values for some eye states being fairly similar to one another making it hard to predict.
Whether the model is trained or not, it will still be prone to errors because of the way the author
scaled the attributes to having only numerical values averaging around 4300. There are not
enough numbers to differentiate one eye state from another uniquely.

Figure 3: Analysis of the Bayesian Network Model

Figure 4 shows us in bar graph form what predictors (attributes) that the Bayesian Network
Model determined as most important for determining the eye state. It can be seen that the AF4,
F8, F4, FC6, T8, P8, O2, O1, P7, and T7 attributes are all equally important or, rather,
unimportant. The reason for all of these attributes being equally (un)important predictors could
be due to the fact that the majority of the values in all 14 attribute columns average roughly
4300.

Figure 4: Importance of the predictors used within the Bayesian Network Model

The graph in Figure 5 shows the actual Bayesian Network Model functionally. It displays to us
the different attributes that form the eye states and sorts them according to overall importance in
a network infrastructure. Note that the attribute notes are all of the same shade of blue because of
their equal (un)importance.

Figure 5: The Bayesian Network Model

Overall, the Bayesian Model did not perform as well as we thought it would and rather
performed very poorly than expected. The attributes are possibly too close together and cannot
have much margin for error in a model in order to accurately group the attributes to the proper
eye state classification.
C5.0 Model
The C5.0 Model constructs a predictive decision tree using the C5.0 algorithm. This algorithm is
a sophisticated data mining tools for discovering patterns that delineate categories, assembling
them into classifiers, and using them to make predictions (Rulequest, 2015).

We can see in Figure 6 that the modeling our data using the C5.0 Model produces a very high
accuracy value of about 91.64%. It shows that the model itself is a very strong predictor for the
data we have been given. A C5.0 model works by splitting the sample based on the field that
provides the maximum information gain. Each subsample defined by the first split is then split
again, usually based on a different field, and the process repeats until the subsamples cannot be
split any further. Finally, the lowest-level splits are re-examined, and those that do not contribute
significantly to the value of the model are removed or pruned (IBM Knowledge Center, 2015).
Figure 6: Analysis of the C5.0 Model

Figure 7 shows us in bar graph form what predictors (attributes) that the C5.0 Model determined
as most important for determining the eye state. It can be seen that the O1, P7, AF3, F3, and AF4
attributes were determined by the model to offer the most information gain at each split and thus

became the most important attributes. The remaining F7, F8, FC6, T7, and FC5 attributes offered
less information gain at each split and so they were less important.
Figure 7: Importance of the predictors used within the C5.0 Model

The graph in Figure 8 shows the actual C5.0 after it has split the data based on information gain
until no further splits were possible. The details of the graph cannot be seen due to it being
extremely large requiring us to shrink it down to its current size as in the figure. However, the
full size image displays to us the different splits that the algorithm made at each node to
determine eye states according to the overall importance and information gain of the attribute at
said node. Figure 9 is Node 0 of the decision tree and features attribute O1 showing the split
between eye states 0 and 1 within it.

Figure 8: The C5.0 Model (zoomed out due to extremely large size)

Figure 9: Node 0 of the above C5.0 Model tree

Neural Network Model

We decided to utilize a Neural Network as one of the models for our dataset because of the
nature of our data. Generally speaking, Neural Networks are useful due to the fact that their
...ability to generalize and learn from data mimics our own ability to learn from experience
(Raahemi, 2015). In other words, they are effective in the sense that we provide the Neural

10

Network input data and in return receive an output in the form of a desired variable or value.
Therefore, real life applications of Neural Networks often involve multiple points of
consideration that are all factored in to a certain degree when determining a final value or output
for this application. A business example of a neural network in action can include a mortgage
valuation. The factors that must be considered include the size, age, and space of a house when
determining what an appropriate appraisal value may be for the house.
As shown in Figure 10, modeling our data using Neural Networks also produces a relatively high
accuracy value of about 85.3%. It shows that the model itself is a very good predictor for the data
we have been given. A neural network can approximate a wide range of predictive models with
minimal demands on model structure and assumption. The form of the relationships is
determined during the learning process. If a linear relationship between the target and predictors
is appropriate, the results of the neural network should closely approximate those of a traditional
linear model. If a nonlinear relationship is more appropriate, the neural network will
automatically approximate the correct model structure (IBM Knowledge Center, 2015).
Figure 10: Analysis of the Neural Network Model

11

Figure 11 shows us in bar graph form what predictors (attributes) that the Neural Network
determined as most important for determining the eye state. It can be seen that the F8 and AF4
attributes were determined by the model to be the most important attributes. The remaining T8,
FC6, F7, O2, P7, O1, F3, and FC5 attributes were relatively equal and less important.
Figure 11: Importance of the predictors used in the Neural Network

As seen in figure 12, the processing units are arranged in layers. There are typically three parts
in a neural network: an input layer, with units representing the input fields; one or more hidden
layers; and an output layer, with a unit or units representing the target field(s). The units are
connected with varying connection strengths (or weights). Input data are presented to the first
layer, and values are propagated from each neuron to every neuron in the next layer. Eventually,
a result is delivered from the output layer (IBM Knowledge Center, 2015).

12

Figure 12: The Neural Network diagram

As mentioned earlier, the correlation between the initial analysis output and the classification in
figure 13 is very strong as the overall percentage of correctness is between 85.3% to 85.5%.

13

Figure 13: Neural Network classification overall percent correct

Results Analysis and Conclusion

According to the experimental results of these three different approaches shown and compared in
the report, the first approach (Bayesian Network Model) obtains the worst and highest
classification error rate of 44.87%, and the C5.0 Predictive Decision Tree method exhibits the
best and lowest classification error rate of 8.36%. In comparison with Rslers experimental
results using multilayer perceptron (2013), where the error rate is more than 30%, the results
derived by the Neural Network and C5.0 Predictive Decision Tree models are much better, as all
of these classification error rates are lower than 30%. This merely indicates that, firstly, the C5.0
Predictive Decision Tree approach can outperform the other methods seen in this experimental
process (with exceptions); and secondly, the mass amount of data involved allows for a more
accurate predictor set, which is very useful in the improvement of the classification results for
time-series problems such as this.
In this paper, a time-series classification approach based on Neural Networks is proposed for
EEG eye state identification. The approach is unique in a way that it firstly extracts features from
the raw data and then sorts these features using SPSS Modelers Type and Partition functions.

14

During the training process, the newly extracted features are imported into the neural predictive
system in a sequential order based on the feature ordering. In comparison with the Bayesian
Network Model, which runs without considering the relation between time-series data, the
experimental results of time-series Neural Networks showed that such a machine learning
approach can not only cope with time-series classification problems, but also improve the
accuracy of the classification results. Although the test indicated an error classification rate
slightly higher than that of the C5.0 Predictive Decision Tree model, the Neural Network
approach is the more suitable and appropriate method to follow for this type of dataset.
Moreover, the experimental results also imply that the relation among time-series data is crucial
to the data analysis in such classification problems.
In the future, some issues may remain as open topics for further research. Due to the high
variability that may arise in time-series data and experimental processes such as the one seen in
this report, the influence existing in the time-series classification process is still unknown.
In general, the feasibility of the C5.0 Predictive Decision Tree approach is difficult to validate
given the type of dataset that EEG Eye State identifies as. Consequently, the Neural Network
model makes most sense as an experiment since the data is time-based. The final results
indicated by IBM SPSS Modeler prove that, although the chosen models have a relatively low
error classification rate and decent acceptance rate, the best type of models and tests to perform
in the future are time-series classification based, such as Incremental Attribute Learning (IAL),
which can be implemented by Neural Networks, and Feature Ordering. Thus, there are still a
number of work items remaining for the future studies of this dataset and the EEG Eye State
topic in general.
In conclusion, this paper demonstrated that it is possible to predict eye state using EEG sensor
input with an accuracy of more than 90% using certain models. The high accuracy and the fact
that no special training is required suggest the use of eye state inferred from EEG signals for
controlling tasks (Rsler & Suendermann, 2013). However, the present study and the results in
15

this report encompass only a single subject (male), which raises the question whether results are
generalizable. To allow for applying the presented technology securely and effectively, the
dependence of eye state prediction accuracy on other activities carried out by the test subject(s)
will have to be explored further. Furthermore, it would be interesting to see whether the number
of sensors can be decreased by means of feature selection without compromising performance.
Also, other techniques for dimensionality reduction, such as linear discriminant analysis, could
be useful to look at (Rsler & Suendermann, 2013). In essence, time-series classification datasets
such as this one prove that many different tests can be performed in order to predict attribute
significance and eye state; however, the type of data mining model used for these dataset types
must be precise and valid in order to output a low classification error rate with high accuracy and
significance.

16

References
Emotiv. (2014. Introducing New EPOC+ Scientific contextual EEG. Retrieved from
http://www.emotiv.com/
IBM Knowledge Center. (2015). IBM Knowledge Center. IBM. Retrieved from http://www01.ibm.com/support/knowledgecenter/SS3RA7_15.0.0/com.ibm.spss.modeler.help/c50no
de_general.htm
IBM Knowledge Center. (2015). IBM Knowledge Center. IBM. Retrieved from http://www01.ibm.com/support/knowledgecenter/SS3RA7_15.0.0/com.ibm.spss.modeler.help/idh_n
euralnet.htm
Raahemi, B. (2015). Neural Networks. Lecture, Ottawa, Canada.
Rsler, O., & Suendermann, D. (2013, September). A First Step towards Eye State Prediction
Using EEG. Baden-Wuerttemberg Cooperative State University (DHBW). Retrieved from
http://suendermann.com/su/pdf/aihls2013.pdf
Rulequest. (2015). Information on See5/C5.0. Rulequest. Retrieved from
http://www.rulequest.com/see5-info.html
Wang, T., Guan, S., Man, K., & Ting, T. (2014, February 25). EEG Eye State Identification
Using Incremental Attribute Learning with Time-Series Classification. Mathematical
Problems In Engineering, 2014, 1-9. doi:10.1155/2014/365101