sensors

Article
Automatic Detection of Driver Fatigue Using Driving
Operation Information for Transportation Safety
Zuojin Li *, Liukui Chen, Jun Peng and Ying Wu *
College of Electrical and Information Engineering, Chongqing University of Science and Technology,
Chongqing 401331, China; cqustclk@126.com (L.C.); pengjun70@126.com (J.P.)
* Correspondence: 2010014@cqust.edu.cn (Z.L.); lwcindycq@163.com (Y.W.); Tel.: +86-23-6502-2333 (Z.L. & Y.W.)

Academic Editor: Simon X. Yang

Received: 19 April 2017; Accepted: 24 May 2017; Published: 25 May 2017

Abstract: Fatigued driving is a major cause of road accidents. For this reason, the method in this
paper is based on the steering wheel angles (SWA) and yaw angles (YA) information under real
driving conditions to detect drivers’ fatigue levels. It analyzes the operation features of SWA and YA
under different fatigue statuses, then calculates the approximate entropy (ApEn) features of a short
sliding window on time series. Using the nonlinear feature construction theory of dynamic time
series, with the fatigue features as input, designs a “2-6-6-3” multi-level back propagation (BP) Neural
Networks classifier to realize the fatigue detection. An approximately 15-h experiment is carried
out on a real road, and the data retrieved are segmented and labeled with three fatigue levels after
expert evaluation, namely “awake”, “drowsy” and “very drowsy”. The average accuracy of 88.02%
in fatigue identification was achieved in the experiment, endorsing the value of the proposed method
for engineering applications.

Keywords: steering wheel angles (SWA); yaw angles (YA); approximate entropy (ApEn); fatigue
detection; back propagation (BP) neural network

1. Introduction
As reported by the World Health Organization, car accidents kill more than 1.3 million people
worldwide every year [1], of which fatigued driving is a major cause. According to other statistics,
it causes thousands of automobile crashes [2] and about 35–45% of vehicle accidents [3,4]. Fatigued
driving usually means the disorder of mental and physical functions after a long-lasting drive,
subsequently leading to a weakening of the driver’s ability to control the vehicle. The technology in
automatic detection of driver fatigue under real driving conditions is meaningful for reducing road
accidents caused by fatigued driving.
The existing detection systems for driver fatigue, according to the source of the surveillance
data, fall into two categories: intrusive and non-intrusive. Intrusive systems use physiological data
of drivers and analyze their rules of change during the driving process, so as to monitor the drivers’
fatigue status. These data mainly include electroencephalogram (EEG) [5–8], electrooculogram (EOG)
and electrocardiograph (ECG) [9–12]. When driving on a real road, data collecting sensors are mounted
on the driver’s body, which may distract him or make him uncomfortable. This greatly restricts the
application of these methods in engineering. Non-intrusive systems acquire information about the
vehicle or the driver without contact with the driver, so the monitoring process will not interfere
with the driver. The data retrieved can better reflect the real behaviors of the driver and the real-time
status of the vehicle [13]. Presently, the latter has become a hot topic in driver fatigue detection
research [14–16]. The system based on SWA and YA is an example of non-intrusive methods.
Literature shows, after long duration driving, due to drowsiness and flagging energy, the driver’s
ability to operate and control the vehicle will decline substantially. This subsequently will decrease the

Sensors 2017, 17, 1212; doi:10.3390/s17061212 www.mdpi.com/journal/sensors

Sensors 2017, 17, 1212 2 of 11

accuracy and frequency of the turning of the steering wheel [17–20]. The driver fatigue status can be
effectively detected by collecting and analyzing SWA data, and constructing an identification model.
Research on drivers’ operation features have obtained valuable results. Fukuda [21], after
analyzing the statistical features of SWA time series, found that the cyclicity of time series is
inherently related to fatigue status. He discovered the on-line detection of driver fatigue levels, with
an accuracy of 76–88%. Andarian [22] analyzed the change rules of drivers’ steering data, established
a model and method for monitoring driver fatigue status. His method demonstrated an accuracy of
85%. Additionally, Bo et al. [14] used statistical analysis of SWA data, extracted 11 feature indexes
reflecting drivers’ fatigue level, and built an SVM based fatigue classification and identification model.
The three-level fatigue detection system boasts an accuracy of 87.7%. Unfortunately, most research is
conducted in a driving simulator, and as a result, the validity in real road conditions remains unproven.
Bittner [23] reported the validity of fatigue features under real road conditions. According to
this study, the standard deviation of steering angles, which saw good performance in a simulation
environment, does not behave well for fatigue detection on a real road. The reason for this is that the
steering features during real road driving relates not only to the driver’s fatigue status, but also the
speed, driving habits and ability, and the road conditions. Meanwhile, on the real road, the stochastic
jiggling due to uneven pavement will mix noise with the driver operation data or vehicle status data,
and greatly increase the probability of drifting. Therefore, the analysis, selection and performance
optimization of steering or yaw angles under real driving conditions become more complex.
This paper presents a fatigue driving detection system using SWA and YA information. At first,
it analyzes the operation features demonstrated with SWA and YA signals at different fatigue levels.
Then, it uses the non-linear feature theory of dynamic time series to calculate the ApEn features
of fixed window SWA and YA time series. At last, with the fatigue features as input, it designs
a “2-6-6-3” multi-level BP Neural Networks classifier for the detection of three fatigue levels during
the driving test.
The rest of this article is structured in the following manner. Section 2 analyzes drivers’ operation
characteristics in steering wheel angles and yaw angles. Section 3 presents the designed fatigue level
classification method with BP Neural Networks classifier based on the ApEn features for a varying
nonlinear parameter to measure the irregularity of steering wheel angles and yaw angles. Section 4
demonstrates the experiment and results on SWA and YA datasets collected from subjects under real
road driving conditions. Discussion and a summary are presented in Sections 5 and 6, respectively.

2. Analysis of Drivers’ Operation Characteristics

The driving process is a typical non-linear dynamic system. Exploring the non-linear features of
the time series of the operation parameters is helpful to analyze and identify drivers’ fatigue status.
In real driving conditions, drivers, to ensure the safety, should judge the vehicle status constantly
during the driving process and make modifications if deviation occurs. When drivers are tired, their
ability to perceive the environment, in order to determine the situation and to keep the vehicle in
control will be diminished. This will lead to a greater number of errors, and lower accuracy in control.
The controlled and status variables of the vehicle will show different fluctuation ranges or frequency.
Previous research [14,15,21,24] shows that tired drivers will show obvious abnormal features in vehicle
operation. For instance, the fluctuation range, frequency and speed of SWA and YA all show deviation
to some extent. It is safe to say that drivers’ operation features reflect their fatigue status, and the
non-linear features and irregularities expressed by SWA and YA data vary widely according to drivers’
fatigue levels.
Waveforms in Figures 1 and 2 are used to visually express how the driver’s fatigue levels affect
his operation features. Figure 1 shows the SWA waveforms at different fatigue levels, from which we
can see that when the driver is sober, as shown in Figure 1a, he will modify the steering wheel angles
frequently in a small range. When he is tired, as shown in Figure 1b, the frequency of modification is
low, as indicated by the waveforms in the yellow block, while the modification amplitude becomes
 Waveforms
Waveforms in in Figures
Figures 11 and
and 22 are
are used
used to
to visually
visually express
express howhow the
the driver’s
driver’s fatigue
fatigue levels
levels affect
affect
his operation features. Figure 1 shows the SWA waveforms at different fatigue
his operation features. Figure 1 shows the SWA waveforms at different fatigue levels, from which levels, from which
we
we can
can see
see that
that when
when thethe driver
driver is
is sober,
sober, as
as shown
shown in in Figure
Figure 1a,1a, he
he will
will modify
modify the the steering
steering wheel
wheel
angles
angles frequently
frequently
Sensors 2017,
in a small range. When he is tired, as shown in Figure
17, 1212 in a small range. When he is tired, as shown in Figure 1b, the frequency
1b, the frequency of
3 of of
11
modification is low, as indicated by the waveforms in the yellow block,
modification is low, as indicated by the waveforms in the yellow block, while the modification while the modification
amplitude
amplitude becomes
becomes larger
larger and
and velocity
velocity higher,
higher, indicated
indicated in in the
the blue
blue block.
block. IfIf the
the driver
driver isis severely
severely
larger as
tired, and velocity higher, indicated in wheel
the blue block.remain
If the unchanged
driver is severely tired, asofshown in
tired, as shown in Figure 1c, steering wheel angles remain unchanged for a period of time,
shown in Figure 1c, steering angles for a period time, as
as
Figure
indicated 1c, steering wheel angles remain unchanged for a period of time, as indicated by the yellow
indicated by by the
the yellow
yellow block,
block, followed
followed by by quick
quick fluctuations
fluctuations withwith big
big amplitude,
amplitude, as as indicated
indicated by
by
block,
the blue followed
block. by quick fluctuations with big amplitude, as indicated by the blue block.
the blue block.

Figure 1. SWA
SWA waveformsunder under different fatigue statuses: (a) Awake; (b) drowsy;
and (c)and
very(c) very
1. SWAwaveforms
Figure 1.
Figure waveforms under different fatigue
different statuses:
fatigue (a) Awake;
statuses: (b) drowsy;
(a) Awake; (b) drowsy; and drowsy.
(c) very
drowsy.
SWA: SWA: wheel
Steering Steering wheel angles.
angles.
drowsy. SWA: Steering wheel angles.

Figure
Figure 22 shows
shows the
the typical
typical YAYA waveforms
waveforms under
under different
different fatigue
fatigue levels.
levels. We
We can see that when
We can see that when
the driver
driver isis
the driver sober,
sober,asas
issober, shown
asshown
shown in Figure 2a, the waveforms fluctuate frequently in
in aa range.
small
small range.
in in Figure
Figure 2a, 2a,
the the waveforms
waveforms fluctuate
fluctuate frequently
frequently in a small range.
When
When
When he is tired,
he isastired, as in
as in Figure 2b, the fluctuations slow down as in the yellow block, and the
he is tired, in Figure 2b,Figure 2b, the fluctuations
the fluctuations slow down slow down
as in the as in
yellow the yellow
block, and the block, and and
amplitude the
amplitude
amplitude and velocity
and velocity increase
increase as shown
as shown in the blue block. When the driver is extremely tired, as
velocity increase as shown in the blue block.in the blue
When block.isWhen
the driver the driver
extremely is extremely
tired, as tired, 2c,
shown in Figure as
shown
shown in
in Figure
Figure 2c,
2c, the waveforms
thesharp
waveforms become
become sharp when the
the SWA
SWAasremains unchanged, as
as under the
the waveforms become when the SWA sharp when
remains unchanged, remains
under theunchanged,
yellow block, under the
and then
yellow
yellow block,
block, and
and then
then quick and
quickoccur huge
and as
huge fluctuations occur
occur as the
the SWA modification become quick
quick and huge fluctuations thefluctuations
SWA modification as
become SWA modification
quick become
and substantial, quick
as under
and
and substantial,
substantial, as under the blue block.
the blue block. as under the blue block.

Figure 2. YA waveforms under different fatigue statuses: (a) Awake; (b) drowsy; and (c) very
Figure 2. YA
Figure 2. YAwaveforms
waveforms under
under different
different fatigue
fatigue statuses:
statuses: (a) Awake;
(a) Awake; (b) drowsy;
(b) drowsy; and (c)and
very(c) very
drowsy.
drowsy. YA: Yaw angles.
drowsy.
YA: Yaw YA: Yaw angles.
angles.

To
To summarize,
summarize, the the drivers’
drivers’ operation
operation features
features under
under fatigue
fatigue mainly
mainly lie lie in
in the
the amplitude,
amplitude,
To summarize,
velocity, the drivers’ operation features under fatigue mainly liefatigue
in the amplitude, velocity,
velocity, and frequency of the changing SWA and YA parameters. As his fatigue grows, the driver’s
and frequency of the changing SWA and YA parameters. As his grows, the driver’s
and frequency
steering of the changing SWA and YA parameters. As his fatigue grows, the driver’s steering
steering modifications
modifications will
will decline
decline in
in frequency,
frequency, rise
rise in
in amplitude,
amplitude, addadd in
in velocity,
velocity, and
and even
even remain
remain
modifications
still willofdecline in frequency, rise in amplitude, add in features
velocity, of and evenfrequency,
remain still for
still for a period of time. The YA parameters also demonstrate features of slow frequency, large
for a period time. The YA parameters also demonstrate slow large
a period
amplitude of time. The YA parameters also demonstrate features of slow frequency, large amplitude and
amplitude andand high
high velocity.
velocity.
high velocity.
3.
3. Methodology
3. Methodology
Methodology
3.1.
3.1. Fatigue
Fatigue Detection
Detection Scheme
Scheme
Figure 3 is
Figure 33isisthethe framework
theframework
framework of
of the
the driver
driver fatigue
fatigue detection
detection algorithm.
algorithm. First
First of
of all,
all, the fatigue
the driving
fatigue
of the driver fatigue detection algorithm. First of all, the fatigue
driving
driving experiment
experiment is conducted
is conducted to acquire
to acquire necessary
necessary datasets,
datasets, which
which are processed
are processed with
with the
the Facial
Facial
experiment is conducted to acquire necessary datasets, which are processed with the Facial Video
Video
Video Expert
Expert Evaluation
Evaluation Method
Method to
to produce
produce graded
graded fatigue
fatigue data.
data. Then,
Then, operation
operation features
features of
of drivers
drivers
Expert Evaluation Method to produce graded fatigue data. Then, operation features of drivers at
at
at different
different fatigue
fatigue grades
grades are
are analyzed
analyzed and the
the ApEns of
of SWA and YA time series are
are calculated to
different fatigue grades are analyzed and and
the ApEns ApEns
of SWA SWA
and YAand YAseries
time timeare
series
calculatedcalculated to
to explore
explore
explore the
the non-linear
non-linear features of
of these
these operation parameters. Finally, aa three-level fatigue online
the non-linear features features
of these operation operation
parameters.parameters.
Finally, aFinally,
three-levelthree-level fatigue
fatigue online online
detection
algorithm is constructed based on the BP Neural Network to complete the driver fatigue detection.
 Sensors 2017, 17, 1212 4 of 12

Sensors 2017,detection
17, 1212 algorithm is constructed based on the BP Neural Network to complete the driver fatigue 4 of 11
detection.

Figure
Figure 3. Frameworkof
3. Framework of fatigue
fatigue detection
detectionsystem.
system.

3.2. Criteria of Fatigue Level Evaluation

3.2. Criteria of Fatigue Level Evaluation
In this paper, we define three fatigue levels: awake, drowsy, and very drowsy. Before
In thisevaluating
paper, we thedefine
validitythree
of thefatigue
fatigue levels:
detection method,
awake, we should
drowsy, anddesign a set of criteria
very drowsy. Beforetoevaluating
determine the fatigue level of the data, which serves as the standard patterns for the identification
the validity of the fatigue detection method, we should design a set of criteria to determine the
of fatigue levels. The facial video expert evaluation is, up to now, the most applicable method for
fatigue level of the
driver data,
fatigue which
status serves as
identification. themethod
This standard patterns
requires a groupfor of the identification
well-trained experts to ofscore
fatigue levels.
The facial the
video expert
fatigue statusevaluation is, up totonow,
of drivers according the most
their facial applicable
expressions, methodand
head positions, for other
driver fatigue status
factors.
Wierwill
identification. Thiset al. [25] were
method the first atogroup
requires introduce of this method in driver
well-trained experts fatigue appreciation.
to score the fatigueTo be status of
specific, this method works as follows: after the drivers’ facial videos are segmented, experts score
drivers according to their facial expressions, head positions, and other factors. Wierwill et al. [25] were
the video clips in random order, according the fatigue features like rubbing eyes, scratching faces,
the first toyawning,
introduce thiseyes,
closing method in driver
and adjusting bodyfatigue appreciation.
positions. The scores range To befrom specific,
0 to 100,thisandmethod
the works
as follows:average
after ofthethedrivers’ facial
scores given by videos
the experts arefor
segmented,
a certain clipexperts
is taken score the video
as its score. clips in random
In application,
however,the
order, according the fatigue
problemsfeatures
of the method
likeused by Wierwill
rubbing eyes,etscratching
al. lie in: (1) faces,
the criteria between closing
yawning, different eyes, and
levels are not easily distinguishable, (2) reasons and methods are not given as to why and how the
adjusting body positions. The scores range from 0 to 100, and the average of the scores given by the
continuous scores are grouped into three discrete levels, and (3) they fail to compare the video
experts forevaluation
a certain clip is
method in taken as its
time series withscore.
that in In application,
random series. Aiming however,
at these the problems
problems, Bo et al.of[26]
the method
used by Wierwill
revised theetevaluation
al. lie in:criteria
(1) the criteria
for driver between
fatigue, as showndifferent
in Table levels
1. They are
also not easily
verified distinguishable,
this method
(2) reasonswithand themethods
statistical consistency
are not given between as to thewhy
methodsandinhowtime series and random series.
the continuous scoresThis arepaper
grouped into
uses the facial video expert evaluation method with the criteria shown in Table 1 to mark drivers’
three discrete levels, and (3) they fail to compare the video evaluation method in time series with that
three-level fatigue.
in random series. Aiming at these problems, Bo et al. [26] revised the evaluation criteria for driver
fatigue, as shown in Table 1. They also
Table verified
1. Driver this
fatigue method
level with
criteria after the statistical consistency between the
evaluation.

methods in timeDrivingseries
Status and random
Fatigue Label series. This paper uses Features the facial video expert evaluation method
The head stays upright, and facial expressions are rich. Attentive to the
with the criteria shown in Table
Awake 1 1 to mark drivers’ three-level fatigue.
environment. Eyes open widely and blink quickly and eyeballs move actively.
Attention to the outside world decreases. Drivers make gestures like scratching
Drowsy 2
Table faces, shaking
1. Driver head,
fatigue winking,
level swallowing,
criteria aftersighing, deep breathing, and yawning.
evaluation.
Eyes tend to close, blink slowly with less eyeball activity.
Eyes close further with eyelids becoming heavier. Eyes are closing for a longer
Very drowsy 3
Driving Status Fatigue Label Features
time. Drivers may nap, nod, slant their heads, and then lose the ability to drive.

The head stays upright, and facial expressions are rich. Attentive to the
Awake 1
environment. Eyes open widely and blink quickly and eyeballs move actively.
Attention to the outside world decreases. Drivers make gestures like scratching
Drowsy 2 faces, shaking head, winking, swallowing, sighing, deep breathing, and yawning.
Eyes tend to close, blink slowly with less eyeball activity.
Eyes close further with eyelids becoming heavier. Eyes are closing for a longer
Very drowsy 3
time. Drivers may nap, nod, slant their heads, and then lose the ability to drive.

3.3. Extraction of ApEn of Time Series of the Operation Parameters

Non-linear property estimation is a complex analysis especially on nonlinear distributions.
Approximate Entropy measures irregularity of time series data, and it is a non-linear dynamic
Sensors 2017, 17, 1212 5 of 11

parameter. In order to present the complexity of a non-linear time series and reflect the occurrence
probability of a new pattern or regulation, ApEn assigns an estimated value of a non-negative fixed
quantity. In previous work on complexity, algorithms of ApEn have been proven as a powerful
estimator, with properties of usability and robusticity for irregularity or predictability evaluation.
Studies [27–29] show that the changes of physiological state of the human body can be characterized
by ApEn. ApEn has been very popular and applied to many static methods to quantify complexity
and regularity. It shows great potential applications to analyze a wide variety of physiological and
unphysiological time series data [30–32]. The ApEn of SWA and YA time series can be used to explore
the irregularity of the driving process. By comparing the change and distribution of ApEn under
different fatigue statuses, we can identify the fatigue level of drivers.
We can use time series data to obtain a robust estimate of ApEn. Consider an inputted time series,
{u(n) = u(1), u(2), u(3), · · · , u( N )}, ApEn is calculated in Equation (1):

N − m +1
1 1 N −m
N − m + 1 i∑ ∑ log(Cim+1 (r))
m
ApEn(m, r, N ) = log ( Ci ( r )) − (1)
=1
N − m i =1

where, Cim (r ) can be obtained through Equation (2).

Bi
Cim (r ) = (2)
N−m+1
Here, Bi represent the number of j for example {d| X (i ), X ( j)| ≤ r }, and both X (i ) and X ( j) are
used to represent inputted time series u(n)’s m-dimensional vectors reconstruction with Equation (3).

X (i ) = [u(i ), u(i + 1), · · · , u(i + m − 1)], X (i ) ∈ Rm

(3)
X ( j) = [u( j), u( j + 1), · · · , u( j + m − 1)], X ( j) ∈ Rm

where d| X (i ), X ( j)| measures the distance between X (i ) and X ( j), i, j = 1 ∼ N − m + 1. To define the
maximum difference between corresponding elements, variable m is set as an embedding dimension,
r is the tolerance parameter as a scale, and N is the amount of time points in phase space. To calculate
ApEn of SWA and YA data, with the increase of the value of m dimensions of embedded vectors and
engineering computing workload, the change of wheel angles will be weakened at the same time,
which will also weaken drowsiness features in SWA and YA data. On the other hand, with reference
to the parameter value in ApEn applications suggested by Yentes [33], this paper chose m = 2, and,
as suggested by Pincus [34], the parameters r are typically chosen as r = 0.2 × SD, (SD represents
standard deviation of the original windowing time series).
As shown in the above equation, in the calculation of the ApEn of time series, vectors should be
circularly constructed to acquire the distances between them, which consume tremendous computing
resources. Usually, construction of a distance matrix is used in engineering applications to reduce the
consumption of resources. Equation (4) expresses a N × N distance matrix.

u(n)0 
  

..

d = [u(n) u(n) · · · u(n)] −  (4)
  
 . 

 u(n) 0 

3.4. Detection of Driver Fatigue Based on BP Neural Network

Established on the basis of modern biological research, an Artificial Neural Network (ANN)
is a computation structure imitating the biological process to reflect some properties of human
brains, and has been widely applied in intelligent information processing [35–37]. ANN classification
is a typical statistical machine learning method, which requires no statistical properties or priori
knowledge in the sample correlation domain. ANN is characteristic of high precision and efficiency in
identification, which justifies its application in driver fatigue identification in this paper.
and has been widely applied in intelligent information processing [35–37]. ANN classification is a
typical statistical machine learning method, which requires no statistical properties or priori
knowledge in the sample correlation domain. ANN is characteristic of high precision and efficiency
in identification, which justifies its application in driver fatigue identification in this paper.
Sensors 2017, 17, 1212 6 of 11
ANN, as a mathematical theory model imitating human brains or behaviors, is a
non-algorithmic, non-linear and self-adaptive system, which is composed of a huge amount of
computation
ANN, asprocessing unitstheory
a mathematical properly connected.
model imitatingNeuron
human is one unit
brains can be expressed
or behaviors, as:
is a non-algorithmic,
non-linear and self-adaptive system, which is composed of a huge amount of computation processing
A  f (WP  B)
units properly connected. Neuron is one unit can be expressed as:
(5)

where W is the weight vector of ANN, B, threshold value vector of the network, P, input vector,
A = f (WP + B) (5)
and f transmission function.
Usually, neural networks need training rules, which involves constant adjustment of their link
where W is the weight vector of ANN, B, threshold value vector of the network, P, input vector, and f
weight to minimize
transmission function.the error between the ideal and real outputs. Training is used to establish
non-linear
Usually,mapping between need
neural networks performance parameters
training rules, and corresponding
which involves operation
constant adjustment condition
of their link
values. Neural network training needs to adjust weight W and threshold value
weight to minimize the error between the ideal and real outputs. Training is used to establish non-linear B through sample
learning,
mappingwhichbetween is done by a specific
performance training
parameters andalgorithm. The development
corresponding of training
operation condition algorithms
values. Neuralis
based on learning rules, which simulate biological learning mechanisms.
network training needs to adjust weight W and threshold value B through sample learning, which Up to now, several types
is
of neural networks with different structures have been reported. This paper
done by a specific training algorithm. The development of training algorithms is based on learning uses the Multi-Layer
Perception
rules, whichNeural
simulate Network (MLPNN)
biological to realize theUpclassification
learning mechanisms. to now, several andtypes
identification of driver
of neural networks
fatigue.
with different structures have been reported. This paper uses the Multi-Layer Perception Neural
MLPNN
Network consists
(MLPNN) of one the
to realize input layer, oneand
classification or more hidden of
identification layers,
driverand one output layer. This
fatigue.
paperMLPNN
adopts aconsists
four-layer network to identify drivers’ fatigue status;
of one input layer, one or more hidden layers, and one output namely, onelayer.
inputThis
layer, two
paper
hidden
adopts layers and one
a four-layer outputtolayer.
network identify Thedrivers’
ApEn features of SWA
fatigue status; and YA
namely, oneduring
inputthe driving
layer, process
two hidden
are the input of the neural network, while the drivers’ fatigue levels, namely
layers and one output layer. The ApEn features of SWA and YA during the driving process are the “awake”, “drowsy”
and
input“very
of thedrowsy”, are the output.
neural network, while theDifferent
drivers’from input
fatigue and namely
levels, output layers,
“awake”, there is no scientific
“drowsy” and “very and
widespread
drowsy”, aremethod to determine
the output. the quantity
Different from input and of output
nodes in the hidden
layers, there is layers. If the and
no scientific node quantity is
widespread
too low, to
method the networkthe
determine cannot show
quantity the difference
of nodes in the hiddenbetween different
layers. models,
If the node quantityandislettoothe
low,fitting
the
results
network approach
cannot showto a linear relation.between
the difference If the node quantity
different is too
models, and large, overfitting
let the mayapproach
fitting results occur in to the
network and lower the generalizability of untrained data. Moreover, with the
a linear relation. If the node quantity is too large, overfitting may occur in the network and lower the increase in node
numbers, the time
generalizability required in
of untrained computing
data. Moreover, will surge,
with makinginthe
the increase nodeengineering
numbers, the application difficult.
time required in
Usually,
computing thewill
most popular
surge, waythe
making to determine
engineeringthe number of
application nods isUsually,
difficult. conducting an experiment.
the most popular way We
determine,
to determine through experiments,
the number of nods isthat the node an
conducting numbers in theWe
experiment. hidden are S1 =experiments,
layersthrough
determine, 6 and S2 =
6. Figure
that 4 shows
the node the designed
numbers ANNlayers
in the hidden structure,
are S1and= 6other
and S2parameters
= 6. Figure of the network
4 shows are set as:
the designed
0.082
ANNfor adaptive
structure, andlearning coefficient,
other parameters of0.95 for momentum
the network coefficient,
are set as: 0.0001 for
0.082 for adaptive MSE, and
learning tangent
coefficient,
0.95 for momentum
sigmoid coefficient,
for the activation 0.0001 for MSE, and tangent sigmoid for the activation function.
function.

Figure 4. BP Network
Network architecture.
architecture.

4. Experiments and Results

4.1. Experiment Setup

To obtain the driver fatigue level evaluation and experiment data, this paper selects the expressway
from Beijing to Qinhuangdao as the driving route. The geographic environment is shown in Figure 5.
The data collecting platforms in the experiment vehicles gather information for SWA, can_braking,
4. Experiments and Results

4.1. Experiment Setup

To obtain the driver fatigue level evaluation and experiment data, this paper selects the
Sensors 2017, 17, from
expressway 1212 Beijing to Qinhuangdao as the driving route. The geographic environment is
7 of 11
shown in Figure 5. The data collecting platforms in the experiment vehicles gather information for
SWA, can_braking, can_thrott, brakeforce, leftsteer, rightsteer, yaw angle, X-accel, Y-accel, and
can_thrott,
synchronized brakeforce,
drivers’ leftsteer, rightsteer,
faces video. yaw angle, X-accel,
The participant drivers Y-accel, and asynchronized
all possess valid driverdrivers’
licensefaces
and
video. The participant drivers all possess a valid driver license and driving experience
driving experience of more than one year. The experiment starts at noon when people are prone to of more than
one year. TheAexperiment
drowsiness. starts
fifteen-minute at noon
trial when people
experiment preparesarethe
prone to drowsiness.
drivers A fifteen-minute
for the operational trial
environment,
experiment prepares the drivers for the operational environment, followed by the ninety
followed by the ninety minute formal experiment. The drivers’ task is to drive at approximately 100 minute formal
experiment.
km/h in the middleThe drivers’
lane. task
Speed is to
anddrive at approximately
position 100 km/h
are kept according in the
to the realmiddle
drivinglane. Speed and
conditions and
position
their own are kept according
driving to the
habits. The realof
length driving conditionsisand
the experiment their ownon
determined driving habits. that
the grounds The usually
length ofa
the experiment
driver becomes is determined
tired after one onhourtheofgrounds that
driving in usually a driver
a monotonous becomes
driving tired after
environment, asone hour of
revealed in
driving
previousinexperiments.
a monotonousNinety drivingminutes
environment, as revealed
of driving in previous
can witness experiments.
the whole process Ninety minutes
from “awake”,
of driving
through can witness
“drowsy”, the whole
to “very process from “awake”, through “drowsy”, to “very drowsy”.
drowsy”.

Figure 5. The
Figure 5. The driving
driving route
route between Beijing and
between Beijing and Qinhuangdao
Qinhuangdao is
is shown
shown on
on the
the blue
blue line.
line.

In a quiet driving environment, cameras mounted in the driving cabins record the drivers’
In a quiet driving environment, cameras mounted in the driving cabins record the drivers’ facial
facial expressions, at a frequency of 15 Hz. The data collection systems in the vehicles record the
expressions, at a frequency of 15 Hz. The data collection systems in the vehicles record the status
status parameters mentioned above. However, as the SWA and YA can reflect drivers’ operation
parameters mentioned above. However, as the SWA and YA can reflect drivers’ operation features more
features more directly than other parameters, they contribute more to driver fatigue level
directly than other parameters, they contribute more to driver fatigue level identification. This paper
identification. This paper chooses the two parameters from them as the objects in fatigue driving
chooses the two parameters from them as the objects in fatigue driving research, with the sampling
research, with the sampling frequency of 100 Hz. Ten drivers in total participated in this experiment,
frequency of 100 Hz. Ten drivers in total participated in this experiment, with an average age of 28 and
with an average age of 28 and 4.3 years of driving experience. The data acquired accumulated to
4.3 years of driving experience. The data acquired accumulated to 14.68 h.
14.68 h.
4.2. Experiment Database
4.2. Experiment Database
Whether to study the operation features of drivers in different fatigue statuses, or to design or
Whether to study the operation features of drivers in different fatigue statuses, or to design or
build driver fatigue detection models, requires a set of sample data with known driver fatigue levels.
build driver fatigue detection models, requires a set of sample data with known driver fatigue levels.
For this reason, a reliably accurate measurement criterion is necessary to estimate the real fatigue levels
For this reason, a reliably accurate measurement criterion is necessary to estimate the real fatigue
of the drivers. After evaluation, the data samples with determined fatigue levels constitute the fatigue
levels of the drivers. After evaluation, the data samples with determined fatigue levels constitute the
sample database.
fatigue sample database.
In this paper, data collected in the experiment was processed with sample segmentation,
In this paper, data collected in the experiment was processed with sample segmentation,
evaluation, and screening to construct the driver fatigue sample database. In sample segmentation,
evaluation, and screening to construct the driver fatigue sample database. In sample segmentation,
the facial videos and operation features are segmented in synchronism. The former are segmented into
the facial videos and operation features are segmented in synchronism. The former are segmented
one minute clips by software in chronological order, according to start time and end time. The operation
into one minute clips by software in chronological order, according to start time and end time. The
feature data are also segmented. The two segments consist of one sample datum. The sample evaluation
operation feature data are also segmented. The two segments consist of one sample datum. The
is conducted with the facial video expert evaluation method, with criteria shown in Table 1. Every facial
sample evaluation is conducted with the facial video expert evaluation method, with criteria shown
video sample is scored in time series by three experts. The experts’ unanimous score is the determined
in Table 1. Every facial video sample is scored in time series by three experts. The experts’
fatigue level of a sample, and in case of disagreement between experts, three experts should negotiate.
unanimous score is the determined fatigue level of a sample, and in case of disagreement between
After negotiation, if agreement is reached, the result is taken as the fatigue level of the sample. If no
experts, three experts should negotiate. After negotiation, if agreement is reached, the result is taken
agreement can be reached, this sample should be discarded. Additionally, samples involving a curved
as the fatigue level of the sample. If no agreement can be reached, this sample should be discarded.
road or lane shift should also be discarded. Finally, the sample datasets labeled with fatigue levels
consist of the three-level driver fatigue sample database, the composition of which in Table 2 is
shown below:
Sensors 2017, 17,2017,
Sensors 121217, 1212 8 of 12 8 of 11

Additionally, samples involving a curved road or lane shift should also be discarded. Finally, the
sample datasets labeled with fatigueTable Sample
levels2.consist database.
of the three-level driver fatigue sample database,
the composition of which in Table 2 is shown below:
Serial No. of Subjects Number of Samples Fatigue Level
Table 2. Sample database.
910_002 34 (0,1)
910_004
Serial No. of Subjects 48of Samples
Number (0,1,2)
Fatigue Level
911_003
910_002 2934 (0,1)
(0,1)
912_007
910_004 2448 (0,1)
(0,1,2)
913_002
911_003 2329 (0,1)
（0,1）
913_004
912_007 5424 (0,1,2)
(0,1)
913_002 23 (0,1)
4.3. Experiment Results 913_004 54 (0,1,2)

Figure 6 demonstrates
4.3. Experiment Results the ApEn distribution of the SWA samples’ time series calculated according
to EquationFigure
(1). The vertical coordinate
6 demonstrates the ApEnrepresents
distributiontheof ApEn
the SWA of SWA
samples’within
time aseries
shortcalculated
period of time
(two seconds)
according and the horizontal
to Equation coordinate
(1). The vertical is the
coordinate time ofthe
represents three
ApEnsamples, each lasting
of SWA within one minute.
a short period
of time
From this (two
chart, weseconds)
can seeandthatthe horizontal
when coordinate
the driver is at is the time fatigue
different of three levels,
samples,theeach
ApEnlasting
of one
SWA time
minute.
series have From this
different chart, we can
distribution see thatexplaining
features, when the driver is at different
the meaning fatigue levels,
of exploring SWA thefatigue
ApEn offeatures.
SWA time series have different distribution features, explaining the meaning of exploring SWA
Similarly, Figure 7 shows similar YA information. These two figures demonstrate that ApEns of SWA
fatigue features. Similarly, Figure 7 shows similar YA information. These two figures demonstrate
and YA that
timeApEns
seriesofcan
SWA fully
and represent drivers’
YA time series operation
can fully representbehavior features,behavior
drivers’ operation and thus identify
features, anddrivers’
fatigue levels.
thus identify drivers’ fatigue levels.

Sensors 2017, 17, 1212 Figure

Figure 6.6.SWA
SWAApEn
ApEn distribution.
distribution. 9 of 12

Figure7.7.YA
Figure YA ApEn distribution.
ApEn distribution.

We conducted a three-level fatigue identification test on 212 samples from six subjects, using
SWA and YA ApEn. The experiment, as demonstrated in Table 3, obtained an accuracy of 88.02%.
Compared with previous research [15], the accuracy only rises by 0.32%, but our data are retrieved
from a real driving environment, so the engineering applicability of this system is actually enhanced.
Sensors 2017, 17, 1212 9 of 11

We conducted a three-level fatigue identification test on 212 samples from six subjects, using SWA
and YA ApEn. The experiment, as demonstrated in Table 3, obtained an accuracy of 88.02%. Compared
with previous research [15], the accuracy only rises by 0.32%, but our data are retrieved from a real
driving environment, so the engineering applicability of this system is actually enhanced.

Table 3. Detection results distributed on confusion matrix for three fatigue levels “0”, “l” and “2”.

Detection Results
“Awake” (Level 0) “Drowsy” (Level 1) “Very drowsy” (Level 2)
“Awake” (Level 0) 92.50% 7.50% 7.00%
Expert
classification “Drowsy”(Level 1) 7.50% 84.60% 14.11%
“Very drowsy”(Level 2) 0.00% 7.90% 78.89%
Samples 112 72 28

4.4. Comparison between Testing Results by Relevant Methods

To prove the superiority of the proposed method in real road driving conditions, we compared the
identification accuracy of driver operation parameters by different testing methods (results are given
in Table 4). Literature [14] makes statistical feature analysis of SWA time series under lab conditions,
and fatigue level identification with the fisher classifier was accurate to 82%. Previous research [15],
with the same statistical features, uses the SVM classifier, improving its accuracy to 87.7%. Other
studies [16,17] based on SWA information obtained in real road driving conditions, and using ApEn
features and a simple classifier, achieved an averaged accuracy of 82.07% and 78.01%, respectively.
Based on these results, this paper introduces in a new type of data, the yaw angle of the vehicle,
and achieves an accuracy of 88.02%.

Table 4. Results comparison between five methods.

Experiment Data Method Average Correct Rate (%)

SWA for laboratory driving conditions [14] Statistical Feature + Fisher 82.00
SWA for laboratory driving conditions [15] Statistical Feature + SVM 87.70
SWA for real driving conditions [16] ApEn Feature + Designed model 82.07
SWA for real driving conditions [17] ApEn Feature + Designed model 78.01
SWA and YA for real driving conditions
ApEn Feature + BP NN 88.02
(presented in this paper)

5. Discussion
The method in this paper uses SWA and YA information to identify the fatigue state of a driver,
on the grounds that drivers’ physical statuses have a direct reflection on their operation behaviors.
The turning of the steering wheel and the yaw of the vehicle are the most sensitive and frequent. Fatigue
identification occurs at three levels: “awake”, “drowsy” and “very drowsy”, boasting a performance of
88.02% accuracy during approximately 15 h of driving on real road. The fatigue identification system
using SWA and YA time series enjoys higher robustness and reliability, which has been proven by the
real road driving test, instead of a lab simulation test. Meanwhile, this system has a very low rate of
false alarms of “drowsy” (7.50%) and “very drowsy” (7.91%), as shown in Table 3. It is robust because
its evaluation criteria, shown in Table 1, have acquired consistent results in evaluation by experts and
in others’ works, which are facial video based rather than SWA information based. This set of criteria
is a combination of drivers’ facial expressions, head positions and physical statuses, whose universal
applicability in driver fatigue evaluation ensures the robustness of this identification system.

6. Conclusions and Future Works

This paper presents a fatigue detection system using SWA and YA information obtained by fixed
sensors. The evaluation of fatigue state is made with the combined information of drivers’ facial
Sensors 2017, 17, 1212 10 of 11

expressions, head positions and physical states, as recorded in videos and with the experts’ agreed
judgments taken as standards. During the detection, ApEn features are used to make the judgment with
the BP neural network classifier. This method is suitable for detection of offline SWA and YA samples,
and fatigue detection at three levels (“awake”, “drowsy” and “very drowsy”). It receives an overall
accuracy of 87.21% in detection. Previous work [26] has proven that SWA, in combination with vehicles’
lateral positions and yaw angles, will improve accuracy in fatigue identification in a lab simulation
environment. Inspired by this, to combination of more drivers’ operational features, and vehicle status
parameters, in fatigue identification to improve the accuracy is the direction of our future work.

Acknowledgments: This work is supported in part by Natural Science Foundation of Chongqing with
cstc2014jcyjA40006, Scientific and Technological Research Program of Chongqing with KJ1601312, Special Project
from Chongqing University of Science and Technology with CK2015Z32. The authors are particularly grateful to
the State Key Laboratory of Automotive Safety and Energy at Tsinghua University for providing experimental
data, as well as the help of Professor Chen Bo and Shengbo Eben Li. Appreciated to sponsor of Key Laboratory of
Process Industry Online Analysis and Advanced Control of Chongqing and Semiconductor Wisdom Lighting
Engineering Technology Research Center.
Author Contributions: Zuojin Li designed experiments and structured this paper; Liukui Chen and Jun Peng
performed the experiments; Ying Wu analyzed the data; Zuojin Li wrote the paper.
Conflicts of Interest: The authors declare no conflicts of interest.

References
1. Sahayadhas, A.; Sundaraj, K.; Murugappan, M. Detecting driver drowsiness based on sensors: Review.
Sensors 2012, 12, 16937–16953. [CrossRef] [PubMed]
2. Williamson, A.; Lombardi, D.A.; Folkard, S.; Stutts, J.; Courtney, T.K.; Connor, J.L. The link between fatigue
and safety. Accid. Anal. Prev. 2011, 432, 498–515. [CrossRef] [PubMed]
3. Connor, J.L. The role of driver sleepiness in car crashes: A review of the epidemiological evidence. In Drugs,
Driving and Traffic Safety; Springer: Basel, Switzerland, 2009; pp. 187–205.
4. Khushaba, R.N.; Kodagoda, S.; Lal, S.; Dissanayake, G. Driver drowsiness classification using fuzzy
wavelet-packet-based feature-extraction algorithm. IEEE Trans. Biomed. Eng. 2011, 58, 121–131. [CrossRef]
[PubMed]
5. Chai, R.; Naik, G.; Nguyen, T.N.; Ling, S.; Tran, Y.; Craig, A. Driver fatigue classification with independent
component by entropy rate bound minimization analysis in an eeg-based system. IEEE J. Biomed.
Health Inform. 2016, 21, 715–724. [CrossRef] [PubMed]
6. Chai, R.; Ling, S.H.; San, P.P.; Naik, G.R.; Nguyen, T.N.; Tran, Y. Improving eeg-based driver fatigue
classification using sparse-deep belief networks. Front. Neurosci. 2017, 11. [CrossRef] [PubMed]
7. Huang, K.C.; Huang, T.Y.; Chuang, C.H.; King, J.T.; Wang, Y.K.; Lin, C.T. An eeg-based fatigue detection and
mitigation system. Int. J. Neural Syst. 2016, 26, 1650018. [CrossRef] [PubMed]
8. Craig, A.; Tran, Y.; Wijesuriya, N.; Nguyen, H. Regional brain wave activity changes associated with fatigue.
Psychophysiology 2012, 49, 574–582. [CrossRef] [PubMed]
9. Picot, A.; Charbonnier, S.; Caplier, A. On-line detection of drowsiness using brain and visual information.
IEEE Trans. Syst. Man Cybern. Part A 2012, 42, 764–775. [CrossRef]
10. Zhang, C.; Wang, H.; Fu, R. Automated detection of driver fatigue based on entropy and complexity
measures. IEEE Trans. Intell. Transp. Syst. 2014, 15, 168–177. [CrossRef]
11. Correa, A.G.; Orosco, L.; Laciar, E. Automatic detection of drowsiness in EEG records based on multimodal
analysis. Med. Eng. Phys. 2014, 36, 244–249. [CrossRef] [PubMed]
12. Jung, S.J.; Shin, H.S.; Chung, W.Y. Driver fatigue and drowsiness monitoring system with embedded
electrocardiogram sensor on steering wheel. IET Intell. Transp. Syst. 2014, 8, 43–50. [CrossRef]
13. Mandal, B.; Li, L.; Wang, G.S.; Lin, J. Towards detection of bus driver fatigue based on robust visual analysis
of eye state. IEEE Trans. Intell. Transp. Syst. 2016, 18, 545–557. [CrossRef]
14. Zhang, X.; Cheng, B.; Feng, R. Real-time detection of driver drowsiness based on steering performance.
J. Tsinghua Univ. 2010, 7, 1072–1076.
15. Qu, X.; Cheng, B.; Lin, Q.; Li, S. Drowsy driving detection based on driver’s steering operation characteristics.
Autom. Eng. 2013, 35, 288–291.
Sensors 2017, 17, 1212 11 of 11

16. Li, Z.J.; Li, S.E.; Li, R.J.; Cheng, B.; Shi, J.L. Driver fatigue detection using approximate entropic features of
steering wheel angle from real driving data. Int. J. Robot. Autom. 2017, 32, in press.
17. Li, Z.J.; Li, S.E.; Li, R.J.; Cheng, B.; Shi, J.L. Online Detection of Driver Fatigue Using Steering Wheel Angles
for Real Driving Conditions. Sensors 2017, 17, 495. [CrossRef] [PubMed]
18. Wang, J.Q.; Lu, M.; Li, K.Q. Characterization of longitudinal driving behavior by measurable parameters.
Transp. Res. Rec. 2010, 2185, 15–23. [CrossRef]
19. Wang, J.Q.; Zhang, L.; Zhang, D.Z. An adaptive longitudinal driving assistance system based on driver
characteristics. IEEE Trans. Intell. Transp. Syst. 2013, 14, 1–12. [CrossRef]
20. Wang, J.Q.; Li, S.E.; Zhang, Y. Longitudinal collision mitigation via coordinated braking of multiple vehicles
using model predictive control. Integr. Comput. Aided Eng. 2015, 22, 171–185.
21. Fukuda, J.; Akutsu, E.; Aoki, K. Estimation of driver’sdrowsiness level using interval of steering adjustment
for lane keeping. JSAE Rev. 1995, 16, 197–199. [CrossRef]
22. Eskandarian, A.; Mortazavi, A. Evaluati on of a smart algorithm for commercial vehicle driver drowsiness
detection. In Proceedings of the 2007 IEEE Intelligent Vehicles Symposium, Istanbul, Turkey, 13–15 July 2007;
pp. 553–559.
23. Bittner, R.; Hana, K.; Pousek, L. Detecting of fatiguestates of a car driver. In Proceedings of the International
Symposium on Biological and Medical Data Analysis, Frankfurt, Germany, 29–30 Septrmber 2000;
pp. 260–273.
24. Krajewski, J.; Golz, M.; Sommer, D. Detecting sleepy drivers by pattern recognition based analysis of steering
wheel behavior. In Proceedings of the Fifth International Driving Symposium on Human Factors in Driver
Assessment, Training and Vehicle Design, Big Sky, MT, USA, 22–25 June 2009; pp. 118–224.
25. Wierwille, W.W.; Ellsworth, L.A. Evaluation of driver drowsiness by trained raters. Accid. Anal. Prev. 1994,
26, 571–581. [CrossRef]
26. Qu, X.; Chen, B. Detection of Driver Drowsiness Based on Steering Operation and Vehicle State.
Master’s Thesis, Tsinghua University, Beijing, China, 2012.
27. Pincus, S.M.; Goldberger, A.L. Physiological time-series analysis: What does regularity quantify?
Am. J. Physiol. 1994, 266, 1643–1656.
28. Richman, J.S.; Moorman, J.R. Physiological time-series analysis using approximate entropy and sample
entropy. Am. J. Physiol. Heart Circ. Physiol. 2000, 278, 2039–2049.
29. Zhang, Z.; Zhou, Y.; Chen, Z.Y.; Tian, X.H.; Du, S.H.; Huang, R.M. Approximate entropy and support vector
machines for electroencephalogram signal classification. Neural Regen. Res. 2013, 8, 1844–1852. [PubMed]
30. Lin, L.; Chu, F.L. Approximate entropy as acoustic emission feature parametric data for crack detection.
Nondestruct. Test. Eval. 2011, 26, 119–128. [CrossRef]
31. He, Y.; Zhang, X. Approximate entropy analysis of the acoustic emission from defects in rolling element
bearings. J. Vib. Acoust. 2012, 134, 969–970. [CrossRef]
32. Kumar, Y.; Dewal, M.L.; Anand, R.S. Epileptic seizure detection using DWT based fuzzy approximate
entropy and support vector machine. Neurocomputing 2014, 133, 271–279. [CrossRef]
33. Yentes, J.M.; Hunt, N.; Schmid, K.K.; Kaipust, J.P.; McGrath, D.; Stergiou, N. The appropriate use of
approximate entropy and sample entropy with short data sets. Ann. Biomed. Eng. 2013, 41, 349–365.
[CrossRef] [PubMed]
34. Pincus, S.M. Approximate entropy as a measure of system complexity. Proc. Natl. Acad. Sci. USA 1991, 88,
2297–2301. [CrossRef] [PubMed]
35. Cao, X.; Zhu, D.; Yang, S.X. Multi-auv target search based on bio inspired aerodynamics model in 3-d
underwater environments. IEEE Trans. Neural Netw. Learn. Syst. 2016, 27, 2364–2374. [CrossRef] [PubMed]
36. Chu, Z.; Zhu, D.; Yang, S.X. Observer-based adaptive neural network trajectory tracking control for remotely
operated vehicle. IEEE Trans. Neural Netw. Learn. Syst. 2016, 1–13. [CrossRef] [PubMed]
37. Ni, J.; Li, X.; Hua, M.; Shen, J. Bio inspired neural network based q-learning approach for robot path planning
in unknown environments. Int. J. Robot. Autom. 2016, 31, 4526–4590.

© 2017 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access
article distributed under the terms and conditions of the Creative Commons Attribution
(CC BY) license (http://creativecommons.org/licenses/by/4.0/).