You are on page 1of 10

Computers in Biology and Medicine 89 (2017) 379–388

Contents lists available at ScienceDirect

Computers in Biology and Medicine


journal homepage: www.elsevier.com/locate/compbiomed

Fuzzy inference model evaluating turn for Parkinson’s disease patients


Christopher Ornelas-Vences a, *, Luis Pastor Sanchez-Fernandez a, Luis Alejandro Sanchez-
Perez a, b, Alejandro Garza-Rodriguez a, Albino Villegas-Bastida c
a
Centro de Investigaci
on en Computacion, Instituto Politecnico Nacional, Juan de Dios B
atiz Avenue, Mexico City, 07738, Mexico
b
Department of Electrical and Computer Engineering, University of Michigan-Dearborn, 4901 Evergreen Road Dearborn, MI, 48128, USA
c
Escuela Nacional de Medicina y Homeopatía, Instituto Politecnico Nacional, 239 Guillermo Massieu Helguera Street, Mexico City, 07320, Mexico

A R T I C L E I N F O A B S T R A C T

Keywords: Parkinson's disease is a chronic illness that affects motor skills. The Unified Parkinson's Disease Rating Scale
Parkinson's disease sponsored by the Movement Disorder Society (MDS-UPDRS) quantifies the current state of the disease based on
Gait clinician's observations. In this scale, turning is part of the gait assessment, yet specific guidelines on which
Turning
features to observe and rate are still unclear. What is more, only visual impairment detection is used as the main
Fuzzy logic
subjective rating tool. In this respect, four biomechanical features are extracted from sensors worn on the lower
limbs in this work. Afterwards, a turning assessment score is computed by means of a fuzzy inference model
constructed based on the examiners knowledge. Overall, 46 patients with varying motor impairment severity
underwent a full MDS-UPDRS motor examination and were monitored using a measurement system that includes
inertial sensors on each ankle. Turning rating scores computed are reasonably consistent with examiners opinions.
Nevertheless, the model proposed herein will always output the same score given the same inputs; whereas the
subjective nature of examiners observations translates into uncertainty and variability in the rating scores.
Furthermore, the continuous scale implemented in this work prevents the floor/ceiling effect inherent of discrete
scales.

1. Introduction assessment.
Similarly, gait analysis and assessment has been widely studied and
Parkinson's disease (PD) is a progressive and irreversible impairment many features and methodologies are available [25–30]. Freezing of Gait
of the nervous system that alters the ability to control movements of the (FoG) [31–35] and its prevalence [36] is one major impairment
limbs, head, jaw and tongue [1,2], as there are tremors, stiffness, aki- numerous works have focused on. In this respect, all the
nesia, postural alterations [3]. This disease affects about 3% of people above-mentioned works propose features to observe during gait but no
over 65 years old [4]. A widely used scale [5–8] for determining the special attention is given to turning exercises during gait. However, many
degree of progression of PD is the Movement Disorder Society - Unified PD patients experience difficulties while turning such as FoG [28,32,34],
Parkinson's Disease Rating Scale (from now on refer as the scale) [9]. This which is a constant falling threat and other mayor injuries [37]. Ap-
includes motor examination exercises supervised and evaluated by an pendix A exhibits and deeply analyzes an exercise performed by a PD
expert. These ratings are assigned subjectively, as they are not performed patient having minor impairments walking in a straight line, but expe-
with specialized measuring instruments [10]. riencing a FoG event during a turn. In the scale, turning is part of the gait
Several works rate specific motor exercises by means of systems of assessment, yet specific guidelines on which features to observe and rate
sensors [11–18], video recordings and mobile devices [19,20]. For are still unclear. For instance, some works analyze turn during daily ac-
instance, a system to detect and assess the severity of Levodopa Induced tivities, where the core evaluation is performed by expert examiners
Dyskinesia (LID) in daily activities is presented [21–24]. Other studies through a video recording system specially installed in the patient's house
introduce monitor systems of PD symptoms such as bradykinesia, LID and [38,39]. They define their own turn categories based on visual charac-
tremor [14–18]. Although results are compared against scale scores, all terization and no rating score is provided. Other works try to detect turns
statistical features are difficult to interpret and link to the final from features extracted from inertial sensors wore by PD patients

* Corresponding author.
E-mail addresses: chrorn9@gmail.com (C. Ornelas-Vences), lsanchez@cic.ipn.mx (L.P. Sanchez-Fernandez), alejand@umich.edu (L.A. Sanchez-Perez), link_agr@hotmail.com (A. Garza-
Rodriguez), albinovillegas_b@hotmail.com (A. Villegas-Bastida).

http://dx.doi.org/10.1016/j.compbiomed.2017.08.026
Received 1 June 2017; Received in revised form 23 August 2017; Accepted 23 August 2017

0010-4825/© 2017 Elsevier Ltd. All rights reserved.


C. Ornelas-Vences et al. Computers in Biology and Medicine 89 (2017) 379–388

[40–42]. In this respect, all the features used are numerical representa- records of 10 control people. The control group is made up of people
tions of inertial signals energy that do not characterize patient's without physiological or neurological impairments that prevent the tasks
impairments. from being performed. The volunteers presented on average twice to
In summary, the following problems are covered in this work: perform the test. Table 1 shows features of the collected database. All
measurements were performed under the supervision of an expert
 Specific guidelines on which features to observe and rate during PD examiner who guided the task and also assigned a rating based on
patients turns to yield rating scores are still unclear. the scale.
 Accurately quantifying features to rate PD patients turns by subjective The data used for this study was only for patients who could walk
observations is not possible. without using devices or the help of a person. The database used in this
 Since currently all turning rating scores are based on subjective fea- work includes more patients than in other studies related with sensor
tures quantification, follow ups are difficult to perform. signals analysis [14–18,20–24,31–35]. The feature “Time of last dose of
Levodopa before test” was asked to the patients and we have no control
In this paper, a fuzzy inference model to evaluate turning in PD pa- about this. It is shown as part of the database collected. Our research
tients is proposed. In addition, four biomechanical features are intro- focuses in the signal analysis during gait and modeling the inference of
duced so as to establish the basis for an objective evaluation of turning the expert examiners. In future works this feature will be considered to
during the gait analysis included in the item 3.10 from the scale, and create a follow up model.
therefore reduce uncertainty. All biomechanical features are extracted
from inertial sensors signals and used as inputs to perform the objective
2.4. Signals pre-processing
rating. The model proposed herein is intended to be part of an integral
model for gait evaluation according to the scale, and to follow up and
The accelerometer used measures all accelerations to which it is
analyze the effectiveness of palliative treatments of PD. This work has
subjected, including acceleration due to gravity, which is added as a
been carried out in accordance to The Code of Ethics of the World Medical
component to the measurements. This is undesirable because this ac-
Association and with Data Protection and Privacy Laws. All data was
celeration prevents the actual components of interest on the sensor from
collected with explicit written patient's consent.
being obtained. To eliminate this effect, it is necessary to calculate the
This work is divided into 4 sections. Section 2 presents Materials and
acceleration components due to gravity in each recorded measurement.
Methods used and describes the measurement system and the database
By the fusion of 3 sensors with the algorithm proposed by Madwick [43],
collected. It also gives the observations and the analysis of the turning to
it is possible to calculate the normalized unitary quaternion, which cal-
establish the elements to consider in order to design the fuzzy inference
culates the gravity acceleration components on the sensor in each mea-
model, presenting the basis of rules that conforms it. Section 3 shows the
surement and eliminates it. More details in Appendix D.
results obtained and is compared with the scores given by the experts,
and finally, in Section 4 the conclusions are given.
2.5. Turn analysis
2. Materials and Methods
The examiner assigns a performance score through observations on
2.1. Measuring units the patient during gait. However, in the turn evaluation, no mention is
made of any items and is left open to the interpretation and opinion of the
Measurements are made with an Inertial Measurement Unit (IMU) examiner. This generates subjectivity in the evaluation because there is
that allows recording the orientation, acceleration and angular velocity no set of elements to follow or measure. This section addresses the
of all movements. It consists of an accelerometer with resolution of 13 problem through an analysis of the patients to determine the elements
bits, range of ±16 g and minimum step of 4mg/LSB, a gyroscope with a that allow the evaluation of the task objectively.
resolution of 16 bits, range of ±2000 /s and minimum step of 0.61 ( /s)/ Table 2 shows the number of people to whom the experts assigned the
LSB and a 12 bits resolution magnetometer, range of ±8 Gauss and different possible scores, both in the gait and turning. The people who
minimum step of 4 m Gauss/LSB. The processing was performed with a participated in the exercise were patients who could do it without the
microcontroller that records the measurements of the 3 sensors with help of a device or a person. So the “3” and “4” scores of the scale do not
communication protocol I2C. Each of these units is provided with a apply in gait assessment. The experts used the 0–4 scale when they were
Bluetooth module that transmits 115,200 bits per second to send the asked to evaluate only the turning after the execution and observe the
measurements to a PC, and a 3.8 V battery. The modules were arranged recordings, where “0” is the best possible rating.
on the patient's lower limbs in order to record the movements while Patients and control subjects were studied for the purpose of
performing the tasks. These modules allow a sampling of sensors up to obtaining the features that show a person with problems to perform this
50 Hz. Appendix B shows a validation of the sampling frequency. task. Patients who did not present problems of postural stiffness or
postural instability, turn in a short time and with a number of steps
2.2. System operation similar to the control group, performed simultaneous movement of the
feet, i.e., began to move the foot slightly when the other foot had not
The system collects information from the modules arranged on the finished making the step, and there was no presence of hesitation on foot
lower limbs of the patient by using a PC, which monitors the modules to
store the information of each sensor. Once the sensors are placed on the
Table 1
patient, they are asked to walk in a straight line 10 m, stop, turn 180 and Statistical information of the database.
return to the starting point. The system allows labeling the activities in
Features Patients Control
real time. In addition, a video recording of the patient is performed
during the activities to verify the results. Appendix C shows more details Age (Mean ± SD) 60.3 ± 7.3 years 59.5 ± 6.1
years
of the system.
Range of age (min - max) 43 - 78 years 47 - 66 years
Total women - men 17–29 3–7
2.3. Database Time with PD (Mean ± SD) 4.6 ± 2.0 years –
Range of Time with PD (min - max) 1–12 –
The database consists of 94 measurements performed in 12 sessions in Time of last dose of Levodopa before test 246.9 ± 278.0 min –
(Mean ± SD)
3 different facilities in a 6-month period: 82 signals of 46 patients and 12

380
C. Ornelas-Vences et al. Computers in Biology and Medicine 89 (2017) 379–388

movements to maintain balance. These patients scored “0” and “1” on the Hesitation Steps, since their magnitude affected directly in the evaluation
expert's assessment. obtained, and it was possible to extract them from the signals registered
Patients with stiffness in the upper and lower limbs take longer to by the measurement modules.
perform the 180 turn because they require a greater number of steps, Fig. 1 shows the functional diagram of the model to extract the
present hesitation at the start of the steps and perform it one at a time. It biomechanical features of the signals recorded during the rotation and
is not possible for them to begin a step until the movement of the other through a fuzzy model, establishes an objective evaluation. The mea-
leg is completely made, which hinders the continuity of the steps. These surement units, previously described in Section 2.1, collect the data of
patients scored between “2” and “3” on the expert's assessment. It was the 3 sensors in 3 inertial axes. The data is pre-processed to change the
observed that the greater the number of hesitations, the lower the score. data frame reference and eliminate the acceleration due to gravity. Signal
Some patients had to take sudden steps of small amplitude to avoid processing is performed to extract the biomechanical features from
falling, and did not resume the rotation until they were completely stable. Table 3, and finally a score is assigned to the turn by using a fuzzy model
People with the worst ratings showed rigidity and also did not show based on a series of rules created through observations and with the
confidence to perform the task. They performed a greater number of steps opinion of the experts.
and took longer than other patients; this is because the steps they per-
formed were short or required moving the feet to not lose balance, and 2.5.1. Total Steps
their leg movements were not continuous. It is observed that the patient This biomechanical feature L1 , is obtained by counting the number of
who only moves the foot until it is completely stable, does not begin a peaks found in the gyroscope signal on the z-axis, since it is in this axis
movement with one leg until the opposite has finished all movement. In where the movement is recorded. The number of Total Steps is the sum of
all these patients, the number of sudden steps necessary or hesitations to the peaks found in both members. The steps required by the patient are
perform the task was greater compared to the other patients. counted, so the hesitation or sudden movements of the foot to avoid
Table 3 shows the biomechanical features observed in each of the losing postural stability are also counted. Fig. 2 shows the algorithm
grades according to the experts. They counted the steps visually and proposed to detect the steps of each member. To determine a signal peak
measured the time with a chronometer. The continuity of the turn and as a patient step, it is necessary to fulfill the conditions defined in Eq. (1).

8
>  i  hi  u1 Þ∧ðwi  u2 Þ∧ðhi  u3 *maxðh
ðρ !  i ji ¼ 1; ⋯; NÞÞ∧ !
>
<    
1  
f ðρi ; ti ; wi ; hi Þ ¼ ∃k k ¼ argmin ti2  tj1  u4 ∨i ¼ 1  f ðρi ; ti ; wi ; hi Þ ¼ 1 (1)
>
>  
: j¼1;⋯;i1 ji > 1
0 otherwise

the presence of hesitations were under their criteria. A direct relationship These conditions only account for steps with prominence ρn near
between Total Steps and Total Time required by the patients with the score height hn with a tolerance u1 ¼ 1, the width of the peak must have a
assigned by the experts in the gait evaluation was observed. It is also minimum duration of u2 ¼ 40 ms and height hn must be at least u2 ¼ 0:2
noted that the number of patients presenting Continuous Steps decreases of the higher peak. Each of the counted peaks counts on a tuple tn ¼
in high scores and increases the presence of Hesitation Steps in the task. ðtn1 ; tn2 Þ that indicates the points in time where the rising edge of the peak
We choose to select biomechanical features to evaluate a person's starts and when the falling edge of the peak ends, respectively. This
180 turn during gait, Total Steps, Total Time, Continuous Steps and the parameter may be composed of peak flanks that did not fulfill the func-
tion defined in Eq. (1), but they form part of a significant peak. The
Table 2
beginning of a new peak for the same foot should have a minimum
People assigned to each score in gait on the scale and turning. separation u4 ¼ 300 ms from the last accounted peak. All tolerances were
calculated by analyzing the signals from the control patients.
Score Gait of patients Turning of Patients
Fig. 3a) shows the steps detection in the gyroscope signal of the right
Expert 1 Expert 2 Expert 1 Expert 2
leg. The measurement of each of the calculated parameters
0 73 73 44 45 ρn ; tn ; wn ; hn is observed and in Fig. 3b) we see the same process but
1 11 12 15 15 with the gyroscope signal of the left leg. Fig. 3c) shows the merge and sort
2 10 9 12 12
3 n/a n/a 14 12 of signals with respect to the parameter tn1 , since the task performed is
4 n/a n/a 9 10 time dependent. It establishes the first biomechanical feature L1 ¼ N that
Total Patients 94 94 94 94 indicates the number of Total Steps required for the patient to finish
n/a: not apply. Just patients that did the test without help. the turn.

Table 3
Value of features in patients according to experts.

Biomechanical features Score given by expert 1 Score given by expert 2

0 1 2 3 4 0 1 2 3 4

Values Values

Total Steps (Mean ± SD) 4.1 ± 0.6 4.9 ± 0.2 6.2 ± 0.6 7.7 ± 1.3 14.7 ± 2.6 4.1 ± 0.6 4.9 ± 0.2 6.4 ± 0.5 7.5 ± 1.2 14.3 ± 2.9
Total Time (Mean ± SD) 2.0 ± 0.4 2.5 ± 0.6 3.1 ± 0.9 3.9 ± 0.8 6.6 ± 2.7 2.1 ± 0.4 2.3 ± 0.6 3.2 ± 0.8 4.0 ± 0.9 6.4 ± 2.7
Continuous steps (Patients observed/Total 43/44 10/15 4/12 0/14 0/9 43/45 10/15 4/12 0/12 0/10
patients)
Hesitation steps (Patients observed/Total 3/44 3/15 7/12 12/14 9/9 2/45 5/15 7/12 10/12 10/10
patients)

381
C. Ornelas-Vences et al. Computers in Biology and Medicine 89 (2017) 379–388

Fig. 1. Functional diagram of the model.

Fig. 2. Steps detection algorithm.

2.5.2. Total Time    1 1 


The biomechanical feature L2 quantifies the time required by the 1 tn1 < tnþ1
1
< tn2 < tnþ1
2 2
∨ tn < tnþ1 < tnþ1 < tn2
gðtn ; tnþ1 Þ ¼ (3)
patient to make the turn. The beginning is when the person moves one of 0 otherwise
the feet and is finished when the person makes a change of approximately
180 in the orientation. In order to check the number of degrees the X
N1

person rotated, a cumulative integration with the trapezoidal method is L3 ¼ ðgðtn ; tnþ1 Þ ¼ 1Þ=L1 (4)
1 n¼1
used from tn¼1 to tN2 as shown in Fig. 4. If the number of degrees is not at
least 150 , then it is necessary to recount the peaks in the signals by To check the continuity of the steps, it is necessary to start the
modifying the tolerances. Eq. (2) defines the biomechanical feature L2 ; movement when the other limb is still in motion. The function defined in
which is the time it took the patient to perform the rotation. Fig. 4 shows Eq. (3) compares the consecutive tuples ti and tiþ1 to find the steps given
an example of the task time quantification with the accumulated integral with continuity. If the foot movement begins after the other foot is
graph. The parameter quantification begins at the moment when the completely static, it is indicative that the person did not have the confi-
patient starts to change their orientation, so that t11 is taken as the starting dence to perform this movement because it could cause them to lose
point because it indicates the beginning of the rising edge for the first postural stability. This biomechanical feature L3 , defined in Eq. (3), takes
step. The task is considered to be completed at the end point of the falling values in the interval [0, 1), since it depends on the Total Steps L1 . An
edge for the last step, sotN2 is considered as this reference. The accu- example of a patient who did the turning continuously is shown in Fig. 5.
mulated integral graph shows the change of orientation of each patient Continuity is observed throughout the turning as each step starts when
lower limbs with respect to the steps that they perform. It is noted that the other foot is still in motion. This indicates that the person had com-
the greatest change of orientation is recorded during the rising edge of plete confidence of not losing postural stability, and did not suffer stiff-
eachpn indicating limb movement. ness in the lower extremities because throughout the entire rotation they
showed a continuous movement of lower limbs.
L2 ¼ tN2  t11 (2)
2.5.4. Hesitation Steps
2.5.3. Continuous steps The fourth and last biomechanical feature L4 , is the percentage of
The third biomechanical feature L3 is the percentage of steps taken steps that are hesitations or sudden leg movements. It was observed that
with one foot while the other still remains in movement. This shows that some people require a slight movement to stabilize and not lose balance.
the person has enough confidence to begin the movement of one foot These hesitations can be at the beginning of a step or immediately after
when the movement of the other foot still does not stop completely. To finishing it. This feature shows the amount of hesitation detected as steps
quantify the steps given under this criterion it is necessary to compare the that were required to maintain balance during task execution. The
tuples tn of each step given by each leg. greater number of small steps required, the greater instability occurred
during the task execution. It takes values in the interval [0, 1), since it

382
C. Ornelas-Vences et al. Computers in Biology and Medicine 89 (2017) 379–388

Fig. 3. Detection and counting of Total Steps performed. A) pn of Gyrz right leg. B) pn of Gyrz left leg. C) Merge and sort ofpn :

depends on the Total Steps L1 . Eq. (5) defines the function that detects the evaluation due to the ease of results interpretation, since a similar range
hesitations in the steps with u6 ¼ 200 ms and Eq. (6) defines the is used as in the scale (0–4), but continuous. Due to the nature of the
biomechanical feature L4 . problem, the fuzzy logic is the one that best adapts because different
   ranges of values exist in the selected parameters, and a boundary is not
1 tn2  tn1 < u6 defined in which a value obtained from a category can be classified. This
hðtn Þ ¼ (5)
0 otherwise logic has the ability to define such boundaries flexibly.
Fuzzy logic behaves similarly to human behavior, since the model
X
N assigns an evaluation to a patient's turn using a series of rules structured
L4 ¼ ðhðtn Þ ¼ 1Þ=L1 (6) with natural language. These rules were established on the basis of ob-
n¼1
servations made on patient videos and expert comments. The member-
Fig. 6 shows a patient who required moving the right foot suddenly ship functions to be used for the fuzzification and parameters of these
twice at the end of the turn. It is a sign that the patient performed the turn functions were defined by observing the control group to establish the
without much control over their lower extremities, since they could not mean ranges of each biomechanical feature, with the data collected from
perform a continuous movement to finish the task completely due to leg the patients, and the score assigned by the experts; other categories were
stiffness, or maybe they had problems maintaining the postural stability established for everyLn :
when changing their body orientation. For Total Steps Ln¼1 , 4 linguistic variables were used, Minimum, Mean,
Bad and Maximum using in all cases trapezoidal membership function
(MF). The Total Time Ln¼2 was fuzzified in 3 linguistic variables using the
2.6. Fuzzy model sigmoid MF for Fast and Slow and a trapezoidal MF for Mean. For the
Continuous Steps Ln¼3 and Hesitation Steps Ln¼4 , a trapezoidal MF was
The fuzzy logic was selected to perform the biomechanical features fuzzified in Low and Mean and into Mean and Large, respectively. More

383
C. Ornelas-Vences et al. Computers in Biology and Medicine 89 (2017) 379–388

Fig. 4. Quantification of start and end time of the turning.

Fig. 5. Time between steps to detect continuous steps.

details in Appendix E.
The model consists of Rn rules where n ¼ 1; …; 21. These rules were
created based on what was expressed by the experts and what was
observed in the videos. Table 4 shows the 21 rules that make up the rule
base that the model uses to assign the evaluation.
In rule R1 it is observed that in order to obtain a Normal evaluation it
is necessary that the patient has a reduced number of steps (L1 is Mini-
mum) and the time does not exceed the mean ðL2 is not Slow). This rule
considers patients who had no problems performing the turn. In case the
time is greater than the meanðL2 is Slow), rule R2 considers the Contin-
uous Steps. If this parameter is average (L3 is Mean) then the evaluation is
Normal despite requiring more time, since the patient makes the turn
with a minimum of steps and continuously. In general, Normal evaluation
is assigned to patients who have a number of steps less than or equal to
the mean and with a high percentage of Continuous Steps.
In rule R5 when Total Steps (L1 is Minimum) and Total Time (L2 is
Slow) are present with Hesitation Steps in high percentage (L4 is Large)
then the evaluation is Slight because the patient performs the turn with
Fig. 6. Hesitations detection in a patient to perform a turn of 180 . insecurity in their steps, and this may mean that the person had problems

384
C. Ornelas-Vences et al. Computers in Biology and Medicine 89 (2017) 379–388

Table 4 high speed of these phenomena.


Rule base of fuzzy model. Table 5 shows a comparison between the model and two experts
1. If (L1 is Minimum) AND (L2 is not Slow) THEN (O1 is Normal) ratings for five patients at different severity. Parameters Total Steps (L1 )
2. If (L1 is Minimum) AND (L2 is Slow) AND (L3 is Mean) THEN (O1 is Normal) and Total Time (L2 ) alone are not fully determinant to assign a rating
3. If (L1 is Mean) AND (L2 is Fast) THEN (O1 is Normal)
score. For example, L1 ¼ 4 for both patient 1 and patient 2 whereas L2
4. If (L1 is Mean) AND (L2 is Mean) AND (L3 is Mean) THEN (O1 is Normal)
only changes by 1.5s, which does not entirely differentiate these two
5. If (L1 is Minimum) AND (L2 is Slow) AND (L4 is Large) THEN (O1 is Slight) cases because L2 < 3. Namely, both patients turn using few steps in a
6. If (L1 is Mean) AND (L2 is Mean) AND (L3 is Low) THEN (O1 is Slight) reasonable time. However, feature L3 (Continuous Steps), which is
7. If (L1 is Mean) AND (L2 is Slow) AND (L4 is Mean) THEN (O1 is Slight) designed to capture how confidence the patient is taking steps during
8. If (L1 is Bad) AND (L2 is Mean) AND (L3 is Mean) AND (L4 is Mean) THEN (O1 is turn, clearly sets these two cases apart. Similarly, although patient 2 and
Slight)
9. If (L1 is Mean) AND (L2 is Mean) AND (L4 is Large) THEN (O1 is Slight)
patient 3 share the same turning Total Time L2 ¼ 2:94, parameter L4
(Hesitation Steps) shows the latter required a higher number of small
10. If (L1 is Mean) AND (L2 is Slow) AND (L4 is Large) THEN (O1 is Mild) stabilizing steps, indicating difficulties to maintain postural stability
11. If (L1 is Bad) AND (L2 is Mean) AND (L3 is Mean) AND (L4 is Large) THEN (O1 is during the task. Interestingly, patient 4 receives a higher score (O1 ¼ 3:2)
Mild) compared to patient 3 (O1 ¼ 2:1) because the Total Time required to
12. If (L1 is Bad) AND (L2 is Mean) AND (L3 is Low) AND (L4 is Mean) THEN (O1 is Mild)
perform the turn was 2 s higher even though values of L1 ; L3 and L4 are
13. If (L1 is Bad) AND (L2 is Slow) AND (L3 is Mean) AND (L4 is Mean) THEN (O1 is
Mild) similar. Patient 5 displays very high values of L1 and L2 , i.e., slow turn
with many steps, so the model assigns a high rating score O1 ¼ 3:7.
14. If (L1 is Bad) AND (L2 is Mean) AND (L3 is Low) AND (L4 is Large) THEN (O1 is Table 5 confirms that theLn biomechanical features proposed in this
Moderate) paper successfully represent turn performance in PD patients since the
15. If (L1 is Bad) AND (L2 is Slow) AND (L3 is Mean) AND (L4 is Large) THEN (O1 is
fuzzy inference model outputs are consistent to the score of expert ex-
Moderate)
aminers. On the one hand, given the subjective nature of human obser-
16. If (L1 is Bad) AND (L2 is Slow) AND (L3 is Low) AND (L4 is Mean) THEN (O1 is vations, variations are always expected in expert's scores as shown for
Moderate) patient 3. On the other hand, the fuzzy inference model proposed herein
17. If (L1 is Maximum) AND (L2 is Mean) AND (L3 is Mean) THEN (O1 is Moderate) will always output the same score given the same biomechanical
18. If (L1 is Maximum) AND (L2 is Slow) AND (L3 is Mean) AND (L4 is Mean) THEN (O1
Ln inputs.
is Moderate)
Table 6 proves the existent variability between ratings assigned by
19. If (L1 is Bad) AND (L2 is Slow) AND (L3 is Low) AND (L4 is Large) THEN (O1 is expert examiners for 20 different PD patients. This is not only due to
Severe) subjective observations but also to the lack of well established guidelines
20. If (L1 is Maximum) AND (L2 is Slow) THEN (O1 is Severe) and features to observe from PD patients. Experts assigned ratings based
21. If (L1 is Maximum) AND (L2 is Mean) AND (L3 is Low) THEN (O1 is Severe) on their own criteria and their general observations of patient confidence
when performing the turn. Regarding patient 3, despite having an average
Total Steps (L1 ) value and a very low Total Time (L2 ), the hesitation
maintaining postural stability during the turn. This evaluation covers observed by expert 1 during the task was sufficient to evaluate the turn
patients who have a Total Time and Total Steps less than or equal to the with 1, whereas expert 2 was not aware of this phenomenon, resulting in
mean but with a low value of Continuous Steps or slightly higher Hesita- a score of 0. However, the fuzzy inference model is able to output a value
tions Steps. in between (O1 ¼ 0:30), taking into consideration all Ln biomechanical
It is observed that in rule R10 the evaluation is Mild, although it features including the observed hesitation captured in L4 ¼ 0:25. All
displays a number of mean steps (L1 is Mean), but registered a Total Time patients with rating scores between 0 and 1 according to the expert ex-
greater than the mean (L2 is Slow) and a high percentage of Steps of aminers, have an average Total Time (L2 ) and a low percentage of
Hesitation (L4 is Large). This may be indicative of requiring steps to Continuous Steps (L3 ). Features and scores for all 94 measurements from
maintain postural stability when performing the task. The Mild assess- the 46 patients and 10 control subjects are shown in Appendix F.
ment is generally for patients with a greater than mean Total Steps and Overall, patients receiving scores in the range 1  O1  2 have a
mean Total Time. decrease in the percentage of Continuous Steps (L3 ) and an increase in the
Rule R14 considers the evaluation Moderate if the patient has a greater Total Time (L2 ) to complete the rotation with respect to patients having
than the mean Total Steps (L1 is Bad), a mean Total Time (L2 is Mean), but lower scores. Patients having scores between 2  O1  3 present a sig-
with a record of unwanted Continuous Steps (L3 is Low) and Hesitation nificant increase in the turning Total Time (L2 ) and Hesitation Steps (L4 ).
Steps (L4 is Large). This is because it registered a number of steps above Moreover, the previous patients, i.e., having 2  O1  3, are still more
the mean, performed quickly but insecure. Due to the fact that it registers confident to perform the turn as the Continuous Steps (L3 ) are higher than
a few of Continuous Steps and a large number of Hesitation Steps, it is in those patients with ratings between 3  O1  4, who exhibit a Hesi-
indicative of problems to maintain postural stability. This evaluation is tation Steps (L4 ) increase as well.
for patients who present a total amount greater than the mean of Total The rating assigned by the model is continuous in the range of (0, 4),
Steps, performed with insecurity or without continuity. which gives more resolution than experts ratings. This allows a better
The evaluation is Severe when a number of steps and time is well representation of the actual turn performance. For example, patient 5 and
above the mean ðL1 is Maximum) and (L2 is Slow) as shown in rule R21 , patient 6 received a score of 0.5 and 0.8, respectively. In this sense, ex-
since the person required a very large number of steps and time to perts did not have such precision, resulting in patients assigned to
complete the task. This assessment is for patients who are more insecure different severity categories. However, the computed Ln biomechanical
or who require pauses to avoid falling. features still capture slight changes in the turn performance, translating
into a more precise representation of the actual difference. This allows
3. Results and discussion precise follow-ups and helps to determine if treatments and medication
have positive effects in PD patients.
The methodology introduced herein allows to successfully extract Unlike other works [38–42], we do not classify the type of turning
Ln biomechanical features of the turn performance for 94 measurements comparing against control subjects; instead four biomechanical features
from 46 different patients and 10 control subjects. Also, it can detect obtained from inertial sensors are used by a fuzzy inference model to
hesitations or sudden movements with a high level of certainty, which is output a turn rating score according to the scale. Using this approach, it is
extremely difficult to consistently observe by expert examiners due to the observed that the model ratings are close to experts opinions, so that

385
C. Ornelas-Vences et al. Computers in Biology and Medicine 89 (2017) 379–388

Table 5
Comparison of evaluation given by the model and expert examiners to five patients at different severity.

Graphical Depiction of Biomechanical Features Features value Expert's score Model Score

Expert 1 Expert 2

L1 ¼4 0 0 0:29
L2 ¼ 1:44
L3 ¼ 0:75
L4 ¼0

L1 ¼4 0 0 1:00
L2 ¼ 2:94
L3 ¼ 0:50
L4 ¼0

L1 ¼7 3 2 2:10
L2 ¼ 2:94
L3 ¼ 0:28
L4 ¼ 0:14

L1 ¼8 3 3 3:24
L2 ¼ 4:48
L3 ¼ 0:25
L4 ¼ 0:12

L1 ¼ 14 4 4 3:70
L2 ¼ 13:2
L3 ¼ 0:07
L4 ¼ 0:64

L1 Total Steps L2 Total Time L3 Continuous Steps L4 Hesitation Steps

Table 6 expert knowledge modeling is achievable. In addition, assessments given


Comparison of scores assigned by experts and assigned by the model to 20 patients. by the proposed model avoid sharp boundaries between categories
Patient Quantified Biomechanical Expert rating Fuzzy model caused by the discrete ranges used in the scale.
Features

L1 L2 L3 L4 Expert 1 Expert 2 O1 Round 4. Conclusions


1 3 1.30 0.33 0 0 0 0.29 0
2 3 3.46 0.66 0 0 0 0.29 0 In this work four biomechanical features are extracted from sensors
3 4 1.62 0.50 0.25 1 0 0.30 0 worn on the lower limbs in order to describe the turning performance in
4 5 2.36 0.40 0 0 0 0.40 0 PD patients: Total Steps, Total Time, Continuous Steps and Hesitation Steps.
5 4 2.44 0.50 0 1 0 0.50 1
These are solely obtained from the inertial sensor signals. Subsequently, a
6 5 2.6 0.40 0.20 0 1 0.88 1
7 5 3.18 0.40 0.20 1 1 1.00 1 turning score is computed by a fuzzy inference model able to capture
8 6 2.56 0.50 0 2 2 1.42 1 examiners knowledge using if-then rules following the same structure
9 6 2.60 0.50 0 2 2 1.57 2 from all the assessment guidelines defined in the scale. This model is
10 6 2.62 0.50 0 2 2 1.64 2 straightforward and scalable since all the relationships between inputs
11 6 2.76 0.66 0.16 2 2 1.93 2
12 7 4.26 0.42 0 2 2 2.00 2
are clearly stated using linguistic terms. Additionally, the continuous
13 6 4.52 0.16 0 2 2 2.84 3 output range of turning scores prevents the floor/ceiling effect inherent
14 7 3.48 0.42 0.57 3 3 2.99 3 of discrete scales such as the one used in the scale (0, 1, 2, 3 and 4).
15 6 3.56 0.16 0.33 3 3 3.05 3 Hence, any given combination of the abovementioned input features will
15 9 3.70 0.33 0.11 3 3 3.20 3
produce the same fuzzy inference model turning score, something diffi-
17 12 5.74 0.16 0.16 4 4 3.50 4
18 11 5.86 0.09 0.18 4 4 3.70 4 cult to achieve applying conventional rating methods given their sub-
19 16 5.86 0.25 0.68 4 4 3.70 4 jective nature. This is particularly useful to perform follow-ups.
20 17 7.42 0.23 0.70 4 4 3.70 4 Overall, a systematic guideline and rating model of turn performance
during gait on PD patients are introduced given the uncertainty of the

386
C. Ornelas-Vences et al. Computers in Biology and Medicine 89 (2017) 379–388

scale in this regard. The proposed model shows that rating a task moni- disease—from the perspective of practical applications, Ann. Transl. Med. 4 (2016),
http://dx.doi.org/10.21037/atm.2016.03.09, 90–90.
tored by wearable sensors is possible, and can be improved by delivering
[11] K.J. Kubota, J.A. Chen, M.A. Little, Machine learning for large-scale wearable
such evaluation in a continuous range of values. The rating model as a sensor data in Parkinson's disease: concepts, promises, pitfalls, and futures, Mov.
whole may set the basis for a more advanced system that considers Disord. 31 (2016) 1314–1326, http://dx.doi.org/10.1002/mds.26693.
additional signals from sensors located on the waist or arms, or could also [12] A. Pantelopoulos, N.G. Bourbakis, A survey on wearable sensor-based systems for
health monitoring and prognosis, IEEE Trans. Syst. Man. Cybern. Part C Appl. Rev.
be part of a follow-up model to evaluate the disease progression. 40 (2010) 1–12, http://dx.doi.org/10.1109/TSMCC.2009.2032660.
[13] A. Darwish, A.E. Hassanien, Wearable and implantable wireless sensor network
solutions for healthcare monitoring, Sensors 11 (2011) 5561–5595, http://
Authors' contribution dx.doi.org/10.3390/s110605561.
[14] M. Pastorino, M.T. Arredondo, J. Cancela, S. Guillen, Wearable sensor network for
COV did most of the data analysis and interpretation and wrote the health monitoring: the case of Parkinson disease, J. Phys. Conf. Ser. 450 (2013)
12055, http://dx.doi.org/10.1088/1742-6596/450/1/012055.
first and final draft of the manuscript. The conception and design of the
[15] S. Patel, K. Lorincz, R. Hughes, N. Huggins, J. Growdon, D. Standaert, M. Akay,
study was made by LPSF and LASP. Also, they worked together with AGR J. Dy, M. Welsh, P. Bonato, Monitoring motor fluctuations in patients with
on data analysis and in the writing and revision of the manuscript. AVB parkinsons disease using wearable sensors, IEEE Trans. Inf. Technol. Biomed. 13
collaborated on the medical conceptual design of the research and in the (2009) 864–873, http://dx.doi.org/10.1109/TITB.2009.2033471.
[16] G. Rigas, A.T. Tzallas, M.G. Tsipouras, P. Bougia, E.E. Tripoliti, D. Baga,
critical revision of the manuscript for important intellectual content. All 5 D.I. Fotiadis, S.G. Tsouli, S. Konitsiotis, Assessment of tremor activity in the
authors contributed in the data acquisition and approved the final parkinsons disease using a set of wearable sensors, IEEE Trans. Inf. Technol.
version of this article. Biomed. 16 (2012) 478–487, http://dx.doi.org/10.1109/TITB.2011.2182616.
[17] A. Salarian, H. Russmann, C. Wider, P.R. Burkhard, F.J.G. Vingerhoets, K. Aminian,
Quantification of tremor and bradykinesia in Parkinson's disease using a novel
Conflicts of interest ambulatory monitoring system, IEEE Trans. Biomed. Eng. 54 (2007) 313–322,
http://dx.doi.org/10.1109/TBME.2006.886670.
[18] D.G.M. Zwartjes, T. Heida, J.P.P. Van Vugt, J.A.G. Geelen, P.H. Veltink, Ambulatory
None to declare. monitoring of activities and motor symptoms in Parkinsons disease, IEEE Trans.
Biomed. Eng. 57 (2010) 2778–2786, http://dx.doi.org/10.1109/
TBME.2010.2049573.
Funding [19] S. Bor-Rong Chen, T. Patel, R. Buckley, D.J.J. Rednic, L. McClure, D. Shih, M. Tarsy,
Welsh, P. Bonato, A web-based system for home monitoring of patients with
This research did not receive any specific grant from funding agencies Parkinson's disease using wearable sensors, Biomed. Eng. IEEE Trans. 58 (2011)
831–836, http://dx.doi.org/10.1109/TBME.2010.2090044.
in the public, commercial, or not-for-profit sectors. [20] D. Pan, R. Dhall, A. Lieberman, D.B. Petitti, A mobile cloud-based Parkinson's
disease assessment system for home-based monitoring,, JMIR mHealth uHealth 3
Acknowledgements (2015) e29, http://dx.doi.org/10.2196/mhealth.3956.
[21] J.I. Hoff, A.A. Plas, E.A.H. Wagemans, J.J. van Hilten, Accelerometric assessment of
levodopa-induced dyskinesias in Parkinson's disease, Mov. Disord. 16 (2001)
We are grateful to the patients and healthcare professionals that 58–61, http://dx.doi.org/10.1002/1531-8257(200101)16, 1<58::AID-
contributed with their participation, ideas and suggestions to accomplish MDS1018>3.0.CO;2–9.
[22] A. Manson, P. Brown, J. O'Sullivan, P. Asselman, D. Buckwell, A. Lees, An
this work. ambulatory dyskinesia monitor, J. Neurol. Neurosurg. Psychiatry 68 (2000)
196–201, http://dx.doi.org/10.1136/jnnp.68.2.196.
[23] M.G. Tsipouras, A.T. Tzallas, G. Rigas, P. Bougia, D.I. Fotiadis, S. Konitsiotis,
Appendix A. Supplementary data
Automated Levodopa-induced dyskinesia assessment, in: Annu. Int. Conf. IEEE Eng.
Med. Biol. Soc. EMBC’10 vol. 2010, 2010, pp. 2411–2414, http://dx.doi.org/
Supplementary data related to this article can be found at http://dx. 10.1109/IEMBS.2010.5626130.
doi.org/10.1016/j.compbiomed.2017.08.026. [24] N.L.W. Keijsers, M.W.I.M. Horstink, S.C.A.M. Gielen, Automatic assessment of
levodopa-induced dyskinesias in daily life by neural networks, Mov. Disord. 18
(2003) 70–80, http://dx.doi.org/10.1002/mds.10310.
References [25] C. Ossig, A. Antonini, C. Buhmann, J. Classen, I. Csoti, B. Falkenburger, M. Schwarz,
J. Winkler, A. Storch, Wearable sensor-based objective assessment of motor
symptoms in Parkinson's disease, J. Neural Transm. 123 (2016) 57–64, http://
[1] L.M. Shulman, A.L. Gruber-Baldini, K.E. Anderson, C.G. Vaughan, S.G. Reich,
dx.doi.org/10.1007/s00702-015-1439-8.
P.S. Fishman, W.J. Weiner, The evolution of disability in Parkinson disease, Mov.
[26] B. Mariani, M.C. Jimenez, F.J.G. Vingerhoets, K. Aminian, On-shoe wearable
Disord. 23 (2008) 790–796, http://dx.doi.org/10.1002/mds.21879.
sensors for gait and turning assessment of patients with parkinson's disease, IEEE
[2] K. Del Tredici, H. Braak, Review: sporadic Parkinson's disease: development and
Trans. Biomed. Eng. 60 (2013) 155–158, http://dx.doi.org/10.1109/
distribution of α -synuclein pathology, Neuropathol. Appl. Neurobiol. 42 (2016)
TBME.2012.2227317.
33–50, http://dx.doi.org/10.1111/nan.12298.
[27] S. Lord, B. Galna, L. Rochester, Moving forward on gait measurement: toward a
[3] J. Jankovic, Parkinson's disease: clinical features and diagnosis, J. Neurol.
more refined approach, Mov. Disord. 28 (2013) 1534–1543, http://dx.doi.org/
Neurosurg. Psychiatry 79 (2008) 368–376, http://dx.doi.org/10.1136/
10.1002/mds.25545.
jnnp.2007.131045.
[28] M.E. Morris, F. Huxham, J. McGinley, K. Dodd, R. Iansek, The biomechanics and
[4] L.M.L. De Lau, P.C.L.M. Giesbergen, M.C. De Rijk, A hofman, P.J. Koudstaal, M.M.B.
motor control of gait in Parkinson disease, Clin. Biomech. 16 (2001) 459–470,
Breteler, incidence of parkinsonism and parkinson disease in a general population
http://dx.doi.org/10.1016/S0268-0033(01)00035-3.
the rotterdam study, Neurology 63 (2004) 1240–1244, http://dx.doi.org/10.1212/
[29] A. Salarian, H. Russmann, F.J.G. Vingerhoets, C. Dehollain, Y. Blanc, P.R. Burkhard,
01.WNL.0000140706.52798.BE.
K. Aminian, Gait assessment in Parkinson's disease: toward an ambulatory system
[5] C. Ramaker, J. Marinus, A.M. Stiggelbout, B.J. van Hilten, Systematic evaluation of
for long-term monitoring, IEEE Trans. Biomed. Eng. 51 (2004) 1434–1443, http://
rating scales for impairment and disability in Parkinson's disease, Mov. Disord. 17
dx.doi.org/10.1109/TBME.2004.827933.
(2002) 867–876, http://dx.doi.org/10.1002/mds.10248.
[30] V. Agosti, C. Vitale, D. Avella, R. Rucco, G. Santangelo, P. Sorrentino, P. Varriale,
[6] G. Parkinson, Study, a controlled, randomized, delayed-start study of rasagiline in
G. Sorrentino, Effects of Global Postural Reeducation on gait kinematics in
early parkinson disease, Arch. Neurol. 61 (2004) 561, http://dx.doi.org/10.1001/
parkinsonian patients: a pilot randomized three-dimensional motion analysis study,
archneur.61.4.561.
Neurol. Sci. 37 (2016) 515–522, http://dx.doi.org/10.1007/s10072-015-2433-5.
[7] O. Rascol, C.J. Fitzer-Attas, R. Hauser, J. Jankovic, et al., A double-blind, delayed-
[31] S.T. Moore, H.G. MacDougall, J.M. Gracies, H.S. Cohen, W.G. Ondo, Long-term
start trial of rasagiline in Parkinson's disease (the ADAGIO study): prespecified and
monitoring of gait in Parkinson's disease, Gait Posture 26 (2007) 200–207, http://
post-hoc analyses of the need for additional therapies, changes in UPDRS scores,
dx.doi.org/10.1016/j.gaitpost.2006.09.011.
and non-motor outcomes, Lancet Neurol. 10 (2011) 415–423, http://dx.doi.org/
[32] M. B€achlin, M. Plotnik, D. Roggen, I. Maidan, J.M. Hausdorff, N. Giladi, G. Tr€ oster,
10.1016/S1474-4422(11)70073-4.
Wearable assistant for Parkinsons disease patients with the freezing of gait
[8] A. Haehner, T. Hummel, C. Hummel, U. Sommer, S. Junghanns, H. Reichmann,
symptom, IEEE Trans. Inf. Technol. Biomed. 14 (2010) 436–446, http://dx.doi.org/
Olfactory loss may be a first sign of idiopathic Parkinson's disease, Mov. Disord. 22
10.1109/TITB.2009.2036165.
(2007) 839–842, http://dx.doi.org/10.1002/mds.21413.
[33] S.T. Moore, H.G. MacDougall, W.G. Ondo, Ambulatory monitoring of freezing of
[9] C.G. Goetz, B.C. Tilley, S.R. Shaftman, G.T. Stebbins, S. Fahn, et al., Movement
gait in Parkinson's disease, J. Neurosci. Methods 167 (2008) 340–348, http://
disorder society-sponsored revision of the unified Parkinson's disease rating scale
dx.doi.org/10.1016/j.jneumeth.2007.08.023.
(MDS-UPDRS): scale presentation and clinimetric testing results, Mov. Disord. 23
[34] J. Spildooren, S. Vercruysse, K. Desloovere, W. Vandenberghe, E. Kerckhofs,
(2008) 2129–2170, http://dx.doi.org/10.1002/mds.22340.
A. Nieuwboer, Freezing of gait in Parkinson's disease: the impact of dual-tasking
[10] K. Yang, W.-X. Xiong, F.-T. Liu, Y.-M. Sun, S. Luo, Z.-T. Ding, J.-J. Wu, J. Wang,
Objective and quantitative assessment of motor function in Parkinson's

387
C. Ornelas-Vences et al. Computers in Biology and Medicine 89 (2017) 379–388

and turning, Mov. Disord. 25 (2010) 2563–2570, http://dx.doi.org/10.1002/ [39] E.L. Stack, A.M. Ashburn, K.E. Jupp, Strategies used by people with Parkinson's
mds.23327. disease who report difficulty turning, Park. Relat. Disord. 12 (2006) 87–92, http://
[35] E.E. Tripoliti, A.T. Tzallas, M.G. Tsipouras, G. Rigas, P. Bougia, M. Leontiou, dx.doi.org/10.1016/j.parkreldis.2005.08.008.
S. Konitsiotis, M. Chondrogiorgi, S. Tsouli, D.I. Fotiadis, Automatic detection of [40] J.E. Visser, N.C. Voermans, L.B.O. Nijhuis, M. van der Eijk, R. Nijk, M. Munneke,
freezing of gait events in patients with Parkinson's disease, Comput. Methods B.R. Bloem, Quantification of trunk rotations during turning and walking in
Programs Biomed. 110 (2013) 12–26, http://dx.doi.org/10.1016/ Parkinson's disease, Clin. Neurophysiol. 118 (2007) 1602–1606, http://dx.doi.org/
j.cmpb.2012.10.016. 10.1016/j.clinph.2007.03.010.
[36] M. Amboni, F. Stocchi, G. Abbruzzese, L. Morgante, M. Onofrj, S. Ruggieri, [41] Y. Higashi, K. Yamakoshi, T. Fujimoto, M. Sekine, T. Tamura, Quantitative
M. Tinazzi, M. Zappia, M. Attar, D. Colombo, L. Simoni, A. Ori, P. Barone, evaluation of movement using the timed up-and-go test, IEEE Eng, Med. Biol. Mag.
A. Antonini, Prevalence and associated features of self-reported freezing of gait in 27 (2008) 38–46, http://dx.doi.org/10.1109/MEMB.2008.919494.
Parkinson disease: the DEEP FOG study, Park. Relat. Disord. 21 (2015) 644–649, [42] A. Salarian, C. Zampieri, F.B. Horak, P. Carlson-Kuhta, J.G. Nutt, K. Aminian,
http://dx.doi.org/10.1016/j.parkreldis.2015.03.028. Analyzing 180 turns using an inertial system reveals early signs of progression of
[37] M. El-Gohary, S. Pearson, J. McNames, M. Mancini, F. Horak, S. Mellone, L. Chiari, Parkinson's disease, in: Proc. 31st Annu. Int. Conf. IEEE Eng. Med. Biol. Soc. Eng.
Continuous monitoring of turning in patients with movement disability, Sensors Futur. Biomed. EMBC 2009, IEEE, 2009, pp. 224–227, http://dx.doi.org/10.1109/
Switz. 14 (2014) 356–369, http://dx.doi.org/10.3390/s140100356. IEMBS.2009.5333970.
[38] E. Stack, K. Jupp, A. Ashburn, Developing methods to evaluate how people with [43] S.O.H. Madgwick, An Efficient Orientation Filter for Inertial and Inertial/magnetic
Parkinson's Disease turn 180 : an activity frequently associated with falls, Disabil. Sensor Arrays, Rep, X-Io Univ. …, 2010, p. 32, http://dx.doi.org/10.1109/
Rehabil. 26 (2004) 478–484, http://dx.doi.org/10.1080/ ICORR.2011.5975346.
09638280410001663085.

388

You might also like