Human Activity Recognition using accelerometers and a neural network

Matthew Robinson December 9th, 2013 Computer Engineering 358 - Computational Intelligence

Author: Matt Robinson is currently an undergraduate at Missouri University of Science and Technology. He will graduate with a Computer Engineering Bachelors degree in May 2014.

Table of Contents
I. Executive Summary ........................................................................... 1! II. Introduction ..................................................................................... 2! III. Project Specifications ...................................................................... 5! IV. Detailed Design ............................................................................... 6! V. Experimental Results ....................................................................... 9! References ........................................................................................... 11! Appendix ............................................................................................. 13! !

I. EXECUTIVE SUMMARY Human Activity Recognition (HAR) is the field that deals with recognizing an agents activity based upon observations of the agents actions. Sensors are becoming more robust lending to an increased use and pervasiveness of wearable sensors. This paper proposes using a neural network in conjunction with four wearable accelerometers to identify users activity classes such as walking, standing, sitting, sitting-down, and standing-up. The neural network used is a 12-X-5 multilayer perceptron (MLP), which has a variable hidden layer count to achieve optimal results. This best hidden layer neuron amount was determined to be 27 through iteration over many combinations and achieved an activity recognition accuracy of 93.43%.

II. INTRODUCTION Human Activity Recognition (HAR) is a field that deals with trying to recognize the actions of one or more agents from observations on the agents actions recorded with various sensors. These sensors can range from accelerometers [5], cameras [11], miniature inertial and magnetic sensors [7], and smart objects such as houses [12] or shoes [19]. Knowledge of these human actions and interactions can be used in a diverse range of applications, including assisting robotic interactions with humans, elderly awareness, athletic pursuits, healthcare, security, sports, and entertainment. The camera and image processing approach for identifying actions does not require the user to attach any equipment onto her body. But, the camera system has limitations such as having to be installed in all rooms, only working where supported cameras are placed, difficulty ensuring proper lighting/image quality, and convincing the user that their privacy will stay intact. Smart homes, aside from being few and far between, only allow the user to capture data when they are in their home, just as the camera system. The use of wearable sensors removes some of these limitations by ensuring that the user can be recording data wherever she intends on going without any installation process. But, the sensors are something that must be worn through extended periods of time. This causes other problems such as concerns about battery life, comfortableness, and attractiveness. The application of HAR to those in need of age or health related care has been an important research topic for many groups over the past few years [1]. The intention is to allow for the better care of the elderly without the need for more caregivers. An HAR system could be placed in the environment that an elderly person occupies in order to ensure their safety by knowing when they fall or become nonresponsive for extended periods of time by enabling remote monitoring and intervention. The data could also be used to analyze their action patterns to increase the healthiness of their day-to-day lives. The recent outbreak of smart phones, watches, and other devices that contain accelerometers and gyroscopes has probably been responsible for the intense growth of HAR research using wearable accelerometers [1]. This has encouraged the convergence of machine learning and pervasive computing. Both

these disciplines can be applied to a wide variety of uses due to the incorporation of these robust sensors into the every day lives of the users. It is feasible to use a single wrist-won accelerometer to accumulate enough data for activity recognition [5]. The activities recognized must be simple and different from the other activities to ensure a high success rate. The single accelerometer systems have issues separating similar tasks, such as sitting and standing depending on where the sensor is located, due to the fact that the movement data can only be correlated at that one point on the body. As the amount of sensors increases, the capability of the system to identify different activities also increases. Between the years of 2006 and 2012, 69 articles were published about body posture or activity recognition and accelerometers or acceleration with most works containing data gathered from four sensors. Due to the benefits and popularity, this paper has elected to use data gathered from a four-sensor system. Human activities can be classified in different ways depending on the application. HAR applied to metabolic energy expenditure estimation benefits from classifying activities in 4 different groups [4]. These groups are stationary, curling a weight, walking/running, and cycling. Once these activities are recognized, the information can be used as input to a different algorithm or network to get more useful information. Sitting, standing-up, standing, walking, and sitting-down are five different activity classes generally used in HAR research [1]. Due to the unavailability of a common dataset until recently, it was hard to compare the results of various methods. This was fixed partially when the HAR Dataset for benchmarking was released in 2012 by Groupware [1]. It contains data measured from 4 sensors correlated to the five different action classes mentioned before. Their aim was to give a data set to benchmark HAR algorithms against for easy comparison. HAR provides a specifically useful application for neural networks. As the amount of sensors and data increases, the patterns being recognized become more difficult and harder to form into concrete steps or linear solutions. Pattern recognition, which is the essential idea behind HAR, is perhaps the most common use for neural networks.

A problem one must answer before applying a neural network to HAR is how the data will be presented to the neural network. The sensors will be outputting instantaneous data consistently but using this isnt the best approach. Accuracy is greatly increased when the sensor data is cut into small time segments (observation windows) as opposed to using standard instantaneous input into the system [13]. Feature extraction from each window is the best approach. The data in this paper was gathered instantaneously but presented to the network in one-second windows to increase the accuracy. Applying a neural network to the field of human activity recognition with wearable accelerometers is the main focus of this paper.

III. PROJECT SPECIFICATIONS

The intention of this project is to construct a neural network to analyze data gathered from four different tri-axies accelerometer devices mounted on the waist, left thigh, right arm, and right ankle of a subject. The neural network will decide which of five specific activity classes her current state is exhibiting. These five activity classes include sitting, sitting-down, standing, standing-up, and walking. A set of training data will be used to determine the best parameters for the neural network in order to get the most accurate results. The goal is to create a system for understanding a users current activity that will be faster and computationally less expensive than a typical iterative approach to run the system on a device with small computational power, such as a microcontroller.

IV. DETAILED DESIGN Data Set and Collection The data used in this paper was recorded by four tri-axial ADXL335 accelerometers mounted onto the users body and connected to an ATmega328V microcontroller by Groupware [1]. The wearable and accelerometers positioning and orientation diagram are illustrated in Figure 1.

Figure 1. Wearable device for data collection [1]

The accelerometers were placed on the waist (1), left thigh (2), right ankle (3), and right arm (4). Before the data collection was done, all accelerometers were calibrated by observing the values returned from the sensors when sitting on a flat surface. These values were considered the zero conditions and then subtracted off of the measured results during data collection. The microcontroller performed data preprocessing on the accelerometer data. The resultant was derivate features of acceleration in axis x, y, and z. These are generated from a one second time window, with 150 ms overlapping. The sensors were attached to four healthy subjects (2 men and 2 women) for a total of eight hours of activities. This resulted in 165634 data records in total. The activities were classified between five different activity classes: sitting, sitting down, standing, standing up, and walking. The protocol was to perform each activity separately. The profile of each subject is shown in Table 1.

Table 1. Characteristics of the participants [1]

The data gathered from the experiment was assembled in a CSV file containing subject name, gender, age, how tall in meters, weight, body mass index, x1, y1, z1, x2, y2, z2, x3, y3, z3, x4, y4, z4, and class. For the purposes of this paper, the accelerations and class were extracted without any of the other data. This was done to simulate a dumb sensor that doesnt have any information about the user aside from the movement. There is potential value in correlating a subjects movements with the subjects information.

Learning Algorithm The scaled conjugate gradient (SCG) learning algorithm was used to update the weights in the neural network. This was chosen in place of the standard backpropogation (BP), the conjugate gradient backpropogation (CGB), and the Broyden-Fletcher-Goldfarb-Shanno memoryless quasi-Newton (BFGS) algorithm. SCG benefits over BP because it doesnt have any user dependent parameters. The fully automated nature of SCG allows it to avoid issues with large-scale problems and other problems that are highly dependent on user parameters learning rate and momentum constant. SCG exhibits speed improvements over CGB and BFGS because it avoids doing a time consuming line-search, which the other two compute in each iteration to determine an appropriate step size. SCG uses a scaling mechanism to calculate the step size instead. As another benefit, SCG is more capable at handling ravine phenomena than BP. These ravine phenomena are exhibited increasingly as the complexity of the neural network decreases with respect to the problem domain. Several experiments have found that SCG is at least an order of magnitude faster than BP, while potentially producing speed ups of over forty times [14].

Neural Network The neural network contained one input layer, one hidden layer, and one output layer. The network was constructed with 12 input neurons to take in the x, y, and z accelerations from each respective accelerometer sensor. The output layer had five output neurons to represent each of the five activity classes. The hidden layer of the network was a variable amount of neurons to decide the optimal number for this application. The scaled conjugate gradient (SCG) algorithm was used to adjust the weights in the network as mentioned above. The data set was split into three portions: 70% of the data was used to train the network, 15% was used to test the network, and 15% was used for validation during training. The validation data is to ensure the neural network doesnt get overtrained. The layout of the neural network is shown in Figure 2.

Figure 2. Neural network representation

V. EXPERIMENTAL RESULTS Hidden Layer Optimization The number of neurons in the hidden layer was tested from 5 to 40 to find the optimal amount. The network was trained for each neuron amount three times and then tested. The accuracy of the test runs was averaged for each number of hidden neurons. The hidden layer neuron amount with the highest average accuracy was determined to be the optimal amount. The optimal number of neurons for the hidden layer was found to be 27. The results for the hidden layer neuron optimization can be seen in Figure 3.

Hidden Layer Neuron Count Calculation

'"! '&! '%! '$! '#! ""! "&! "%! "$! "#! (! )! *! +! '! ((! ()! (*! (+! ('! $(! $)! $*! $+! $'! )(! ))! )*! Neuron # Accuracy (%)

Figure 3. Hidden layer count calculation

Activity Recognition Results After finding the optimal amount of hidden neurons, the network was both trained and ran on the test data eight separate times. The accuracies of these eight trials were averaged to find that the accuracy of the neural network was 93.43%. The results of the trials can be seen in Figure 4. The Matlab code used to parse the data input and create a neural network can be found in the Appendix.

Activity Recognization Accuracy

($#! (##! Accuracy (%) "#! &#! %#! $#! #! ,-..-/01234/! ,.5/2-/0167! ,.5/2-/0! Activity 4589-/0! ,-..-/0!

Figure 4. Activity recognition accuracy

The neural network had the greatest difficulty identifying the in-between stages of the activity class, sitting down and standing up. This is common amongst many of the papers dealing with HAR because of the similarity in the data with other activity classes. Ignoring the first two activity classes in Figure 4, the recognition rate for the standing, walking, and sitting activity classes was 96.03%. The distribution of the error and where the network has difficulties can be seen in greater detail in the confusion matrix in Figure 5.

Figure 5. Confusion matrix for 12-27-5 MLP!

REFERENCES
[1] Ugulino, W.; Cardador, C.; Vega, K.; Velloso, E.; Milidiu, R.; Fuks, H., Wearable Computing: Accelerometers Data Classification of Body Postures and st Movements, Proceedings of 21 Notes in Computer Science, pp. 52-61, Curitiba, PR: Springer Berlin / Heidelberg, 2012. Hongqing Fang; Lei He, "BP Neural Network for Human Activity Recognition in Smart Home," Computer Science & Service System (CSSS), 2012 International Conference on , vol., no., pp.1034,1037, 11-13 Aug. 2012. Rednic, R.; Gaura, E.; Kemp, J.; Brusey, J., "Fielded Autonomous Posture Classification Systems: Design and Realistic Evaluation," Software Engineering, Artificial Intelligence, Networking and Parallel/Distributed Computing (SNPD), 2013 14th ACIS International Conference on , vol., no., pp.635,640, 1-3 July 2013. Bo Dong; Biswas, S.; Montoye, A.; Pfeiffer, K., "Comparing metabolic energy expenditure estimation using wearable multi-sensor network and single accelerometer," Engineering in Medicine and Biology Society (EMBC), 2013 35th Annual International Conference of the IEEE , vol., no., pp.2866,2869, 3-7 July 2013. Chernbumroong, S.; Atkins, A.S.; Hongnian Yu, "Activity classification using a single wrist-worn accelerometer," Software, Knowledge Information, Industrial Management and Applications (SKIMA), 2011 5th International Conference on , vol., no., pp.1,6, 8-11 Sept. 2011. Ahsan, M.R.; Ibrahimy, M.I.; Khalifa, O.O., "Electromygraphy (EMG) signal based hand gesture recognition using artificial neural network (ANN)," Mechatronics (ICOM), 2011 4th International Conference On , vol., no., pp.1,6, 17-19 May 2011. Yuksek, M.C.; Barshan, B., "Human activity classification with miniature inertial and magnetic sensors," Signal Processing and Communications Applications (SIU), 2011 IEEE 19th Conference on , vol., no., pp.1052,1055, 20-22 April 2011. Zhang, G.P., "Avoiding Pitfalls in Neural Network Research," Systems, Man, and Cybernetics, Part C: Applications and Reviews, IEEE Transactions on , vol.37, no.1, pp.3,16, Jan. 2007. Yoonseon Song; Seungchul Shin; Seunghwan Kim; Doheon Lee; Lee, K.H., "Speed Estimation From a Tri-axial Accelerometer Using Neural Networks," Engineering in Medicine and Biology Society, 2007. EMBS 2007. 29th Annual International Conference of the IEEE , vol., no., pp.3224,3227, 22-26 Aug. 2007. Edgar, S.R.; Fulk, G.D.; Sazonov, E.S., "Recognition of household and athletic activities using smartshoe," Engineering in Medicine and Biology Society (EMBC), 2012 Annual International Conference of the IEEE , vol., no., pp.6382,6385, Aug. 28 2012-Sept. 1 2012.

[2]

[3]

[4]

[5]

[6]

[7]

[8]

[9]

[10]

[11]

[12]

[13]

[14]

Dragan, M.-A.; Mocanu, I., "Human Activity Recognition in Smart Environments," Control Systems and Computer Science (CSCS), 2013 19th International Conference on , vol., no., pp.495,502, 29-31 May 2013. Huiru Zheng; Haiying Wang; Black, Norman, "Human Activity Detection in Smart Home Environment with Self-Adaptive Neural Networks," Networking, Sensing and Control, 2008. ICNSC 2008. IEEE International Conference on , vol., no., pp.1505,1510, 6-8 April 2008. Zhelong Wang; Ming Jiang; Yaohua Hu; Hongyi Li, "An Incremental Learning Method Based on Probabilistic Neural Networks and Adjustable Fuzzy Clustering for Human Activity Recognition by Using Wearable Sensors," Information Technology in Biomedicine, IEEE Transactions on , vol.16, no.4, pp.691,699, July 2012. Mller, Martin Fodslette. "A scaled conjugate gradient algorithm for fast supervised learning." Neural networks 6.4 (1993): 525-533.

APPENDIX
csv_to_var.m
% Read dataset file into Matlab structure % Author: Matt Robinson % Date: 12-09-13 fileID = fopen('dataset-har-PUC-Rio-ugulino_smaller.csv'); C = textscan(fileID,'%s %s %f %s %f %s %f %f %f %f %f %f %f %f %f %f %f %f %s','delimiter',';'); fclose(fileID); %user ;gender;age; how_tall_in_meters;weight;body_mass_index; %debora;Woman ;46 ; 1,62 ;75 ;28,6 ; %x1;y1;z1 ;x2 ;y2;z2 ;x3;y3 ;z3 ;x4 ;y4 ;z4 ;class %-3;92;-63;-23;18;-19;5 ;104;-92;-150;-103;-147;sitting % user = C{1} % gender = C{2} % age = C{3} % tall = C{4} % weight = C{5} % massindex = C{6} x1 = C{7}; y1 = C{8}; x2 = C{10}; y2 = C{11}; x3 = C{13}; y3 = C{14}; x4 = C{16}; y4 = C{17}; class = C{19};

z1 z2 z3 z4

= = = =

C{9}; C{12}; C{15}; C{18};

temp = zeros(length(class),5); for n = 1:length(class) if( strcmp( class(n,1), 'sittingdown' ) ) temp(n,1) = 1; elseif( strcmp( class(n,1), 'standingup' ) ) temp(n,2) = 1; elseif( strcmp( class(n,1), 'standing' ) ) temp(n,3) = 1; elseif( strcmp( class(n,1), 'walking' ) ) temp(n,4) = 1; elseif( strcmp( class(n,1), 'sitting' ) ) temp(n,5) = 1; end end; nn_input_mat = horzcat(x1, y1, z1, x2, y2, z2, x3, y3, z3, x4, y4, z4); nn_output_mat = temp;

nn_iterator_script.m
% % % % % % % % Solve a Pattern Recognition Problem with a Neural Network Author: Matt Robinson Date: 12-09-13 This script assumes these variables are defined: nn_input_mat - input data. nn_output_mat - target data.

for i = 5:40 inputs = nn_input_mat'; targets = nn_output_mat'; % Create a Pattern Recognition Network hiddenLayerSize = i; net = patternnet(hiddenLayerSize); % Setup Division of Data for Training, Validation, Testing net.divideParam.trainRatio = 70/100; net.divideParam.valRatio = 15/100; net.divideParam.testRatio = 15/100; % Train the Network [net,tr] = train(net,inputs,targets); % Test the Network outputs = net(inputs); errors = gsubtract(targets,outputs); performance = perform(net,targets,outputs); % View the Network view(net); % Plots % Uncomment these lines to enable various plots. %figure, plotperform(tr) perform_plot = plotperform(tr); output_p_file = sprintf('12-%d-55_perform_%d_final.bmp',hiddenLayerSize, i); print(perform_plot,'-dbmp',output_p_file); %figure, plottrainstate(tr) %figure, plotconfusion(targets,outputs) confusion_plot = plotconfusion(targets,outputs); output_c_file = sprintf('12-%d-55_confus_%d_final.bmp',hiddenLayerSize, i); print(confusion_plot,'-dbmp',output_c_file); %figure, ploterrhist(errors) end;