Enhancing human activity recognition using deep learning and time series augmented data

Human activity recognition is concerned with detecting different types of human movements and actions using data gathered from various types of sensors. Deep learning approaches, when applied on time series data, offer promising results over intensive handcrafted feature extraction techniques that are highly reliant on the quality of defined domain parameters. In this paper, we investigate the benefits of time series data augmentation in improving the accuracy of several deep learning models on human activity data gathered from mobile phone accelerometers. More specifically, we compare the performance of the Vanilla, Long-Short Term Memory, and Gated Recurrent Units neural network models on three open-source datasets. We use two time series data augmentation techniques and study their impact on the accuracy of the target models. The experiments show that using gated recurrent units achieves the best results in terms of accuracy and training time followed by the long-short term memory technique. Furthermore, the results show that using data augmentation significantly enhances recognition quality.

[1]  Lei Zhang,et al.  DanHAR: Dual Attention Network For Multimodal Human Activity Recognition Using Wearable Sensors , 2020, Appl. Soft Comput..

[2]  Anthony G. Cohn,et al.  Egocentric Activity Monitoring and Recovery , 2012, ACCV.

[3]  Stefanos Zafeiriou,et al.  ArcFace: Additive Angular Margin Loss for Deep Face Recognition , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[4]  John E. Freund,et al.  Probability and statistics for engineers , 1965 .

[5]  P. F. Vasconcelos,et al.  In situ immune response and mechanisms of cell damage in central nervous system of fatal cases microcephaly by Zika virus , 2018, Scientific Reports.

[6]  Lei Zhang,et al.  The Layer-Wise Training Convolutional Neural Networks Using Local Loss for Sensor-Based Human Activity Recognition , 2020, IEEE Sensors Journal.

[7]  Venet Osmani,et al.  Human activity recognition in pervasive health-care: Supporting efficient remote collaboration , 2008, J. Netw. Comput. Appl..

[8]  Bo Yu,et al.  Convolutional Neural Networks for human activity recognition using mobile sensors , 2014, 6th International Conference on Mobile Computing, Applications and Services.

[9]  Mahmoud Al-Ayyoub,et al.  Bidirectional Gated Recurrent Units For Human Activity Recognition Using Accelerometer Data , 2019, 2019 IEEE SENSORS.

[10]  William Robson Schwartz,et al.  Latent HyperNet: Exploring the Layers of Convolutional Neural Networks , 2018, 2018 International Joint Conference on Neural Networks (IJCNN).

[11]  Andreas Stolcke,et al.  Recurrent neural network and LSTM models for lexical utterance classification , 2015, INTERSPEECH.

[12]  Claudio Savaglio,et al.  A body sensor data fusion and deep recurrent neural network-based behavior recognition approach for robust healthcare , 2020, Inf. Fusion.

[13]  Jae-Young Pyun,et al.  Deep Recurrent Neural Networks for Human Activity Recognition , 2017, Sensors.

[14]  Kevin K Dobbin,et al.  Optimally splitting cases for training and testing high dimensional classifiers , 2011, BMC Medical Genomics.

[15]  Davide Anguita,et al.  A Public Domain Dataset for Human Activity Recognition using Smartphones , 2013, ESANN.

[16]  Yan Liu,et al.  Recurrent Neural Networks for Multivariate Time Series with Missing Values , 2016, Scientific Reports.

[17]  Xiaohui Peng,et al.  Deep Learning for Sensor-based Activity Recognition: A Survey , 2017, Pattern Recognit. Lett..

[18]  Peter Stagge,et al.  Recurrent neural networks for time series classification , 2003, Neurocomputing.

[19]  Slim Abdennadher,et al.  Human Activity Recognition - Using Sensor Data of Smartphones and Smartwatches , 2016, ICAART.

[20]  Faouzi Alaya Cheikh,et al.  Stacked Lstm Network for Human Activity Recognition Using Smartphone Data , 2019, 2019 8th European Workshop on Visual Information Processing (EUVIP).

[21]  Peter Loos,et al.  Time Series Classification using Deep Learning for Process Planning: A Case from the Process Industry , 2017 .

[22]  Maria Trocan,et al.  Deep learning of smartphone sensor data for personal health assistance , 2018, Microelectron. J..

[23]  Hanyu Wang,et al.  LSTM-CNN Architecture for Human Activity Recognition , 2020, IEEE Access.

[24]  Zhaozheng Yin,et al.  Human Activity Recognition Using Wearable Sensors by Deep Convolutional Neural Networks , 2015, ACM Multimedia.

[25]  Nitish Srivastava,et al.  Dropout: a simple way to prevent neural networks from overfitting , 2014, J. Mach. Learn. Res..

[26]  Yoshua Bengio,et al.  Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling , 2014, ArXiv.

[27]  Yuwen Chen,et al.  LSTM Networks for Mobile Human Activity Recognition , 2016 .

[28]  Magnus Snorrason,et al.  Learning patterns of human activity for anomaly detection , 2007, SPIE Defense + Commercial Sensing.

[29]  Emanuele Frontoni,et al.  A sequential deep learning application for recognising human activities in smart homes , 2020, Neurocomputing.

[30]  Gary M. Weiss,et al.  Activity recognition using cell phone accelerometers , 2011, SKDD.

[31]  Thomas Plötz,et al.  Deep, Convolutional, and Recurrent Models for Human Activity Recognition Using Wearables , 2016, IJCAI.

[32]  Rose Qingyang Hu,et al.  Sensor-Based Human Activity Recognition for Smart Healthcare: A Semi-supervised Machine Learning , 2019, Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering.

[33]  Deba Prasad Dash,et al.  Hidden Markov Model based human activity recognition using shape and optical flow based features , 2016, 2016 IEEE Region 10 Conference (TENCON).

[34]  Debotosh Bhattacharjee,et al.  EnsemConvNet: a deep learning approach for human activity recognition using smartphone sensors for healthcare applications , 2020, Multimedia Tools and Applications.

[35]  Sang Min Yoon,et al.  Divide and Conquer-Based 1D CNN Human Activity Recognition Using Test Data Sharpening † , 2018, Sensors.

[36]  Sung-Bae Cho,et al.  Human activity recognition with smartphone sensors using deep learning neural networks , 2016, Expert Syst. Appl..

[37]  Giancarlo Fortino,et al.  A facial expression recognition system using robust face features from depth videos and deep learning , 2017, Comput. Electr. Eng..

[38]  Shuwan Xue,et al.  Portable Preimpact Fall Detector With Inertial Sensors , 2008, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[39]  Thomas George,et al.  An effective approach for human activity recognition on smartphone , 2015, 2015 IEEE International Conference on Engineering and Technology (ICETECH).

[40]  Daniela Micucci,et al.  On the Personalization of Classification Models for Human Activity Recognition , 2020, IEEE Access.

[41]  Daijin Kim,et al.  A Depth Video Sensor-Based Life-Logging Human Activity Recognition System for Elderly Care in Smart Indoor Environments , 2014, Sensors.

[42]  Nirmalya Roy,et al.  Recent trends in machine learning for human activity recognition—A survey , 2018, WIREs Data Mining Knowl. Discov..

[43]  C. Ha,et al.  Genetic inhibition of an ATP synthase subunit extends lifespan in C. elegans , 2018, Scientific Reports.

[44]  R. B. Woodruff,et al.  Know Your Customer: New Approaches to Understanding Customer Value and Satisfaction , 1996 .

[45]  Jianjun Xu,et al.  Deep Learning with Gated Recurrent Unit Networks for Financial Sequence Predictions , 2018 .

[46]  Ronald R. Yager,et al.  Time Series Smoothing and OWA Aggregation , 2008, IEEE Transactions on Fuzzy Systems.

[47]  Kuldip K. Paliwal,et al.  Bidirectional recurrent neural networks , 1997, IEEE Trans. Signal Process..

[48]  Mei Song,et al.  Margin-Based Deep Learning Networks for Human Activity Recognition , 2020, Sensors.

[49]  Jia-Ching Wang,et al.  Self-Gated Recurrent Neural Networks for Human Activity Recognition on Wearable Devices , 2017, ACM Multimedia.

[50]  Daniela Micucci,et al.  UniMiB SHAR: a new dataset for human activity recognition using acceleration data from smartphones , 2016, ArXiv.

[51]  A. Zakaria,et al.  Activity recognition using accelerometer sensor and machine learning classifiers , 2018, 2018 IEEE 14th International Colloquium on Signal Processing & Its Applications (CSPA).

[52]  Ahmad Almogren,et al.  A robust human activity recognition system using smartphone sensors and deep learning , 2018, Future Gener. Comput. Syst..

[53]  L. Klingbeil,et al.  Detecting walking activity in cardiac rehabilitation by using accelerometer , 2007, 2007 3rd International Conference on Intelligent Sensors, Sensor Networks and Information.

[54]  Yeng Chai Soh,et al.  Robust Human Activity Recognition Using Smartphone Sensors via CT-PCA and Online SVM , 2017, IEEE Transactions on Industrial Informatics.

[55]  Jozsef Suto,et al.  Human activity recognition using neural networks , 2014, Proceedings of the 2014 15th International Carpathian Control Conference (ICCC).

[56]  Ian Craddock,et al.  A Human Activity Recognition Framework for Healthcare Applications: Ontology, Labelling Strategies, and Best Practice , 2016, IoTBD.

[57]  Ying Wah Teh,et al.  Deep learning algorithms for human activity recognition using mobile and wearable sensor networks: State of the art and research challenges , 2018, Expert Syst. Appl..

[58]  Thinagaran Perumal,et al.  Activity recognition based on accelerometer sensor using combinational classifiers , 2015, 2015 IEEE Conference on Open Systems (ICOS).

[59]  Sajal K. Das,et al.  A-Wristocracy: Deep learning on wrist-worn sensing for recognition of user complex activities , 2015, 2015 IEEE 12th International Conference on Wearable and Implantable Body Sensor Networks (BSN).

[60]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[61]  Friedrich Foerster,et al.  Detection of posture and motion by accelerometry : a validation study in ambulatory monitoring , 1999 .

[62]  Cesar Torres-Huitzil,et al.  Accelerometer-Based Human Activity Recognition in Smartphones for Healthcare Services , 2015 .

[63]  Yoshua Bengio,et al.  Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation , 2014, EMNLP.

[64]  Andrew Zisserman,et al.  Two-Stream Convolutional Networks for Action Recognition in Videos , 2014, NIPS.

[65]  Guo-Jun Qi,et al.  Differential Recurrent Neural Networks for Action Recognition , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[66]  Rob J. Hyndman,et al.  Forecasting with Exponential Smoothing , 2008 .

[67]  M.A. Hanson,et al.  A Wearable Inertial Sensing Technology for Clinical Assessment of Tremor , 2007, 2007 IEEE Biomedical Circuits and Systems Conference.

[68]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[69]  Kimiaki Shirahama,et al.  Comparison of Feature Learning Methods for Human Activity Recognition Using Wearable Sensors , 2018, Sensors.

[70]  Jürgen Schmidhuber,et al.  Learning Precise Timing with LSTM Recurrent Networks , 2003, J. Mach. Learn. Res..

[71]  Guang-Zhong Yang,et al.  A Deep Learning Approach to on-Node Sensor Data Analytics for Mobile or Wearable Devices , 2017, IEEE Journal of Biomedical and Health Informatics.

[72]  Rob J Hyndman,et al.  Forecasting with Exponential Smoothing: The State Space Approach , 2008 .

[73]  Andreas Stolcke,et al.  A comparative study of recurrent neural network models for lexical domain classification , 2016, 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[74]  Yee Whye Teh,et al.  A Fast Learning Algorithm for Deep Belief Nets , 2006, Neural Computation.

[75]  Angelo M. Sabatini,et al.  Assessment of walking features from foot inertial sensing , 2005, IEEE Transactions on Biomedical Engineering.