Multitask LSTM Model for Human Activity Recognition and Intensity Estimation Using Wearable Sensor Data

Human activity recognition (HAR) and measuring the intensity of activity are increasingly important for healthcare applications, such as fitness tracking and patient monitoring. However, these two tasks have been performed separately, leading to delays and expensive implementations. In this article, we propose a holistic approach to achieve both HAR and intensity estimation simultaneously. We introduce a new data set with both activity types and activity intensities, and a multitask long short-term memory (LSTM) model to accurately classify the activity types and estimate the intensity of each activity. In addition, we evaluate our proposed neural network with other publicly available data sets and show that including activity intensities in the data set help multitask models perform comparably to two separate single-task models.

[1]  Long Wang,et al.  A Novel Human Activity Recognition Scheme for Smart Health Using Multilayer Extreme Learning Machine , 2019, IEEE Internet of Things Journal.

[2]  Nikolaos G. Bourbakis,et al.  A Survey on Wearable Sensor-Based Systems for Health Monitoring and Prognosis , 2010, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[3]  Klaas R. Westerterp,et al.  Assessment of physical activity: a critical appraisal , 2009, European Journal of Applied Physiology.

[4]  Björn Eskofier,et al.  Performance Comparison of Two Step Segmentation Algorithms Using Different Step Activities , 2014, 2014 11th International Conference on Wearable and Implantable Body Sensor Networks.

[5]  Scott E Crouter,et al.  A novel method for using accelerometer data to predict energy expenditure. , 2006, Journal of applied physiology.

[6]  Thomas Plötz,et al.  Deep, Convolutional, and Recurrent Models for Human Activity Recognition Using Wearables , 2016, IJCAI.

[7]  Zhen-Hua Ling,et al.  Enhancing and Combining Sequential and Tree LSTM for Natural Language Inference , 2016, ArXiv.

[8]  Thomas Plötz,et al.  Ensembles of Deep LSTM Learners for Activity Recognition using Wearables , 2017, Proc. ACM Interact. Mob. Wearable Ubiquitous Technol..

[9]  Yu Zhao,et al.  Deep Residual Bidir-LSTM for Human Activity Recognition Using Wearable Sensors , 2017, Mathematical Problems in Engineering.

[10]  John Staudenmayer,et al.  An artificial neural network to estimate physical activity energy expenditure and identify physical activity type from an accelerometer. , 2009, Journal of applied physiology.

[11]  Roozbeh Jafari,et al.  Enabling Effective Programming and Flexible Management of Efficient Body Sensor Network Applications , 2013, IEEE Transactions on Human-Machine Systems.

[12]  Paolo Fornacciari,et al.  IoT Wearable Sensor and Deep Learning: An Integrated Approach for Personalized Human Activity Recognition in a Smart Home Environment , 2019, IEEE Internet of Things Journal.

[13]  Ling Bao,et al.  Activity Recognition from User-Annotated Acceleration Data , 2004, Pervasive.

[14]  Davide Anguita,et al.  A Public Domain Dataset for Human Activity Recognition using Smartphones , 2013, ESANN.

[15]  Daniel Roggen,et al.  Deep Convolutional and LSTM Recurrent Neural Networks for Multimodal Wearable Activity Recognition , 2016, Sensors.

[16]  L B Sardinha,et al.  Comparison of equations for predicting energy expenditure from accelerometer counts in children , 2008, Scandinavian journal of medicine & science in sports.

[17]  Wei Lu,et al.  Wearable Computing for Internet of Things: A Discriminant Approach for Human Activity Recognition , 2019, IEEE Internet of Things Journal.

[18]  Ming Zeng,et al.  Understanding and improving recurrent networks for human activity recognition by continuous attention , 2018, UbiComp.

[19]  Li Fei-Fei,et al.  Jointly Learning Energy Expenditures and Activities Using Egocentric Multimodal Signals , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[20]  Majid Sarrafzadeh,et al.  Dynamic Computation Offloading for Low-Power Wearable Health Monitoring Systems , 2017, IEEE Transactions on Biomedical Engineering.

[21]  Thomas Plötz,et al.  On attention models for human activity recognition , 2018, UbiComp.

[22]  P S Freedson,et al.  Calibration of the Computer Science and Applications, Inc. accelerometer. , 1998, Medicine and science in sports and exercise.

[23]  Aydin Alatan,et al.  Geospatial object detection using deep networks , 2019, Optical Engineering + Applications.

[24]  Hassan Ghasemzadeh,et al.  Multi-sensor fusion in body sensor networks: State-of-the-art and research challenges , 2017, Inf. Fusion.

[25]  Giancarlo Fortino,et al.  A framework for collaborative computing and multi-sensor data fusion in body sensor networks , 2015, Inf. Fusion.

[26]  Marcus Edel,et al.  Binarized-BLSTM-RNN based Human Activity Recognition , 2016, 2016 International Conference on Indoor Positioning and Indoor Navigation (IPIN).