A hybrid deep convolutional and recurrent neural network for complex activity recognition using multimodal sensors

Abstract Complex activities refer to users' activities performed in their daily lives (e.g., having dinner, shopping, etc.). Complex activity recognition is a valuable issue in wearable and mobile computing. The time-series sensory data from multimodal sensors have sophisticated relationships to characterize the complex activities (e.g., intra-sensor relationships, inter-sensor relationships, and temporal relationships), making the traditional methods based on manually designed features ineffective. To this end, we propose HConvRNN, an end-to-end deep neural network for complex activity recognition using multimodal sensors by integrating convolutional neural network (CNN) and recurrent neural network (RNN). To be specific, it uses a hierarchical CNN to exploit the intra-sensor relationships among similar sensors and merge intra-sensor relationships of different sensor modalities into inter-sensor relationships, and uses a RNN to model the temporal relationships of signal dynamics. The experiments based on real-world datasets show that HConvRNN outperforms the existing complex activity recognition methods.

[1]  Thomas Plötz,et al.  Deep, Convolutional, and Recurrent Models for Human Activity Recognition Using Wearables , 2016, IJCAI.

[2]  Nobuhiko Nishio,et al.  A recognition method for continuous gestures with an accelerometer , 2016, UbiComp Adjunct.

[3]  Bernt Schiele,et al.  Daily Routine Recognition through Activity Spotting , 2009, LoCA.

[4]  Zhijun Zhang,et al.  A New Varying-Parameter Recurrent Neural-Network for Online Solution of Time-Varying Sylvester Equation , 2018, IEEE Transactions on Cybernetics.

[5]  Mitja Lustrek,et al.  Recognition of high-level activities with a smartphone , 2015, UbiComp/ISWC Adjunct.

[6]  Davide Anguita,et al.  Transition-Aware Human Activity Recognition Using Smartphones , 2016, Neurocomputing.

[7]  Diane J. Cook,et al.  Simple and Complex Activity Recognition through Smart Phones , 2012, 2012 Eighth International Conference on Intelligent Environments.

[8]  Archan Misra,et al.  An exploration with online complex activity recognition using cellphone accelerometer , 2013, UbiComp.

[9]  Sung-Bae Cho,et al.  Deep Convolutional Neural Networks for Human Activity Recognition with Smartphone Sensors , 2015, ICONIP.

[10]  Thomas Plötz,et al.  Ensembles of Deep LSTM Learners for Activity Recognition using Wearables , 2017, Proc. ACM Interact. Mob. Wearable Ubiquitous Technol..

[11]  VALENTIN RADU,et al.  Multimodal Deep Learning for Activity and Context Recognition , 2018, Proc. ACM Interact. Mob. Wearable Ubiquitous Technol..

[12]  Jae-Young Pyun,et al.  Deep Recurrent Neural Networks for Human Activity Recognition , 2017, Sensors.

[13]  Seungjin Choi,et al.  Multi-modal Convolutional Neural Networks for Activity Recognition , 2015, 2015 IEEE International Conference on Systems, Man, and Cybernetics.

[14]  Tim Dallas,et al.  Feature Selection and Activity Recognition System Using a Single Triaxial Accelerometer , 2014, IEEE Transactions on Biomedical Engineering.

[15]  Bin Guo,et al.  Cyber-physical-social collaborative sensing: from single space to cross-space , 2017, Frontiers of Computer Science.

[16]  René Mayrhofer,et al.  Smartphone-Based Gait Recognition: From Authentication to Imitation , 2017, IEEE Transactions on Mobile Computing.

[17]  Marcus Edel,et al.  Binarized-BLSTM-RNN based Human Activity Recognition , 2016, 2016 International Conference on Indoor Positioning and Indoor Navigation (IPIN).

[18]  Takeshi Nishida,et al.  Deep recurrent neural network for mobile human activity recognition with high throughput , 2017, Artificial Life and Robotics.

[19]  Ming Liu,et al.  Sensor-based human activity recognition system with a multilayered model using time series shapelets , 2015, Knowl. Based Syst..

[20]  Keechul Jung,et al.  Recognition-based gesture spotting in video games , 2004, Pattern Recognit. Lett..

[21]  Sajal K. Das,et al.  A-Wristocracy: Deep learning on wrist-worn sensing for recognition of user complex activities , 2015, 2015 IEEE 12th International Conference on Wearable and Implantable Body Sensor Networks (BSN).

[22]  Gerhard Tröster,et al.  Discovery of activity composites using topic models: An analysis of unsupervised methods , 2014, Pervasive Mob. Comput..

[23]  Zhijun Zhang,et al.  A Varying-Parameter Convergent-Differential Neural Network for Solving Joint-Angular-Drift Problems of Redundant Robot Manipulators , 2018, IEEE/ASME Transactions on Mechatronics.

[24]  Tara N. Sainath,et al.  Convolutional, Long Short-Term Memory, fully connected Deep Neural Networks , 2015, 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[25]  Didier Stricker,et al.  Introducing a New Benchmarked Dataset for Activity Monitoring , 2012, 2012 16th International Symposium on Wearable Computers.

[26]  Ling Chen,et al.  Smartphone-based activity recognition independent of device orientation and placement , 2016, Int. J. Commun. Syst..

[27]  Daniel Roggen,et al.  Deep Convolutional and LSTM Recurrent Neural Networks for Multimodal Wearable Activity Recognition , 2016, Sensors.

[28]  Ling Chen,et al.  Bi-View Semi-Supervised Learning Based Semantic Human Activity Recognition Using Accelerometers , 2018, IEEE Transactions on Mobile Computing.

[29]  Seungjin Choi,et al.  Convolutional neural networks for human activity recognition using multiple accelerometer and gyroscope sensors , 2016, 2016 International Joint Conference on Neural Networks (IJCNN).

[30]  Xiaojie Wu,et al.  Hierarchical Complex Activity Representation and Recognition Using Topic Model and Classifier Level Fusion , 2017, IEEE Transactions on Biomedical Engineering.

[31]  Yuqing Chen,et al.  A Deep Learning Approach to Human Activity Recognition Based on Single Accelerometer , 2015, 2015 IEEE International Conference on Systems, Man, and Cybernetics.