A spatiotemporal multi-feature extraction framework with space and channel based squeeze-and-excitation blocks for human activity recognition

Human activity recognition (HAR) is an active field in ubiquitous computing and body area network (BAN), which has been widely applied in medical care, sport and smart home. In recent years, a lot of methods based on deep learning show great performance on HAR. In consideration of the temporal and spatial dependencies of time series, the extracted features of traditional methods are not comprehensive. In this paper, we propose a new activity recognition framework based on spatiotemporal multi-feature extraction with space and channel based squeeze-and-excitation blocks (SCbSE-SMFE). The framework includes a temporal feature extraction layer composed of gated recurrent unit (GRU) blocks, a spatial feature extraction layer composed of convolutional neural networks (CNN) blocks with SCbSE blocks, a statistical feature extraction layer and an output layer. Meanwhile, regarding the actual needs for recognizing aggressive activities, we simulate the prison environment and collect an aggressive activity dataset (AAD). What’s more, aiming at the characteristics of aggressive activities, a threshold-based aggressive activity detection method is proposed to reduce the computational complexity. The proposed framework is evaluated on the public dataset WISDM and the collected dataset AAD, and the results prove that the proposed SCbSE-SMFE framework can effectively improve the accuracy and distinguish similar activities better. The proposed aggressive activity detection method based on threshold can simplify the model and improve the recognition speed while ensuring the recognition accuracy.

[1]  Shuo Wang,et al.  Overview of deep learning , 2016, 2016 31st Youth Academic Annual Conference of Chinese Association of Automation (YAC).

[2]  Nitish Srivastava,et al.  Dropout: a simple way to prevent neural networks from overfitting , 2014, J. Mach. Learn. Res..

[3]  Rahul Sukthankar,et al.  Violence Detection in Video Using Computer Vision Techniques , 2011, CAIP.

[4]  Cheng Xu,et al.  InnoHAR: A Deep Neural Network for Complex Human Activity Recognition , 2019, IEEE Access.

[5]  Yu Zhao,et al.  Deep Residual Bidir-LSTM for Human Activity Recognition Using Wearable Sensors , 2017, Mathematical Problems in Engineering.

[6]  Hong Qu,et al.  Deep Dilation on Multimodality Time Series for Human Activity Recognition , 2018, IEEE Access.

[7]  Peng Wang,et al.  A Time-Distributed Spatiotemporal Feature Learning Method for Machine Health Monitoring with Multi-Sensor Time Series , 2018, Sensors.

[8]  Abdulhamit Subasi,et al.  Sensor Based Human Activity Recognition Using Adaboost Ensemble Classifier , 2018 .

[9]  Donghui Guo,et al.  Improving the Generalization Ability of Deep Neural Networks for Cross-Domain Visual Recognition , 2020 .

[10]  Liming Chen,et al.  Combining ontological and temporal formalisms for composite activity modelling and recognition in smart homes , 2014, Future Gener. Comput. Syst..

[11]  Andrey Ignatov,et al.  Real-time human activity recognition from accelerometer data using Convolutional Neural Networks , 2018, Appl. Soft Comput..

[12]  Miguel A. Labrador,et al.  A Survey on Human Activity Recognition using Wearable Sensors , 2013, IEEE Communications Surveys & Tutorials.

[13]  Koushik Maharatna,et al.  Rehab-Net: Deep Learning Framework for Arm Movement Classification Using Wearable Sensors for Stroke Rehabilitation , 2019, IEEE Transactions on Biomedical Engineering.

[14]  Zhaozheng Yin,et al.  Human Activity Recognition Using Wearable Sensors by Deep Convolutional Neural Networks , 2015, ACM Multimedia.

[15]  Long Wang,et al.  A Novel Human Activity Recognition Scheme for Smart Health Using Multilayer Extreme Learning Machine , 2019, IEEE Internet of Things Journal.

[16]  Ning Zhang,et al.  Learning-Aided User Identification Using Smartphone Sensors for Smart Homes , 2019, IEEE Internet of Things Journal.

[17]  Sang Min Yoon,et al.  Divide and Conquer-Based 1D CNN Human Activity Recognition Using Test Data Sharpening † , 2018, Sensors.

[18]  Weiping Li,et al.  Review of Deep Learning , 2018, ArXiv.

[19]  Guang-Zhong Yang,et al.  A Deep Learning Approach to on-Node Sensor Data Analytics for Mobile or Wearable Devices , 2017, IEEE Journal of Biomedical and Health Informatics.

[20]  LuKun Wang,et al.  Human Activity Recognition Based on Wearable Sensor Using Hierarchical Deep LSTM Networks , 2019, Circuits, Systems, and Signal Processing.

[21]  Xiaohui Peng,et al.  Deep Learning for Sensor-based Activity Recognition: A Survey , 2017, Pattern Recognit. Lett..

[22]  Hanyu Wang,et al.  LSTM-CNN Architecture for Human Activity Recognition , 2020, IEEE Access.

[23]  Qian Huang,et al.  The Development of Artificial Intelligence (AI) Algorithms to Avoid Potential Baby Sleep Hazards in Smart Buildings , 2020 .

[24]  Jürgen Schmidhuber,et al.  Learning to Forget: Continual Prediction with LSTM , 2000, Neural Computation.

[25]  Haoxi Zhang,et al.  A Novel IoT-Perceptive Human Activity Recognition (HAR) Approach Using Multihead Convolutional Attention , 2020, IEEE Internet of Things Journal.

[26]  Belkacem Fergani,et al.  Integrating Prior Knowledge in Weighted SVM for Human Activity Recognition in Smart Home , 2017, ICOST.

[27]  Gary M. Weiss,et al.  Applications of mobile activity recognition , 2012, UbiComp.