Real-Time Human Activity Recognition System Based on Capsule and LoRa

Human activity recognition (HAR) has become a research hotspot in the field of artificial intelligence and pattern recognition. However, the HAR system still has some deficiencies in the aspects of platform algorithms and wireless access technologies. On the one hand, some state-of-the-art frameworks such as convolutional neural network (CNN) and recurrent neural network (RNN) have been proven successfully in classification tasks of HAR, while those frameworks just identify the feature data of activity but ignore the spatial relationship among features, which may lead to incorrect recognition. On the other hand, some existing transmission modes, such as Bluetooth and 4G, are difficult to realize real-time transmission in the case of a large range and low-power consumption. In this paper, a real-time human activity recognition system based on capsule and “long range” (LoRa) is presented, which pioneers the application of capsule to HAR. The capsule framework encapsulates the multiple convolution layers in parallel to solve the defect that current frameworks cannot identify the spatial relationship among features. Simultaneously, the combination of long-distance transmission and low-power consumption is achieved by using LoRa networking technology instead of other existing transmission modes. The experiments are performed on the dataset WISDM that is collected by the Wireless Sensor Data Mining Lab in Fordham University, and the results demonstrate that the proposed capsule framework achieves a higher classification result than CNN and RNN, and the proposed system makes the real-time HAR based on intelligent sensor devices possible in some special scenarios such as smart prison.

[1]  Jürgen Schmidhuber,et al.  Learning to Forget: Continual Prediction with LSTM , 2000, Neural Computation.

[2]  Serge J. Belongie,et al.  Behavior recognition via sparse spatio-temporal features , 2005, 2005 IEEE International Workshop on Visual Surveillance and Performance Evaluation of Tracking and Surveillance.

[3]  Maria Trocan,et al.  Personal Health Indicators by Deep Learning of Smart Phone Sensor Data , 2017, 2017 3rd IEEE International Conference on Cybernetics (CYBCON).

[4]  Billur Barshan,et al.  Comparative study on classifying human activities with miniature inertial and magnetic sensors , 2010, Pattern Recognit..

[5]  Yuqing Chen,et al.  A Deep Learning Approach to Human Activity Recognition Based on Single Accelerometer , 2015, 2015 IEEE International Conference on Systems, Man, and Cybernetics.

[6]  Stewart Massie,et al.  kNN Sampling for Personalised Human Activity Recognition , 2017, ICCBR.

[7]  Xiaohui Peng,et al.  Deep Learning for Sensor-based Activity Recognition: A Survey , 2017, Pattern Recognit. Lett..

[8]  Angelo M. Sabatini,et al.  Machine Learning Methods for Classifying Human Physical Activity from On-Body Accelerometers , 2010, Sensors.

[9]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[10]  Wen-Chang Cheng,et al.  Triaxial Accelerometer-Based Fall Detection Method Using a Self-Constructing Cascade-AdaBoost-SVM Classifier , 2013, IEEE Journal of Biomedical and Health Informatics.

[11]  Honglak Lee,et al.  Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations , 2009, ICML '09.

[12]  Hongnian Yu,et al.  A Data Fusion-Based Hybrid Sensory System for Older People’s Daily Activity and Daily Routine Recognition , 2018, IEEE Sensors Journal.

[13]  Subhas Chandra Mukhopadhyay,et al.  Wearable Sensors for Human Activity Monitoring: A Review , 2015, IEEE Sensors Journal.

[14]  Tahmina Zebin,et al.  Human activity recognition with inertial sensors using a deep learning approach , 2016, 2016 IEEE SENSORS.

[15]  Friedrich Foerster,et al.  Detection of posture and motion by accelerometry : a validation study in ambulatory monitoring , 1999 .

[16]  Xiaoli Li,et al.  Deep Convolutional Neural Networks on Multichannel Time Series for Human Activity Recognition , 2015, IJCAI.

[17]  Geoffrey E. Hinton,et al.  Using Deep Belief Nets to Learn Covariance Kernels for Gaussian Processes , 2007, NIPS.

[18]  Andrey Ignatov,et al.  Real-time human activity recognition from accelerometer data using Convolutional Neural Networks , 2018, Appl. Soft Comput..

[19]  Simon A. Dobson,et al.  KCAR: A knowledge-driven approach for concurrent activity recognition , 2015, Pervasive Mob. Comput..

[20]  Miguel A. Labrador,et al.  A Survey on Human Activity Recognition using Wearable Sensors , 2013, IEEE Communications Surveys & Tutorials.

[21]  Svetha Venkatesh,et al.  Activity recognition and abnormality detection with the switching hidden semi-Markov model , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[22]  Cheng Xu,et al.  InnoHAR: A Deep Neural Network for Complex Human Activity Recognition , 2019, IEEE Access.

[23]  Geoffrey E. Hinton,et al.  Dynamic Routing Between Capsules , 2017, NIPS.

[24]  Engin Erzin,et al.  Monitoring Infant's Emotional Cry in Domestic Environments Using the Capsule Network Architecture , 2018, INTERSPEECH.

[25]  Hermann Ney,et al.  LSTM Neural Networks for Language Modeling , 2012, INTERSPEECH.

[26]  Seungjin Choi,et al.  Convolutional neural networks for human activity recognition using multiple accelerometer and gyroscope sensors , 2016, 2016 International Joint Conference on Neural Networks (IJCNN).

[27]  Daniel Roggen,et al.  Deep Convolutional and LSTM Recurrent Neural Networks for Multimodal Wearable Activity Recognition , 2016, Sensors.

[28]  Ning Zhang,et al.  Learning-Aided User Identification Using Smartphone Sensors for Smart Homes , 2019, IEEE Internet of Things Journal.

[29]  Andrew W. Senior,et al.  Long short-term memory recurrent neural network architectures for large scale acoustic modeling , 2014, INTERSPEECH.

[30]  Roger K. Moore,et al.  Learning Capsules for Vehicle Logo Recognition , 2018, 2018 21st International Conference on Information Fusion (FUSION).

[31]  Basel Kikhia,et al.  Optimal Placement of Accelerometers for the Detection of Everyday Activities , 2013, Sensors.

[32]  K. V. N. Sunitha,et al.  DEEP RNN BASED HUMAN ACTIVITY RECOGNITION USING LSTM ARCHITECTURE ON SMARTPHONE SENSOR DATA , 2018 .

[33]  Yuwen Chen,et al.  LSTM Networks for Mobile Human Activity Recognition , 2016 .

[34]  Sang Min Yoon,et al.  Human activity recognition from accelerometer data using Convolutional Neural Network , 2017, 2017 IEEE International Conference on Big Data and Smart Computing (BigComp).

[35]  Sung-Bae Cho,et al.  Human activity recognition with smartphone sensors using deep learning neural networks , 2016, Expert Syst. Appl..

[36]  Cecilia Mascolo,et al.  Opportunities for smartphones in clinical care: the future of mobile mood monitoring. , 2016, The Journal of clinical psychiatry.

[37]  Petia Radeva,et al.  Human Activity Recognition from Accelerometer Data Using a Wearable Device , 2011, IbPRIA.

[38]  Nicholas D. Lane,et al.  From smart to deep: Robust activity recognition on smartwatches using deep learning , 2016, 2016 IEEE International Conference on Pervasive Computing and Communication Workshops (PerCom Workshops).

[39]  Timo Sztyler,et al.  Position-aware activity recognition with wearable devices , 2017, Pervasive Mob. Comput..

[40]  Qiang Yang,et al.  Sensor-Based Abnormal Human-Activity Detection , 2008, IEEE Transactions on Knowledge and Data Engineering.

[41]  David Howard,et al.  A Comparison of Feature Extraction Methods for the Classification of Dynamic Activities From Accelerometer Data , 2009, IEEE Transactions on Biomedical Engineering.

[42]  Tara N. Sainath,et al.  Deep Neural Networks for Acoustic Modeling in Speech Recognition: The Shared Views of Four Research Groups , 2012, IEEE Signal Processing Magazine.

[43]  Yang Jin,et al.  Capsule Network Performance on Complex Data , 2017, ArXiv.

[44]  Zebin Tahmina,et al.  Human activity recognition with inertial sensors using a deep learning approach , 2016 .

[45]  Olivier Seller,et al.  IoT: The era of LPWAN is starting now , 2016, ESSCIRC Conference 2016: 42nd European Solid-State Circuits Conference.

[46]  Shahrokh Valaee,et al.  A Survey on Behavior Recognition Using WiFi Channel State Information , 2017, IEEE Communications Magazine.

[47]  Bo Yu,et al.  Convolutional Neural Networks for human activity recognition using mobile sensors , 2014, 6th International Conference on Mobile Computing, Applications and Services.

[48]  Akhter Raza,et al.  Study of Multi-Classification of Advanced Daily Life Activities on SHIMMER Sensor Dataset , 2016, Int. J. Commun. Networks Inf. Secur..