Recent Progress in Sensing and Computing Techniques for Human Activity Recognition and Motion Analysis

The recent scientific and technical advances in Internet of Things (IoT) based pervasive sensing and computing have created opportunities for the continuous monitoring of human activities for different purposes. The topic of human activity recognition (HAR) and motion analysis, due to its potentiality in human–machine interaction (HMI), medical care, sports analysis, physical rehabilitation, assisted daily living (ADL), children and elderly care, has recently gained increasing attention. The emergence of some novel sensing devices featuring miniature size, a light weight, and wireless data transmission, the availability of wireless communication infrastructure, the progress of machine learning and deep learning algorithms, and the widespread IoT applications has promised new opportunities for a significant progress in this particular field. Motivated by a great demand for HAR-related applications and the lack of a timely report of the recent contributions to knowledge in this area, this investigation aims to provide a comprehensive survey and in-depth analysis of the recent advances in the diverse techniques and methods of human activity recognition and motion analysis. The focus of this investigation falls on the fundamental theories, the innovative applications with their underlying sensing techniques, data fusion and processing, and human activity classification methods. Based on the state-of-the-art, the technical challenges are identified, and future perspectives on the future rich, sensing, intelligent IoT world are given in order to provide a reference for the research and practices in the related fields.

[1]  David Howard,et al.  A Comparison of Feature Extraction Methods for the Classification of Dynamic Activities From Accelerometer Data , 2009, IEEE Transactions on Biomedical Engineering.

[2]  Eduardo Souto,et al.  A Smartphone Lightweight Method for Human Activity Recognition Based on Information Theory , 2020, Sensors.

[3]  Nasir Rashid,et al.  Human activity recognition using 2D skeleton data and supervised machine learning , 2019, IET Image Process..

[4]  Andreas W. Kempa-Liehr,et al.  Time Series FeatuRe Extraction on basis of Scalable Hypothesis tests (tsfresh - A Python package) , 2018, Neurocomputing.

[5]  Joan Condell,et al.  IMU Sensor-Based Electronic Goniometric Glove for Clinical Finger Movement Analysis , 2018, IEEE Sensors Journal.

[6]  Luciano Lavagno,et al.  Neural Networks for Indoor Human Activity Reconstructions , 2020, IEEE Sensors Journal.

[7]  Nicholas P. Fey,et al.  Lower Limb Motion Estimation Using Ultrasound Imaging: A Framework for Assistive Device Control , 2019, IEEE Journal of Biomedical and Health Informatics.

[8]  Valentin Peretroukhin,et al.  Robust Data-Driven Zero-Velocity Detection for Foot-Mounted Inertial Navigation , 2020, IEEE Sensors Journal.

[9]  Zhelong Wang,et al.  Using Wearable Sensors to Capture Posture of the Human Lumbar Spine in Competitive Swimming , 2019, IEEE Transactions on Human-Machine Systems.

[10]  Mohammad Ghavami,et al.  Non-Contact Human Gait Identification Through IR-UWB Edge-Based Monitoring Sensor , 2019, IEEE Sensors Journal.

[11]  Daqing Zhang,et al.  RT-Fall: A Real-Time and Contactless Fall Detection System with Commodity WiFi Devices , 2017, IEEE Transactions on Mobile Computing.

[12]  Jonathan Rodriguez,et al.  SmartWall: Novel RFID-Enabled Ambient Human Activity Recognition Using Machine Learning for Unobtrusive Health Monitoring , 2019, IEEE Access.

[13]  Avik Santra,et al.  Continuous Human Activity Classification With Unscented Kalman Filter Tracking Using FMCW Radar , 2020, IEEE Sensors Letters.

[14]  Paola Pierleoni,et al.  A Wireless Body Sensor Network for Clinical Assessment of the Flexion-Relaxation Phenomenon , 2020, Electronics.

[15]  Weihua Sheng,et al.  Wearable Sensor-Based Hand Gesture and Daily Activity Recognition for Robot-Assisted Living , 2011, IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans.

[16]  Yunyoung Nam,et al.  Analysis of Machine Learning-Based Assessment for Elbow Spasticity Using Inertial Sensors , 2020, Sensors.

[17]  I-Ming Chen,et al.  Localization and velocity tracking of human via 3 IMU sensors , 2014 .

[18]  Robert Bergevin,et al.  Semantic human activity recognition: A literature review , 2015, Pattern Recognit..

[19]  Xiaoxiang Zhou,et al.  Wearable health monitoring system based on human motion state recognition , 2020, Comput. Commun..

[20]  Domenico Prattichizzo,et al.  Upper Body Pose Estimation Using Wearable Inertial Sensors and Multiplicative Kalman Filter , 2019, IEEE Sensors Journal.

[21]  Fan Zhang Human–Computer Interactive Gesture Feature Capture and Recognition in Virtual Reality: , 2020 .

[22]  Wanxin Xu,et al.  Human body reshaping and its application using multiple RGB-D sensors , 2019, Signal Process. Image Commun..

[23]  Yuanjin Zheng,et al.  An FMCW Radar Transceiver Chip for Object Positioning and Human Limb Motion Detection , 2017, IEEE Sensors Journal.

[24]  Guillermo Cámara Chávez,et al.  Multimodal hand gesture recognition combining temporal and pose information based on CNN descriptors and histogram of cumulative magnitudes , 2020, J. Vis. Commun. Image Represent..

[25]  Luis González Abril,et al.  Mobile activity recognition and fall detection system for elderly people using Ameva algorithm , 2017, Pervasive Mob. Comput..

[26]  Chunyu Li,et al.  Learning skeleton information for human action analysis using Kinect , 2020, Signal Process. Image Commun..

[27]  Tajana Rosing,et al.  Human Behavior Aware Energy Management in Residential Cyber-Physical Systems , 2020, IEEE Transactions on Emerging Topics in Computing.

[28]  Qingguo Li,et al.  Two Shank-Mounted IMUs-Based Gait Analysis and Classification for Neurological Disease Patients , 2020, IEEE Robotics and Automation Letters.

[29]  Tengyue Li,et al.  Fusing wearable and remote sensing data streams by fast incremental learning with swarm decision table for human activity recognition , 2020, Inf. Fusion.

[30]  John L. Crassidis,et al.  Hip and Trunk Kinematics Estimation in Gait Through Kalman Filter Using IMU Data at the Ankle , 2018, IEEE Sensors Journal.

[31]  Chi Cuong Vu,et al.  Human Motion Recognition by Textile Sensors Based on Machine Learning Algorithms , 2018, Sensors.

[32]  Yichun Liu,et al.  Flexible, high-sensitive, and wearable strain sensor based on organic crystal for human motion detection , 2018, Organic Electronics.

[33]  Nima Enayati,et al.  A Quaternion-Based Unscented Kalman Filter for Robust Optical/Inertial Motion Tracking in Computer-Assisted Surgery , 2015, IEEE Transactions on Instrumentation and Measurement.

[34]  Paolo Fornacciari,et al.  IoT Wearable Sensor and Deep Learning: An Integrated Approach for Personalized Human Activity Recognition in a Smart Home Environment , 2019, IEEE Internet of Things Journal.

[35]  Xiaohua Zhu,et al.  Non-Contact Human Motion Recognition Based on UWB Radar , 2018, IEEE Journal on Emerging and Selected Topics in Circuits and Systems.

[36]  Martin Schönfelder,et al.  In-Ear Accelerometer-Based Sensor for Gait Classification , 2020, IEEE Sensors Journal.

[37]  Kenneth Meijer,et al.  Activity identification using body-mounted sensors—a review of classification techniques , 2009, Physiological measurement.

[38]  Hideo Saito,et al.  Gait Measurement at Home Using A Single RGB Camera. , 2019, Gait & posture.

[39]  Yufeng Shu,et al.  Interactive design of intelligent machine vision based on human-computer interaction mode , 2020, Microprocess. Microsystems.

[40]  Slimane Larabi,et al.  User-independent system for sign language finger spelling recognition , 2014, J. Vis. Commun. Image Represent..

[41]  Mohammed Sadgal,et al.  Skeleton-based human activity recognition for elderly monitoring systems , 2018, IET Comput. Vis..

[42]  Hazem Wannous,et al.  Heterogeneous hand gesture recognition using 3D dynamic skeletal data , 2019, Comput. Vis. Image Underst..

[43]  Seung Ju Han,et al.  Wireless Epidermal Six-Axis Inertial Measurement Units for Real-Time Joint Angle Estimation , 2020 .

[44]  EMMA VILLENEUVE,et al.  Reconstruction of Angular Kinematics From Wrist-Worn Inertial Sensor Data for Smart Home Healthcare , 2017, IEEE Access.

[45]  Francesco Fioranelli,et al.  Continuous Human Activity Classification From FMCW Radar With Bi-LSTM Networks , 2020, IEEE Sensors Journal.

[46]  Subhas Chandra Mukhopadhyay,et al.  Wearable Sensors for Human Activity Monitoring: A Review , 2015, IEEE Sensors Journal.

[47]  Nathaniel Rossol,et al.  A Multisensor Technique for Gesture Recognition Through Intelligent Skeletal Pose Analysis , 2016, IEEE Transactions on Human-Machine Systems.

[48]  Yu Guan,et al.  Deep Learning for Human Activity Recognition in Mobile Computing , 2018, Computer.

[49]  N. Troje,et al.  Objectively Differentiating Movement Patterns between Elite and Novice Athletes , 2018, Medicine and science in sports and exercise.

[50]  Cheng Xu,et al.  InnoHAR: A Deep Neural Network for Complex Human Activity Recognition , 2019, IEEE Access.

[51]  Noureddine Manamanni,et al.  Complementary Observer for Body Segments Motion Capturing by Inertial and Magnetic Sensors , 2014, IEEE/ASME Transactions on Mechatronics.

[52]  Hairui Jia,et al.  Integrated data and knowledge driven methodology for human activity recognition , 2020, Inf. Sci..

[53]  Minho Lee,et al.  Human motion based intent recognition using a deep dynamic neural model , 2015, Robotics Auton. Syst..

[54]  Sunghoon Kim,et al.  Golf swing analysis system with a dual band and motion analysis algorithm , 2017, IEEE Transactions on Consumer Electronics.

[55]  G. Anitha,et al.  Posture based health monitoring and unusual behavior recognition system for elderly using dynamic Bayesian network , 2018, Cluster Computing.

[56]  Alberto Del Bimbo,et al.  Motion segment decomposition of RGB-D sequences for human behavior understanding , 2017, Pattern Recognit..

[57]  Diego H. Milone,et al.  Assessment of Homomorphic Analysis for Human Activity Recognition From Acceleration Signals , 2018, IEEE Journal of Biomedical and Health Informatics.

[58]  Thomas Seel,et al.  Alignment-free, self-calibrating elbow angles measurement using inertial sensors , 2017, 2016 IEEE-EMBS International Conference on Biomedical and Health Informatics (BHI).

[59]  Daniela Micucci,et al.  On the Personalization of Classification Models for Human Activity Recognition , 2020, IEEE Access.

[60]  Patrick Boissy,et al.  Using Inertial Sensors to Automatically Detect and Segment Activities of Daily Living in People With Parkinson’s Disease , 2018, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[61]  Jesse Hoey,et al.  Sensor-Based Activity Recognition , 2012, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[62]  Francesco Fioranelli,et al.  Bi-LSTM Network for Multimodal Continuous Human Activity Recognition and Fall Detection , 2020, IEEE Sensors Journal.

[63]  Reza Vahidnia,et al.  Wearables and the Internet of Things (IoT), Applications, Opportunities, and Challenges: A Survey , 2020, IEEE Access.

[64]  Kai-Hsiang Ke,et al.  Monitoring of Large-Area IoT Sensors Using a LoRa Wireless Mesh Network System: Design and Evaluation , 2018, IEEE Transactions on Instrumentation and Measurement.

[65]  Javed Imran,et al.  Combining CNN streams of RGB-D and skeletal data for human activity recognition , 2018, Pattern Recognit. Lett..

[66]  Kamiar Aminian,et al.  Gaussian process framework for pervasive estimation of swimming velocity with body-worn IMU , 2013 .

[67]  Reynald Hoskinson,et al.  A cascaded Kalman filter-based GPS/MEMS-IMU integration for sports applications , 2015 .

[68]  Doik Kim,et al.  Flexible Piezoelectric Sensor-Based Gait Recognition , 2018, Sensors.

[69]  Yixiang Lim,et al.  Experimental characterisation of eye-tracking sensors for adaptive human-machine systems , 2019, Measurement.

[70]  Hejun Wu,et al.  Real-Time Continuous Action Recognition Using Pose Contexts With Depth Sensors , 2018, IEEE Access.

[71]  Guangyan Huang,et al.  A novel Nest-based scheduling method for mobile wireless body area networks , 2020 .

[72]  Yu-Liang Hsu,et al.  Human Daily and Sport Activity Recognition Using a Wearable Inertial Sensor Network , 2018, IEEE Access.

[73]  Zong Woo Geem,et al.  A Novel Human Respiration Pattern Recognition Using Signals of Ultra-Wideband Radar Sensor , 2019, Sensors.

[74]  Miguel A. Labrador,et al.  A Survey on Human Activity Recognition using Wearable Sensors , 2013, IEEE Communications Surveys & Tutorials.

[75]  Paolo Dario,et al.  A Human Activity Recognition System Based on Dynamic Clustering of Skeleton Data , 2017, Sensors.

[76]  Chin-Ling Chen,et al.  Accurate Human Gesture Sensing With Coarse-Grained RF Signatures , 2019, IEEE Access.

[77]  Meng Chen,et al.  IoT for Next-Generation Racket Sports Training , 2018, IEEE Internet of Things Journal.

[78]  Ioannis A. Kakadiaris,et al.  A Review of Human Activity Recognition Methods , 2015, Front. Robot. AI.

[79]  John McAllister,et al.  Real-Time Embedded EMG Signal Analysis for Wrist-Hand Pose Identification , 2020, IEEE Transactions on Signal Processing.

[80]  Zhaofeng Li,et al.  Adaptive EKF Based on HMM Recognizer for Attitude Estimation Using MEMS MARG Sensors , 2018, IEEE Sensors Journal.

[81]  C. Nicol,et al.  Classification of Phantom Finger, Hand, Wrist, and Elbow Voluntary Gestures in Transhumeral Amputees With sEMG , 2017, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[82]  Rezvan Kianifar,et al.  Automated Assessment of Dynamic Knee Valgus and Risk of Knee Injury During the Single Leg Squat , 2017, IEEE Journal of Translational Engineering in Health and Medicine.

[83]  Wei Gong,et al.  Multi-Adversarial In-Car Activity Recognition Using RFIDs , 2020, IEEE Transactions on Mobile Computing.

[84]  S. J. M. Bamberg,et al.  A Wireless Sensory Feedback Device for Real-Time Gait Feedback and Training , 2012, IEEE/ASME Transactions on Mechatronics.

[85]  Koushik Maharatna,et al.  Rehab-Net: Deep Learning Framework for Arm Movement Classification Using Wearable Sensors for Stroke Rehabilitation , 2019, IEEE Transactions on Biomedical Engineering.

[86]  Zhichao Cao,et al.  WiHF: Gesture and User Recognition With WiFi , 2022, IEEE Transactions on Mobile Computing.

[87]  Xiaorong Guan,et al.  Estimation of Knee Movement from Surface EMG Using Random Forest with Principal Component Analysis , 2019, Electronics.

[88]  Yan Luo,et al.  Multitask LSTM Model for Human Activity Recognition and Intensity Estimation Using Wearable Sensor Data , 2020, IEEE Internet of Things Journal.

[89]  Shilei Lv,et al.  A light-weight on-line action detection with hand trajectories for industrial surveillance , 2020 .

[90]  Shui Yu,et al.  WiReader: Adaptive Air Handwriting Recognition Based on Commercial WiFi Signal , 2020, IEEE Internet of Things Journal.

[91]  Zhihua Wang,et al.  Efficient High Cross-User Recognition Rate Ultrasonic Hand Gesture Recognition System , 2020, IEEE Sensors Journal.

[92]  Bernt Schiele,et al.  Weakly Supervised Recognition of Daily Life Activities with Wearable Sensors , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[93]  Chen Feng,et al.  Upper limb motion tracking with the integration of IMU and Kinect , 2015, Neurocomputing.

[94]  Yue Liu,et al.  In-Air Handwriting by Passive Gesture Tracking Using Commodity WiFi , 2020, IEEE Communications Letters.

[95]  João Gama,et al.  Human Activity Recognition Using Inertial Sensors in a Smartphone: An Overview , 2019, Sensors.

[96]  Matthias Pätzold,et al.  A Machine Learning Approach for Fall Detection and Daily Living Activity Recognition , 2019, IEEE Access.

[97]  Yanjun Xiao,et al.  Cost-Effective Wearable Indoor Localization and Motion Analysis via the Integration of UWB and IMU , 2020, Sensors.

[98]  Sandro Fioretti,et al.  Recognition of Gait Phases with a Single Knee Electrogoniometer: A Deep Learning Approach , 2020, Electronics.

[99]  Quan Pan,et al.  Whole-Body Pose Estimation in Human Bicycle Riding Using a Small Set of Wearable Sensors , 2016, IEEE/ASME Transactions on Mechatronics.

[100]  Amitava Chatterjee,et al.  Recognition of Human Behavior for Assisted Living Using Dictionary Learning Approach , 2018, IEEE Sensors Journal.