A Survey of Vision-Based Transfer Learning in Human Activity Recognition

Human activity recognition (HAR) and transfer learning (TL) are two broad areas widely studied in computational intelligence (CI) and artificial intelligence (AI) applications. Much effort has been put into developing suitable solutions to advance the current performance of existing systems. However, challenges are facing the existing methods of HAR. In HAR, the variations in data required in HAR systems pose challenges to many existing solutions. The type of sensory information used could play an important role in overcoming some of these challenges. Vision-based information in 3D acquired using RGB-D cameras is one type. Furthermore, with the successes encountered in TL, HAR stands to benefit from TL to address challenges to existing methods. Therefore, it is important to review the current state-of-the-art related to both areas. This paper presents a comprehensive survey of vision-based HAR using different methods with a focus on the incorporation of TL in HAR methods. It also discusses the limitations, challenges and possible future directions for more research.

[1]  Bart Selman,et al.  Human Activity Detection from RGBD Images , 2011, Plan, Activity, and Intent Recognition.

[2]  Jie Lu,et al.  Fuzzy Refinement Domain Adaptation for Long Term Prediction in Banking Ecosystem , 2014, IEEE Transactions on Industrial Informatics.

[3]  Gang Wang,et al.  NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understanding , 2019, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[4]  Ronald Poppe,et al.  A survey on vision-based human action recognition , 2010, Image Vis. Comput..

[5]  Stefan Wermter,et al.  Self-organizing neural integration of pose-motion features for human action recognition , 2015, Front. Neurorobot..

[6]  Chih-Min Lin,et al.  Use of human gestures for controlling a mobile robot via adaptive CMAC network and fuzzy logic controller , 2017, Neurocomputing.

[7]  Huosheng Hu,et al.  A robot calligraphy system: From simple to complex writing by human gestures , 2017, Eng. Appl. Artif. Intell..

[8]  Caroline S. Langensiepen,et al.  A multi-level refinement approach towards the classification of quotidian activities using accelerometer data , 2019, J. Ambient Intell. Humaniz. Comput..

[9]  Paolo Dario,et al.  Two-person activity recognition using skeleton data , 2018, IET Comput. Vis..

[10]  Taghi M. Khoshgoftaar,et al.  A survey on heterogeneous transfer learning , 2017, Journal of Big Data.

[11]  Peng Hao,et al.  Transfer learning using computational intelligence: A survey , 2015, Knowl. Based Syst..

[12]  Diane J. Cook,et al.  Transfer learning for activity recognition: a survey , 2013, Knowledge and Information Systems.

[13]  Anthony G. Cohn,et al.  CLAD: A Complex and Long Activities Dataset with Rich Crowdsourced Annotations , 2017, ArXiv.

[14]  Plamen P. Angelov,et al.  Human Activity Recognition Based on Evolving Fuzzy Systems , 2010, Int. J. Neural Syst..

[15]  SelfHAR , 2021 .

[16]  Simon Coupland,et al.  Fuzzy Transfer Learning: Methodology and application , 2015, Inf. Sci..

[17]  Qiang Yang,et al.  A Survey on Transfer Learning , 2010, IEEE Transactions on Knowledge and Data Engineering.

[18]  Jie Lu,et al.  Heterogeneous Unsupervised Cross-domain Transfer Learning , 2017, ArXiv.

[19]  Jake K. Aggarwal,et al.  Human activity recognition from 3D data: A review , 2014, Pattern Recognit. Lett..

[20]  Witold Pedrycz,et al.  Fuzzy Rule-Based Domain Adaptation in Homogeneous and Heterogeneous Spaces , 2019, IEEE Transactions on Fuzzy Systems.

[21]  Chee Seng Chan,et al.  Fuzzy qualitative human model for viewpoint identification , 2016, Neural Computing and Applications.

[22]  Diane J. Cook,et al.  Transfer Learning across Feature-Rich Heterogeneous Feature Spaces via Feature-Space Remapping (FSR) , 2015, ACM Trans. Intell. Syst. Technol..

[23]  Cecilia Mascolo,et al.  SelfHAR: Improving Human Activity Recognition through Self-training with Unlabeled Data , 2021, Proc. ACM Interact. Mob. Wearable Ubiquitous Technol..

[24]  Hongnian Yu,et al.  A survey on wearable sensor modality centred human activity recognition in health care , 2019, Expert Syst. Appl..

[25]  Tao Gu,et al.  Towards a Diffraction-based Sensing Approach on Human Activity Recognition , 2019, Proc. ACM Interact. Mob. Wearable Ubiquitous Technol..

[26]  H. Shimodaira,et al.  Improving predictive inference under covariate shift by weighting the log-likelihood function , 2000 .

[27]  Ivor W. Tsang,et al.  Domain Adaptation via Transfer Component Analysis , 2009, IEEE Transactions on Neural Networks.

[28]  Ahmet Arslan,et al.  Genetic transfer learning , 2010, Expert Syst. Appl..

[29]  Elnaz Soleimani,et al.  Cross-Subject Transfer Learning in Human Activity Recognition Systems using Generative Adversarial Networks , 2019, Neurocomputing.

[30]  Chao Yang,et al.  A Survey on Deep Transfer Learning , 2018, ICANN.

[31]  Ahmad Lotfi,et al.  The human behaviour indicator: A measure of behavioural evolution , 2019, Expert Syst. Appl..

[32]  Taghi M. Khoshgoftaar,et al.  A survey of transfer learning , 2016, Journal of Big Data.

[33]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[34]  Xiaodong Yang,et al.  Effective 3D action recognition using EigenJoints , 2014, J. Vis. Commun. Image Represent..

[35]  Ennio Gambi,et al.  A Human Activity Recognition System Using Skeleton Data from RGBD Sensors , 2016, Comput. Intell. Neurosci..

[36]  Barbara Caputo,et al.  Learning Categories From Few Examples With Multi Model Knowledge Transfer , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[37]  Wen Zhou,et al.  Human Action Recognition With Multiple-Instance Markov Model , 2014, IEEE Transactions on Information Forensics and Security.

[38]  Hani Hagras,et al.  A fuzzy logic-based system for the automation of human behavior recognition using machine vision in intelligent environments , 2015, Soft Comput..

[39]  Csaba Benedek,et al.  Lidar-Based Gait Analysis and Activity Recognition in a 4D Surveillance System , 2018, IEEE Transactions on Circuits and Systems for Video Technology.

[40]  Guodong Guo,et al.  Evaluating spatiotemporal interest point features for depth-based action recognition , 2014, Image Vis. Comput..

[41]  Edward D. Lemaire,et al.  Feature Selection for Wearable Smartphone-Based Human Activity Recognition with Able bodied, Elderly, and Stroke Patients , 2015, PloS one.

[42]  Miguel A. Labrador,et al.  A Survey on Human Activity Recognition using Wearable Sensors , 2013, IEEE Communications Surveys & Tutorials.

[43]  Xin Zhang,et al.  Human pose estimation method based on single depth image , 2018, IET Comput. Vis..

[44]  Mirko Wächter,et al.  Learning Object-Action Relations from Bimanual Human Demonstration Using Graph Networks , 2019, IEEE Robotics and Automation Letters.

[45]  Nassir Navab,et al.  3D Pictorial Structures Revisited: Multiple Human Pose Estimation , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[46]  Yong Pei,et al.  Multilevel Depth and Image Fusion for Human Activity Detection , 2013, IEEE Transactions on Cybernetics.

[47]  Hsinchun Chen,et al.  A sequence-to-sequence model-based deep learning approach for recognizing activity of daily living for senior care , 2018, J. Biomed. Informatics.

[48]  Philip S. Yu,et al.  Deep Learning of Transferable Representation for Scalable Domain Adaptation , 2016, IEEE Transactions on Knowledge and Data Engineering.

[49]  Antonio Torralba,et al.  Through-Wall Human Pose Estimation Using Radio Signals , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[50]  Olcay Sahin,et al.  Transfer Learning Using Deep Neural Networks for Classification of Truck Body Types Based on Side-Fire Lidar Data , 2019, Journal of Big Data Analytics in Transportation.

[51]  Marco Morana,et al.  Human Activity Recognition Process Using 3-D Posture Data , 2015, IEEE Transactions on Human-Machine Systems.

[52]  Yi Wang,et al.  Robust Indoor Human Activity Recognition Using Wireless Signals , 2015, Sensors.

[53]  Jie Lu,et al.  Fuzzy bridged Refinement Domain Adaptation: Long-Term Bank Failure Prediction , 2013, Int. J. Comput. Intell. Appl..

[54]  Fei Han,et al.  Space-Time Representation of People Based on 3D Skeletal Data: A Review , 2016, Comput. Vis. Image Underst..

[55]  Tian Wang,et al.  Human Pose Estimation by Exploiting Spatial and Temporal Constraints in Body-Part Configurations , 2017, IEEE Access.

[56]  Chalavadi Krishna Mohan,et al.  Human action recognition in RGB-D videos using motion sequence information and deep learning , 2017, Pattern Recognit..

[57]  Aydin Kaya,et al.  Analysis of transfer learning for deep neural network based plant classification models , 2019, Comput. Electron. Agric..

[58]  Paulo Peixoto,et al.  A human activity recognition framework using max-min features and key poses with differential evolution random forests classifier , 2017, Pattern Recognit. Lett..

[59]  Gavriel Salomon,et al.  T RANSFER OF LEARNING , 1992 .