Modeling Two-Person Segmentation and Locomotion for Stereoscopic Action Identification: A Sustainable Video Surveillance System

Due to the constantly increasing demand for automatic tracking and recognition systems, there is a need for more proficient, intelligent and sustainable human activity tracking. The main purpose of this study is to develop an accurate and sustainable human action tracking system that is capable of error-free identification of human movements irrespective of the environment in which those actions are performed. Therefore, in this paper we propose a stereoscopic Human Action Recognition (HAR) system based on the fusion of RGB (red, green, blue) and depth sensors. These sensors give an extra depth of information which enables the three-dimensional (3D) tracking of each and every movement performed by humans. Human actions are tracked according to four features, namely, (1) geodesic distance; (2) 3D Cartesian-plane features; (3) joints Motion Capture (MOCAP) features and (4) way-points trajectory generation. In order to represent these features in an optimized form, Particle Swarm Optimization (PSO) is applied. After optimization, a neuro-fuzzy classifier is used for classification and recognition. Extensive experimentation is performed on three challenging datasets: A Nanyang Technological University (NTU) RGB+D dataset; a UoL (University of Lincoln) 3D social activity dataset and a Collective Activity Dataset (CAD). Evaluation experiments on the proposed system proved that a fusion of vision sensors along with our unique features is an efficient approach towards developing a robust HAR system, having achieved a mean accuracy of 93.5% with the NTU RGB+D dataset, 92.2% with the UoL dataset and 89.6% with the Collective Activity dataset. The developed system can play a significant role in many computer vision-based applications, such as intelligent homes, offices and hospitals, and surveillance systems.

[1]  Caifeng Shan,et al.  Sensors, vision and networks: From video surveillance to activity recognition and health monitoring , 2019, J. Ambient Intell. Smart Environ..

[2]  Elsayed E. Hemayed,et al.  Human action recognition using trajectory-based representation , 2015 .

[3]  Mohand Saïd Allili,et al.  Group-of-features relevance in multinomial kernel logistic regression and application to human interaction recognition , 2020, Expert Syst. Appl..

[4]  Silvio Savarese,et al.  A Unified Framework for Multi-target Tracking and Collective Activity Recognition , 2012, ECCV.

[5]  Aun Irtaza,et al.  Robust Human Activity Recognition Using Multimodal Feature-Level Fusion , 2019, IEEE Access.

[6]  Xu Yong,et al.  Three-stage network for age estimation , 2019 .

[7]  Kibum Kim,et al.  Automated Sustainable Multi-Object Segmentation and Recognition via Modified Sampling Consensus and Kernel Sliding Perceptron , 2020, Symmetry.

[8]  Santiago Garrido,et al.  Marine Applications of the Fast Marching Method , 2020, Frontiers in Robotics and AI.

[9]  Daijin Kim,et al.  Shape and Motion Features Approach for Activity Tracking and Recognition from Kinect Video Camera , 2015, 2015 IEEE 29th International Conference on Advanced Information Networking and Applications Workshops.

[10]  Xuan-Tu Tran,et al.  A Novel Hardware Architecture for Human Detection using HOG-SVM Co-Optimization , 2019, 2019 IEEE Asia Pacific Conference on Circuits and Systems (APCCAS).

[11]  Antonio Ortega,et al.  Graph Based Skeleton Modeling for Human Activity Analysis , 2019, 2019 IEEE International Conference on Image Processing (ICIP).

[12]  Eran Treister,et al.  A fast marching algorithm for the factored eikonal equation , 2016, J. Comput. Phys..

[13]  Sanghoon Kim,et al.  Automatic image thresholding using Otsu’s method and entropy weighting scheme for surface defect detection , 2017, Soft Computing.

[14]  Suixian Yang,et al.  Learning Attention Representation with a Multi-Scale CNN for Gear Fault Diagnosis under Different Working Conditions , 2020, Sensors.

[15]  Meng Li,et al.  Multi-view depth-based pairwise feature learning for person-person interaction recognition , 2019, Multimedia Tools and Applications.

[16]  Wenbing Zhao,et al.  Rule based realtime motion assessment for rehabilitation exercises , 2014, 2014 IEEE Symposium on Computational Intelligence in Healthcare and e-health (CICARE).

[17]  Md. Zia Uddin,et al.  A Depth Camera-based Human Activity Recognition via Deep Learning Recurrent Neural Network for Health and Social Care Services , 2016, CENTERIS/ProjMAN/HCist.

[18]  Amir Nadeem,et al.  Human Actions Tracking and Recognition Based on Body Parts Detection via Artificial Neural Network , 2020, 2020 3rd International Conference on Advancements in Computational Sciences (ICACS).

[19]  Ahmad Lotfi,et al.  Human activities recognition based on neuro-fuzzy finite state machine , 2018 .

[20]  Wei Liang,et al.  Recognising human interaction from videos by a discriminative model , 2014, IET Comput. Vis..

[21]  Jiang Wu,et al.  Image Encryption Based on Pixel-Level Diffusion with Dynamic Filtering and DNA-Level Permutation with 3D Latin Cubes , 2019, Entropy.

[22]  Unsang Park,et al.  Compositional interaction descriptor for human interaction recognition , 2017, Neurocomputing.

[23]  Kibum Kim,et al.  RGB-D Images for Object Segmentation, Localization and Recognition in Indoor Scenes using Feature Descriptor and Hough Voting , 2020, 2020 17th International Bhurban Conference on Applied Sciences and Technology (IBCAST).

[24]  Balachandra,et al.  Enhanced Human Action Recognition Using Fusion of Skeletal Joint Dynamics and Structural Features , 2020, J. Robotics.

[25]  Ahmad Jalal,et al.  Robust Spatio-Temporal Features for Human Interaction Recognition Via Artificial Neural Network , 2018, 2018 International Conference on Frontiers of Information Technology (FIT).

[26]  Cesar A. Azurdia-Meza,et al.  Depth Maps-Based Human Segmentation and Action Recognition Using Full-Body Plus Body Color Cues Via Recognizer Engine , 2019, Journal of Electrical Engineering & Technology.

[27]  Yan Song,et al.  Concurrence-Aware Long Short-Term Sub-Memories for Person-Person Action Recognition , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[28]  Ahmad Jalal,et al.  Sustainable Wearable System: Human Behavior Modeling for Life-Logging Activities Using K-Ary Tree Hashing Classifier , 2020, Sustainability.

[29]  Hong Cheng,et al.  Learning contrastive feature distribution model for interaction recognition , 2015, J. Vis. Commun. Image Represent..

[30]  Duoqian Miao,et al.  Influence of kernel clustering on an RBFN , 2019, CAAI Trans. Intell. Technol..

[31]  Yongmei Zhang,et al.  Human Interaction Recognition Based on Whole-Individual Detection , 2020, Sensors.

[32]  Bummo Ahn,et al.  Real-Time Human Action Recognition with a Low-Cost RGB Camera and Mobile Robot Platform , 2020, Sensors.

[33]  Serhan Cosar,et al.  Social Activity Recognition on Continuous RGB-D Video Sequences , 2020, Int. J. Soc. Robotics.

[34]  Mubarak Shah,et al.  Recognizing 50 human action categories of web videos , 2012, Machine Vision and Applications.

[35]  Mathieu Barnachon,et al.  Ongoing human action recognition with motion capture , 2014, Pattern Recognit..

[36]  Daijin Kim,et al.  Robust human activity recognition from depth video using spatiotemporal multi-fused features , 2017, Pattern Recognit..

[37]  Yi Lin,et al.  Skeleton based action recognition using translation-scale invariant image mapping and multi-scale deep CNN , 2017, 2017 IEEE International Conference on Multimedia & Expo Workshops (ICMEW).

[38]  Kibum Kim,et al.  Automatic Recognition of Human Interaction via Hybrid Descriptors and Maximum Entropy Markov Model Using Depth Sensors , 2020, Entropy.

[39]  Paolo Dario,et al.  Two-person activity recognition using skeleton data , 2018, IET Comput. Vis..

[40]  Nasser Kehtarnavaz,et al.  A survey of depth and inertial sensor fusion for human action recognition , 2015, Multimedia Tools and Applications.

[41]  Jeng-Fung Chen,et al.  A Neuro-Fuzzy Approach in the Classification of Students' Academic Performance , 2013, Comput. Intell. Neurosci..

[42]  Atsuo Yoshitaka,et al.  Human Interaction Recognition Using Hierarchical Invariant Features , 2015, Int. J. Semantic Comput..

[43]  Aliaa A. A. Youssif,et al.  Depth-based human activity recognition: A comparative perspective study on feature extraction , 2017, Future Computing and Informatics Journal.

[44]  David Picard,et al.  2D/3D Pose Estimation and Action Recognition Using Multitask Deep Learning , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[45]  S. Rüping,et al.  Advanced Sensing and Human Activity Recognition in Early Intervention and Rehabilitation of Elderly People , 2020, Journal of Population Ageing.

[46]  Amir Nadeem,et al.  Human Body Parts Estimation and Detection for Physical Sports Movements , 2019, 2019 2nd International Conference on Communication, Computing and Digital systems (C-CODE).

[47]  Yong Du,et al.  Hierarchical recurrent neural network for skeleton based action recognition , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[48]  Muhammad Younus Javed,et al.  A framework of human detection and action recognition based on uniform segmentation and combination of Euclidean distance and joint entropy-based features selection , 2017, EURASIP J. Image Video Process..

[49]  Muhammad Sher,et al.  Automated multi-feature human interaction recognition in complex environment , 2018, Comput. Ind..

[50]  Björn E. Ottersten,et al.  Localized Trajectories for 2D and 3D Action Recognition † , 2019, Sensors.

[51]  Wei-Shi Zheng,et al.  Learning Person–Person Interaction in Collective Activity Recognition , 2015, IEEE Transactions on Image Processing.

[52]  Huaijun Wang,et al.  Segmentation and Recognition of Basic and Transitional Activities for Continuous Physical Human Activity , 2019, IEEE Access.

[53]  Yepeng Guan,et al.  Image Enhancement via Subimage Histogram Equalization Based on Mean and Variance , 2017, Comput. Intell. Neurosci..

[54]  Mala John,et al.  Particle swarm optimization with deep learning for human action recognition , 2020, Multimedia Tools and Applications.

[55]  Gang Wang,et al.  NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysis , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[56]  Maria Mahmood,et al.  WHITE STAG model: wise human interaction tracking and estimation (WHITE) using spatio-temporal and angular-geometric (STAG) descriptors , 2019, Multimedia Tools and Applications.

[57]  Kibum Kim,et al.  An Accurate Facial Expression Detector using Multi-Landmarks Selection and Local Transform Features , 2020, 2020 3rd International Conference on Advancements in Computational Sciences (ICACS).

[58]  Ifeyinwa E. Achumba,et al.  Sensor Data Acquisition and Processing Parameters for Human Activity Classification , 2014, Sensors.

[59]  Kibum Kim,et al.  A Wrist Worn Acceleration Based Human Motion Analysis and Classification for Ambient Smart Home System , 2019, Journal of Electrical Engineering & Technology.

[60]  Daijin Kim,et al.  Human Depth Sensors-Based Activity Recognition Using Spatiotemporal Features and Hidden Markov Model for Smart Environments , 2016, J. Comput. Networks Commun..

[61]  Ying Sun,et al.  Intelligent human computer interaction based on non redundant EMG signal , 2020, Alexandria Engineering Journal.

[62]  Narasimhan Sundararajan,et al.  A Metacognitive Neuro-Fuzzy Inference System (McFIS) for Sequential Classification Problems , 2013, IEEE Transactions on Fuzzy Systems.

[63]  Lei Liu,et al.  Particle swarm optimization algorithm: an overview , 2017, Soft Computing.

[64]  S. Chitrakala,et al.  Recognition of human-human interaction using CWDTW , 2016, 2016 International Conference on Circuit, Power and Computing Technologies (ICCPCT).

[65]  Zhaojie Ju,et al.  A New Framework of Human Interaction Recognition Based on Multiple Stage Probability Fusion , 2017 .

[66]  M. A. Saleem Durai,et al.  Intelligent video surveillance: a review through deep learning techniques for crowd analysis , 2019, Journal of Big Data.

[67]  Maria Mahmood,et al.  Multi-features descriptors for Human Activity Tracking and Recognition in Indoor-Outdoor Environments , 2019, 2019 16th International Bhurban Conference on Applied Sciences and Technology (IBCAST).

[68]  Abdenour Hadid,et al.  Vision-based human activity recognition: a survey , 2020, Multimedia Tools and Applications.

[69]  Dong-Ho Lee,et al.  An adaptive dynamically weighted median filter for impulse noise removal , 2017, EURASIP J. Adv. Signal Process..

[70]  Silvio Savarese,et al.  What are they doing? : Collective activity classification using spatio-temporal relationship among people , 2009, 2009 IEEE 12th International Conference on Computer Vision Workshops, ICCV Workshops.

[71]  Jian Yang,et al.  Hierarchical Long Short-Term Concurrent Memory for Human Interaction Recognition , 2018, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[72]  Magudeeswaran Veluchamy,et al.  Image contrast and color enhancement using adaptive gamma correction and histogram equalization , 2019, Optik.

[73]  Ahmad Jalal,et al.  Wearable Inertial Sensors for Daily Activity Analysis Based on Adam Optimization and the Maximum Entropy Markov Model , 2020, Entropy.

[74]  Omer Faruk Ince,et al.  Human activity recognition with analysis of angles between skeletal joints using a RGB‐depth sensor , 2019, ETRI Journal.

[75]  Mohamed Atri,et al.  Human action recognition using RGB data , 2016, 2016 11th International Design & Test Symposium (IDT).

[76]  Gang Wang,et al.  Deep Multimodal Feature Analysis for Action Recognition in RGB+D Videos , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[77]  Serhan Cosar,et al.  Automatic detection of human interactions from RGB-D data for social activity classification , 2017, 2017 26th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN).

[78]  Xiaofei Xu,et al.  Activity Recognition Method for Home-Based Elderly Care Service Based on Random Forest and Activity Similarity , 2019, IEEE Access.

[79]  Ahmad Jalal,et al.  Human Posture Estimation and Sustainable Events Classification via Pseudo-2D Stick Model and K-ary Tree Hashing , 2020, Sustainability.

[80]  Andreas M. Kunz,et al.  Tracking human locomotion by relative positional feet tracking , 2015, 2015 IEEE Virtual Reality (VR).

[81]  Serdar Enginoglu,et al.  Different applied median filter in salt and pepper noise , 2018, Comput. Electr. Eng..

[82]  Jürgen Weber,et al.  Analytical analysis of single-stage pressure relief valves , 2019, International Journal of Hydromechatronics.

[83]  Nicola Bellotto,et al.  Social activity recognition based on probabilistic merging of skeleton features with proximity priors from RGB-D data , 2016, 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[84]  Ahmad Jalal,et al.  Sensors Technologies for Human Activity Analysis Based on SVM Optimized by PSO Algorithm , 2019, 2019 International Conference on Applied and Engineering Mathematics (ICAEM).

[85]  Greg Mori,et al.  Structure Inference Machines: Recurrent Neural Networks for Analyzing Relations in Group Activity Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[86]  Seba Susan,et al.  New shape descriptor in the context of edge continuity , 2019, CAAI Trans. Intell. Technol..

[87]  Kibum Kim,et al.  A Study of Accelerometer and Gyroscope Measurements in Physical Life-Log Activities Detection Systems , 2020, Sensors.

[88]  Thuong Le-Tien,et al.  PAM-based flexible generative topic model for 3D interactive activity recognition , 2015, 2015 International Conference on Advanced Technologies for Communications (ATC).

[89]  Kwanghee Ko,et al.  Depth edge detection by image-based smoothing and morphological operations , 2016, J. Comput. Des. Eng..

[90]  Kibum Kim,et al.  A Novel Statistical Method for Scene Classification Based on Multi-Object Categorization and Logistic Regression , 2020, Sensors.

[91]  Ahmad Jalal,et al.  Accurate Physical Activity Recognition using Multidimensional Features and Markov Model for Smart Health Fitness , 2020, Symmetry.

[92]  Dietrich Paulus,et al.  Local Feature Extraction from RGB and Depth Videos for Human Action Recognition , 2018, International Journal of Machine Learning and Computing.

[93]  Daijin Kim,et al.  A Depth Video Sensor-Based Life-Logging Human Activity Recognition System for Elderly Care in Smart Indoor Environments , 2014, Sensors.