Improving posture classification accuracy for depth sensor-based human activity monitoring in smart environments

A new posture classification framework for Kinect is proposed.Accuracy in classifying noisy postures is improved by considering the reliability of each joint.Reliability of a joint can be evaluated by the consistency in different aspects over time.Performance of classifier is improved by learning the weights of reliability terms. Smart environments and monitoring systems are popular research areas nowadays due to its potential to enhance the quality of life. Applications such as human behavior analysis and workspace ergonomics monitoring are automated, thereby improving well-being of individuals with minimal running cost. The central problem of smart environments is to understand what the user is doing in order to provide the appropriate support. While it is difficult to obtain information of full body movement in the past, depth camera based motion sensing technology such as Kinect has made it possible to obtain 3D posture without complex setup. This has fused a large number of research projects to apply Kinect in smart environments. The common bottleneck of these researches is the high amount of errors in the detected joint positions, which would result in inaccurate analysis and false alarms. In this paper, we propose a framework that accurately classifies the nature of the 3D postures obtained by Kinect using a max-margin classifier. Different from previous work in the area, we integrate the information about the reliability of the tracked joints in order to enhance the accuracy and robustness of our framework. As a result, apart from general classifying activity of different movement context, our proposed method can classify the subtle differences between correctly performed and incorrectly performed movement in the same context. We demonstrate how our framework can be applied to evaluate the user's posture and identify the postures that may result in musculoskeletal disorders. Such a system can be used in workplace such as offices and factories to reduce risk of injury. Experimental results have shown that our method consistently outperforms existing algorithms in both activity classification and posture healthiness classification. Due to the low cost and the easy deployment process of depth camera based motion sensors, our framework can be applied widely in home and office to facilitate smart environments.

[1]  David A. McAllester,et al.  Object Detection with Discriminatively Trained Part Based Models , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[2]  Lasitha Piyathilaka,et al.  Gaussian mixture based HMM for human daily activity recognition using 3D skeleton features , 2013, 2013 IEEE 8th Conference on Industrial Electronics and Applications (ICIEA).

[3]  Daniel P. Huttenlocher,et al.  Pictorial Structures for Object Recognition , 2004, International Journal of Computer Vision.

[4]  Mohammad H. Mahoor,et al.  Human activity recognition using multi-features and multiple kernel learning , 2014, Pattern Recognit..

[5]  Hubert P. H. Shum,et al.  Real-Time Posture Reconstruction for Microsoft Kinect , 2013, IEEE Transactions on Cybernetics.

[6]  Jake K. Aggarwal,et al.  View invariant human action recognition using histograms of 3D joints , 2012, 2012 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops.

[7]  Angelika Garz,et al.  Affective computing for wearable diary and lifelogging systems: An overview , 2011 .

[8]  Vladimir N. Vapnik,et al.  The Nature of Statistical Learning Theory , 2000, Statistics for Engineering and Information Science.

[9]  Jessica K. Hodgins,et al.  Performance animation from low-dimensional control signals , 2005, SIGGRAPH 2005.

[10]  Mikhail Belkin,et al.  Automatic Annotation of Daily Activity from Smartphone-Based Multisensory Streams , 2012, MobiCASE.

[11]  Tae-Seong Kim,et al.  Depth video-based human activity recognition system using translation and scaling invariant features for life logging at smart home , 2012, IEEE Transactions on Consumer Electronics.

[12]  Hubert P. H. Shum,et al.  Real-time physical modelling of character movements with microsoft kinect , 2012, VRST '12.

[13]  Larry S. Davis,et al.  Observing Human-Object Interactions: Using Spatial and Functional Compatibility for Recognition , 2009, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[14]  Alberto Del Bimbo,et al.  Submitted to Ieee Transactions on Cybernetics 1 3d Human Action Recognition by Shape Analysis of Motion Trajectories on Riemannian Manifold , 2022 .

[16]  Marwan Torki,et al.  Human Action Recognition Using a Temporal Hierarchy of Covariance Descriptors on 3D Joint Locations , 2013, IJCAI.

[17]  Cristian Sminchisescu,et al.  The Moving Pose: An Efficient 3D Kinematics Descriptor for Low-Latency Action Recognition and Detection , 2013, 2013 IEEE International Conference on Computer Vision.

[18]  Alexei A. Efros,et al.  People Watching: Human Actions as a Cue for Single View Geometry , 2012, International Journal of Computer Vision.

[19]  Xiaodong Yang,et al.  EigenJoints-based action recognition using Naïve-Bayes-Nearest-Neighbor , 2012, 2012 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops.

[20]  Guodong Guo,et al.  Fusing Spatiotemporal Features and Joints for 3D Action Recognition , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition Workshops.

[21]  Shaharyar Kamal,et al.  Real-time life logging via a depth silhouette-based human activity recognition system for smart home services , 2014, 2014 11th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS).

[22]  A. Uribe-Quevedo,et al.  Seated Tracking for Correcting Computer Work Postures , 2013, 2013 29th Southern Biomedical Engineering Conference.

[23]  Hans-Peter Seidel,et al.  A data-driven approach for real-time full body pose reconstruction from a depth camera , 2011, 2011 International Conference on Computer Vision.

[24]  Wanqing Li,et al.  Action recognition based on a bag of 3D points , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Workshops.

[25]  Chakarida Nukoolkit,et al.  Prolonged sitting detection for office workers syndrome prevention using kinect , 2014, 2014 11th International Conference on Electrical Engineering/Electronics, Computer, Telecommunications and Information Technology (ECTI-CON).

[26]  Rushil Anirudh,et al.  Elastic functional coding of human actions: From vector-fields to latent variables , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[27]  Miguel A. Labrador,et al.  A Survey on Human Activity Recognition using Wearable Sensors , 2013, IEEE Communications Surveys & Tutorials.

[28]  D. Sharma,et al.  Senior health monitoring using Kinect , 2012, 2012 Fourth International Conference on Communications and Electronics (ICCE).

[29]  Cheng Hong,et al.  RGB-Depth feature for 3D human activity recognition , 2013, China Communications.

[30]  Tae-Seong Kim,et al.  Human Activity Recognition via the Features of Labeled Depth Body Parts , 2012, ICOST.

[31]  Octavian Postolache,et al.  Kinect based system and Artificial Neural Networks classifiers for physiotherapy assessment , 2014, 2014 IEEE International Symposium on Medical Measurements and Applications (MeMeA).

[32]  Hala H. Zayed,et al.  Human Activity Recognition for Surveillance Applications , 2015, ICIT 2015.

[33]  Jong Bae Kim,et al.  Efficient region-based motion segmentation for a video monitoring system , 2003, Pattern Recognit. Lett..

[34]  Tido Röder,et al.  Efficient content-based retrieval of motion capture data , 2005, SIGGRAPH 2005.

[35]  Meinard Müller,et al.  Motion templates for automatic classification and retrieval of motion capture data , 2006, SCA '06.

[36]  Wenbing Zhao,et al.  A feasibility study of using a single Kinect sensor for rehabilitation exercises monitoring: A rule based approach , 2014, 2014 IEEE Symposium on Computational Intelligence in Healthcare and e-health (CICARE).

[37]  Paul A. Viola,et al.  Learning silhouette features for control of human motion , 2004, SIGGRAPH '04.

[38]  Daijin Kim,et al.  Depth map-based human activity tracking and recognition using body joints features and Self-Organized Map , 2014, Fifth International Conference on Computing, Communications and Networking Technologies (ICCCNT).

[39]  Victor B. Zordan,et al.  Mapping optical motion capture data to skeletal motion using a physical model , 2003, SCA '03.

[40]  Paul Lukowicz,et al.  Activity Recognition of Assembly Tasks Using Body-Worn Microphones and Accelerometers , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[41]  Daijin Kim,et al.  Shape and Motion Features Approach for Activity Tracking and Recognition from Kinect Video Camera , 2015, 2015 IEEE 29th International Conference on Advanced Information Networking and Applications Workshops.

[42]  Allen Y. Yang,et al.  Distributed segmentation and classification of human actions using a wearable motion sensor network , 2008, 2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops.

[43]  Taku Komura,et al.  Indexing and Retrieving Motions of Characters in Close Contact , 2009, IEEE Transactions on Visualization and Computer Graphics.

[44]  G. G. Stokes.  "J." , 1890, The New Yale Book of Quotations.

[45]  Hubert P. H. Shum,et al.  Posture reconstruction using Kinect with a probabilistic model , 2014, VRST '14.

[46]  Ahmad Jalal,et al.  Dense depth maps-based human pose tracking and recognition in dynamic scenes using ridge data , 2014, 2014 11th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS).

[47]  Shaharyar Kamal,et al.  Dense RGB-D Map-Based Human Tracking and Activity Recognition using Skin Joints Features and Self-Organizing Map , 2015, KSII Trans. Internet Inf. Syst..

[48]  Pieter Abbeel,et al.  Max-margin Classification of Data with Absent Features , 2008, J. Mach. Learn. Res..

[49]  Mohan M. Trivedi,et al.  Joint Angles Similarities and HOG2 for Action Recognition , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition Workshops.

[50]  Tae-Seong Kim,et al.  Human Activity Recognition via Recognized Body Parts of Human Depth Silhouettes for Residents Monitoring Services at Smart Home , 2013 .

[51]  Marc Pollefeys,et al.  Joint Camera Pose Estimation and 3D Human Pose Estimation in a Multi-camera Setup , 2014, ACCV.

[52]  Rama Chellappa,et al.  Human Action Recognition by Representing 3D Skeletons as Points in a Lie Group , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[53]  Shuicheng Yan,et al.  Body Surface Context: A New Robust Feature for Action Recognition From Depth Videos , 2014, IEEE Transactions on Circuits and Systems for Video Technology.

[54]  Ying Wu,et al.  Mining actionlet ensemble for action recognition with depth cameras , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[55]  Alberto Del Bimbo,et al.  Recognizing Actions from Depth Cameras as Weakly Aligned Multi-part Bag-of-Poses , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition Workshops.

[56]  Ruigang Yang,et al.  Accurate 3D pose estimation from a single depth image , 2011, 2011 International Conference on Computer Vision.

[57]  Alan L. Yuille,et al.  An Approach to Pose-Based Action Recognition , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[58]  Daijin Kim,et al.  A Depth Video Sensor-Based Life-Logging Human Activity Recognition System for Elderly Care in Smart Indoor Environments , 2014, Sensors.

[59]  Jessica K. Hodgins,et al.  Automatic Joint Parameter Estimation from Magnetic Motion Capture Data , 2023, Graphics Interface.

[60]  Taku Komura,et al.  Emulating human perception of motion similarity , 2008, Comput. Animat. Virtual Worlds.

[61]  Toby Sharp,et al.  Real-time human pose recognition in parts from single depth images , 2011, CVPR.

[62]  Hubert P. H. Shum,et al.  Emulating human perception of motion similarity , 2008 .

[63]  Sean Kean,et al.  Meet the Kinect: An Introduction to Programming Natural User Interfaces , 2011 .

[64]  Taku Komura,et al.  Fast accelerometer-based motion recognition with a dual buffer framework , 2011 .

[65]  Hans-Peter Seidel,et al.  Motion reconstruction using sparse accelerometer data , 2011, TOGS.

[66]  Jessica K. Hodgins,et al.  Action capture with accelerometers , 2008, SCA '08.

[67]  Andrew Blake,et al.  Efficient Human Pose Estimation from Single Depth Images , 2013, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[68]  Masahiro Okuda,et al.  Embedded Key-Frame Extraction for CG Animation by Frame Decimation , 2005, 2005 IEEE International Conference on Multimedia and Expo.

[69]  C. C. Martin,et al.  A real-time ergonomic monitoring system using the Microsoft Kinect , 2012, 2012 IEEE Systems and Information Engineering Design Symposium.

[70]  Daijin Kim,et al.  Ridge body parts features for human pose estimation and recognition from RGB-D video data , 2014, Fifth International Conference on Computing, Communications and Networking Technologies (ICCCNT).