A novel framework of continuous human-activity recognition using Kinect

Abstract Automatic human activity recognition is being studied widely by researchers for various applications. However, majority of the existing work are limited to recognition of isolated activities, though human activities are inherently continuous in nature with spatial and temporal transitions between various segments. Therefore, there are scopes to develop a robust and continuous Human Activity Recognition (HAR) system. In this paper, we present a novel Coarse-to-Fine framework for continuous HAR using Microsoft Kinect. The activity sequences are captured in the form of 3D skeleton trajectories consisting of 3D positions of 20 joints estimated from the depth data. The recorded sequences are first coarsely grouped into two activity sequences performed during sitting and standing. Next, the activities present in the segmented sequences are recognized into fine-level activities. Activity classification in both stages are performed using Bidirectional Long Short-Term Memory Neural Network (BLSTM-NN) classifier. A total of 1110 continuous activity sequences have been recorded using a combination of 24 isolated human activities. Recognition rates of 68.9% and 64.45% have been recorded using BLSTM-NN classifier when tested using length-modeling and without length-modeling, respectively. We have also computed results for isolated activity recognition performance. Finally, the performance has been compared with existing approaches.

[1]  Dimitrios Makris,et al.  G3D: A gaming action dataset and real time action recognition evaluation framework , 2012, 2012 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops.

[2]  Ioannis A. Kakadiaris,et al.  Fusion of Human Posture Features for Continuous Action Recognition , 2010, ECCV Workshops.

[3]  Mubarak Shah,et al.  Action recognition in videos acquired by a moving camera using motion decomposition of Lagrangian particle trajectories , 2011, 2011 International Conference on Computer Vision.

[4]  Jian-Huang Lai,et al.  Jointly Learning Heterogeneous Features for RGB-D Activity Recognition , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[5]  Lukun Wang,et al.  Recognition of Human Activities Using Continuous Autoencoders with Wearable Sensors , 2016, Sensors.

[6]  Tal Hassner,et al.  Motion Interchange Patterns for Action Recognition in Unconstrained Videos , 2012, ECCV.

[7]  Debi Prosad Dogra,et al.  3D text segmentation and recognition using leap motion , 2017, Multimedia Tools and Applications.

[8]  Douglas A. Reynolds,et al.  Gaussian Mixture Models , 2018, Encyclopedia of Biometrics.

[9]  Lawrence R. Rabiner,et al.  A tutorial on hidden Markov models and selected applications in speech recognition , 1989, Proc. IEEE.

[10]  Nanning Zheng,et al.  Modeling 4D Human-Object Interactions for Event and Object Recognition , 2013, 2013 IEEE International Conference on Computer Vision.

[11]  Jürgen Schmidhuber,et al.  Framewise phoneme classification with bidirectional LSTM and other neural network architectures , 2005, Neural Networks.

[12]  Jake K. Aggarwal,et al.  View invariant human action recognition using histograms of 3D joints , 2012, 2012 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops.

[13]  Horst Bunke,et al.  Hidden Markov model length optimization for handwriting recognition systems , 2002, Proceedings Eighth International Workshop on Frontiers in Handwriting Recognition.

[14]  Nasser Kehtarnavaz,et al.  A Real-Time Human Action Recognition System Using Depth and Inertial Sensor Fusion , 2016, IEEE Sensors Journal.

[15]  Debi Prosad Dogra,et al.  Study of Text Segmentation and Recognition Using Leap Motion Sensor , 2017, IEEE Sensors Journal.

[16]  Christopher Joseph Pal,et al.  Activity recognition using the velocity histories of tracked keypoints , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[17]  Zicheng Liu,et al.  HON4D: Histogram of Oriented 4D Normals for Activity Recognition from Depth Sequences , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[18]  Nasser Kehtarnavaz,et al.  Improving Human Action Recognition Using Fusion of Depth Camera and Inertial Sensors , 2015, IEEE Transactions on Human-Machine Systems.

[19]  Tae-Seong Kim,et al.  Depth video-based human activity recognition system using translation and scaling invariant features for life logging at smart home , 2012, IEEE Transactions on Consumer Electronics.

[20]  Dan Schonfeld,et al.  Object Trajectory-Based Activity Classification and Recognition Using Hidden Markov Models , 2007, IEEE Transactions on Image Processing.

[21]  Xiaodong Yang,et al.  Super Normal Vector for Activity Recognition Using Depth Sequences , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[22]  Ying Wu,et al.  Mining actionlet ensemble for action recognition with depth cameras , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[23]  Zhengyou Zhang,et al.  Microsoft Kinect Sensor and Its Effect , 2012, IEEE Multim..

[24]  Debi Prosad Dogra,et al.  Virtual trainer with real-time feedback using kinect sensor , 2017, 2017 IEEE Region 10 Symposium (TENSYMP).

[25]  Václav Hlavác,et al.  Pose primitive based human action recognition in videos or still images , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[26]  Georgios Evangelidis,et al.  Continuous Action Recognition Based on Sequence Alignment , 2014, International Journal of Computer Vision.

[27]  Debi Prosad Dogra,et al.  A multimodal framework for sensor based sign language recognition , 2017, Neurocomputing.

[28]  Luis Enrique Sucar,et al.  Continuous activity recognition with missing data , 2002, Object recognition supported by user interaction for service robots.

[29]  Nasser Kehtarnavaz,et al.  Action Recognition from Depth Sequences Using Depth Motion Maps-Based Local Binary Patterns , 2015, 2015 IEEE Winter Conference on Applications of Computer Vision.

[30]  Mubarak Shah,et al.  Human Action Recognition in Videos Using Kinematic Features and Multiple Instance Learning , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[31]  Junsong Yuan,et al.  Learning Actionlet Ensemble for 3D Human Action Recognition , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[32]  Alberto Del Bimbo,et al.  Space-Time Pose Representation for 3D Human Action Recognition , 2013, ICIAP Workshops.

[33]  Vigneshwaran Subbaraju,et al.  Energy-Efficient Continuous Activity Recognition on Mobile Phones: An Activity-Adaptive Approach , 2012, 2012 16th International Symposium on Wearable Computers.

[34]  Juan Song,et al.  An Online Continuous Human Action Recognition Algorithm Based on the Kinect Sensor , 2016, Sensors.

[35]  Marco Morana,et al.  Human Activity Recognition Process Using 3-D Posture Data , 2015, IEEE Transactions on Human-Machine Systems.

[36]  Bernt Schiele,et al.  A database for fine grained activity detection of cooking activities , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[37]  Paul Lukowicz,et al.  Activity Recognition of Assembly Tasks Using Body-Worn Microphones and Accelerometers , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[38]  Samsu Sempena,et al.  Human action recognition using Dynamic Time Warping , 2011, Proceedings of the 2011 International Conference on Electrical Engineering and Informatics.

[39]  Amit K. Roy-Chowdhury,et al.  A Continuous Learning Framework for Activity Recognition Using Deep Hybrid Feature Models , 2015, IEEE Transactions on Multimedia.

[40]  Ying Wu,et al.  Discriminative subvolume search for efficient action detection , 2009, CVPR.

[41]  Jake K. Aggarwal,et al.  Spatio-temporal relationship match: Video structure comparison for recognition of complex human activities , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[42]  Li-Chen Fu,et al.  Daily activity recognition using the informative features from skeletal and depth data , 2016, 2016 IEEE International Conference on Robotics and Automation (ICRA).