Early detection of human actions - A hybrid approach

Graphical abstractDisplay Omitted Early detection of human actions is essential in a wide spectrum of applications ranging from video surveillance to health-care. While human action recognition has been extensively studied, little attention is paid to the problem of detecting ongoing human action early, i.e. detecting an action as soon as it begins, but before it finishes. This study aims at training a detector to be capable of recognizing a human action when only partial action sample is seen. To do so, a hybrid technique is proposed in this work which combines the benefits of computer vision as well as fuzzy set theory based on the fuzzy Bandler and Kohout's sub-triangle product (BK subproduct). The novelty lies in the construction of a frame-by-frame membership function for each kind of possible movement. Detection is triggered when a pre-defined threshold is reached in a suitable way. Experimental results on a publicly available dataset demonstrate the benefits and effectiveness of the proposed method.

[1]  Carsten Rother,et al.  Weakly supervised discriminative localization and classification: a joint learning process , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[2]  Amir-Masoud Eftekhari-Moghadam,et al.  Fuzzy rule-based reasoning approach for event detection and annotation of broadcast soccer video , 2013, Appl. Soft Comput..

[3]  Ian D. Reid,et al.  High Five: Recognising human interactions in TV shows , 2010, BMVC.

[4]  Anastasios Tefas,et al.  Combining Fuzzy Vector Quantization With Linear Discriminant Analysis for Continuous Human Movement Recognition , 2008, IEEE Transactions on Circuits and Systems for Video Technology.

[5]  Jake K. Aggarwal,et al.  Articulated and elastic non-rigid motion: a review , 1994, Proceedings of 1994 IEEE Workshop on Motion of Non-rigid and Articulated Objects.

[6]  Ladislav J. Kohout,et al.  Relational-product architectures for information processing , 1985, Inf. Sci..

[7]  William Brendel,et al.  Learning spatiotemporal graphs of human activities , 2011, 2011 International Conference on Computer Vision.

[8]  Yun Fu,et al.  ARMA-HMM: A new approach for early recognition of human activity , 2012, Proceedings of the 21st International Conference on Pattern Recognition (ICPR2012).

[9]  Honghai Liu,et al.  Fuzzy Qualitative Trigonometry , 2005, 2005 IEEE International Conference on Systems, Man and Cybernetics.

[10]  Zhihai He,et al.  Recognizing Falls from Silhouettes , 2006, 2006 International Conference of the IEEE Engineering in Medicine and Biology Society.

[11]  Honghai Liu,et al.  Fuzzy Qualitative Human Motion Analysis , 2009, IEEE Transactions on Fuzzy Systems.

[12]  Chan Chee Seng,et al.  A BK Subproduct Approach for Scene Classification , 2012 .

[13]  Jake K. Aggarwal,et al.  Semantic Representation and Recognition of Continued and Recursive Human Activities , 2009, International Journal of Computer Vision.

[14]  Ekta Vats,et al.  An improved BK sub-triangle product approach for scene classification , 2015, J. Intell. Fuzzy Syst..

[15]  Alessio Del Bue,et al.  Human behavior analysis in video surveillance: A Social Signal Processing perspective , 2013, Neurocomputing.

[16]  L. Kohout,et al.  FUZZY POWER SETS AND FUZZY IMPLICATION OPERATORS , 1980 .

[17]  Dariu Gavrila,et al.  The Visual Analysis of Human Movement: A Survey , 1999, Comput. Vis. Image Underst..

[18]  Gracián Triviño,et al.  Walking pattern classification using a granular linguistic analysis , 2015, Appl. Soft Comput..

[19]  Hani Hagras,et al.  A fuzzy logic-based system for the automation of human behavior recognition using machine vision in intelligent environments , 2015, Soft Comput..

[20]  Nasrollah Moghaddam Charkari,et al.  A Novel Fuzzy HMM Approach for Human Action Recognition in Video , 2011, KTW.

[21]  Gang Yu,et al.  Predicting human activities using spatio-temporal structure of interest points , 2012, ACM Multimedia.

[22]  Michael S. Ryoo,et al.  Human activity prediction: Early recognition of ongoing activities from streaming videos , 2011, 2011 International Conference on Computer Vision.

[23]  Fernando De la Torre,et al.  Joint segmentation and classification of human actions in video , 2011, CVPR 2011.

[24]  Kaoru Hirota,et al.  Detection algorithm for color dynamic images by multiple surveillance cameras under low luminance conditions based on fuzzy corresponding map , 2008, Appl. Soft Comput..

[25]  Chee Seng Chan,et al.  Logical Connectives and Operativeness of BK Sub-triangle Product in Fuzzy Inferencing , 2011 .

[26]  Fatih Murat Porikli,et al.  Covariance Tracking using Model Update Based on Lie Algebra , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[27]  L. J. Kohout,et al.  Interval-based reasoning in medical diagnosis , 1997, Proceedings Intelligent Information Systems. IIS'97.

[28]  Mubarak Shah,et al.  Action MACH a spatio-temporal Maximum Average Correlation Height filter for action recognition , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[29]  Ladislav J. Kohout,et al.  Semantics of implication operators and fuzzy relational products , 1980 .

[30]  Ekta Vats,et al.  Early human actions detection using BK sub-triangle product , 2015, 2015 IEEE International Conference on Fuzzy Systems (FUZZ-IEEE).

[31]  Kejun Wang,et al.  Video-Based Abnormal Human Behavior Recognition—A Review , 2012, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[32]  Honghai Liu,et al.  Fuzzy qualitative complex actions recognition , 2010, International Conference on Fuzzy Systems.

[33]  Ángel Carmona Poyato,et al.  Comparing evolutionary algorithms and particle filters for Markerless Human Motion Capture , 2014, Appl. Soft Comput..

[34]  Sven J. Dickinson,et al.  Recognize Human Activities from Partially Observed Videos , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[35]  Etienne Kerre,et al.  Fuzzy relational calculus in land evaluation. , 1997 .

[36]  Ruediger Dillmann,et al.  Human Motion Analysis: A Review , 1997 .

[37]  Ekta Vats,et al.  Fuzzy human motion analysis: A review , 2014, Pattern Recognit..

[38]  Mubarak Shah,et al.  Recognizing 50 human action categories of web videos , 2012, Machine Vision and Applications.

[39]  Alessia Saggese,et al.  Combining Neural Networks and Fuzzy Systems for Human Behavior Understanding , 2012, 2012 IEEE Ninth International Conference on Advanced Video and Signal-Based Surveillance.

[40]  Yun Fu,et al.  A Discriminative Model with Multiple Temporal Scales for Action Prediction , 2014, ECCV.

[41]  Larry S. Davis,et al.  Event Modeling and Recognition Using Markov Logic Networks , 2008, ECCV.

[42]  Fernando De la Torre,et al.  Max-Margin Early Event Detectors , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[43]  Chee Seng Chan,et al.  A weighted inference engine based on interval-valued fuzzy relational theory , 2015, Expert Syst. Appl..

[44]  Jean-Marie Le Yaouanc,et al.  A Fuzzy Spatio-temporal-Based Approach for Activity Recognition , 2012, ER Workshops.

[45]  Jean Ponce,et al.  Automatic annotation of human actions in video , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[46]  Thomas Hofmann,et al.  Large Margin Methods for Structured and Interdependent Output Variables , 2005, J. Mach. Learn. Res..

[47]  Jin Young Choi,et al.  Intelligent visual surveillance — A survey , 2010 .

[48]  Nobuyuki Arai,et al.  DNA based computing for understanding complex shapes , 2014, Biosyst..

[49]  Jake K. Aggarwal,et al.  Human motion analysis: a review , 1997, Proceedings IEEE Nonrigid and Articulated Motion Workshop.

[50]  Yong-Gi Kim,et al.  An obstacle-avoidance technique for autonomous underwater vehicles based on BK-products of fuzzy relation , 2006, Fuzzy Sets Syst..

[51]  Alex Pentland,et al.  Pfinder: Real-Time Tracking of the Human Body , 1997, IEEE Trans. Pattern Anal. Mach. Intell..

[52]  James M. Keller,et al.  Modeling Human Activity From Voxel Person Using Fuzzy Logic , 2009, IEEE Transactions on Fuzzy Systems.

[53]  Mubarak Shah,et al.  Motion-based recognition a survey , 1995, Image Vis. Comput..

[54]  Ronen Basri,et al.  Actions as Space-Time Shapes , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.