Incorporating frequent pattern analysis into multimodal HMM event classification for baseball videos

Data mining and frequent pattern analysis have recently become a popular way of discovering new knowledge from a data set. However, it is rarely applied to video semantic analysis. Therefore, this paper introduces two methods: frequent-pattern trained HMM and frequent-pattern tailored HMM to incorporate frequent pattern analysis into multimodal HMM event classification for baseball videos. Besides, different symbol coding methods including temporal sequence coding and co-occurrence symbol coding for multimodal HMM classification are compared. The results of our experiments on baseball video event classification demonstrate that integration of frequent pattern analysis could help to improve event classification performances.

[1]  Yen-Liang Chen,et al.  Mining Nonambiguous Temporal Patterns for Interval-Based Events , 2007, IEEE Transactions on Knowledge and Data Engineering.

[2]  Chien-Li Chou,et al.  HMM-based ball hitting event exploration system for broadcast baseball video , 2012, J. Vis. Commun. Image Represent..

[3]  Marcel Worring,et al.  Multimedia event-based video indexing using time intervals , 2005, IEEE Transactions on Multimedia.

[4]  Mubarak Shah,et al.  High-level event recognition in unconstrained videos , 2013, International Journal of Multimedia Information Retrieval.

[5]  Qiang Ji,et al.  Modeling Temporal Interactions with Interval Temporal Bayesian Networks for Complex Activity Recognition , 2013, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[6]  Mei Han,et al.  Extract highlights from baseball game video with hidden Markov models , 2002, Proceedings. International Conference on Image Processing.

[7]  Hui Cheng,et al.  Video event recognition using concept attributes , 2013, 2013 IEEE Workshop on Applications of Computer Vision (WACV).

[8]  Alberto Del Bimbo,et al.  Semantic annotation of soccer videos by visual instance clustering and spatial/temporal reasoning in ontologies , 2010, Multimedia Tools and Applications.

[9]  Alberto Del Bimbo,et al.  Event detection and recognition for semantic annotation of video , 2010, Multimedia Tools and Applications.

[10]  Mei Han,et al.  Maximum entropy model-based baseball highlight detection and classification , 2004, Comput. Vis. Image Underst..

[11]  Xiaofeng Wang,et al.  Ice hockey shooting event modeling with mixture hidden Markov model , 2010, Multimedia Tools and Applications.

[12]  Gang Hua,et al.  Semantic Model Vectors for Complex Video Event Recognition , 2012, IEEE Transactions on Multimedia.

[13]  Lexing Xie,et al.  Event Mining in Multimedia Streams , 2008, Proceedings of the IEEE.

[14]  Deb Roy,et al.  Temporal feature induction for baseball highlight classification , 2007, ACM Multimedia.

[15]  Deb Roy,et al.  Unsupervised content-based indexing of sports video , 2007, MIR '07.

[16]  Imran N. Junejo,et al.  Dynamic scene understanding using temporal association rules , 2014, Image Vis. Comput..

[17]  Alan Fern,et al.  Probabilistic event logic for interval-based event recognition , 2011, CVPR 2011.

[18]  Patrick Bouthemy,et al.  A unified approach to shot change detection and camera motion characterization , 1999, IEEE Trans. Circuits Syst. Video Technol..

[19]  Marcel Worring,et al.  A review on multimodal video indexing , 2002, Proceedings. IEEE International Conference on Multimedia and Expo.

[20]  Xueming Qian,et al.  Video content categorization using the double decomposition , 2012, Multimedia Tools and Applications.

[21]  James F. Allen Maintaining knowledge about temporal intervals , 1983, CACM.

[22]  Alberto Del Bimbo,et al.  Video event classification using string kernels , 2010, Multimedia Tools and Applications.

[23]  Amjad Rehman,et al.  Features extraction for soccer video semantic analysis: current achievements and remaining issues , 2012, Artificial Intelligence Review.

[24]  Christine Solnon,et al.  Classification of Images Based on Hidden Markov Models , 2009, 2009 Seventh International Workshop on Content-Based Multimedia Indexing.

[25]  Min Chen,et al.  Hierarchical Temporal Association Mining for Video Event Detection in Video Databases , 2007, 2007 IEEE 23rd International Conference on Data Engineering Workshop.

[26]  John H. L. Hansen,et al.  Multi-modal highlight generation for sports videos using an information-theoretic excitability measure , 2013, EURASIP J. Adv. Signal Process..

[27]  Ramesh C. Jain,et al.  A comprehensive study of visual event computing , 2011, Multimedia Tools and Applications.

[28]  Del BimboAlberto,et al.  Video event classification using string kernels , 2010 .

[29]  Yong Man Ro,et al.  Semantic Event Detection in Structured Video Using Hybrid HMM/SVM , 2005, CIVR.

[30]  Matthieu Guillaumin,et al.  Event Recognition in Photo Collections with a Stopwatch HMM , 2013, 2013 IEEE International Conference on Computer Vision.

[31]  Min Chen,et al.  Semantic event detection via multimodal data mining , 2006, IEEE Signal Processing Magazine.

[32]  Lawrence R. Rabiner,et al.  A tutorial on hidden Markov models and selected applications in speech recognition , 1989, Proc. IEEE.

[33]  Amir-Masoud Eftekhari-Moghadam,et al.  Multimodal feature extraction and fusion for semantic mining of soccer video: a survey , 2012, Artificial Intelligence Review.

[34]  Junqing Yu,et al.  Affection arousal based highlight extraction for soccer video , 2013, Multimedia Tools and Applications.

[35]  Wen-Jiin Tsai,et al.  A framework for video event classification by modeling temporal context of multimodal features using HMM , 2014, J. Vis. Commun. Image Represent..

[36]  Jianping Fan,et al.  A semi-supervised incremental learning method based on adaptive probabilistic hypergraph for video semantic detection , 2014, Multimedia Tools and Applications.

[37]  Christopher D. Carothers,et al.  VOGUE: A variable order hidden Markov model with duration based on frequent sequence mining , 2010, TKDD.

[38]  Mubarak Shah,et al.  Video Classification Using Semantic Concept Co-occurrences , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[39]  Deb Roy,et al.  Mining temporal patterns of movement for video content classification , 2006, MIR '06.

[40]  Noboru Babaguchi,et al.  A new spatio-temporal method for event detection and personalized retrieval of sports video , 2010, Multimedia Tools and Applications.

[41]  Barry Vercoe,et al.  Folk Music Classification Using Hidden Markov Models , 2001 .

[42]  Xindong Wu,et al.  Video data mining: semantic indexing and event detection from the association perspective , 2005, IEEE Transactions on Knowledge and Data Engineering.

[43]  Xueming Qian,et al.  HMM based soccer video event detection using enhanced mid-level semantic , 2011, Multimedia Tools and Applications.

[44]  Larry S. Davis,et al.  Clauselets: Leveraging Temporally Related Actions for Video Event Analysis , 2015, 2015 IEEE Winter Conference on Applications of Computer Vision.

[45]  Christopher D. Carothers,et al.  VOGUE: A Novel Variable Order-Gap State Machine for Modeling Sequences , 2006, PKDD.

[46]  Chang-Hsing Lee,et al.  Scene-based event detection for baseball videos , 2007, J. Vis. Commun. Image Represent..

[47]  Jenny Benois-Pineau,et al.  Hierarchical Hidden Markov Model in detecting activities of daily living in wearable videos for studies of dementia , 2011, Multimedia Tools and Applications.