Video key frame extraction through dynamic Delaunay clustering with a structural constraint

Key frame based video summarization has emerged as an important area of research for the multimedia community. Video key frames enable an user to access any video in a friendly and meaningful way. In this paper, we propose an automated method of video key frame extraction using dynamic Delaunay graph clustering via an iterative edge pruning strategy. A structural constraint in form of a lower limit on the deviation ratio of the graph vertices further improves the video summary. We also employ an information-theoretic pre-sampling where significant valleys in the mutual information profile of the successive frames in a video are used to capture more informative frames. Various video key frame visualization techniques for efficient video browsing and navigation purposes are incorporated. A comprehensive evaluation on 100 videos from the Open Video and YouTube databases using both objective and subjective measures demonstrate the superiority of our key frame extraction method.

[1]  Ioannis Pitas,et al.  Information theory-based shot cut/fade detection and video summarization , 2006, IEEE Transactions on Circuits and Systems for Video Technology.

[2]  Andreas Girgensohn,et al.  Stained-glass visualization for highly condensed video summaries , 2004, 2004 IEEE International Conference on Multimedia and Expo (ICME) (IEEE Cat. No.04TH8763).

[3]  Alex Pothen,et al.  Computing the block triangular form of a sparse matrix , 1990, TOMS.

[4]  Sang Uk Lee,et al.  Efficient video indexing scheme for content-based retrieval , 1999, IEEE Trans. Circuits Syst. Video Technol..

[5]  Tie-Yan Liu,et al.  Shot reconstruction degree: a novel criterion for key frame selection , 2004, Pattern Recognit. Lett..

[6]  Malcolm Slaney,et al.  Precision-Recall Is Wrong for Multimedia , 2011, IEEE MultiMedia.

[7]  Stephen R. Gulliver,et al.  Introduction to special issue on eye-tracking applications in multimedia systems , 2007, TOMCCAP.

[8]  Avideh Zakhor,et al.  Content analysis of video using principal components , 1998, IEEE Trans. Circuits Syst. Video Technol..

[9]  Alan Hanjalic,et al.  An integrated scheme for automated video abstraction based on unsupervised cluster-validity analysis , 1999, IEEE Trans. Circuits Syst. Video Technol..

[10]  Marco Pellegrini,et al.  STIMO: STIll and MOving video storyboard for the web scenario , 2009, Multimedia Tools and Applications.

[11]  Gary Marchionini,et al.  Key frame preview techniques for video browsing , 1998, DL '98.

[12]  Yoshinobu Tonomura,et al.  VideoMAP and VideoSpaceIcon: tools for anatomizing video content , 1993, INTERCHI.

[13]  Tao Mei,et al.  Video Collage: A Novel Presentation of Video Sequence , 2007, 2007 IEEE International Conference on Multimedia and Expo.

[14]  Ioannis Pitas,et al.  Entropy metrics used for video summarization , 2002, SCCG '02.

[15]  Aggelos K. Katsaggelos,et al.  MINMAX optimal video summarization , 2005, IEEE Transactions on Circuits and Systems for Video Technology.

[16]  Xin Liu,et al.  Video summarization and retrieval using singular value decomposition , 2003, Multimedia Systems.

[17]  Raj Jain,et al.  The art of computer systems performance analysis - techniques for experimental design, measurement, simulation, and modeling , 1991, Wiley professional computing.

[18]  Shingo Uchihashi,et al.  Video Manga: generating semantically meaningful video summaries , 1999, MULTIMEDIA '99.

[19]  Clayton Brian Atkins.  Blocked recursive image composition , 2008, ACM Multimedia.

[20]  Wei-Ying Ma,et al.  Graph based multi-modality learning , 2005, ACM Multimedia.

[21]  Michael Ian Shamos,et al.  Computational geometry: an introduction , 1985 .

[22]  Satu Elisa Schaeffer,et al.  Graph Clustering , 2017, Encyclopedia of Machine Learning and Data Mining.

[23]  B. S. Manjunath,et al.  Color and texture descriptors , 2001, IEEE Trans. Circuits Syst. Video Technol..

[24]  Dragutin Petkovic,et al.  Key to effective video retrieval: effective cataloging and browsing , 1998, MULTIMEDIA '98.

[25]  Ba Tu Truong,et al.  Video abstraction: A systematic review and classification , 2007, TOMCCAP.

[26]  Arnaldo de Albuquerque Araújo,et al.  VSUMM: A mechanism designed to produce static video summaries and a novel evaluation method , 2011, Pattern Recognit. Lett..

[27]  Jian Yang,et al.  Feature fusion: parallel strategy vs. serial strategy , 2003, Pattern Recognit..

[28]  Yan Liu,et al.  A new method of feature fusion and its application in image recognition , 2005, Pattern Recognit..

[29]  Yelena Yesha,et al.  Keyframe-based video summarization using Delaunay clustering , 2006, International Journal on Digital Libraries.

[30]  Jurandy Almeida,et al.  Online video summarization on compressed domain , 2013, J. Vis. Commun. Image Represent..

[31]  Tie-Yan Liu,et al.  Frame interpolation scheme using inertia motion prediction , 2003, Signal Process. Image Commun..

[32]  Michael J. Swain,et al.  Color indexing , 1991, International Journal of Computer Vision.

[33]  Georgios S. Paschos,et al.  Perceptually uniform color spaces for color texture analysis: an empirical evaluation , 2001, IEEE Trans. Image Process..

[34]  José María Martínez Sanchez,et al.  An efficient summarization algorithm based on clustering and bitstream extraction , 2009, 2009 IEEE International Conference on Multimedia and Expo.

[35]  Boon-Lock Yeo,et al.  Video visualization for compact presentation and fast browsing of pictorial content , 1997, IEEE Trans. Circuits Syst. Video Technol..

[36]  Harry W. Agius,et al.  Video summarisation: A conceptual framework and survey of the state of the art , 2008, J. Vis. Commun. Image Represent..

[37]  Ananda S. Chowdhury,et al.  Video storyboard design using Delaunay graphs , 2012, Proceedings of the 21st International Conference on Pattern Recognition (ICPR2012).

[38]  Jurandy Almeida,et al.  VISON: VIdeo Summarization for ONline applications , 2012, Pattern Recognit. Lett..

[39]  Charu C. Aggarwal,et al.  Graph Clustering , 2010, Encyclopedia of Machine Learning and Data Mining.

[40]  Gary Marchionini,et al.  Open video: A framework for a test collection , 2000, J. Netw. Comput. Appl..

[41]  Sing-Tze Bow,et al.  Pattern recognition and image preprocessing , 1992 .

[42]  J. O´Rourke,et al.  Computational Geometry in C: Arrangements , 1998 .

[43]  Raimondo Schettini,et al.  Erratum to: An innovative algorithm for key frame extraction in video summarization , 2006, Journal of Real-Time Image Processing.

[44]  Jian Yang,et al.  Generalized K-L transform based combined feature extraction , 2002, Pattern Recognit..

[45]  Joseph O'Rourke,et al.  Computational Geometry in C. , 1995 .

[46]  Raj Jain,et al.  The Art of Computer Systems Performance Analysis : Tech-niques for Experimental Design , 1991 .

[47]  Shih-Fu Chang,et al.  A fully automated content-based video search engine supporting spatiotemporal queries , 1998, IEEE Trans. Circuits Syst. Video Technol..

[48]  Mohan S. Kankanhalli,et al.  Semantic video summarization in compressed domain MPEG video , 2003, 2003 International Conference on Multimedia and Expo. ICME '03. Proceedings (Cat. No.03TH8698).