VSUMM: A mechanism designed to produce static video summaries and a novel evaluation method

The fast evolution of digital video has brought many new multimedia applications and, as a consequence, has increased the amount of research into new technologies that aim at improving the effectiveness and efficiency of video acquisition, archiving, cataloging and indexing, as well as increasing the usability of stored videos. Among possible research areas, video summarization is an important topic that potentially enables faster browsing of large video collections and also more efficient content indexing and access. Essentially, this research area consists of automatically generating a short summary of a video, which can either be a static summary or a dynamic summary. In this paper, we present VSUMM, a methodology for the production of static video summaries. The method is based on color feature extraction from video frames and k-means clustering algorithm. As an additional contribution, we also develop a novel approach for the evaluation of video static summaries. In this evaluation methodology, video summaries are manually created by users. Then, several user-created summaries are compared both to our approach and also to a number of different techniques in the literature. Experimental results show - with a confidence level of 98% - that the proposed solution provided static video summaries with superior quality relative to the approaches to which it was compared.

[1]  Chia-Hung Yeh,et al.  Techniques for movie content analysis and skimming: tutorial and overview on video abstraction techniques , 2006, IEEE Signal Processing Magazine.

[2]  Hans-Peter Kriegel,et al.  A Density-Based Algorithm for Discovering Clusters in Large Spatial Databases with Noise , 1996, KDD.

[3]  Arnaldo de Albuquerque Araújo,et al.  Video segmentation based on 2D image analysis , 2003, Pattern Recognit. Lett..

[4]  Harry W. Agius,et al.  Video summarisation: A conceptual framework and survey of the state of the art , 2008, J. Vis. Commun. Image Represent..

[5]  A. Murat Tekalp,et al.  Automatic Soccer Video Analysis and Summarization , 2003, IS&T/SPIE Electronic Imaging.

[6]  José María Martínez Sanchez,et al.  An efficient summarization algorithm based on clustering and bitstream extraction , 2009, 2009 IEEE International Conference on Multimedia and Expo.

[7]  Ioannis Pitas,et al.  Information theory-based shot cut/fade detection and video summarization , 2006, IEEE Transactions on Circuits and Systems for Video Technology.

[8]  J. Rissanen,et al.  Modeling By Shortest Data Description* , 1978, Autom..

[9]  Raj Jain,et al.  The art of computer systems performance analysis - techniques for experimental design, measurement, simulation, and modeling , 1991, Wiley professional computing.

[10]  H. Akaike A new look at the statistical model identification , 1974 .

[11]  Shingo Uchihashi,et al.  Video Manga: generating semantically meaningful video summaries , 1999, MULTIMEDIA '99.

[12]  Alain Trémeau,et al.  Color in Image and Video Processing: Most Recent Trends and Future Research Directions , 2008, EURASIP J. Image Video Process..

[13]  Benoit Huet,et al.  Automatic video summarization , 2006 .

[14]  Bernard Mérialdo,et al.  Multi-document video summarization , 2009, 2009 IEEE International Conference on Multimedia and Expo.

[15]  Alan Hanjalic,et al.  An integrated scheme for automated video abstraction based on unsupervised cluster-validity analysis , 1999, IEEE Trans. Circuits Syst. Video Technol..

[16]  S. Jørgensen The art of computer systems performance analysis: Techniques for Experimental Design, Measurement, Simulation and Modeling. Raj Jain. John Wiley, New York. Hardcover, 720 p. U.S. $52.95. , 1992 .

[17]  David S. Doermann,et al.  Video summarization by curve simplification , 1998, MULTIMEDIA '98.

[18]  J. MacQueen Some methods for classification and analysis of multivariate observations , 1967 .

[19]  Sansanee Auephanwiriyakul,et al.  On fuzzy clustering and content based access to networked video databases , 1998, Proceedings Eighth International Workshop on Research Issues in Data Engineering. Continuous-Media Databases and Applications.

[20]  Aggelos K. Katsaggelos,et al.  Temporal rate-distortion based optimal video summary generation , 2003, 2003 International Conference on Multimedia and Expo. ICME '03. Proceedings (Cat. No.03TH8698).

[21]  Irena Koprinska,et al.  Temporal video segmentation: A survey , 2001, Signal Process. Image Commun..

[22]  David G. Stork,et al.  Pattern Classification , 1973 .

[23]  ARTHUR G. MONEY,et al.  Video Summarisation : A Conceptual Framework and Survey , 2010 .

[24]  Avideh Zakhor,et al.  Efficient video similarity measurement with video signature , 2002, Proceedings. International Conference on Image Processing.

[25]  Ilangko Balasingham,et al.  Are the Wavelet Transforms the Best Filter Banks for Image Compression? , 2008, EURASIP J. Image Video Process..

[26]  Yelena Yesha,et al.  Keyframe-based video summarization using Delaunay clustering , 2006, International Journal on Digital Libraries.

[27]  A. Murat Tekalp,et al.  Automatic soccer video analysis and summarization , 2003, IEEE Trans. Image Process..

[28]  Janko Calic,et al.  Efficient Layout of Comic-Like Video Summaries , 2007, IEEE Transactions on Circuits and Systems for Video Technology.

[29]  Qi Tian,et al.  Multilevel video representation with application to keyframe extraction , 2004, 10th International Multimedia Modelling Conference, 2004. Proceedings..

[30]  Arnold W. M. Smeulders,et al.  Image Databases and Multi-Media Search , 1998, Image Databases and Multi-Media Search.

[31]  Ioannis Pitas,et al.  Video shot detection and condensed representation. a review , 2006, IEEE Signal Processing Magazine.

[32]  Rainer Lienhart,et al.  Comparison of automatic shot boundary detection algorithms , 1998, Electronic Imaging.

[33]  Michael J. Swain,et al.  Color indexing , 1991, International Journal of Computer Vision.

[34]  Matthieu Cord,et al.  VSUMM: An Approach for Automatic Video Summarization and Quantitative Evaluation , 2008, 2008 XXI Brazilian Symposium on Computer Graphics and Image Processing.

[35]  N. Nikolaidis,et al.  Video shot detection and condensed representation. a review , 2006, IEEE Signal Processing Magazine.

[36]  Avideh Zakhor,et al.  Efficient video similarity measurement with video signature , 2003, IEEE Trans. Circuits Syst. Video Technol..

[37]  Youssef Hadi,et al.  Video summarization by k-medoid clustering , 2006, SAC '06.

[38]  Roland Wilson,et al.  Unsupervised learning and clustering using a random field approach , 2007 .

[39]  Marco Pellegrini,et al.  STIMO: STIll and MOving video storyboard for the web scenario , 2009, Multimedia Tools and Applications.

[40]  Hans Weda,et al.  Video Segmentation , 2009, Encyclopedia of Database Systems.

[41]  Patrick Pérez,et al.  Rapid Summarisation and Browsing of Video Sequences , 2002, BMVC.

[42]  Nathalie Guyader,et al.  Video Summarization Based on Camera Motion and a Subjective Evaluation Method , 2007, EURASIP J. Image Video Process..

[43]  Aggelos K. Katsaggelos,et al.  MINMAX optimal video summarization , 2005, IEEE Transactions on Circuits and Systems for Video Technology.

[44]  Yueting Zhuang,et al.  Adaptive key frame extraction using unsupervised clustering , 1998, Proceedings 1998 International Conference on Image Processing. ICIP98 (Cat. No.98CB36269).

[45]  Tao Mei,et al.  Video Collage: A Novel Presentation of Video Sequence , 2007, 2007 IEEE International Conference on Multimedia and Expo.

[46]  Jhing-Fa Wang,et al.  A Novel Video Summarization Based on Mining the Story-Structure and Semantic Relations Among Concept Entities , 2009, IEEE Transactions on Multimedia.

[47]  B. S. Manjunath,et al.  Color and texture descriptors , 2001, IEEE Trans. Circuits Syst. Video Technol..

[48]  Alan Hanjalic,et al.  A New Method for Key Frame Based Video Content Representation , 1998, Image Databases and Multi-Media Search.

[49]  Ba Tu Truong,et al.  Video abstraction: A systematic review and classification , 2007, TOMCCAP.

[50]  H. L. Le Roy,et al.  Proceedings of the Fifth Berkeley Symposium on Mathematical Statistics and Probability; Vol. IV , 1969 .

[51]  Boon-Lock Yeo,et al.  Video visualization for compact presentation and fast browsing of pictorial content , 1997, IEEE Trans. Circuits Syst. Video Technol..

[52]  I-Cheng Chang,et al.  Content-Selection Based Video Summarization , 2007, 2007 Digest of Technical Papers International Conference on Consumer Electronics.

[53]  Tie-Yan Liu,et al.  Dynamic selection and effective compression of key frames for video abstraction , 2003, Pattern Recognit. Lett..

[54]  S. de Avila,et al.  VSUMM: A simple and efficient approach for automatic video summarization , 2008, IWSSIP 2008.

[55]  Jianping Fan,et al.  Exploring video content structure for hierarchical summarization , 2004, Multimedia Systems.

[56]  Xin Liu,et al.  Video summarization using singular value decomposition , 2000, Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.PR00662).

[57]  Wolfgang Effelsberg,et al.  Abstracting Digital Movies Automatically , 1996, J. Vis. Commun. Image Represent..