Automatic Performance Evaluation for Video Summarization

Abstract : This paper describes a system for automated performance evaluation of video summarization algorithms. We call it SUPERSIEV (System for Unsupervised Performance Evaluation of Ranked Summarization in Extended Videos). It is primarily designed for evaluating video summarization algorithms that perform frame ranking. The task of summarization is viewed as a kind of database retrieval, and we adopt some of the concepts developed for performance evaluation of retrieval in database systems. First, ground truth summaries are gathered in a user study from many assessors and for several video sequences. For each video sequence, these summaries are combined to generate a single reference file that represents the majority of assessors opinions. Then the system determines the best target reference frame for each frame of the whole video sequence and computes matching scores to form a lookup table that rates each frame. Given a summary from a candidate summarization algorithm, the system can then evaluate this summary from different aspects by computing recall, cumulated average precision, redundancy rate and average closeness. With this evaluation system, we can not only grade the quality of a video summary, but also (1) compare different automatic summarization algorithms and (2) make stepwise improvements on algorithms, without the need for new user feedback.

[1]  Ellen M. Voorhees,et al.  Variations in relevance judgments and the measurement of retrieval effectiveness , 1998, SIGIR '98.

[2]  Mark S. Drew,et al.  Video keyframe production by efficient clustering of compressed chromaticity signatures (poster session) , 2000, ACM Multimedia.

[3]  David S. Doermann,et al.  Video summarization by curve simplification , 1998, MULTIMEDIA '98.

[4]  Anoop Gupta,et al.  Auto-summarization of audio-video presentations , 1999, MULTIMEDIA '99.

[5]  Yihong Gong,et al.  Lessons Learned from Building a Terabyte Digital Video Library , 1999, Computer.

[6]  Salim Roukos,et al.  Bleu: a Method for Automatic Evaluation of Machine Translation , 2002, ACL.

[7]  Horacio Saggion,et al.  Colouring Summaries BLEU , 2003 .

[8]  Christus,et al.  A General Method Applicable to the Search for Similarities in the Amino Acid Sequence of Two Proteins , 2022 .

[9]  David S. Doermann,et al.  Event detection from MPEG video in the compressed domain , 2000, Proceedings 15th International Conference on Pattern Recognition. ICPR-2000.

[10]  Ying Li,et al.  An Overview of Video Abstraction Techniques , 2001 .

[11]  Malcolm Slaney,et al.  Understanding the Semantics of Media , 2003 .

[12]  Shingo Uchihashi,et al.  Video Manga: generating semantically meaningful video summaries , 1999, MULTIMEDIA '99.

[13]  Shih-Fu Chang,et al.  A utility framework for the automatic generation of audio-visual skims , 2002, MULTIMEDIA '02.

[14]  Alan Hanjalic,et al.  An integrated scheme for automated video abstraction based on unsupervised cluster-validity analysis , 1999, IEEE Trans. Circuits Syst. Video Technol..

[15]  Regunathan Radhakrishnan,et al.  Video Summarization Using Mpeg-7 Motion Activity and Audio Descriptors , 2003 .

[16]  Xin Liu,et al.  Video summarization using singular value decomposition , 2000, Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.PR00662).

[17]  Azriel Rosenfeld,et al.  Relevance Ranking of Video Data using Hidden Markov Model Distances and Polygon Simplification , 2000, VISUAL.

[18]  Patrick Pérez,et al.  Rapid Summarisation and Browsing of Video Sequences , 2002, BMVC.