A narrative-based abstraction framework for story-oriented video

This article proposes a novel video abstraction framework for online review services of story-oriented videos such as dramas. Among the many genres of TV programs, a drama is one of the most popularly watched on the Web. The abstracts generated by the proposed framework not only give a summary of a video but also effectively help viewers understand the overall story. In addition, our method is duration-flexible. We get clues about human understanding of a story from scenario writing rules and editorial techniques that are popularly used in the process of video production to explicitly express a narrative, and propose a new video abstraction model, called a Narrative Abstraction Model. The model effectively captures the narrative structure embedded in a story-oriented video and articulates the progress of the story in a weighted directed graph, called a Narrative Structure Graph (NSG). The model provides a basis for a flexible framework for abstract generation using the NSG as the intermediary representation of a video. Different abstracts can be appropriately generated based upon different user requirements. To show the effectiveness of the proposed model and method, we developed a video abstraction system realizing the framework, and successfully applied it to large volumes of TV dramas. The evaluation results show that the proposed framework is a feasible solution for online review services.

[1]  John S. Boreczky,et al.  Comparison of video shot boundary detection techniques , 1996, J. Electronic Imaging.

[2]  Michael A. Smith,et al.  Video skimming and characterization through the combination of image and language understanding techniques , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[3]  D. Arijon,et al.  Grammar of Film Language , 1976 .

[4]  Svetha Venkatesh,et al.  Toward automatic extraction of expressive elements from motion pictures: tempo , 2002, IEEE Trans. Multim..

[5]  Jongho Nang,et al.  An Authoring Tool Generating Various Video Abstractions Semi-automatically , 2002, IEEE Pacific Rim Conference on Multimedia.

[6]  Francine Chen,et al.  A trainable document summarizer , 1995, SIGIR '95.

[7]  Shih-Fu Chang,et al.  Scene change detection in an MPEG-compressed video sequence , 1995, Electronic Imaging.

[8]  David Bordwell,et al.  Film Art: An Introduction , 1979 .

[9]  Boon-Lock Yeo,et al.  On the extraction of DC sequence from MPEG compressed video , 1995, Proceedings., International Conference on Image Processing.

[10]  Alan Hanjalic,et al.  An integrated scheme for automated video abstraction based on unsupervised cluster-validity analysis , 1999, IEEE Trans. Circuits Syst. Video Technol..

[11]  Newton Lee,et al.  ACM Transactions on Multimedia Computing, Communications and Applications (ACM TOMCCAP) , 2007, CIE.

[12]  Steve Renals,et al.  Indexing and retrieval of broadcast news , 2000, Speech Commun..

[13]  Shingo Uchihashi,et al.  Summarizing video using a shot importance measure and a frame-packing algorithm , 1999, 1999 IEEE International Conference on Acoustics, Speech, and Signal Processing. Proceedings. ICASSP99 (Cat. No.99CH36258).

[14]  Nuno Vasconcelos,et al.  A spatiotemporal motion model for video summarization , 1998, Proceedings. 1998 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No.98CB36231).

[15]  S. Chatman.  Story and Discourse: Narrative Structure in Fiction and Film , 1980 .

[16]  Boon-Lock Yeo,et al.  Video visualization for compact presentation and fast browsing of pictorial content , 1997, IEEE Trans. Circuits Syst. Video Technol..

[17]  Stefan Sharff.  The Elements of Cinema: Toward a Theory of Cinesthetic Impact , 1982 .

[18]  Stephen W. Smoliar,et al.  Video parsing, retrieval and browsing: an integrated and content-based solution , 1997, MULTIMEDIA '95.

[19]  Shih-Fu Chang,et al.  Computable scenes and structures in films , 2002, IEEE Trans. Multim..

[20]  Junehwa Song,et al.  Narrative abstraction model for story-oriented video , 2004, MULTIMEDIA '04.

[21]  Boon-Lock Yeo,et al.  Extracting story units from long programs for video browsing and navigation , 1996, Proceedings of the Third IEEE International Conference on Multimedia Computing and Systems.

[22]  Boon-Lock Yeo,et al.  Fast extraction of spatially reduced image sequences from MPEG-2 compressed video , 1999, IEEE Trans. Circuits Syst. Video Technol..

[23]  Boon-Lock Yeo,et al.  A fast algorithm for DCT-domain inverse motion compensation based on shared information in a macroblock , 2000, IEEE Trans. Circuits Syst. Video Technol..

[24]  John R. Kender,et al.  Video scene segmentation via continuous video coherence , 1998, Proceedings. 1998 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No.98CB36231).

[25]  Alan Hanjalic,et al.  Automated high-level movie segmentation for advanced video-retrieval systems , 1999, IEEE Trans. Circuits Syst. Video Technol..

[26]  Kevin M. Brooks.  Do story agents use rocking chairs? The theory and implementation of one model for computational narrative , 1997, MULTIMEDIA '96.

[27]  Tanveer F. Syeda-Mahmood,et al.  Learning video browsing behavior and its application in the generation of video previews , 2001, MULTIMEDIA '01.

[28]  Boon-Lock Yeo,et al.  A unified approach to temporal segmentation of motion JPEG and MPEG compressed video , 1995, Proceedings of the International Conference on Multimedia Computing and Systems.

[29]  Rainer Lienhart,et al.  Scene Determination Based on Video and Audio Features , 2004, Multimedia Tools and Applications.

[30]  Wolfgang Effelsberg,et al.  Abstracting Digital Movies Automatically , 1996, J. Vis. Commun. Image Represent..

[31]  Boon-Lock Yeo,et al.  Video browsing using clustering and scene transitions on compressed sequences , 1995, Electronic Imaging.

[32]  Anoop Gupta,et al.  Auto-summarization of audio-video presentations , 1999, MULTIMEDIA '99.

[33]  Jeho Nam,et al.  Video abstract of video , 1999, 1999 IEEE Third Workshop on Multimedia Signal Processing (Cat. No.99TH8451).

[34]  Alan Hanjalic,et al.  Indexing and retrieval of TV broadcast news using DANCERS , 2001, J. Electronic Imaging.

[35]  Boon-Lock Yeo,et al.  Rapid scene analysis on compressed video , 1995, IEEE Trans. Circuits Syst. Video Technol..

[36]  Wolfgang Effelsberg,et al.  Scene Determination Based on Video and Audio Features , 1999, Proceedings IEEE International Conference on Multimedia Computing and Systems.

[37]  Anoop Gupta,et al.  Time-compression: systems concerns, usage, and benefits , 1999, CHI '99.

[38]  Masahito Hirakawa,et al.  Content-based retrieval of video data by the grammar of film , 1997, Proceedings. 1997 IEEE Symposium on Visual Languages (Cat. No.97TB100180).

[39]  Noboru Babaguchi,et al.  Event based indexing of broadcasted sports video by intermodal collaboration , 2002, IEEE Trans. Multim..

[40]  Edward Branigan.  Narrative Comprehension and Film , 1992 .

[41]  Shih-Fu Chang,et al.  A utility framework for the automatic generation of audio-visual skims , 2002, MULTIMEDIA '02.

[42]  Nilesh V. Patel,et al.  Audio characterization for video indexing , 1996, Electronic Imaging.