A video retrieval and sequencing system

Video is an effective medium for capturing the events in the real world around us, and a vast amount of video materials exists, covering a wide range of applications. However, widespread use of video in computer applications is often impeded by the lack of effective tools to manage video information systematically. This article discusses the design and implementation of a frame-based video retrieval and sequencing system (VRSS). The system is designed to support the entire process of video information management: segmenting, indexing, retrieving, and sequencing of video data. A semiautomatic tool is developed to divide video sequences into meaningful shots. Each video shot is logged using text descriptions, audio dialogue, and cinematic attributes. A two-layered, concept-based model is used as the basis for accurately retrieving relevant video shots based on users' free-text queries. A cinematic, rule-based, virtual editing tool is also developed to sequence the video shots retrieved for presentation within a specified time constraint. The system has been tested on a video documentary on the NUS (National University of Singapore) engineering faculty. The results of video retrieval experiments are encouraging.

[1]  Gerald Millerson.  The technique of television production , 1961 .

[2]  Richard Dyer MacCann.  Film: A Montage of Theories , 1966 .

[3]  Marvin Minsky,et al.  A framework for representing knowledge , 1974 .

[4]  Marvin Minsky,et al.  A framework for representing knowledge" in the psychology of computer vision , 1975 .

[5]  Terry Noreault,et al.  Automatic ranked output from boolean searches in SIRE , 1977, J. Am. Soc. Inf. Sci..

[6]  Roger C. Schank,et al.  Scripts, plans, goals and understanding: an inquiry into human knowledge structures , 1978 .

[7]  Andrew Lippman,et al.  Movie-maps: An application of the optical videodisc to computer graphics , 1980, SIGGRAPH '80.

[8]  Michael McGill,et al.  Introduction to Modern Information Retrieval , 1983 .

[9]  Richard Kuehn Belew,et al.  Adaptive information retrieval: machine learning in associative networks (connectionist, free-text, browsing, feedback) , 1986 .

[10]  Richard M. Tong,et al.  Conceptual information retrieval using RUBRIC , 1987, SIGIR '87.

[11]  W. Bruce Croft,et al.  I 3 R: a new approach to the design of document retrieval systems , 1987 .

[12]  Philip J. Smith,et al.  EP-X: a demonstration of semantically based search of bibliographic databases , 1987, SIGIR '87.

[13]  E. Frisse Mark,et al.  Searching for information in a hypertext medical handbook , 1988 .

[14]  Gilles R. Bloch.  From Concepts to Film Sequences , 1988, RIAO.

[15]  Gerard Salton,et al.  Term-Weighting Approaches in Automatic Text Retrieval , 1988, Inf. Process. Manag..

[16]  Jeff Conklin,et al.  Hypertext: an introduction and survey (Reprint) , 1988 .

[17]  Kui-Lam Kwok.  A neural network for probabilistic information retrieval , 1989, SIGIR '89.

[18]  Wendy E. Mackay,et al.  Virtual video editing in interactive multimedia applications , 1989, CACM.

[19]  Scott M. Stevens.  Intelligent interactive video simulation of a code inspection , 1989, CACM.

[20]  G. Davenport,et al.  Structured content modeling for cinematic information , 1989, SGCH.

[21]  Gerard Salton,et al.  Improving retrieval performance by relevance feedback , 1997, J. Am. Soc. Inf. Sci..

[22]  Bruno Feijó,et al.  Intelligent Hypertext for Normative Knowledge in Engineering , 1990, ECHT.

[23]  Dario Lucarella,et al.  A Model for Hypertext-Based Information Retrieval , 1992, ECHT.

[24]  Yoshinobu Tonomura,et al.  Video handling based on structured information for hypermedia systems , 1991 .

[25]  Akio Nagasaka,et al.  Automatic Video Indexing and Full-Video Search for Object Appearances , 1991, VDB.

[26]  Amy Bruckman.  The electronic scrapbook : towards an intelligent home-video editing system , 1991 .

[27]  Simon J. Gibbs.  Composite multimedia and active objects , 1991, OOPSLA 1991.

[28]  Kristina Woolsey.  Multimedia scouting , 1991, IEEE Computer Graphics and Applications.

[29]  Natalio Pincever,et al.  Parsing Movies in Context , 1991, USENIX Summer.

[30]  Composite multimedia and active objects , 1991, OOPSLA '91.

[31]  Glorianna Davenport,et al.  Cinematic primitives for multimedia , 1991, IEEE Computer Graphics and Applications.

[32]  Tat-Seng Chua,et al.  Supporting Composition in a Hypermedia Environment , 1991, Hypermedia.

[33]  Paul Kahn,et al.  IRIS hypermedia services , 1992, CACM.

[34]  Tat-Seng Chua,et al.  Applying relevance feedback to a photo archival system , 1992, J. Inf. Sci..

[35]  Marc Davis,et al.  IDIC: assembling video sequences from story plans and content annotations , 1994, 1994 Proceedings of IEEE International Conference on Multimedia Computing and Systems.

[36]  Tat-Seng Chua,et al.  A concept-based image retrieval system , 1994, 1994 Proceedings of the Twenty-Seventh Hawaii International Conference on System Sciences.

[37]  Alex Pentland,et al.  Video and Image Semantics: Advanced Tools for Telecommunications , 1994, IEEE Multim..