A fully automated content-based video search engine supporting spatiotemporal queries

The rapidity with which digital information, particularly video, is being generated has necessitated the development of tools for efficient search of these media. Content-based visual queries have been primarily focused on still image retrieval. In this paper, we propose a novel, interactive system on the Web, based on the visual paradigm, with spatiotemporal attributes playing a key role in video retrieval. We have developed innovative algorithms for automated video object segmentation and tracking, and use real-time video editing techniques while responding to user queries. The resulting system, called VideoQ , is the first on-line video search engine supporting automatic object-based indexing and spatiotemporal queries. The system performs well, with the user being able to retrieve complex video clips such as those of skiers and baseball players with ease.

[1]  Shih-Fu Chang,et al.  CVEPS - a compressed video editing and parsing system , 1997, MULTIMEDIA '96.

[2]  Alberto Del Bimbo,et al.  Visual information retrieval , 1999 .

[3]  Svetha Venkatesh,et al.  Spatial Indexing for Video Databases , 1996, J. Vis. Commun. Image Represent..

[4]  Emanuele Trucco,et al.  Geometric Invariance in Computer Vision , 1995 .

[5]  Forouzan Golshani,et al.  Rx for semantic video database retrieval , 1994, MULTIMEDIA '94.

[6]  K. Wakimoto,et al.  Efficient and Effective Querying by Image Content , 1994 .

[7]  Boon-Lock Yeo,et al.  Video content characterization and compaction for digital library applications , 1997, Electronic Imaging.

[8]  Dragutin Petkovic,et al.  Query by Image and Video Content: The QBIC System , 1995, Computer.

[9]  Edward H. Adelson,et al.  Representing moving images with layers , 1994, IEEE Trans. Image Process..

[10]  A. Murat Tekalp,et al.  Region-Based Shape Matching for Automatic Image Annotation and Query-by-Example , 1997, J. Vis. Commun. Image Represent..

[11]  Shih-Fu Chang,et al.  Finding Images/Video in Large Archives: Columbia's Content-Based Visual Query Project , 1997, D Lib Mag..

[12]  David B. Cooper,et al.  Describing Complicated Objects by Implicit Polynomials , 1994, IEEE Trans. Pattern Anal. Mach. Intell..

[13]  Esther M. Arkin,et al.  An efficiently computable metric for comparing polygonal shapes , 1991, SODA '90.

[14]  M. Bierling,et al.  Displacement Estimation By Hierarchical Blockmatching , 1988, Other Conferences.

[15]  Thomas P. Minka,et al.  An image database browser that learns from user interaction , 1996 .

[16]  Karen Spärck Jones,et al.  Open-vocabulary speech indexing for voice and video mail retrieval , 1997, MULTIMEDIA '96.

[17]  Shi-Kuo Chang,et al.  Iconic Indexing by 2-D Strings , 1987, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[18]  Toshikazu Kato,et al.  Query by Visual Example - Content based Image Retrieval , 1992, EDBT.

[19]  Behzad Shahraray,et al.  Automatic generation of pictorial transcripts of video programs , 1995, Electronic Imaging.

[20]  Shi-Kuo Chang,et al.  Image sequence compression by iconic indexing , 1989, [Proceedings] 1989 IEEE Workshop on Visual Languages.

[21]  David B. Cooper,et al.  Computationally fast Bayesian recognition of complex objects based on mutual algebraic invariants , 1995, Proceedings., International Conference on Image Processing.

[22]  Alexander G. Hauptmann,et al.  Text, Speech, and Vision for Video Segmentation: The InformediaTM Project , 1995 .

[23]  A. Murat Tekalp,et al.  Motion segmentation by multistage affine classification , 1997, IEEE Trans. Image Process..

[24]  Shih-Fu Chang,et al.  Scene change detection in an MPEG-compressed video sequence , 1995, Electronic Imaging.

[25]  Alberto Del Bimbo,et al.  Visual Image Retrieval by Elastic Matching of User Sketches , 1997, IEEE Trans. Pattern Anal. Mach. Intell..

[26]  Hideyuki Tamura,et al.  Textural Features Corresponding to Visual Perception , 1978, IEEE Transactions on Systems, Man, and Cybernetics.

[27]  Rakesh Mohan,et al.  Text-based search of TV news stories , 1996, Other Conferences.

[28]  Shih-Fu Chang,et al.  Video object model and segmentation for content-based video indexing , 1997, Proceedings of 1997 IEEE International Symposium on Circuits and Systems. Circuits and Systems in the Information Age ISCAS '97.

[29]  Stephen W. Smoliar,et al.  Content based video indexing and retrieval , 1994, IEEE MultiMedia.

[30]  A. Murat Tekalp,et al.  Fusion of color and edge information for improved segmentation and edge linking , 1997, Image Vis. Comput..

[31]  Shih-Fu Chang,et al.  VisualSEEk: a fully automated content-based image query system , 1997, MULTIMEDIA '96.

[32]  Amarnath Gupta,et al.  Virage video engine , 1997, Electronic Imaging.

[33]  David Salesin,et al.  Fast multiresolution image querying , 1995, SIGGRAPH.

[34]  Touradj Ebrahimi,et al.  Morphological moving object segmentation and tracking for content-based video coding , 1995 .

[35]  Harpreet S. Sawhney,et al.  Model-based 2D&3D dominant motion estimation for mosaicing and video representation , 1995, Proceedings of IEEE International Conference on Computer Vision.