Query by video clip

Abstract. Typical digital video search is based on queries involving a single shot. We generalize this problem by allowing queries that involve a video clip (say, a 10-s video segment). We propose two schemes: (i) retrieval based on key frames follows the traditional approach of identifying shots, computing key frames from a video, and then extracting image features around the key frames. For each key frame in the query, a similarity value (using color, texture, and motion) is obtained with respect to the key frames in the database video. Consecutive key frames in the database video that are highly similar to the query key frames are then used to generate the set of retrieved video clips. (ii) In retrieval using sub-sampled frames, we uniformly sub-sample the query clip as well as the database video. Retrieval is based on matching color and texture features of the sub-sampled frames. Initial experiments on two video databases (basketball video with approximately 16,000 frames and a CNN news video with approximately 20,000 frames) show promising results. Additional experiments using segments from one basketball video as query and a different basketball video as the database show the effectiveness of feature representation and matching schemes.

[1]  Wayne H. Wolf,et al.  Key frame selection by motion analysis , 1996, 1996 IEEE International Conference on Acoustics, Speech, and Signal Processing Conference Proceedings.

[2]  Ramesh C. Jain,et al.  Digital video segmentation , 1994, MULTIMEDIA '94.

[3]  Shih-Fu Chang,et al.  VideoQ: an automated content based video search system using visual cues , 1997, MULTIMEDIA '97.

[4]  A. Murat Tekalp,et al.  Video indexing through integration of syntactic and semantic features , 1996, Proceedings Third IEEE Workshop on Applications of Computer Vision. WACV'96.

[5]  Boon-Lock Yeo,et al.  Time-constrained clustering for segmentation of video into story units , 1996, Proceedings of 13th International Conference on Pattern Recognition.

[6]  Wei Xiong,et al.  Automatic video data structuring through shot partitioning and key-frame computing , 1997, Machine Vision and Applications.

[7]  Rakesh Mohan,et al.  Text-based search of TV news stories , 1996, Other Conferences.

[8]  Shih-Fu Chang,et al.  Video object model and segmentation for content-based video indexing , 1997, Proceedings of 1997 IEEE International Symposium on Circuits and Systems. Circuits and Systems in the Information Age ISCAS '97.

[9]  Boon-Lock Yeo,et al.  Rapid scene analysis on compressed video , 1995, IEEE Trans. Circuits Syst. Video Technol..

[10]  Anil K. Jain,et al.  Image retrieval using color and shape , 1996, Pattern Recognit..

[11]  Stephen W. Smoliar,et al.  An integrated system for content-based video retrieval and browsing , 1997, Pattern Recognit..

[12]  K. Wakimoto,et al.  Efficient and Effective Querying by Image Content , 1994 .

[13]  Jonathan D. Courtney Automatic video indexing via object motion analysis , 1997, Pattern Recognit..

[14]  Amarnath Gupta,et al.  Virage video engine , 1997, Electronic Imaging.

[15]  Yücel Altunbasak,et al.  Content-based video retrieval and compression: a unified solution , 1997, Proceedings of International Conference on Image Processing.

[16]  Anil K. Jain,et al.  Algorithms for Clustering Data , 1988 .

[17]  Yihong Gong,et al.  Automatic parsing of news video , 1994, 1994 Proceedings of IEEE International Conference on Multimedia Computing and Systems.

[18]  Stephen W. Smoliar,et al.  Content-based video browsing tools , 1995, Electronic Imaging.

[19]  Shih-Fu Chang,et al.  Clustering methods for video browsing and annotation , 1996, Electronic Imaging.