A survey on the use of pattern recognition methods for abstraction, indexing and retrieval of images and video

The need for content-based access to image and video information from media archives has captured the attention of researchers in recent years. Research e0orts have led to the development of methods that provide access to image and video data. These methods have their roots in pattern recognition. The methods are used to determine the similarity in the visual information content extracted from low level features. These features are then clustered for generation of database indices. This paper presents a comprehensive surveyon the use of these pattern recognition methods which enable image and video retrieval bycontent. ? 2002 Pattern Recognition Society. Published by Elsevier Science Ltd. All rights reserved.

[1]  Akio Nagasaka,et al.  Automatic Video Indexing and Full-Video Search for Object Appearances , 1991, VDB.

[2]  Shih-Fu Chang,et al.  Quad-tree segmentation for texture-based image query , 1994, MULTIMEDIA '94.

[3]  Alberto Del Bimbo,et al.  Retrieval of commercials by video semantics , 1998, Proceedings. 1998 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No.98CB36231).

[4]  Yihong Gong,et al.  Video parsing using compressed data , 1994, Electronic Imaging.

[5]  Alan Hanjalic,et al.  Optimal shot boundary detection based on robust statistical models , 1999, Proceedings IEEE International Conference on Multimedia Computing and Systems.

[6]  Nilesh V. Patel,et al.  Video shot detection and characterization for video databases , 1997, Pattern Recognit..

[7]  Alan Hanjalic,et al.  Automatically Segmenting Movies into Logical Story Units , 1999, VISUAL.

[8]  Jia Wang,et al.  Efficient access to and retrieval from a shape image database , 1998, Proceedings. IEEE Workshop on Content-Based Access of Image and Video Libraries (Cat. No.98EX173).

[9]  A. Murat Tekalp,et al.  Shape similarity matching for query-by-example , 1998, Pattern Recognit..

[10]  Euripides G. M. Petrakis,et al.  Efficient retrieval by shape content , 1999, Proceedings IEEE International Conference on Multimedia Computing and Systems.

[11]  Mohan S. Kankanhalli,et al.  Cluster-based color matching for image retrieval , 1996, Pattern Recognit..

[12]  William I. Grosky,et al.  Spatial color indexing: a novel approach for content-based image retrieval , 1999, Proceedings IEEE International Conference on Multimedia Computing and Systems.

[13]  Anil K. Jain,et al.  Object localization using color, texture and shape , 2000, Pattern Recognit..

[14]  Dominique Barba,et al.  Binkey: a system for video content analysis "on the fly" , 1999, Proceedings IEEE International Conference on Multimedia Computing and Systems.

[15]  Alberto Del Bimbo,et al.  Visual Querying By Color Perceptive Regions , 1998, Pattern Recognit..

[16]  Linda G. Shapiro,et al.  A Flexible Image Database System for Content-Based Retrieval , 1999, Comput. Vis. Image Underst..

[17]  Chahab Nastar,et al.  Relevance feedback and category search in image databases , 1999, Proceedings IEEE International Conference on Multimedia Computing and Systems.

[18]  Shi-Nine Yang,et al.  Color image retrieval based on hidden Markov models , 1997, IEEE Trans. Image Process..

[19]  Nuno Vasconcelos,et al.  Statistical models of video structure for content analysis and characterization , 2000, IEEE Trans. Image Process..

[20]  Wolfgang Effelsberg,et al.  Video abstracting , 1997, CACM.

[21]  Michael Stonebraker,et al.  Chabot: Retrieval from a Relational Database of Images , 1995, Computer.

[22]  Yueting Zhuang,et al.  Adaptive key frame extraction using unsupervised clustering , 1998, Proceedings 1998 International Conference on Image Processing. ICIP98 (Cat. No.98CB36269).

[23]  Alberto Del Bimbo,et al.  Effective image retrieval using deformable templates , 1996, Proceedings of 13th International Conference on Pattern Recognition.

[24]  Simone Santini,et al.  Image retrieval by shape and texture , 1999, Pattern Recognit..

[25]  Aleksandra Mojsilovic,et al.  Matching and retrieval based on the vocabulary and grammar of color patterns , 1999, Proceedings IEEE International Conference on Multimedia Computing and Systems.

[26]  Rangasami L. Kashyap,et al.  Models for motion-based video indexing and retrieval , 2000, IEEE Trans. Image Process..

[27]  Shih-Fu Chang,et al.  Visual information retrieval from large distributed online repositories , 1997, CACM.

[28]  Gerhard Rigoll,et al.  Multimedia database retrieval using hand-drawn sketches , 1999, Proceedings of the Fifth International Conference on Document Analysis and Recognition. ICDAR '99 (Cat. No.PR00318).

[29]  Patrick Bouthemy,et al.  Scene Segmentation and Image Feature Extraction for Video Indexing and Retrieval , 1999, VISUAL.

[30]  Sang Uk Lee,et al.  Efficient video indexing scheme for content-based retrieval , 1999, IEEE Trans. Circuits Syst. Video Technol..

[31]  Sharad Mehrotra,et al.  Query reformulation for content based multimedia retrieval in MARS , 1999, Proceedings IEEE International Conference on Multimedia Computing and Systems.

[32]  Ralf Steinmetz,et al.  Automatic Recognition of Camera Zooms , 1999, VISUAL.

[33]  Eugenio Di Sciascio,et al.  Content-Based Image Retrieval over the Web Using Query by Sketch and Relevance Feedback , 1999, VISUAL.

[34]  Patrick Bouthemy,et al.  Motion-Based Feature Extraction and Ascendant Hierarchical Classification for Video Indexing and Retrieval , 1999, VISUAL.

[35]  Raj Acharya,et al.  Color clustering techniques for color-content-based image retrieval from image databases , 1997, Proceedings of IEEE International Conference on Multimedia Computing and Systems.

[36]  Michael J. Swain,et al.  Color indexing , 1991, International Journal of Computer Vision.

[37]  Simone Santini,et al.  Beyond query by example , 1998, MULTIMEDIA '98.

[38]  W PicardRosalind,et al.  Periodicity, Directionality, and Randomness , 1996 .

[39]  U. Gargi,et al.  Image database querying using a multi-scale localized color representation , 1999, Proceedings IEEE Workshop on Content-Based Access of Image and Video Libraries (CBAIVL'99).

[40]  Carole A. Goble,et al.  The Manchester Multimedia Information System , 1992, EDBT.

[41]  No Value,et al.  IEEE International Conference on Image Processing , 2003 .

[42]  Benjamin B. Kimia,et al.  Symmetry-Based Indexing of Image Databases , 1998, J. Vis. Commun. Image Represent..

[43]  Yan Gong,et al.  Intelligent image databases - towards advanced image retrieval , 1997, The Kluwer international series in engineering and computer science.

[44]  Konstantinos N. Plataniotis,et al.  Distance measures for color image retrieval , 1998, Proceedings 1998 International Conference on Image Processing. ICIP98 (Cat. No.98CB36269).

[45]  M. Lew.  Content Based Image Retrieval : KLT , Projections , or Templates , 1996 .

[46]  Hideo Hashimoto,et al.  Video indexing using motion vectors , 1992, Other Conferences.

[47]  John S. Boreczky,et al.  Comparison of video shot boundary detection techniques , 1996, J. Electronic Imaging.

[48]  Thomas S. Huang,et al.  Exploring video structure beyond the shots , 1998, Proceedings. IEEE International Conference on Multimedia Computing and Systems (Cat. No.98TB100241).

[49]  Sougata Mukherjea,et al.  Integrating image matching and classification for multimedia retrieval on the Web , 1999, Proceedings IEEE International Conference on Multimedia Computing and Systems.

[50]  Stefanos D. Kollias,et al.  A Stochastic Framework for Optimal Key Frame Extraction from MPEG Video Databases , 1999, Comput. Vis. Image Underst..

[51]  Babu M. Mehtre,et al.  Content-based retrieval for trademark registration , 1996, Multimedia Tools and Applications.

[52]  Avideh Zakhor,et al.  Content analysis of video using principal components , 1998, IEEE Trans. Circuits Syst. Video Technol..

[53]  Raimondo Schettini,et al.  Multiresolution wavelet transform and supervised learning for content-based image retrieval , 1999, Proceedings IEEE International Conference on Multimedia Computing and Systems.

[54]  Wolfgang Effelsberg,et al.  VisualGREP: A Systematic Method to Compare and Retrieve Video Sequences , 2004, Multimedia Tools and Applications.

[55]  James Lee Hafner,et al.  Efficient Color Histogram Indexing for Quadratic Form Distance Functions , 1995, IEEE Trans. Pattern Anal. Mach. Intell..

[56]  Aya Soffer.  Image categorization using N x M grams , 1997, Electronic Imaging.

[57]  Boon-Lock Yeo,et al.  A unified approach to temporal segmentation of motion JPEG and MPEG compressed video , 1995, Proceedings of the International Conference on Multimedia Computing and Systems.

[58]  Rainer Lienhart,et al.  Comparison of automatic shot boundary detection algorithms , 1998, Electronic Imaging.

[59]  Alex Pentland,et al.  Photobook: Content-based manipulation of image databases , 1996, International Journal of Computer Vision.

[60]  Sethuraman Panchanathan,et al.  Video segmentation in the wavelet domain , 1998, Other Conferences.

[61]  Anil K. Jain,et al.  Shape-Based Retrieval: A Case Study With Trademark Image Databases , 1998, Pattern Recognit..

[62]  Yihong Gong.  An accurate and robust method for detecting video shot boundaries , 1999, Proceedings IEEE International Conference on Multimedia Computing and Systems.

[63]  Stan Sclaroff,et al.  Deformable prototypes for encoding shape categories in image databases , 1995, Pattern Recognit..

[64]  Ken Kennedy,et al.  A nationwide parallel computing environment , 1997, CACM.

[65]  Ingemar J. Cox,et al.  The Bayesian image retrieval system, PicHunter: theory, implementation, and psychophysical experiments , 2000, IEEE Trans. Image Process..

[66]  Gregory L. Zick,et al.  Scene decomposition of MPEG-compressed video , 1995, Electronic Imaging.

[67]  Wolfgang Effelsberg,et al.  On the detection and recognition of television commercials , 1997, Proceedings of IEEE International Conference on Multimedia Computing and Systems.

[68]  Charles A. Bouman,et al.  Storage and Retrieval for Image and Video Databases VII , 1998 .

[69]  Michael S. Lew,et al.  IRUS: image retrieval using shape , 1999, Proceedings IEEE International Conference on Multimedia Computing and Systems.

[70]  Shih-Fu Chang,et al.  Scene change detection in an MPEG-compressed video sequence , 1995, Electronic Imaging.

[71]  Ullas Gargi,et al.  Performance characterization of video-shot-change detection methods , 2000, IEEE Trans. Circuits Syst. Video Technol..

[72]  Ruggero Milanese,et al.  A Rotation, Translation, and Scale-Invariant Approach to Content-Based Image Retrieval , 1999, J. Vis. Commun. Image Represent..

[73]  Mourad Cherfaoui,et al.  Temporal segmentation of videos: a new approach , 1995, Electronic Imaging.

[74]  Jing Huang,et al.  Spatial Color Indexing and Applications , 2004, International Journal of Computer Vision.

[75]  Tom Minka,et al.  Interactive learning with a "society of models" , 1997, Pattern Recognit..

[76]  Ullas Gargi,et al.  Performance characterization and comparison of video indexing algorithms , 1998, Proceedings. 1998 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No.98CB36231).

[77]  Sethuraman Panchanathan,et al.  A scene change detection algorithm for MPEG compressed video sequences , 1995, Proceedings 1995 Canadian Conference on Electrical and Computer Engineering.

[78]  Luigi Cinque,et al.  Color-based image retrieval using spatial-chromatic histograms , 1999, Proceedings IEEE International Conference on Multimedia Computing and Systems.

[79]  Faouzi Ghorbel,et al.  Invariant content-based image retrieval using a complete set of Fourier-Mellin descriptors , 1999, Proceedings IEEE International Conference on Multimedia Computing and Systems.

[80]  A. Murat Tekalp,et al.  Content-based video abstraction , 1998, Proceedings 1998 International Conference on Image Processing. ICIP98 (Cat. No.98CB36269).

[81]  Thomas S. Huang,et al.  Automatic Matching Tool Selection Using Relevance Feedback In Mars , 1997 .

[82]  Dragutin Petkovic,et al.  Key to effective video retrieval: effective cataloging and browsing , 1998, MULTIMEDIA '98.

[83]  Raimondo Schettini,et al.  Using a Relevance Feedback Mechanism to Improve Content-Based Image Retrieval , 1999, VISUAL.

[84]  Anil K. Jain,et al.  On image classification: city images vs. landscapes , 1998, Pattern Recognit..

[85]  John P. Oakley,et al.  Storage and Retrieval for Image and Video Databases , 1993 .

[86]  Aya Sooer.  Image Categorization Using N M-grams , 1997 .

[87]  Arnold W. M. Smeulders,et al.  PicToSeek: combining color and shape invariant features for image retrieval , 2000, IEEE Trans. Image Process..

[88]  Ehud Rivlin,et al.  Invariant-Based Shape Retrieval in Pictorial Databases , 1998, Comput. Vis. Image Underst..

[89]  John R. Smith,et al.  Image Classification and Querying Using Composite Region Templates , 1999, Comput. Vis. Image Underst..

[90]  Boon-Lock Yeo,et al.  Segmentation of Video by Clustering and Graph Analysis , 1998, Comput. Vis. Image Underst..

[91]  Mohan S. Kankanhalli,et al.  Color matching for image retrieval , 1995, Pattern Recognit. Lett..

[92]  Liming Chen,et al.  Efficient content-based image retrieval based on color homogeneous objects segmentation and their spatial relationship characterization , 1999, Proceedings IEEE International Conference on Multimedia Computing and Systems.

[93]  Boon-Lock Yeo,et al.  Video visualization for compact presentation and fast browsing of pictorial content , 1997, IEEE Trans. Circuits Syst. Video Technol..

[94]  Chung-Sheng Li,et al.  MMAP: modified maximum a posteriori algorithm for image segmentation in large image/video databases , 1997, Proceedings of International Conference on Image Processing.

[95]  S. Sclaroff,et al.  Combining textual and visual cues for content-based image retrieval on the World Wide Web , 1998, Proceedings. IEEE Workshop on Content-Based Access of Image and Video Libraries (Cat. No.98EX173).

[96]  Jitendra Malik,et al.  Blobworld: A System for Region-Based Image Indexing and Retrieval , 1999, VISUAL.

[97]  Jianying Hu,et al.  Matching and retrieval based on the vocabulary and grammar of color patterns , 2000, IEEE Trans. Image Process..

[98]  Paul Scheunders,et al.  A comparison of clustering algorithms applied to color image quantization , 1997, Pattern Recognit. Lett..

[99]  Ze-Nian Li,et al.  Illumination Invariance and Object Model in Content-Based Image and Video Retrieval , 1999, J. Vis. Commun. Image Represent..

[100]  Ramin Zabih,et al.  A feature-based algorithm for detecting and classifying scene breaks , 1995, MULTIMEDIA '95.

[101]  Boon-Lock Yeo,et al.  Rapid scene analysis on compressed video , 1995, IEEE Trans. Circuits Syst. Video Technol..

[102]  Arding Hsu,et al.  Feature management for large video databases , 1993, Electronic Imaging.

[103]  Bir Bhanu,et al.  Probabilistic Feature Relevance Learning for Content-Based Image Retrieval , 1999, Comput. Vis. Image Underst..

[104]  David B. Cooper,et al.  Object signature curve and invariant shape patches for geometric indexing into pictorial databases , 1997, Other Conferences.

[105]  Samuel Moon-Ho Song,et al.  Morphological approach to scene change detection and digital video storage and retrieval , 1998, Electronic Imaging.

[106]  David G. Stork,et al.  Pattern Classification , 1973 .

[107]  Simone Santini,et al.  In search of information in visual media , 1997, CACM.

[108]  Mohan S. Kankanhalli,et al.  Color and spatial feature for content-based image retrieval , 1999, Pattern Recognit. Lett..

[109]  Behzad Shahraray,et al.  Scene change detection and content-based sampling of video sequences , 1995, Electronic Imaging.

[110]  B. S. Manjunath,et al.  Texture Features for Browsing and Retrieval of Image Data , 1996, IEEE Trans. Pattern Anal. Mach. Intell..

[111]  Thomas D. C. Little,et al.  A Survey of Technologies for Parsing and Indexing Digital Video1 , 1996, J. Vis. Commun. Image Represent..

[112]  F. Arman,et al.  A Statistical Approach to Scene Change Detection , 1995 .

[113]  Eli Upfal,et al.  Updates to the QBIC system , 1997, Electronic Imaging.

[114]  Alberto Del Bimbo,et al.  Structured representation and automatic indexing of movie information content , 1998, Pattern Recognit..

[115]  Simone Santini,et al.  Similarity Measures , 1999, IEEE Trans. Pattern Anal. Mach. Intell..

[116]  Anil K. Jain,et al.  Statistical Pattern Recognition: A Review , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[117]  Edward J. Delp,et al.  A fast algorithm for video parsing using MPEG compressed sequences , 1995, Proceedings., International Conference on Image Processing.

[118]  Ramesh C. Jain,et al.  Pattern Recognition Methods in Image and Video Databases: Past, Present and Future , 1998, SSPR/SPR.

[119]  Alan Hanjalic,et al.  An integrated scheme for automated video abstraction based on unsupervised cluster-validity analysis , 1999, IEEE Trans. Circuits Syst. Video Technol..

[120]  Keshi Chen,et al.  Similarity-based retrieval of images using color histograms , 1998, Electronic Imaging.

[121]  Raimondo Schettini,et al.  Color-based image retrieval using spatial-chromatic histograms , 2001, Image Vis. Comput..

[122]  Ramin Zabih,et al.  Comparing images using joint histograms , 1999, Multimedia Systems.

[123]  Ralph M. Ford.  Quantitative comparison of shot boundary detection metrics , 1998, Electronic Imaging.

[124]  David S. Doermann,et al.  Special-effect edit detection using VideoTrails: a comparison with existing techniques , 1998, Electronic Imaging.

[125]  Shih-Fu Chang,et al.  An integrated approach for content-based video object segmentation and retrieval , 1999, IEEE Trans. Circuits Syst. Video Technol..

[126]  Alberto Del Bimbo,et al.  Color-induced image representation and retrieval , 1999, Pattern Recognit..

[127]  Henning Müller,et al.  Relevance Feedback and Term Weighting Schemes for Content-Based Image Retrieval , 1999, VISUAL.

[128]  Roberto Brunelli,et al.  On the use of histograms for image retrieval , 1999, Proceedings IEEE International Conference on Multimedia Computing and Systems.

[129]  Hong Heather Yu,et al.  A Hierarchical Multiresolution Video Shot Transition Detection Scheme , 1999, Comput. Vis. Image Underst..

[130]  Alberto Del Bimbo,et al.  Semantics in Visual Information Retrieval , 1999, IEEE Multim..

[131]  M. Smith,et al.  Video Skimming for Quick Browsing based on Audio and Image Characterization , 1995 .

[132]  Mohamed Abdel-Mottaleb,et al.  A scalable algorithm for image retrieval by color , 1998, Proceedings 1998 International Conference on Image Processing. ICIP98 (Cat. No.98CB36269).

[133]  R. Brunelli,et al.  A Survey on the Automatic Indexing of Video Data, , 1999, J. Vis. Commun. Image Represent..

[134]  Min Wu,et al.  An algorithm for wipe detection , 1998, Proceedings 1998 International Conference on Image Processing. ICIP98 (Cat. No.98CB36269).

[135]  Paul D. Gader,et al.  Image content retrieval from image databases using feature integration by Choquet integral , 1998, Electronic Imaging.

[136]  Joachim M. Buhmann,et al.  Non-parametric similarity measures for unsupervised texture segmentation and image retrieval , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[137]  Alberto Del Bimbo,et al.  Visual information retrieval , 1999 .

[138]  Sethuraman Panchanathan,et al.  Fast Wavelet Histogram Techniques for Image Indexing , 1999, Comput. Vis. Image Underst..

[139]  Edoardo Ardizzone,et al.  Video indexing using MPEG motion compensation vectors , 1999, Proceedings IEEE International Conference on Multimedia Computing and Systems.

[140]  Markus A. Stricker.  Bounds for the discrimination power of color indexing techniques , 1994, Electronic Imaging.

[141]  Marco La Cascia,et al.  Unifying Textual and Visual Cues for Content-Based Image Retrieval on the World Wide Web , 1999, Comput. Vis. Image Underst..

[142]  Sadegh Abbasi,et al.  Retrieval of Similar Shapes under Affine Transform , 1999, VISUAL.

[143]  Wei Xiong,et al.  Query by video clip , 1998, Proceedings. Fourteenth International Conference on Pattern Recognition (Cat. No.98EX170).

[144]  Sethuraman Panchanathan,et al.  Optical flow based model for scene cut detection , 1996, Proceedings of 1996 Canadian Conference on Electrical and Computer Engineering.

[145]  Sethuraman Panchanathan,et al.  Review of Image and Video Indexing Techniques , 1997, J. Vis. Commun. Image Represent..

[146]  Ramesh Jain,et al.  Storage and Retrieval for Image and Video Databases III , 1995 .

[147]  Charles A. Bouman,et al.  ViBE: a video indexing and browsing environment , 1999, Optics East.

[148]  Fang Liu,et al.  Periodicity, Directionality, and Randomness: Wold Features for Image Modeling and Retrieval , 1996, IEEE Trans. Pattern Anal. Mach. Intell..

[149]  Nilesh V. Patel,et al.  Statistical approach to scene change detection , 1995, Electronic Imaging.

[150]  Christos Faloutsos,et al.  Compressed-domain video indexing techniques using DCT and motion vector information in MPEG video , 1997, Electronic Imaging.

[151]  Hain-Ching Liu,et al.  Automatic determination of scene changes in MPEG compressed video , 1995, Proceedings of ISCAS'95 - International Symposium on Circuits and Systems.

[152]  Wolfgang Effelsberg,et al.  VisualGREP: a systematic method to compare and retrieve video sequences , 1997, Electronic Imaging.

[153]  Vincenzo Di Lecce,et al.  An Evaluation of the Effectiveness of Image Features for Image Retrieval , 1999, J. Vis. Commun. Image Represent..

[154]  Konstantinos N. Plataniotis,et al.  A Novel Vector-Based Approach to Color Image Retrieval Using a Vector Angular-Based Distance Measure , 1999, Comput. Vis. Image Underst..

[155]  Ronald-Bryan O. Alferez,et al.  Image indexing and retrieval using image-derived, geometrically and illumination invariant features , 1999, Proceedings IEEE International Conference on Multimedia Computing and Systems.

[156]  Jordi Vitrià,et al.  Local Color Analysis for Scene Break Detection Applied to TV Commercials Recognition , 1999, VISUAL.

[157]  Shih-Fu Chang,et al.  Image Retrieval: Current Techniques, Promising Directions, and Open Issues , 1999, J. Vis. Commun. Image Represent..

[158]  B. S. Manjunath,et al.  NeTra-V: toward an object-based video representation , 1998, IEEE Trans. Circuits Syst. Video Technol..

[159]  William J. Christmas,et al.  Combining multiple experts for classifying shot changes in video sequences , 1999, Proceedings IEEE International Conference on Multimedia Computing and Systems.

[160]  Sethuraman Panchanathan,et al.  A critical evaluation of image and video indexing techniques in the compressed domain , 1999, Image Vis. Comput..

[161]  A. Murat Tekalp,et al.  A high-performance shot boundary detection algorithm using multiple cues , 1998, Proceedings 1998 International Conference on Image Processing. ICIP98 (Cat. No.98CB36269).