Image Retrieval: Current Techniques, Promising Directions, and Open Issues

This paper provides a comprehensive survey of the technical achievements in the research area of image retrieval, especially content-based image retrieval, an area that has been so active and prosperous in the past few years. The survey includes 100+ papers covering the research aspects of image feature representation and extraction, multidimensional indexing, and system design, three of the fundamental bases of content-based image retrieval. Furthermore, based on the state-of-the-art technology available now and the demand from real-world applications, open research issues are identified and future promising research directions are suggested.

[1]  Ming-Kuei Hu,et al.  Visual pattern recognition by moment invariants , 1962, IRE Trans. Inf. Theory.

[2]  Ralph Roskies,et al.  Fourier Descriptors for Plane Closed Curves , 1972, IEEE Transactions on Computers.

[3]  Robert M. Haralick,et al.  Textural Features for Image Classification , 1973, IEEE Trans. Syst. Man Cybern..

[4]  Richard O. Duda,et al.  Pattern classification and scene analysis , 1974, A Wiley-Interscience publication.

[5]  Azriel Rosenfeld,et al.  A Comparative Study of Texture Measures for Terrain Classification , 1975, IEEE Transactions on Systems, Man, and Cybernetics.

[6]  Robert C. Bolles,et al.  Parametric Correspondence and Chamfer Matching: Two New Techniques for Image Matching , 1977, IJCAI.

[7]  King-Sun Fu,et al.  Shape Discrimination Using Fourier Descriptors , 1977, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[8]  Hideyuki Tamura,et al.  Textural Features Corresponding to Visual Perception , 1978, IEEE Transactions on Systems, Man, and Cybernetics.

[9]  Ning-San Chang,et al.  A Relational Database System for Images , 1980, Pictorial Information Systems.

[10]  P. Wintz,et al.  An efficient three-dimensional aircraft recognition algorithm using normalized fourier descriptors , 1980 .

[11]  Shi-Kuo Chang,et al.  Pictorial Data-Base Systems , 1981, Computer.

[12]  Keinosuke Fukunaga,et al.  Three-Dimensional Shape Analysis Using Local Shape Descriptors , 1981, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[13]  King-Sun Fu,et al.  Picture Query Languages for Pictorial Data-Base Systems , 1981, Computer.

[14]  Michael McGill,et al.  Introduction to Modern Information Retrieval , 1983 .

[15]  Anil K. Jain,et al.  Markov Random Field Texture Models , 1983, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[16]  A. Guttmma,et al.  R-trees: a dynamic index structure for spatial searching , 1984 .

[17]  Alex Pentland,et al.  Fractal-Based Description of Natural Scenes , 1984, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[18]  A. Guttman,et al.  A Dynamic Index Structure for Spatial Searching , 1984, SIGMOD 1984.

[19]  Hideyuki Tamura,et al.  Image database systems: A survey , 1984, Pattern Recognit..

[20]  Stan Matwin,et al.  A Logic-Based Knowledge Source System for Natural Language Document , 1985, Data Knowl. Eng..

[21]  Christos Faloutsos,et al.  The R+-Tree: A Dynamic Index for Multi-Dimensional Objects , 1987, VLDB.

[22]  Gunilla Borgefors,et al.  Hierarchical Chamfer Matching: A Parametric Edge Matching Algorithm , 1988, IEEE Trans. Pattern Anal. Mach. Intell..

[23]  Makoto Miyahara,et al.  Mathematical Transform Of (R, G, B) Color Data To Munsell (H, V, C) Color Data , 1988, Other Conferences.

[24]  Gerard Salton,et al.  Term-Weighting Approaches in Automatic Text Retrieval , 1988, Inf. Process. Manag..

[25]  Shi-Kuo Chang,et al.  An Intelligent Image Database System , 1988, IEEE Trans. Software Eng..

[26]  Diane Greene,et al.  An implementation and performance analysis of spatial data access methods , 1989, [1989] Proceedings. Fifth International Conference on Data Engineering.

[27]  Calvin C. Gotlieb,et al.  Texture descriptors based on co-occurrence matrices , 1990, Comput. Vis. Graph. Image Process..

[28]  Hans-Peter Kriegel,et al.  The R*-tree: an efficient and robust access method for points and rectangles , 1990, SIGMOD '90.

[29]  Esther M. Arkin,et al.  An efficiently computable metric for comparing polygonal shapes , 1991, SODA '90.

[30]  David B. Cooper,et al.  Recognition and positioning of rigid objects using algebraic moment invariants , 1991, Optics & Photonics.

[31]  Jian Fan,et al.  Texture Classification by Wavelet Packet Signatures , 1993, MVA.

[32]  Richard C. Dubes,et al.  Performance evaluation for four classes of textural features , 1992, Pattern Recognit..

[33]  Shi-Kuo Chang,et al.  Image Information Systems: Where Do We Go From Here? , 1992, IEEE Trans. Knowl. Data Eng..

[34]  W. Bruce Croft,et al.  The INQUERY Retrieval System , 1992, DEXA.

[35]  Jia-Lin Chen,et al.  Texture classification using QMF bank-based subband decomposition , 1992, CVGIP Graph. Model. Image Process..

[36]  Toshikazu Kato,et al.  Query by Visual Example - Content based Image Retrieval , 1992, EDBT.

[37]  Freddy Fierens,et al.  Interactive outlining: an improved approach using active contours , 1993, Electronic Imaging.

[38]  James Dowe,et al.  Content-based retrieval in multimedia imaging , 1993, Electronic Imaging.

[39]  C.-C. Jay Kuo,et al.  Texture analysis and classification with tree-structured wavelet transform , 1993, IEEE Trans. Image Process..

[40]  Ramin Samadani,et al.  Computer-assisted extraction of boundaries from images , 1993, Electronic Imaging.

[41]  Ramesh C. Jain,et al.  NSF workshop on visual information management systems: workshop report , 1993, Electronic Imaging.

[42]  Christos Faloutsos,et al.  QBIC project: querying images by content, using color, texture, and shape , 1993, Electronic Imaging.

[43]  Michael J. Swain,et al.  Interactive indexing into image databases , 1993, Electronic Imaging.

[44]  John P. Oakley,et al.  Storage and Retrieval for Image and Video Databases , 1993 .

[45]  Biing-Hwang Juang,et al.  Fundamentals of speech recognition , 1993, Prentice Hall signal processing series.

[46]  Markus H. Gross,et al.  Multiscale image texture analysis in wavelet spaces , 1994, Proceedings of 1st International Conference on Image Processing.

[47]  Matthew Lybanon,et al.  Segmentation of diverse image types using opening and closing , 1994, Proceedings of 12th International Conference on Pattern Recognition.

[48]  Brian Scassellati,et al.  Retrieving images by 2D shape: a comparison of computation methods with human perceptual judgments , 1994, Electronic Imaging.

[49]  H. D. Cheng,et al.  A fuzzy logic approach to image segmentation , 1994, Proceedings of 12th International Conference on Pattern Recognition.

[50]  K. S. Thyagarajan,et al.  A maximum likelihood approach to texture classification using wavelet transform , 1994, Proceedings of 1st International Conference on Image Processing.

[51]  William I. Grosky,et al.  Multimedia information systems , 1994, IEEE MultiMedia.

[52]  Dragutin Petkovic,et al.  Indexing for complex queries on a query-by-content image database , 1994, Proceedings of 12th International Conference on Pattern Recognition.

[53]  Shih-Fu Chang,et al.  Transform features for texture classification and discrimination in large image databases , 1994, Proceedings of 1st International Conference on Image Processing.

[54]  Ian H. Witten,et al.  Managing Gigabytes: Compressing and Indexing Documents and Images , 1999 .

[55]  William E. Higgins,et al.  Watershed-driven relaxation labeling for image segmentation , 1994, Proceedings of 1st International Conference on Image Processing.

[56]  Songde Ma,et al.  On the relation between region and contour representation , 1994, Proceedings of 12th International Conference on Pattern Recognition.

[57]  Yihong Gong,et al.  An image database system with content capturing and fast image indexing abilities , 1994, 1994 Proceedings of IEEE International Conference on Multimedia Computing and Systems.

[58]  K. Wakimoto,et al.  Efficient and Effective Querying by Image Content , 1994 .

[59]  Theo Gevers,et al.  Image segmentation by directed region subdivision , 1994, Proceedings of 12th International Conference on Pattern Recognition.

[60]  Thomas S. Huang,et al.  Segmentation of road scenes using color and fractal-based texture classification , 1994, Proceedings of 1st International Conference on Image Processing.

[61]  David B. Cooper,et al.  On Representation and Invariant Recognition of Complex Objects Based on Patches and Parts , 1994, Object Representation in Computer Vision.

[62]  Beng Chin Ooi,et al.  Efficient Image Retrieval By Color Contents , 1994, ADB.

[63]  Fritz Albregtsen,et al.  Fast computation of invariant geometric moments: a new method giving correct results , 1994, Proceedings of 12th International Conference on Pattern Recognition.

[64]  David B. Cooper,et al.  Computationally fast Bayesian recognition of complex objects based on mutual algebraic invariants , 1995, Proceedings., International Conference on Image Processing.

[65]  Rohini K. Srihari,et al.  Automatic Indexing and Content-Based Retrieval of Captioned Images , 1995, Computer.

[66]  Rosalind W. Picard.  Computer learning of subjectivity , 1995, CSUR.

[67]  Markus A. Stricker,et al.  Similarity of color images , 1995, Electronic Imaging.

[68]  James Allan,et al.  Relevance feedback with too much data , 1995, SIGIR '95.

[69]  Shih-Fu Chang,et al.  Extracting multidimensional signal features for content-based visual query , 1995, Other Conferences.

[70]  Shih-Fu Chang,et al.  Single color extraction and image query , 1995, Proceedings., International Conference on Image Processing.

[71]  HongJiang Zhang,et al.  Scheme for visual feature-based image indexing , 1995, Electronic Imaging.

[72]  Shih-Fu Chang,et al.  Compressed-domain techniques for image/video indexing and manipulation , 1995, Proceedings., International Conference on Image Processing.

[73]  William M. Shaw,et al.  Termrelevance Computations and Perfect Retrieval Performance , 1995, Inf. Process. Manag..

[74]  B. S. Manjunath,et al.  Adaptive filtering and indexing for image databases , 1995, Electronic Imaging.

[75]  B. S. Manjunath,et al.  A comparison of wavelet transform features for texture image annotation , 1995, Proceedings., International Conference on Image Processing.

[76]  B. S. Manjunath,et al.  Image indexing using a texture dictionary , 1995, Other Conferences.

[77]  Gerard Salton,et al.  Optimization of relevance feedback weights , 1995, SIGIR '95.

[78]  David Salesin,et al.  Fast multiresolution image querying , 1995, SIGGRAPH.

[79]  Rosalind W. Picard.  Toward a Visual Thesaurus , 1995, MIRO.

[80]  Y. N. Lakshman,et al.  Computing invariants using elimination methods , 1995, Proceedings of International Symposium on Computer Vision - ISCV.

[81]  Juyang Weng,et al.  Efficient content-based image retrieval using automatic feature selection , 1995, Proceedings of International Symposium on Computer Vision - ISCV.

[82]  Dragutin Petkovic,et al.  The query by image content (QBIC) system , 1995, SIGMOD '95.

[83]  Michael Stonebraker,et al.  Chabot: Retrieval from a Relational Database of Images , 1995, Computer.

[84]  Christos Faloutsos,et al.  FastMap: a fast algorithm for indexing, data-mining and visualization of traditional and multimedia datasets , 1995, SIGMOD '95.

[85]  Dragutin Petkovic,et al.  Query by Image and Video Content: The QBIC System , 1995, Computer.

[86]  Thomas S. Huang,et al.  Automated region segmentation using attraction-based grouping in spatial-color-texture space , 1996, Proceedings of 3rd IEEE International Conference on Image Processing.

[87]  Shih-Fu Chang,et al.  Automated binary texture feature sets for image retrieval , 1996, 1996 IEEE International Conference on Acoustics, Speech, and Signal Processing Conference Proceedings.

[88]  Michael J. Swain,et al.  WebSeer: An Image Search Engine for the World Wide Web , 1996 .

[89]  Shih-Fu Chang,et al.  Tools and techniques for color image retrieval , 1996, Electronic Imaging.

[90]  B. S. Manjunath,et al.  Texture Features for Browsing and Retrieval of Image Data , 1996, IEEE Trans. Pattern Anal. Mach. Intell..

[91]  Tom Minka,et al.  Modeling user subjectivity in image libraries , 1996, Proceedings of 3rd IEEE International Conference on Image Processing.

[92]  Hsinchun Chen,et al.  Building Large-Scale Digital Libraries - Guest Editors' Introduction , 1996, Computer.

[93]  B. S. Manjunath,et al.  Texture features and learning similarity , 1996, Proceedings CVPR IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[94]  Raymond T. Ng,et al.  Evaluating multidimensional indexing structures for images transformed by principal component analysis , 1996, Electronic Imaging.

[95]  Markus A. Stricker,et al.  Color indexing with weak spatial constraints , 1996, Electronic Imaging.

[96]  Aidong Zhang,et al.  Approaches to image retrieval based on compressed data for multimedia database systems , 1996 .

[97]  T. John Stonham,et al.  Content-based image retrieval using color tuple histograms , 1996, Electronic Imaging.

[98]  Ingemar J. Cox,et al.  Target testing and the PicHunter Bayesian multimedia retrieval system , 1996, Proceedings of the Third Forum on Research and Technology Advances in Digital Libraries,.

[99]  Ramesh C. Jain,et al.  Similarity indexing: algorithms and performance , 1996, Electronic Imaging.

[100]  Kannan Ramchandran,et al.  Multimedia Analysis and Retrieval System (MARS) Project , 1996, Data Processing Clinic.

[101]  Amarnath Gupta,et al.  Virage image search engine: an open framework for image management , 1996, Electronic Imaging.

[102]  Shih-Fu Chang.  Compressed-Domain Content-Based Image and Video Retrieval , 1996 .

[103]  Masao Sakauchi,et al.  A flexible content-based image retrieval system with combined scene description keyword , 1996, Proceedings of the Third IEEE International Conference on Multimedia Computing and Systems.

[104]  Tom Minka,et al.  Interactive learning with a "Society of Models" , 1996, Proceedings CVPR IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[105]  Shih-Fu Chang,et al.  Local color and texture extraction and spatial query , 1996, Proceedings of 3rd IEEE International Conference on Image Processing.

[106]  Fang Liu,et al.  Periodicity, Directionality, and Randomness: Wold Features for Image Modeling and Retrieval , 1996, IEEE Trans. Pattern Anal. Mach. Intell..

[107]  Thomas S. Huang,et al.  Modified Fourier Descriptors for Shape Representation - A Practical Approach , 1996 .

[108]  Ramin Zabih,et al.  Histogram refinement for content-based image retrieval , 1996, Proceedings Third IEEE Workshop on Applications of Computer Vision. WACV'96.

[109]  Rosalind W. Picard.  A Society of Models for Video and Image Libraries , 1996, IBM Syst. J..

[110]  C.-C. Jay Kuo,et al.  Wavelet descriptor of planar curves: theory and applications , 1996, IEEE Trans. Image Process..

[111]  Raj Jain,et al.  Algorithms and strategies for similarity retrieval , 1996 .

[112]  Rosalind W. Picard,et al.  Interactive Learning Using a "Society of Models" , 2017, CVPR 1996.

[113]  Ingemar J. Cox,et al.  PicHunter: Bayesian relevance feedback for image retrieval , 1996, Proceedings of 13th International Conference on Pattern Recognition.

[114]  B. S. Manjunath,et al.  A Pattern Thesaurus for Browsing Large Aerial Photographs , 1996 .

[115]  Shih-Fu Chang,et al.  Image and video search engine for the World Wide Web , 1997, Electronic Imaging.

[116]  Rajeev Motwani,et al.  Incremental clustering and dynamic information retrieval , 1997, STOC '97.

[117]  T.S. Huang,et al.  A relevance feedback architecture for content-based multimedia information retrieval systems , 1997, 1997 Proceedings IEEE Workshop on Content-Based Access of Image and Video Libraries.

[118]  Joshua R. Smith,et al.  Multi-stage classi cation of images from features and related text , 1997 .

[119]  Shih-Fu Chang,et al.  Querying by color regions using VisualSEEk content-based visual query system , 1997 .

[120]  Shih-Fu Chang,et al.  MetaSEEk: a content-based metasearch engine for images , 1997, Electronic Imaging.

[121]  C. Frankel,et al.  Distinguishing photographs and graphics on the World Wide Web , 1997, 1997 Proceedings IEEE Workshop on Content-Based Access of Image and Video Libraries.

[122]  Shih-Fu Chang,et al.  Enhancing image search engines in visual information environments , 1997, Proceedings of First Signal Processing Society Workshop on Multimedia Signal Processing.

[123]  Shih-Fu Chang,et al.  Visually Searching the Web for Content , 1997, IEEE Multim..

[124]  Hong Heather Yu,et al.  A hierarchical, multi-resolution method for dictionary-driven content-based image retrieval , 1997, Proceedings of International Conference on Image Processing.

[125]  S. Sclaroff,et al.  ImageRover: a content-based image browser for the World Wide Web , 1997, 1997 Proceedings IEEE Workshop on Content-Based Access of Image and Video Libraries.

[126]  Shih-Fu Chang,et al.  Visual information retrieval from large distributed online repositories , 1997, CACM.

[127]  Linda G. Shapiro,et al.  Efficient image retrieval with multiple distance measures , 1997, Electronic Imaging.

[128]  Thomas S. Huang,et al.  Content-based image retrieval with relevance feedback in MARS , 1997, Proceedings of International Conference on Image Processing.

[129]  Thomas S. Huang,et al.  Supporting similarity queries in MARS , 1997, MULTIMEDIA '97.

[130]  M. Lew,et al.  Webcrawling Using Sketches , 1997 .

[131]  Thomas S. Huang,et al.  Automatic Matching Tool Selection Using Relevance Feedback In Mars , 1997 .

[132]  Thomas S. Huang,et al.  Supporting content-based queries over images in MARS , 1997, Proceedings of IEEE International Conference on Multimedia Computing and Systems.

[133]  Serge J. Belongie,et al.  Region-based image querying , 1997, 1997 Proceedings IEEE Workshop on Content-Based Access of Image and Video Libraries.

[134]  Raj Acharya,et al.  Color clustering techniques for color-content-based image retrieval from image databases , 1997, Proceedings of IEEE International Conference on Multimedia Computing and Systems.

[135]  B. S. Manjunath,et al.  Edge flow: A framework of boundary detection and image segmentation , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[136]  Yong Rui,et al.  Multimedia Analysis and Retrieval System , 1997 .

[137]  H.D. Tagare.  Increasing retrieval efficiency by index tree adaptation , 1997, 1997 Proceedings IEEE Workshop on Content-Based Access of Image and Video Libraries.

[138]  Jing Huang,et al.  Image indexing using color correlograms , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[139]  B. S. Manjunath,et al.  An Eigenspace Update Algorithm for Image Analysis , 1997, CVGIP Graph. Model. Image Process..

[140]  Ramin Zabih,et al.  Comparing images using color coherence vectors , 1997, MULTIMEDIA '96.

[141]  Shih-Fu Chang,et al.  Finding Images/Video in Large Archives: Columbia's Content-Based Visual Query Project , 1997, D Lib Mag..

[142]  Stuart Weibel,et al.  Image Description on the Internet: A Summary of the CNI/OCLC Image Metadata Workshop September 24 - 25, 1996, Dublin, Ohio , 1997, D Lib Mag..

[143]  Yali Amit,et al.  Shape Quantization and Recognition with Randomized Trees , 1997, Neural Computation.

[144]  Aidong Zhang,et al.  WebView: A Multimedia Database Resource Integration and Search System over Web , 1997, WebNet.

[145]  Beng Chin Ooi,et al.  Fast signature-based color-spatial image retrieval , 1997, Proceedings of IEEE International Conference on Multimedia Computing and Systems.

[146]  Ronald Fagin,et al.  Incorporating User Preferences in Multimedia Queries , 1997, ICDT.

[147]  Shih-Fu Chang,et al.  VisualSEEk: a fully automated content-based image query system , 1997, MULTIMEDIA '96.

[148]  Mohan S. Kankanhalli,et al.  Shape Measures for Content Based Image Retrieval: A Comparison , 1997, Inf. Process. Manag..

[149]  Thomas S. Huang,et al.  Relevance feedback: a power tool for interactive content-based image retrieval , 1998, IEEE Trans. Circuits Syst. Video Technol..

[150]  Ingemar J. Cox,et al.  Psychophysical studies of the performance of an image database retrieval system , 1998, Electronic Imaging.

[151]  Shih-Fu Chang,et al.  Next-generation content representation, creation, and searching for new-media applications in education , 1998 .

[152]  Yishay Mansour,et al.  A Fast, Bottom-Up Decision Tree Pruning Algorithm with Near-Optimal Generalization , 1998, ICML.

[153]  Charles A. Bouman,et al.  Perceptual image similarity experiments , 1998, Electronic Imaging.

[154]  Ingemar J. Cox,et al.  An optimized interaction strategy for Bayesian relevance feedback , 1998, Proceedings. 1998 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No.98CB36231).

[155]  B. S. Manjunath,et al.  NeTra: A toolbox for navigating large image databases , 1997, Multimedia Systems.

[156]  Alberto Del Bimbo,et al.  Visual information retrieval , 1999 .

[157]  Eugenio Di Sciascio,et al.  Content-Based Image Retrieval over the Web Using Query by Sketch and Relevance Feedback , 1999, VISUAL.

[158]  Marcel Worring,et al.  Content-Based Image Retrieval at the End of the Early Years , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[159]  Remco C. Veltkamp,et al.  Content-based image retrieval systems: A survey , 2000 .

[160]  Luciano da Fontoura Costa,et al.  Shape Analysis and Classification: Theory and Practice , 2000 .

[161]  Anil K. Jain,et al.  Image classification for content-based indexing , 2001, IEEE Trans. Image Process..