Improving image retrieval by using spatial relations

In this paper we proposed the use of spatial relations as a way of improving annotation-based image retrieval. We analyzed different types of spatial relations and selected the most adequate ones for image retrieval. We developed an image comparison and retrieval method based on conceptual graphs, which incorporates spatial relations. Additionally, we proposed an alternative term-weighting scheme and explored the use of more than one sample image for retrieval using several late fusion techniques. Our methods were evaluated with a rich and complex image dataset, based on the 39 topics developed for the ImageCLEF 2008 photo retrieval task. Results show that: (i) incorporating spatial relations produces a significant increase in performance, (ii) the label weighting scheme we proposed obtains better results than other traditional schemes, and (iii) the combination of several sample images using late fusion produces an additional improvement in retrieval according to several metrics.

[1]  Shengli Wu,et al.  Performance prediction of data fusion for information retrieval , 2006, Inf. Process. Manag..

[2]  Bo Zhang,et al.  Exploiting spatial context constraints for automatic image region annotation , 2007, ACM Multimedia.

[3]  David A. Forsyth,et al.  Object Recognition as Machine Translation: Learning a Lexicon for a Fixed Image Vocabulary , 2002, ECCV.

[4]  Maneesha Singh,et al.  IMAGE RETRIEVAL USING SPATIAL CONTEXT , 2002 .

[5]  David M. Mark,et al.  Natural-Language Spatial Relations Between Linear and Areal Objects: The Topology and Metric of English-Language Terms , 1998, Int. J. Geogr. Inf. Sci..

[6]  M. Egenhofer,et al.  Point-Set Topological Spatial Relations , 2001 .

[7]  Jong-Hak Lee,et al.  Analyses of multiple evidence combination , 1997, SIGIR '97.

[8]  Shih-Fu Chang,et al.  Image Retrieval: Current Techniques, Promising Directions, and Open Issues , 1999, J. Vis. Commun. Image Represent..

[9]  John F. Sowa,et al.  Conceptual Structures: Information Processing in Mind and Machine , 1983 .

[10]  Carlos Arturo Hernández-Gracidas,et al.  Data fusion and label weighting for image retrieval based on spatio-conceptual information , 2010, RIAO.

[11]  Gustavo Carneiro,et al.  Supervised Learning of Semantic Classes for Image Annotation and Retrieval , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[12]  Abby Goodrum,et al.  Image Information Retrieval: An Overview of Current Research , 2000, Informing Sci. Int. J. an Emerg. Transdiscipl..

[13]  Michael Grubinger,et al.  Analysis and evaluation of visual information systems performance , 2007 .

[14]  Tom Minka,et al.  Vision texture for annotation , 1995, Multimedia Systems.

[15]  Hugo Jair Escalante,et al.  The segmented and annotated IAPR TC-12 benchmark , 2010, Comput. Vis. Image Underst..

[16]  James Ze Wang,et al.  Image retrieval: Ideas, influences, and trends of the new age , 2008, CSUR.

[17]  Paul B. Kantor,et al.  Decision Level Data Fusion for Routing of Documents in the TREC3 Context: A Base Case Analysis of Worst Case Results , 1994, TREC.

[18]  Max J. Egenhofer,et al.  A Formal Definition of Binary Topological Relationships , 1989, FODO.

[19]  Javed A. Aslam,et al.  Models for metasearch , 2001, SIGIR '01.

[20]  Johann van der Merwe,et al.  A survey on peer-to-peer key management for mobile ad hoc networks , 2007, CSUR.

[21]  Andrew U. Frank,et al.  Qualitative spatial reasoning about distances and directions in geographic space , 1992, J. Vis. Lang. Comput..

[22]  Mohammed Belkhatir,et al.  A signal/semantic framework for image retrieval , 2005, Proceedings of the 5th ACM/IEEE-CS Joint Conference on Digital Libraries (JCDL '05).

[23]  Clement T. Yu,et al.  Reasoning About Spatial Relationships in Picture Retrieval Systems , 1994, VLDB.

[24]  Arun K. Majumdar,et al.  Content Based Image Search over the World Wide Web , 2002, ICVGIP.

[25]  Alberto Del Bimbo,et al.  Weighted walkthroughs between extended entities for retrieval by spatial arrangement , 2003, IEEE Trans. Multim..

[26]  Edward A. Fox,et al.  Combination of Multiple Searches , 1993, TREC.

[27]  Marcel Worring,et al.  Content-Based Image Retrieval at the End of the Early Years , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[28]  Karen Spärck Jones.  A statistical interpretation of term specificity and its application in retrieval , 2021, J. Documentation.

[29]  David E. Millard,et al.  Mind the semantic gap , 2005, HYPERTEXT '05.

[30]  R. K. Goyal,et al.  Similarity assessment for cardinal directions between extended spatial objects , 2000 .

[31]  Gopal Kanji,et al.  100 Statistical Tests , 1994 .

[32]  J. Chenb,et al.  A MODEL FOR TOPOLOGICAL RELATIONSHIPS BASED ON EULER NUMBER , 2005 .

[33]  Garrison W. Cottrell,et al.  Fusion Via a Linear Combination of Scores , 1999, Information Retrieval.

[34]  C. Gold,et al.  DESCRIBING TOPOLOGICAL RELATIONS WITH VORONOI-BASED 9-INTERSECTION MODEL , 2003 .

[35]  Shi-Kuo Chang,et al.  Iconic Indexing by 2-D Strings , 1987, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[36]  Manuel Montes-y-Gómez,et al.  Modeling Spatial Relations for Image Retrieval by Conceptual Graphs , 2009 .

[37]  Yves Chiaramella,et al.  Using a Conceptual Graph Framework for Image Retrieval , 1995 .

[38]  Stephen S.-T. Yau,et al.  On Interactability of Spatial Relationships in Content-Based Image Database Systems , 2004, Commun. Inf. Syst..

[39]  Alexander F. Gelbukh,et al.  Comparison of Conceptual Graphs , 2000, MICAI.

[40]  James Ze Wang,et al.  Learning-based linguistic indexing of pictures with 2--d MHMMs , 2002, MULTIMEDIA '02.

[41]  Alberto Del Bimbo,et al.  Efficient Matching and Indexing of Graph Models in Content-Based Retrieval , 2001, IEEE Trans. Pattern Anal. Mach. Intell..

[42]  Carlos Arturo Hernández-Gracidas,et al.  Markov Random Fields and Spatial Information to Improve Automatic Image Annotation , 2007, PSIVT.

[43]  Max J. Egenhofer,et al.  Definitions of Line-Line Relations for Geographic Databases , 1993, IEEE Data Eng. Bull..