Fusion Techniques for Combining Textual and Visual Information Retrieval

This chapter describes several approaches for information fusion that have been used in ImageCLEF over the past seven years. In this context, the fusion of information is mainly meant to combine textual and visual retrieval. Data fusion techniques from 116 papers (62% of ImageCLEF working notes) are categorized, described and discussed. It was observed that three general approaches were used for retrieval that can be categorized based on the system level chosen for combining modalities: 1) at the input of the system with inter–media query expansion, 2) internally to the system with early fusion and 3) at the output of the system with late fusion which is by far the most widely used fusion strategy.

[1]  Fernando Llopis,et al.  A Multimodal Approach to the Medical Retrieval Task using IR-n , 2008, CLEF.

[2]  Christophe Moulin,et al.  UJM at ImageCLEFwiki 2008 , 2008, CLEF.

[3]  Hichem Sahbi,et al.  TELECOMParisTech at ImageClefphoto 2008: Bi-Modal Text and Image Retrieval with Diversity Enhancement , 2008, CLEF.

[4]  Edward A. Fox,et al.  Combination of Multiple Searches , 1993, TREC.

[5]  José Luis Martínez-Fernández,et al.  MIRACLE at ImageCLEFphoto 2007: Evaluation of Merging Strategies for Multilingual and Multimedia Information Retrieval , 2008, CLEF.

[6]  Stefan M. Rüger,et al.  Exploring Image, Text and Geographic Evidences in ImageCLEF 2007 , 2007, CLEF.

[7]  Marcel Worring,et al.  Content-Based Image Retrieval at the End of the Early Years , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[8]  Jong-Hak Lee,et al.  Analyses of multiple evidence combination , 1997, SIGIR '97.

[9]  Hermann Ney,et al.  Image Retrieval and Annotation Using Maximum Entropy , 2006, CLEF.

[10]  William R. Hersh,et al.  Medical Image Retrieval and Automated Annotation: OHSU at ImageCLEF 2006 , 2006, CLEF.

[11]  Wei-Pang Yang,et al.  NCTU_DBLAB@ImageCLEFmed 2005: Medical Image Retrieval Task , 2005, CLEF.

[12]  Paul B. Kantor,et al.  A study of information seeking and retrieving. II. Users, questions, and effectiveness , 1988 .

[13]  Halina Kwasnicka,et al.  Multimodal Photo Retrieval through Finding Similar Documents Enhanced with Visual Clues - a Baseline Method , 2009, CLEF.

[14]  Joo-Hwee Lim,et al.  VisMed: A Visual Vocabulary Approach for Medical Image Indexing and Retrieval , 2005, AIRS.

[15]  Georges Quénot,et al.  MRIM-LIG at ImageCLEF 2009: Photo Retrieval and Photo Annotation Tasks , 2009, CLEF.

[16]  Miguel Ángel García Cumbreras,et al.  SINAI at ImageCLEFmed 2008 , 2008, CLEF.

[17]  A. J. Bell.  THE CO-INFORMATION LATTICE , 2003 .

[18]  Ana M. García-Serrano,et al.  MIRACLE-FI at ImageCLEFphoto 2008: Experiences in merging Text-based and Content-based Retrievals , 2008, CLEF.

[19]  Lambert Schomaker,et al.  Variants of the Borda count method for combining ranked classifier hypotheses , 2000 .

[20]  Henning Müller,et al.  Information Fusion for Combining Visual and Textual Image Retrieval , 2010, 2010 20th International Conference on Pattern Recognition.

[21]  Hugo Jair Escalante,et al.  TIA-INAOE's Participation at ImageCLEF 2007 , 2007 .

[22]  Xin Zhou,et al.  The MedGIFT Group at ImageCLEF 2008 , 2009, CLEF.

[23]  Alan F. Smeaton,et al.  DCU at ImageCLEFPhoto 2008 , 2008, CLEF.

[24]  Hsin-Hsi Chen,et al.  Approaches Using a Word-Image Ontology and an Annotated Image Corpus as Intermedia for Cross-Language Image Retrieval , 2006, CLEF.

[25]  Theodore Kalamboukis,et al.  Combined Content based and Semantic Image Retrieval , 2009, CLEF.

[26]  William R. Hersh,et al.  Medical Image Retrieval and Automatic Annotation: OHSU at ImageCLEF 2007 , 2007, CLEF.

[27]  Miguel E. Ruiz,et al.  UB at CLEF 2005: Medical Image Retrieval Task , 2005, CLEF.

[28]  Joo-Hwee Lim,et al.  IPAL at ImageClef 2007 Mixing Features, Models and Knowledge , 2007, CLEF.

[29]  Henning Müller,et al.  The MedGIFT Group at ImageCLEF 2009 , 2008, CLEF.

[30]  Leila Kosseim,et al.  CLaC at ImageCLEFPhoto 2008 , 2008, CLEF.

[31]  István Petrás,et al.  SZTAKI @ ImageCLEF 2009 , 2009, CLEF.

[32]  Ana M. García-Serrano,et al.  MIRACLE (FI) at ImageCLEFphoto 2009 , 2009, CLEF.

[33]  Rui Hu,et al.  MMIS at ImageCLEF 2008: Experiments combining Different Evidence Sources , 2008, CLEF.

[34]  Miguel Ángel García Cumbreras,et al.  SINAI at ImageCLEF 2006 , 2006, CLEF.

[35]  Massih-Reza Amini,et al.  UPMC/LIP6 at ImageCLEF's WikipediaMM: An Image-Annotation Model for an Image Search-Engine , 2008, CLEF.

[36]  Joon Ho Lee,et al.  Combining multiple evidence from different properties of weighting schemes , 1995, SIGIR '95.

[37]  Hermann Ney,et al.  FIRE in ImageCLEF 2005: Combining Content-based Image Retrieval with Textual Information Retrieval , 2005, CLEF.

[38]  Joo-Hwee Lim,et al.  Using Ontology Dimensions and Negative Expansion to solve Precise Queries in the ImageCLEF Medical Task , 2005, CLEF.

[39]  Joo-Hwee Lim,et al.  IPAL Knowledge-based Medical Image Retrieval in ImageCLEFmed 2006 , 2006, CLEF.

[40]  Philipp Daumke,et al.  Morphosaurus in ImageCLEF 2006: The Effect of Subwords on Biomedical IR , 2006, CLEF.

[41]  Christian Lovis,et al.  Using medGIFT and easyIR for the ImageCLEF 2005 Evaluation Tasks , 2005, CLEF.

[42]  Patrick Ruch,et al.  Query and Document Translation by Automatic Text Categorization: A Simple Approach to Establish a Strong Textual Baseline for ImageCLEFmed 2006 , 2006, CLEF.

[43]  Gabriela Csurka,et al.  XRCE's Participation in ImageCLEF 2009 , 2009, CLEF.

[44]  Masashi Inoue,et al.  Effects of Visual Concept-based Post-retrieval Clustering in ImageCLEFphoto 2008 , 2008, CLEF.

[45]  Nicholas J. Belkin,et al.  Combining Evidence for Information Retrieval , 1993, TREC.

[46]  Kieran McDonald,et al.  Dublin City University at CLEF 2004: Experiments with the ImageCLEF St Andrew's Collection , 2004, CLEF.

[47]  Leila Kosseim,et al.  Experiments with Clustering the Collection at ImageCLEF 2007 , 2007, CLEF.

[48]  Philippe Mulhem.  LIG at ImageCLEFphoto 2008 , 2008, CLEF.

[49]  William R. Hersh,et al.  Manual Query Modification and Automated Translation to Improve Cross-Language Medical Image Retrieval , 2005, CLEF.

[50]  José Carlos González,et al.  MIRACLE at ImageCLEFmed 2007: Merging Textual and Visual Strategies to Improve Medical Image Retrieval , 2007, CLEF.

[51]  Massih-Reza Amini,et al.  UPMC/LIP6 at ImageCLEFphoto 2008: on the Exploitation of Visual Concepts (VCDT) , 2008, CLEF.

[52]  Nicholas J. Belkin,et al.  The effect multiple query representations on information retrieval system performance , 1993, SIGIR.

[53]  C. Schmid,et al.  INRIA-LEARs participation to ImageCLEF 2009 , 2009 .

[54]  Bipin C. Desai,et al.  CINDI at ImageCLEF 2006: Image Retrieval & Annotation Tasks for the General Photographic and Medical Image Collections , 2006, CLEF.

[55]  Javed A. Aslam,et al.  Relevance score normalization for metasearch , 2001, CIKM '01.

[56]  András A. Benczúr,et al.  Cross-modal Retrieval by Text and Image Feature Biclustering , 2007, CLEF.

[57]  Romaric Besançon,et al.  Merging Results from Different Media: Lic2m Experiments at ImageCLEF 2005 , 2005, CLEF.

[58]  Steven C. H. Hoi.  Cross-Language and Cross-Media Image Retrieval: An Empirical Study at ImageCLEF2007 , 2007, CLEF.

[59]  J. Ross Quinlan,et al.  Induction of Decision Trees , 1986, Machine Learning.

[60]  Alan F. Smeaton,et al.  DCU and UTA at ImageCLEFPhoto 2007 , 2008, CLEF.

[61]  Martin Halvey,et al.  The University of Glasgow at ImageCLEFPhoto 2009 , 2009, CLEF.

[62]  William R. Hersh,et al.  Multimodal Medical Image Retrieval: OHSU at ImageCLEF 2008 , 2008, CLEF.

[63]  Mark Sanderson,et al.  Diversity in Photo Retrieval: Overview of the ImageCLEFPhoto Task 2009 , 2009, CLEF.

[64]  Maximilian Eibl,et al.  ImageCLEF 2006 Experiments at the Chemnitz Technical University , 2006, CLEF.

[65]  Hervé Glotin,et al.  Concept Content Based Wikipedia WEB Image Retrieval using CLEF VCDT 2008 , 2008, CLEF.

[66]  Miguel Ángel García Cumbreras,et al.  Using Information Gain to Improve the ImageCLEF 2006 Collection , 2006, CLEF.

[67]  Antoine Geissbühler,et al.  ImageCLEF 2004: Combining Image and Multi-lingual Search for Medical Image Retrieval , 2004, CLEF.

[68]  Henning Müller,et al.  MedGIFT at ImageCLEF 2008 , 2008, CLEF.

[69]  Adil Alpkocak,et al.  DEU at ImageCLEFmed 2009: Evaluating a Re-ranking and Integrated Retrieval Model , 2009, CLEF.

[70]  Adrian Popescu,et al.  Conceptual Image Retrieval over the Wikipedia Corpus , 2008, CLEF.

[71]  Leila Kosseim,et al.  Image Retrieval by Inter-media Fusion and Pseudo-relevance Feedback , 2008, CLEF.

[72]  Miguel E. Ruiz.  UNT at ImageCLEFmed 2009 , 2009, CLEF.

[73]  D. Frank Hsu,et al.  Comparing Rank and Score Combination Methods for Data Fusion in Information Retrieval , 2005, Information Retrieval.

[74]  Wei-Pang Yang,et al.  NCTU-ISU's Evaluation for the User-Centered Search Task at ImageCLEF 2004 , 2004, CLEF.

[75]  José Carlos González,et al.  MIRACLE's Combination of Visual and Textual Queries for Medical Image Retrieval , 2005, CLEF.

[76]  Hsin-Hsi Chen,et al.  Increasing Relevance and Diversity in Photo Retrieval by Result Fusion , 2008, CLEF.

[77]  Wei-Pang Yang,et al.  KIDS's Evaluation in the Medical Image Retrieval Task at ImageCLEF 2004 , 2004, CLEF.

[78]  Miguel Ángel García Cumbreras,et al.  SINAI at ImageCLEF 2005 , 2005, CLEF.

[79]  Hsin-Hsi Chen,et al.  From Text to Image: Generating Visual Query for Image Retrieval , 2004, CLEF.

[80]  Miguel E. Ruiz,et al.  UB at CLEF2004 Cross Language Medical Image Retrieval , 2004, CLEF.

[81]  Miguel E. Ruiz,et al.  UB at CLEF2004: Part 2 - Cross Language Medical Image Retrieval , 2004, CLEF.

[82]  Romaric Besançon,et al.  Data Fusion of Retrieval Results from Different Media: Experiments at ImageCLEF 2005 , 2005, CLEF.

[83]  Christian Lovis,et al.  The Use of MedGIFT and EasyIR for ImageCLEF 2005 , 2005, CLEF.

[84]  Hervé Glotin,et al.  Affinity Propagation promoting Diversity in Visuo-entropic and Text Features for CLEF Photo Retrieval 2008 Campaign , 2008, CLEF.

[85]  Mohand Boughanem,et al.  Using Pseudo-relevance Feedback to Improve Image Retrieval Results , 2008, CLEF.

[86]  Jonathon S. Hare,et al.  IAM@ImageCLEFphoto 2009: Experiments on Maximising Diversity using Image Features , 2009, CLEF.

[87]  Wen Gao,et al.  PKU at ImageCLEF 2008: Experiments with Query Extension Techniques for Text-Based and Content-Based Image Retrieval , 2008, CLEF.

[88]  Antoine Geissbühler,et al.  Report on the CLEF Experiment: Combining Image and Multilingual Search for Medical Image Retrieval , 2004, CLEF.

[89]  José Luis Martínez-Fernández,et al.  MIRACLE at ImageCLEF 2004 , 2004, CLEF.

[90]  Jian-Yun Nie,et al.  Toward Cross-Language and Cross-Media Image Retrieval , 2004, CLEF.

[91]  Tomohiro Takagi,et al.  Meiji University at ImageCLEF2008 Photo Retrieval Task: Evaluation of Image Retrieval Methods Integrating Different Media , 2008, CLEF.

[92]  Bipin C. Desai,et al.  Multi-Modal Interactive Approach to ImageCLEF 2007 Photographic and Medical Retrieval Tasks by CINDI , 2007, CLEF.

[93]  Adrian Popescu,et al.  Visual Reranking for Image Retrieval over the Wikipedia Corpus , 2009, CLEF.

[94]  Fernando Llopis,et al.  Evaluating Fusion Techniques at Different Domains in ImageCLEF Subtasks , 2009, CLEF.

[95]  Guido C. H. E. de Croon,et al.  FINT: Find Images aNd Text , 2004, CLEF.

[96]  Romaric Besançon,et al.  LIC2M Experiments at ImageCLEF 2004 , 2004, CLEF.

[97]  Joo-Hwee Lim,et al.  A Structured Learning Approach for Medical Image Indexing and Retrieval , 2005, CLEF.

[98]  Christophe Moulin,et al.  Combining Text/Image in WikipediaMM Task 2009 , 2009, CLEF.

[99]  Joo-Hwee Lim,et al.  IPAL Inter-Media Pseudo-Relevance Feedback Approach to ImageCLEF 2006 Photo Retrieval , 2006, CLEF.

[100]  Joo-Hwee Lim,et al.  Inter-media Pseudo-relevance Feedback Application to ImageCLEF 2006 Photo Retrieval , 2006, CLEF.

[101]  Henning Müller,et al.  Fusing visual and clinical information for lung tissue classification in high-resolution computed tomography , 2010, Artif. Intell. Medicine.

[102]  Patrick Ruch,et al.  University and Hospitals of Geneva at ImageCLEF 2007 , 2007, CLEF.

[103]  Garrison W. Cottrell,et al.  Fusion Via a Linear Combination of Scores , 1999, Information Retrieval.

[104]  Hermann Ney,et al.  FIRE in ImageCLEF 2007 , 2007, CLEF.

[105]  Hsin-Hsi Chen,et al.  Experiment for Using Web Information to do Query and Document Expansion , 2007, CLEF.

[106]  Miguel E. Ruiz,et al.  UB at CLEF 2005: Bilingual CLIR and Medical Image Retrieval Tasks , 2005, CLEF.

[107]  Hongfei Lin,et al.  Towards A Better Performance for Medical Image Retrieval Using An Integrated Approach , 2009, CLEF.

[108]  Fernando Llopis,et al.  Different Multimodal Approaches using IR-n in ImageCLEFphoto 2008 , 2008, CLEF.

[109]  Gabriela Csurka,et al.  XRCE's Participation to ImageCLEFphoto 2007 , 2007, CLEF.

[110]  Diana Inkpen,et al.  Clustering for Photo Retrieval at Image CLEF 2008 , 2008, CLEF.

[111]  William R. Hersh,et al.  Multimodal Medical Image Retrieval: Improving Precision at ImageCLEF 2009 , 2009, CLEF.

[112]  Michael R. Lyu,et al.  CUHK Experiments with ImageCLEF 2005 , 2005, CLEF.

[113]  Hatice Gunes,et al.  Affect recognition from face and body: early fusion vs. late fusion , 2005, 2005 IEEE International Conference on Systems, Man and Cybernetics.

[114]  Sargur N. Srihari,et al.  Decision Combination in Multiple Classifier Systems , 1994, IEEE Trans. Pattern Anal. Mach. Intell..

[115]  András A. Benczúr,et al.  Increasing Cluster Recall of Cross-modal Image Retrieval , 2008, CLEF.

[116]  Patrick Ruch,et al.  University and Hospitals of Geneva Participating at ImageCLEF 2007 , 2008, CLEF.

[117]  L. A. Ureña-López,et al.  SINAI at ImageCLEF 2007 , 2007 .

[118]  Gerard Salton,et al.  Term-Weighting Approaches in Automatic Text Retrieval , 1988, Inf. Process. Manag..

[119]  Maximilian Eibl,et al.  Experiments for the ImageCLEF 2007 Photographic Retrieval Task , 2007, CLEF.

[120]  Hermann Ney,et al.  FIRE in ImageCLEF 2007: Support Vector Machines and Logistic Models to Fuse Image Descriptors for Photo Retrieval , 2008, CLEF.

[121]  Stéfan Jacques Darmoni,et al.  MedIC/CISMeF at ImageCLEF 2006: Image Annotation and Retrieval Tasks , 2006, CLEF.

[122]  Joo-Hwee Lim,et al.  IPAL at CLEF 2008: Mixed-Modality based Image Search, Novelty based Re-ranking and Extended Matching , 2008, CLEF.

[123]  Mohand Boughanem,et al.  Methods for combining Content-based and Textual-based Approaches in Medical Image Retrieval , 2008, CLEF.

[124]  Martin Halvey,et al.  University of Glasgow at ImageCLEFPhoto 2009: Optimising Similarity and Diversity in Image Retrieval , 2009, CLEF.

[125]  Andy Way,et al.  Dublin City University at CLEF 2004: Experiments with the ImageCLEF St. Andrew's Collection , 2004, CLEF.

[126]  Joo-Hwee Lim,et al.  I2R at ImageCLEF Photo Retrieval 2009 , 2009, CLEF.