An empirical investigation of user term feedback in text-based targeted image search

Text queries are natural and intuitive for users to describe their information needs. However, text-based image retrieval faces many challenges. Traditional text retrieval techniques on image descriptions have not been very successful. This is mainly due to the inconsistent textual descriptions and the discrepancies between user queries and terms in the descriptions. To investigate strategies to alleviate this vocabulary problem, this article examines the role of user term feedback in targeted image search that is based on text-based image retrieval. Term feedback refers to the feedback from a user on specific terms regarding their relevance to a target image. Previous studies have indicated the effectiveness of term feedback in interactive text retrieval. However, in our experiments on text-based image retrieval, the term feedback has not been shown to be effective. Our results indicate that, although term feedback has a positive effect by allowing users to identify more relevant terms, it also has a strong negative effect by providing more opportunities for users to specify irrelevant terms. To understand these different effects and their implications, this article further analyzes important factors that contribute to the utility of term feedback and discusses the outlook of term feedback in interactive text-based image retrieval.

[1]  Wei-Ying Ma,et al.  Learning a semantic space from user's relevance feedback for image retrieval , 2003, IEEE Trans. Circuits Syst. Video Technol..

[2]  Nicholas J. Belkin,et al.  Interface issues and interaction strategies for information retrieval systems , 1995, CHI '95.

[3]  Takeo Kanade,et al.  Intelligent Access to Digital Video: Informedia Project , 1996, Computer.

[4]  Sara Shatford Layne,et al.  Some Issues in the Indexing of Images , 1994, J. Am. Soc. Inf. Sci..

[5]  Nicholas J. Belkin,et al.  Determining the functionality features of an intelligent interface to an information retrieval system , 1989, SIGIR '90.

[6]  Beng Chin Ooi,et al.  Giving meanings to WWW images , 2000, MM 2000.

[7]  Simone Santini,et al.  Exploratory Image Databases: Content-Based Retrieval , 2001 .

[8]  R. Manmatha,et al.  Retrieving images by appearance , 1998, Sixth International Conference on Computer Vision (IEEE Cat. No.98CH36271).

[9]  Daniel Gatica-Perez,et al.  On image auto-annotation with latent space models , 2003, ACM Multimedia.

[10]  Paul Clough,et al.  A proposal for the CLEF Cross-Language Image Retrieval Track 2004 , 2004 .

[11]  Edward Y. Chang,et al.  Statistical learning for effective visual information retrieval , 2003, Proceedings 2003 International Conference on Image Processing (Cat. No.03CH37429).

[12]  Peter G. B. Enser,et al.  Progress in Documentation Pictorial Information Retrieval , 1995, J. Documentation.

[13]  R. Manmatha,et al.  Automatic image annotation and retrieval using cross-media relevance models , 2003, SIGIR.

[14]  Thomas S. Huang,et al.  Relevance feedback: a power tool for interactive content-based image retrieval , 1998, IEEE Trans. Circuits Syst. Video Technol..

[15]  Alex Pentland,et al.  Photobook: tools for content-based manipulation of image databases , 1994, Other Conferences.

[16]  Mark Sanderson,et al.  Relevance Feedback for Cross Language Image Retrieval , 2004, ECIR.

[17]  Shi-Kuo Chang,et al.  Image Information Systems: Where Do We Go From Here? , 1992, IEEE Trans. Knowl. Data Eng..

[18]  W. Bruce Croft,et al.  A language modeling approach to information retrieval , 1998, SIGIR '98.

[19]  Shih-Fu Chang,et al.  Image Retrieval: Current Techniques, Promising Directions, and Open Issues , 1999, J. Vis. Commun. Image Represent..

[20]  Tomaso A. Poggio,et al.  Image Representations and Feature Selection for Multimedia Database Search , 2003, IEEE Trans. Knowl. Data Eng..

[21]  James Ze Wang,et al.  SIMPLIcity: Semantics-Sensitive Integrated Matching for Picture LIbraries , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[22]  Michael Stonebraker,et al.  Chabot: Retrieval from a Relational Database of Images , 1995, Computer.

[23]  Marcel Worring,et al.  Content-Based Image Retrieval at the End of the Early Years , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[24]  Mark Sanderson,et al.  The CLEF 2004 Cross-Language Image Retrieval Track , 2004, CLEF.

[25]  James Ze Wang,et al.  Automatic Linguistic Indexing of Pictures by a Statistical Modeling Approach , 2003, IEEE Trans. Pattern Anal. Mach. Intell..

[26]  Chen Zhang,et al.  User term feedback in interactive text-based image retrieval , 2005, SIGIR '05.

[27]  W. Bruce Croft,et al.  Relevance-Based Language Models , 2001, SIGIR '01.

[28]  Michael I. Jordan,et al.  Modeling annotated data , 2003, SIGIR.

[29]  R. Manmatha,et al.  A Model for Learning the Semantics of Pictures , 2003, NIPS.

[30]  John D. Lafferty,et al.  Two-stage language models for information retrieval , 2002, SIGIR '02.

[31]  Nicholas J. Belkin,et al.  Using Relevance Feedback and Ranking in Interactive Searching , 1995, TREC.

[32]  John D. Lafferty,et al.  Information Retrieval as Statistical Translation , 2017 .

[33]  Peter G. Anick,et al.  The paraphrase search assistant: terminological feedback for iterative information seeking , 1999, SIGIR '99.

[34]  K SrihariRohini,et al.  Intelligent Indexing and Semantic Retrieval of Multimodal Documents , 2000 .

[35]  Edie M. Rasmussen,et al.  Searching for images: The analysis of users' queries for image retrieval in American history , 2003, J. Assoc. Inf. Sci. Technol..

[36]  Hideyuki Tamura,et al.  Image database systems: A survey , 1984, Pattern Recognit..

[37]  Hayit Greenspan,et al.  Finding Pictures of Objects in Large Collections of Images , 1996, Object Representation in Computer Vision.

[38]  Marti A. Hearst.  Using Categories to Provide Context for Full-Text Retrieval Results , 1994, RIAO.

[39]  Michael J. Swain,et al.  WebSeer: An Image Search Engine for the World Wide Web , 1996 .

[40]  K. Wakimoto,et al.  Efficient and Effective Querying by Image Content , 1994 .

[41]  Bella Hass Weinberg,et al.  Challenges in indexing electronic text and images , 1994 .

[42]  Alex Pentland,et al.  Photobook: Content-based manipulation of image databases , 1996, International Journal of Computer Vision.

[43]  Peter G. B. Enser.  Pictorial information retrieval , 1995 .

[44]  Daniela Petrelli,et al.  Concept Hierarchy across Languages in Text-Based Image Retrieval: A User Evaluation , 2005, CLEF.

[45]  Nicholas J. Belkin,et al.  A case for interaction: a study of interactive information retrieval behavior and effectiveness , 1996, CHI.

[46]  Yixin Chen,et al.  CLUE: cluster-based retrieval of images by unsupervised learning , 2005, IEEE Transactions on Image Processing.

[47]  Luo Si,et al.  An automatic weighting scheme for collaborative filtering , 2004, SIGIR '04.

[48]  Corinne Jörgensen,et al.  Indexing Images: Testing an Image Description Template. , 1996 .

[49]  Michael R. Lyu,et al.  A novel log-based relevance feedback technique in content-based image retrieval , 2004, MULTIMEDIA '04.

[50]  Marcel Worring,et al.  Filter Image Browsing: Interactive Image Retrieval by Using Database Overviews , 2001, Multimedia Tools and Applications.

[51]  John Tait,et al.  Search strategies in content-based image retrieval , 2003, SIGIR.

[52]  Rohini K. Srihari,et al.  Intelligent Indexing and Semantic Retrieval of Multimodal Documents , 2004, Information Retrieval.

[53]  Ben Shneiderman,et al.  Direct annotation: a drag-and-drop strategy for labeling photos , 2000, 2000 IEEE Conference on Information Visualization. An International Conference on Computer Visualization and Graphics.

[54]  KanadeTakeo,et al.  Intelligent Access to Digital Video , 1996 .

[55]  Wei-Ying Ma,et al.  Learning an image manifold for retrieval , 2004, MULTIMEDIA '04.

[56]  Richard M. Schwartz,et al.  A hidden Markov model information retrieval system , 1999, SIGIR '99.

[57]  Y ChaiJoyce,et al.  An empirical investigation of user term feedback in text-based targeted image search , 2007 .