A survey on the use of relevance feedback for information access systems

Users of online search engines often find it difficult to express their need for information in the form of a query. However, if the user can identify examples of the kind of documents they require then they can employ a technique known as relevance feedback. Relevance feedback covers a range of techniques intended to improve a user's query and facilitate retrieval of information relevant to a user's information need. In this paper we survey relevance feedback techniques. We study both automatic techniques, in which the system modifies the user's query, and interactive techniques, in which the user has control over query modification. We also consider specific interfaces to relevance feedback systems and characteristics of searchers that can affect the use and success of relevance feedback systems.

[1]  C. J. van Rijsbergen,et al.  The selection of good search terms , 1981, Inf. Process. Manag..

[2]  Jacob Shapiro,et al.  Boolean Search: Current State and Perspectives , 1999, J. Am. Soc. Inf. Sci..

[3]  Gerard Salton,et al.  Improving retrieval performance by relevance feedback , 1997, J. Am. Soc. Inf. Sci..

[4]  Iain Campbell,et al.  Interactive Evaluation of the Ostensive Model Using a New Test Collection of Images with Multiple Relevance Assessments , 2000, Information Retrieval.

[5]  Iain Campbell,et al.  Supporting Information Needs by Ostensive Definition in an Adaptive Information Space , 1995, MIRO.

[6]  Nicholas J. Belkin,et al.  An Overview of Results from Rutgers' Investigations of Interactive Information Retrieval , 1998 .

[7]  Theo Huibers,et al.  An axiomatic theory for information retrieval , 1996 .

[8]  Alan F. Smeaton.  Independence of Contributing Retrieval Strategies in Data Fusion for Effective Information Retrieval , 1998, BCS-IRSG Annual Colloquium on IR Research.

[9]  Valerie Galpin,et al.  Relevance feedback in a public access catalogue for a research library: MUSCAT at the Scott Polar Research Institute Library , 1988 .

[10]  Nicholas J. Belkin,et al.  Using Relevance Feedback and Ranking in Interactive Searching , 1995, TREC.

[11]  Efthimis N. Efthimiadis,et al.  A user-centred evaluation of ranking algorithms for interactive query expansion , 1993, SIGIR.

[12]  Mark D. Dunlop.  The effect of accessing nonmatching documents on relevance feedback , 1997, TOIS.

[13]  Fabrizio Sebastiani,et al.  A probabilistic terminological logic for modelling information retrieval , 1994, SIGIR '94.

[14]  Stephen E. Robertson,et al.  Okapi at TREC-3 , 1994, TREC.

[15]  Catherine Berrut,et al.  An Image Retrieval System Based on the Visualization of System Relevance via Documents , 1997, DEXA.

[16]  W. Bruce Croft,et al.  Relevance feedback and inference networks , 1993, SIGIR.

[17]  IJsbrand Jan Aalbersberg,et al.  Incremental relevance feedback , 1992, SIGIR '92.

[18]  Carol Collier Kuhlthau.  Inside the Search Process: Information Seeking from the User's Perspective. , 1991 .

[19]  Alan F. Smeaton,et al.  The Retrieval Effects of Query Expansion on a Feedback Document Retrieval System , 1983, Comput. J..

[20]  Ophir Frieder,et al.  Improving relevance feedback in the vector space model , 1997, CIKM '97.

[21]  Sanjiv K. Bhatia,et al.  Selection of search terms based on user profile , 1992, SAC '92.

[22]  Peter Ingwersen,et al.  Cognitive Perspectives of Information Retrieval Interaction: Elements of a Cognitive IR Theory , 1996, J. Documentation.

[23]  Stephen E. Robertson,et al.  Some simple effective approximations to the 2-Poisson model for probabilistic weighted retrieval , 1994, SIGIR '94.

[24]  Mary Elizabeth Stevens,et al.  Statistical Association Methods for Mechanized Documentation. , 1967 .

[25]  Raya Fidel,et al.  Ranking expansion terms using partial and ostensive evidence , 2002 .

[26]  Carol L. Barry,et al.  Order Effects: A Study of the Possible Influence of Presentation Order on User Judgments of Document Relevance. , 1988 .

[27]  Joon Ho Lee,et al.  Combining the Evidence of Different Relevance Feedback Methods for Information Retrieval , 1998, Inf. Process. Manag..

[28]  Ingrid Hsieh-Yee,et al.  Effects of Search Experience and Subject Knowledge on the Search Tactics of Novice and Experienced Searchers. , 1993 .

[29]  Makoto Iwayama,et al.  Relevance feedback with a small number of relevance judgements: incremental relevance feedback vs. document clustering , 2000, SIGIR '00.

[30]  Ingrid Hsieh-Yee,et al.  Effects of Search Experience and Subject Knowledge on the Search Tactics of Novice and Experienced Searchers , 1993, J. Am. Soc. Inf. Sci..

[31]  Stephen E. Robertson,et al.  A probabilistic model of information retrieval: development and comparative experiments - Part 2 , 2000, Inf. Process. Manag..

[32]  Efthimis N. Efthimiadis,et al.  UCLA-Okapi at TREC-2: Query Expansion Experiments , 1993, TREC.

[33]  Mark Magennis,et al.  The potential and actual effectiveness of interactive query expansion , 1997, SIGIR '97.

[34]  Peter Ingwersen,et al.  Information Retrieval Interaction , 1992 .

[35]  M. E. Maron,et al.  On Relevance, Probabilistic Indexing and Information Retrieval , 1960, JACM.

[36]  Nicholas J. Belkin,et al.  New tools and old habits : the interactive searching behavior of expert online searchers using INQUERY , 1994 .

[37]  Nicholas J. Belkin,et al.  Ranking in Principle , 1978, J. Documentation.

[38]  Patrick van Bommel,et al.  Augmenting a characterization network with semantic information , 1997, Inf. Process. Manag..

[39]  Donna K. Harman,et al.  Overview of the Fifth Text REtrieval Conference (TREC-5) , 1996, TREC.

[40]  Donna K. Harman,et al.  Relevance feedback revisited , 1992, SIGIR '92.

[41]  S. E. Robertson,et al.  On Relevance weight estimation and Query Expansion , 1986, J. Documentation.

[42]  Raya Fidel.  Searchers' selection of search keys: I. The selection routine , 1991 .

[43]  Jian-Yun Nie,et al.  An information retrieval model based on modal logic , 1989, Inf. Process. Manag..

[44]  Carol Collier Kuhlthau,et al.  A Principle of Uncertainty for Information seeking , 1993, J. Documentation.

[45]  Efthimis N. Efthimiadis,et al.  User Choices: A new Yardstick for the Evaluation of Ranking Algorithms for Interactive Query Expansion , 1995, Inf. Process. Manag..

[46]  Chris Buckley,et al.  Improving automatic query expansion , 1998, SIGIR '98.

[47]  Christine L. Borgman,et al.  All users of information retrieval systems are not created equal: An exploration into individual differences , 1989, Inf. Process. Manag..

[48]  Donna K. Harman,et al.  Overview of the Second Text REtrieval Conference (TREC-2) , 1994, HLT.

[49]  T. Bayes.  An essay towards solving a problem in the doctrine of chances , 2003 .

[50]  Amanda Spink,et al.  Toward a Theoretical Framework for Information Retrieval (IR) Evaluation in an Information Seeking Context , 1999, MIRA.

[51]  Nicholas J. Belkin,et al.  A case for interaction: a study of interactive information retrieval behavior and effectiveness , 1996, CHI.

[52]  Stephen E. Robertson,et al.  A probabilistic model of information retrieval: development and comparative experiments - Part 1 , 2000, Inf. Process. Manag..

[53]  Peter Bruza,et al.  Users' models of the information space: the case for two search models , 1995, SIGIR '95.

[54]  Stephen E. Robertson,et al.  On Term Selection for Query Expansion , 1991, J. Documentation.

[55]  Vladimir G. Voiskunskii,et al.  Boolean Search: Current State and Perspectives , 1999, J. Am. Soc. Inf. Sci..

[56]  Carol Collier Kuhlthau,et al.  Inside the search process: Information seeking from the user's perspective , 1991, J. Am. Soc. Inf. Sci..

[57]  Michael Eisenberg,et al.  Order effects: A study of the possible influence of presentation order on user judgments of document relevance , 1988, J. Am. Soc. Inf. Sci..

[58]  William M. Shaw,et al.  Interactive Retrieval using IRIS: TREC-6 Experiments , 1997, TREC.

[59]  Marcia J. Bates,et al.  Where should the person stop and the information search interface start? , 1990, Inf. Process. Manag..

[60]  Carol L. Barry,et al.  Users' Criteria for Relevance Evaluation: A Cross-situational Comparison , 1998, Inf. Process. Manag..

[61]  Micheline Hancock-Beaulieu,et al.  Interactive searching and interface issues in the Okapi best match probabilistic retrieval system , 1998, Interact. Comput..

[62]  Karen Spärck Jones.  A statistical interpretation of term specificity and its application in retrieval , 2021, J. Documentation.

[63]  Fredric C. Gey,et al.  The Relationship between Recall and Precision , 1994, J. Am. Soc. Inf. Sci..

[64]  Christine L. Borgman,et al.  Why are online catalogs still hard to use , 1996 .

[65]  Peter Willett,et al.  The limitations of term co-occurrence data for query expansion in document retrieval systems , 1991, J. Am. Soc. Inf. Sci..

[66]  Mounia Lalmas,et al.  Information Retrieval: Uncertainty and Logics: Advanced Models for the Representation and Retrieval of Information , 1998 .

[67]  Ryen W. White,et al.  A task-oriented study on the influencing effects of query-biased summarisation in web searching , 2003, Inf. Process. Manag..

[68]  James Allan,et al.  Automatic Query Expansion Using SMART: TREC 3 , 1994, TREC.

[69]  Joseph W. Janes,et al.  Relevance judgments and the incremental presentation of document representations , 1991, Inf. Process. Manag..

[70]  W. Bruce Croft,et al.  The INQUERY Retrieval System , 1992, DEXA.

[71]  M. F. Porter,et al.  An algorithm for suffix stripping , 1997 .

[72]  Peter Ingwersen,et al.  The Application of Work Tasks in Connection with the Evaluation of Interactive Information Retrieval Systems: Empirical Results , 1999, MIRA.

[73]  Ian Ruthven.  On the Use of Explanations as Mediating Device for Relevance Feedback , 2002, ECDL.

[74]  Christopher S. G. Khoo,et al.  An Expert Approach to Online Catalog Subject Searching , 1994, Inf. Process. Manag..

[75]  Donna K. Harman,et al.  Relevance Feedback and Other Query Modification Techniques , 1992, Information retrieval (Boston).

[76]  David Ellis,et al.  A Behavioural Approach to Information Retrieval System Design , 1989, J. Documentation.

[77]  Pertti Vakkari,et al.  Cognition and changes of search terms and tactics during task performance: A longitudinal case study , 2000, RIAO.

[78]  C. J. van Rijsbergen,et al.  A Non-Classical Logic for Information Retrieval , 1997, Comput. J..

[79]  Donna K. Harman,et al.  Overview of the Sixth Text REtrieval Conference (TREC-6) , 1997, Inf. Process. Manag..

[80]  Nicholas J. Belkin,et al.  Rutgers' TREC 2001 Interactive Track Experience , 2001, TREC.

[81]  C. J. van Rijsbergen,et al.  Empirical investigations on query modification using abductive explanations , 2001, SIGIR '01.

[82]  Raya Fidel,et al.  Searchers' selection of search keys: II. Controlled vocabulary or free‐text searching , 1991 .

[83]  S. Robertson.  The probability ranking principle in IR , 1997 .

[84]  Pertti Vakkari,et al.  Relevance and contributing information types of searched documents in task performance , 2000, SIGIR '00.

[85]  Micheline Beaulieu,et al.  Experiments on interfaces to support query expansion , 1997, J. Documentation.

[86]  Donna K. Harman,et al.  Overview of the First Text REtrieval Conference (TREC-1) , 1992, TREC.

[87]  E. A. Fox,et al.  Combining the Evidence of Multiple Query Representations for Information Retrieval , 1995, Inf. Process. Manag..

[88]  Amanda Spink,et al.  Interaction in Information Retrieval: Selection and Effectiveness of Search Terms , 1997, J. Am. Soc. Inf. Sci..

[89]  Fabio Crestani,et al.  The Troubles with Using a Logical Model of IR on a Large Collection of Documents , 1995, TREC.

[90]  Gary Marchionini,et al.  Information processing in the context of medical care , 1995, SIGIR '95.

[91]  Louise T. Su.  The Relevance of Recall and Precision in User Evaluation , 1994, J. Am. Soc. Inf. Sci..

[92]  Isola Ajiferuke,et al.  A total relevance and document interaction effects model for the evaluation of information retrieval processes , 1988, Inf. Process. Manag..

[93]  Amanda Spink,et al.  Study of Interactive Feedback During Mediated Information Retrieval , 1997, J. Am. Soc. Inf. Sci..

[94]  Amanda Spink,et al.  From Highly Relevant to Not Relevant: Examining Different Regions of Relevance , 1998, Inf. Process. Manag..

[95]  Stephen E. Robertson,et al.  Relevance weighting of search terms , 1976, J. Am. Soc. Inf. Sci..

[96]  Tom Peters,et al.  When Smart People Fail: An Analysis of the Transaction Log of an Online Public Access Catalog. , 1989 .

[97]  Joemon M. Jose,et al.  A study on the use of summaries and summary-based query expansion for a question-answering task , 2001 .

[98]  Peter Schäuble,et al.  The Perils of Interpreting Recall and Precision Values , 1991, Information Retrieval.

[99]  Amanda Spink,et al.  Multiple Search Sessions Model of End-User Behavior: An Exploratory Study , 1996, J. Am. Soc. Inf. Sci..

[100]  Nicholas J. Belkin,et al.  Rutgers Interactive Track at TREC-5 , 1996, TREC.

[101]  Nicholas J. Belkin,et al.  Rutgers' TREC-6 Interactive Track Experience , 1997, TREC.

[102]  Mounia Lalmas,et al.  The use of logic in information retrieval modelling , 1998, The Knowledge Engineering Review.

[103]  C. J. van Rijsbergen,et al.  Incorporating user search behavior into relevance feedback , 2003, J. Assoc. Inf. Sci. Technol..

[104]  Karen Spärck Jones.  Search Term Relevance Weighting given Little Relevance Information , 1997, J. Documentation.

[105]  J. J. Rocchio,et al.  Relevance feedback in information retrieval , 1971 .

[106]  Peter Ingwersen,et al.  Polyrepresentation of information needs and semantic entities: elements of a cognitive theory for information retrieval interaction , 1994, SIGIR '94.

[107]  Peter Ingwersen,et al.  The development of a method for the evaluation of interactive information retrieval systems , 1997, J. Documentation.

[108]  Chris Buckley,et al.  A probabilistic learning approach for document indexing , 1991, TOIS.

[109]  Gerard Salton,et al.  The SMART Retrieval System—Experiments in Automatic Document Processing , 1971 .

[110]  C. J. van Rijsbergen,et al.  Combining and selecting characteristics of information use , 2002, J. Assoc. Inf. Sci. Technol..