Relevance: A review of the literature and a framework for thinking on the notion in information science. Part III: Behavior and effects of relevance

All is flux. —Plato on Knowledge in the Theaetetus (about 369 BC) Relevance is a, if not even the, key notion in information science in general and information retrieval in particular. This two-part critical review traces and synthesizes the scholarship on relevance over the past 30 years or so and provides an updated framework within which the still widely dissonant ideas and works about relevance might be interpreted and related. It is a continuation and update of a similar review that appeared in 1975 under the same title, considered here as being Part I. The present review is organized in two parts: Part II addresses the questions related to nature and manifestations of relevance, and Part III addresses questions related to relevance behavior and effects. In Part II, the nature of relevance is discussed in terms of meaning ascribed to relevance, theories used or proposed, and models that have been developed. The manifestations of relevance are classified as to several kinds of relevance that form an interdependent system of relevancies. In Part III, relevance behavior and effects are synthesized using experimental and observational works that incorporated data. In both parts, each section concludes with a summary that in effect provides an interpretation and synthesis of contemporary thinking on the topic treated or suggests hypotheses for future research. Analyses of some of the major trends that shape relevance work are offered in conclusions. © 2007 Wiley Periodicals, Inc.

[1]  Peng Dong,et al.  Relevance similarity: an alternative means to monitor information retrieval systems , 2005, Biomedical digital libraries.

[2]  Joseph Janes,et al.  Relevance Judgments of Actual Users and Secondary Judges: A Comparative Study , 1992, The Library Quarterly.

[3]  Soo Young Rieh,et al.  Analysis of multiple query reformulations on the web: The interactive information retrieval context , 2006, Information Processing & Management.

[4]  John J. Regazzi Performance measures for information retrieval systems ― an experimental approach , 1988 .

[5]  Javed Mostafa,et al.  An experiment in building profiles in information filtering: the role of context of user relevance feedback , 2002, Inf. Process. Manag..

[6]  Mu-Hsuan Huang,et al.  The influence of document presentation order and number of documents judged on users' judgments of relevance , 2004, J. Assoc. Inf. Sci. Technol..

[7]  Ian Ruthven,et al.  Integrating approaches to relevance , 2005 .

[8]  Amanda Spink,et al.  Regions and levels: Measuring and mapping users' relevance judgments , 2001, J. Assoc. Inf. Sci. Technol..

[9]  David Ellis,et al.  The Dilemma of Measurement in Information Retrieval Research , 1996, J. Am. Soc. Inf. Sci..

[10]  Nicholas J. Belkin,et al.  A case for interaction: a study of interactive information retrieval behavior and effectiveness , 1996, CHI.

[11]  David Levy,et al.  Digital Libraries and the Problem of Purpose , 2000, D Lib Mag..

[12]  Katherine W. McCain,et al.  Visualizing a discipline: an author co-citation analysis of information science, 1972–1995 , 1998 .

[13]  Ellen M. Voorhees,et al.  Evaluation by highly relevant documents , 2001, SIGIR '01.

[14]  Yunjie Xu,et al.  Relevance judgment: What do information users consider beyond topicality? , 2006 .

[15]  Don R. Swanson,et al.  Some Unexplained Aspects of the Cranfield Tests of Indexing Performance Factors , 1971, The Library Quarterly.

[16]  Harry Bruce A cognitive view of the situational dynamism of user-centered relevance estimation , 1994 .

[17]  Carol A. Bean,et al.  Topical Relevance Relationships. II. An Exploratory Study and Preliminary Typology , 1995, J. Am. Soc. Inf. Sci..

[18]  Neil R. Smalheiser,et al.  Artificial Intelligence An interactive system for finding complementary literatures : a stimulus to scientific discovery , 1995 .

[19]  David Davidson,et al.  The effect of individual differences of cognitive style on judgments of document relevance , 1977, J. Am. Soc. Inf. Sci..

[20]  Amanda Spink,et al.  New Directions in Cognitive Information Retrieval , 2005 .

[21]  Tefko Saracevic,et al.  Individual Differences in Organizing, Searching and Retrieving Information. , 1991 .

[22]  Myke Gluck,et al.  Exploring the Relationship between User Satisfaction and Relevance in Information Systems , 1996, Inf. Process. Manag..

[23]  Michael E. Lesk,et al.  Relevance assessments and retrieval system evaluation , 1968, Inf. Storage Retr..

[24]  Ellen M. Voorhees Variations in relevance judgments and the measurement of retrieval effectiveness , 2000, Inf. Process. Manag..

[25]  Linda Schamber Relevance and Information Behavior. , 1994 .

[26]  Robert Burgin Variations in Relevance Judgments and the Evaluation of Retrieval Performance , 1992, Inf. Process. Manag..

[27]  Fabrizio Sebastiani,et al.  Trends in ... a Critical Review: On the Role of Logic in Information Retrieval , 1998, Inf. Process. Manag..

[28]  Robert E. Johnson,et al.  Does Order of Presentation Affect Users' Judgment of Documents?. , 1990 .

[29]  Helen R. Tibbo,et al.  The Cystic Fibrosis Database: Content and Research Opportunities. , 1991 .

[30]  William Goffman,et al.  On relevance as a measure , 1964, Inf. Storage Retr..

[31]  Carol L. Barry,et al.  Users' Criteria for Relevance Evaluation: A Cross-situational Comparison , 1998, Inf. Process. Manag..

[32]  Preben Hansen,et al.  Effects of foreign language and task scenario on relevance assessment , 2005, J. Documentation.

[33]  Carol Collier Kuhlthau,et al.  Seeking Meaning: a process approach to library and information services" Ablex Publishing , 2003 .

[34]  Gerhard Lakemeyer,et al.  Relevance from an Epistemic Perspective , 1997, Artif. Intell..

[35]  Amanda Spink,et al.  From Highly Relevant to Not Relevant: Examining Different Regions of Relevance , 1998, Inf. Process. Manag..

[36]  Nicholas J. Belkin,et al.  The TREC Interactive Tracks: Putting the User into Search , 2005 .

[37]  Raya Fidel,et al.  Users' perception of the performance of a filtering system , 1997, SIGIR '97.

[38]  Amanda Spink,et al.  Human-computer interaction in information retrieval: nature and manifestations of feedback , 1998, Interact. Comput..

[39]  Nuel D. Belnap,et al.  Entailment : the logic of relevance and necessity , 1975 .

[40]  Thomas J. Froehlich,et al.  Relevance reconsidered—towards an agenda for the 21st century: introduction to special topic issue on relevance research , 1994 .

[41]  Fabian A. Ehikhamenor,et al.  Socio-economic factors in the application of information and communication technologies in Nigerian print media , 2002, J. Assoc. Inf. Sci. Technol..

[42]  Paul B. Kantor,et al.  Studying the Value of Library and Information Services. Part I: Establishing a Theoretical Framework. , 1997 .

[43]  Joseph W. Janes On the Distribution of Relevance Judgments. , 1993 .

[44]  D. Sperber,et al.  Relevance: Communication and cognition, 2nd ed. , 1995 .

[45]  M E Williams,et al.  Data bases on-line in 1979. , 1979, Bulletin of the American Society for Information Science.

[46]  William R. Hersh Relevance and retrieval evaluation: perspectives from medicine , 1994 .

[47]  Peter Ingwersen,et al.  Cognitive Perspectives of Information Retrieval Interaction: Elements of a Cognitive IR Theory , 1996, J. Documentation.

[48]  Myke Gluck Understanding performance in information systems: blending relevance and competence , 1995 .

[49]  Calvin N. Mooers,et al.  Zatocoding applied to mechanical organization of knowledge , 1951 .

[50]  Michael B. Eisenberg Measuring relevance judgments , 1988, Inf. Process. Manag..

[51]  Dara Lee Howard Pertinence as reflected in personal constructs , 1994 .

[52]  Peter Ingwersen,et al.  The Turn - Integration of Information Seeking and Retrieval in Context , 2005, The Kluwer International Series on Information Retrieval.

[53]  John M. Budd,et al.  Relevance: Language, Semantics, Philosophy , 2004, Libr. Trends.

[54]  J. Janes Other people's judgments: a comparison of users' and others' judgments of document relevance, topicality, and utility , 1994 .

[55]  Stephen E. Robertson,et al.  On the Evaluation of IR Systems , 1992, Inf. Process. Manag..

[56]  Birger Hjørland,et al.  Epistemology and the socio-cognitive perspective in information science , 2002, J. Assoc. Inf. Sci. Technol..

[57]  Nicholas J. Belkin,et al.  Rutgers Information Retrieval Evaluation Project on IR Performance on Different Precision Levels , 2006 .

[58]  T. Park The Nature of Relevance in Information Retrieval: An Empirical Study , 1993, The Library Quarterly.

[59]  Judy Bateman Changes in Relevance Criteria: A Longitudinal Study. , 1998 .

[60]  Yunjie Xu Relevance judgment in epistemic and hedonic information searches , 2007 .

[61]  Michael B. Eisenberg,et al.  DICHOTOMOUS RELEVANCE JUDGMENTS AND THE EVALUATION OF INFORMATION SYSTEMS. , 1987 .

[62]  Dagobert Soergel Indexing and retrieval performance: the logical evidence , 1994 .

[63]  José Luis Vicedo González,et al.  TREC: Experiment and evaluation in information retrieval , 2007, J. Assoc. Inf. Sci. Technol..

[64]  Carol L. Barry User-defined relevance criteria: an exploratory study , 1994 .

[65]  Tefko Saracevic,et al.  RELEVANCE: A review of and a framework for the thinking on the notion in information science , 1997, J. Am. Soc. Inf. Sci..

[66]  Amanda Spink,et al.  A multitasking framework for cognitive information retrieval , 2005 .

[67]  T. Saracevic,et al.  Relevance: A review of the literature and a framework for thinking on the notion in information science. Part II: nature and manifestations of relevance , 2007, J. Assoc. Inf. Sci. Technol..

[68]  James A. Thom,et al.  Relevance Judgments for Assessing Recall , 1996, Inf. Process. Manag..

[69]  Carol L. Barry,et al.  Order Effects: A Study of the Possible Influence of Presentation Order on User Judgments of Document Relevance. , 1988 .

[70]  Cyril Cleverdon,et al.  The Cranfield tests on index language devices , 1997 .

[71]  Diane Kelly,et al.  IMPLICIT FEEDBACK: USING BEHAVIOR TO INFER RELEVANCE , 2005 .

[72]  A. Schutz,et al.  Reflections On The Problem Of Relevance , 1971 .

[73]  Pertti Vakkari,et al.  The influence of relevance levels on the effectiveness of interactive information retrieval , 2004, J. Assoc. Inf. Sci. Technol..

[74]  Mark Sanderson,et al.  Advantages of query biased summaries in information retrieval , 1998, SIGIR '98.

[75]  K. A. McKibbon,et al.  Online access to MEDLINE in clinical settings. A study of use and usefulness. , 1990, Annals of internal medicine.

[76]  Amanda Spink,et al.  Median measure: an approach to IR systems evaluation , 2001, Inf. Process. Manag..

[77]  Taemin Kim Park Toward a theory of user-based relevance: a call for a new paradigm of inquiry , 1994 .

[78]  Joseph W. Janes,et al.  Relevance judgments and the incremental presentation of document representations , 1991, Inf. Process. Manag..

[79]  Stephen P. Harter Variations in relevance assessments and the measurement of retrieval effectiveness , 1996 .

[80]  Howard D. White,et al.  Combining bibliometrics, information retrieval, and relevance theory, Part 2: Some implications for information science , 2007, J. Assoc. Inf. Sci. Technol..

[81]  Theresa Dirndorfer Anderson,et al.  Relevance as process: judgements in the context of scholarly research , 2005, Inf. Res..

[82]  S. Hirsh Children's Relevance Criteria and Information Seeking on Electronic Resources , 1999, J. Am. Soc. Inf. Sci..

[83]  Giyeong Kim,et al.  Relationship between index term specificity and relevance judgment , 2006, Inf. Process. Manag..

[84]  Judea Pearl,et al.  Axioms of Causal Relevance , 1997, Artif. Intell..

[85]  C. J. van Rijsbergen,et al.  A Non-Classical Logic for Information Retrieval , 1997, Comput. J..

[86]  Paul Solomon,et al.  Toward an Understanding of the Dynamics of Relevance Judgment: An Analysis of One Person's Search Behavior , 1998, Inf. Process. Manag..

[87]  Amanda Spink,et al.  Real life, real users, and real needs: a study and analysis of user queries on the web , 2000, Inf. Process. Manag..

[88]  Stephen P. Harter,et al.  The Cranfield II Relevance Assessments: A Critical Evaluation , 1971, The Library Quarterly.

[89]  Edie M. Rasmussen,et al.  Users' relevance criteria in image retrieval in American history , 2002, Inf. Process. Manag..

[90]  C. J. van Rijsbergen,et al.  Incorporating user search behavior into relevance feedback , 2003, J. Assoc. Inf. Sci. Technol..

[91]  Joemon M. Jose,et al.  How users assess Web pages for information seeking , 2005, J. Assoc. Inf. Sci. Technol..

[92]  T. Saracevic Relevance: A Review of the Literature and a Framework for Thinking on the Notion in Information Science. Part II , 2006 .

[93]  Tefko Saracevic,et al.  The Stratified Model of Information Retrieval Interaction: Extension and Applications , 1997 .

[94]  Justin Zobel,et al.  How reliable are the results of large-scale information retrieval experiments? , 1998, SIGIR '98.

[95]  Michael B. Eisenberg,et al.  A re-examination of relevance: toward a dynamic, situational definition , 1990, Inf. Process. Manag..

[96]  William S. Cooper,et al.  A definition of relevance for information retrieval , 1971, Inf. Storage Retr..

[97]  Louise T. Su Evaluation Measures for Interactive Information Retrieval , 1992, Inf. Process. Manag..

[98]  Tefko Saracevic,et al.  Relevance: A review of the literature and a framework for thinking on the notion in information science. Part III: Behavior and effects of relevance , 2007, J. Assoc. Inf. Sci. Technol..

[99]  Roger Lindsay,et al.  The roots of relevance , 1993 .

[100]  George Kingsley Zipf,et al.  Human behavior and the principle of least effort , 1949 .

[101]  John R. Searle,et al.  Intentionality and its place in nature , 1984, Synthese.

[102]  Mounia Lalmas,et al.  Logical Models in Information Retrieval: Introduction and Overview , 1998, Inf. Process. Manag..

[103]  A. Schutz The Structures of the Life World , 1973 .

[104]  Linda Schamber,et al.  Relevance Criteria Uses and Importance: Progress in Development of a Measurement Scale. , 1999 .

[105]  Pia Borlund,et al.  The concept of relevance in IR , 2003, J. Assoc. Inf. Sci. Technol..

[106]  Jian-Yun Nie,et al.  Information Retrieval as Counterfactual , 1995, Comput. J..

[107]  Peter Ingwersen,et al.  Dimensions of relevance , 2000, Inf. Process. Manag..

[108]  Stefano Mizzaro Relevance: the whole history , 1997 .

[109]  Joseph Janes The Binary Nature of Continuous Relevance Judgments: A Study of Users' Perceptions. , 1991 .

[110]  Soo Young Rieh,et al.  Interaction on the Web: Scholars' Judgment of Information Quality and Cognitive Authority , 2000 .

[111]  Pertti Vakkari,et al.  Changes in relevance criteria and problem stages in task performance , 2000, J. Documentation.

[112]  Elaine Toms,et al.  Searching for Relevance in the Relevance of Search , 2005, CoLIS.

[113]  Rebecca Green,et al.  Topical Relevance Relationships. I. Why Topic Matching Fails , 1995, J. Am. Soc. Inf. Sci..

[114]  Neil R. Smalheiser,et al.  Implicit Text Linkages between Medline Records: Using Arrowsmith as an Aid to Scientific Discovery , 1999, Libr. Trends.

[115]  C. D. Gull Seven years of work on the organization of materials in the special library , 1956 .

[116]  D. Sperber,et al.  Relevance: Communication and Cognition , 1989 .

[117]  K. Popper Objective Knowledge: An Evolutionary Approach , 1972 .

[118]  Stefano Mizzaro,et al.  How many relevances in information retrieval? , 1998, Interact. Comput..

[119]  Carol L. Barry Document representations and clues to document relevance , 1998 .

[120]  Russell Greiner,et al.  The Relevance of Relevance (Editorial) , 1997, Artif. Intell..

[121]  Don R. Swanson,et al.  Subjective versus Objective Relevance in Bibliographic Retrieval Systems , 1986, The Library Quarterly.

[122]  Paul B. Kantor,et al.  A Study of Information Seeking and Retrieving. III. Searchers, Searches, and Overlap* , 1988 .

[123]  Peiling Wang,et al.  A Cognitive Model of Document Use During a Research Project. Study II. Decisions at the Reading and Citing Stages , 1999, J. Am. Soc. Inf. Sci..

[124]  Steve Smithson,et al.  Information Retrieval Evaluation in Practice: A Case Study Approach , 1994, Inf. Process. Manag..

[125]  Chad Galloway,et al.  Relevance judging, evaluation, and decision making in virtual libraries: A descriptive study , 2001, J. Assoc. Inf. Sci. Technol..

[126]  Howard D. White,et al.  Combining bibliometrics, information retrieval, and relevance theory, Part 1: First examples of a synthesis , 2007, J. Assoc. Inf. Sci. Technol..

[127]  Diane Kelly Measuring online information seeking context, Part 1: Background and method , 2006 .

[128]  Peiling Wang,et al.  A cognitive model of document use during a research project. Study I. document selection , 1998 .

[129]  Peiling Wang,et al.  Document Use during a Research Project: A Longitudinal Study. , 1995 .

[130]  Pertti Vakkari,et al.  Changes in Search Tactics and Relevance Judgements when Preparing a Research Proposal A Summary of the Findings of a Longitudinal Study , 2001, Information Retrieval.

[131]  Diane H. Sonnenwald,et al.  User perspectives on relevance criteria: A comparison among relevant, partially relevant, and not-relevant judgments , 2002, J. Assoc. Inf. Sci. Technol..

[132]  Howard Greisdorf,et al.  Relevance thresholds: a multi-stage predictive model of how users evaluate information , 2003, Inf. Process. Manag..

[133]  Pam J. Mayhew,et al.  An Alternative Prototyping Classification , 1987, Comput. J..

[134]  Allen Kent,et al.  Machine literature searching VIII. Operational criteria for designing information retrieval systems , 1955 .

[135]  Rong Tang,et al.  Use of relevance criteria across stages of document evaluation: on the complementarity of experimental and naturalistic studies , 2001 .

[136]  S. P. Harter Psychological relevance and information science , 1992 .