Relevance: An improved framework for explicating the notion

Synthesizing and building on many ideas from the literature, this article presents an improved conceptual framework that clarifies the notion of relevance with its many elements, variables, criteria, and situational factors. Relevance is defined as a Relationship (R) between an Information Object (I) and an Information Need (N) (which consists of Topic, User, Problem/Task, and Situation/Context) with focus on R. This defines Relevance-as-is (conceptual relevance, strong relevance). To determine relevance, an Agent A (a person or system) operates on a representation I′ of the information object and a representation N′ of the information need, resulting in relevance-as-determined (operational measure of relevance, weak relevance, an approximation). Retrieval tests compare relevance-as-determined by different agents. This article discusses and compares two major approaches to conceptualizing relevance: the entity-focused approach (focus on elaborating the entities involved in relevance) and the relationship-focused approach (focus on explicating the relational nature of relevance). The article argues that because relevance is fundamentally a relational construct the relationship-focused approach deserves a higher priority and more attention than it has received. The article further elaborates on the elements of the framework with a focus on clarifying several critical issues on the discourse on relevance. © 2013 Wiley Periodicals, Inc.

[1]  Nicholas J. Belkin,et al.  Retrieval techniques , 1987 .

[2]  Peter Ingwersen,et al.  The Turn - Integration of Information Seeking and Retrieval in Context , 2005, The Kluwer International Series on Information Retrieval.

[3]  Stefano Mizzaro,et al.  Relevance: The Whole History , 1997, J. Am. Soc. Inf. Sci..

[4]  Patrick Wilson,et al.  Situational relevance , 1973, Inf. Storage Retr..

[5]  Taemin Kim Park,et al.  Toward a Theory of User-Based Relevance: A Call for a New Paradigm of Inquiry , 1994, J. Am. Soc. Inf. Sci..

[6]  Judy Bateman Changes in Relevance Criteria: A Longitudinal Study. , 1998 .

[7]  Stephen P. Harter,et al.  Psychological Relevance and Information Science , 1992, J. Am. Soc. Inf. Sci..

[8]  Paul B. Kantor,et al.  Cross-Evaluation: A new model for information system evaluation , 2006, J. Assoc. Inf. Sci. Technol..

[9]  Anita Komlodi Task management support in information seeking: a case for search histories , 2004, Comput. Hum. Behav..

[10]  Xiaoli Huang,et al.  Developing a Cross-Disciplinary Typology of Topical Relevance Relationships as the Basis for a Topic-Oriented Information Architecture , 2009 .

[11]  Don R. Swanson,et al.  Information Retrieval as a Trial-And-Error Process , 1977, The Library Quarterly.

[12]  José Luis Vicedo González,et al.  TREC: Experiment and evaluation in information retrieval , 2007, J. Assoc. Inf. Sci. Technol..

[13]  Harry W. Bruce,et al.  A Cognitive View of the Situational Dynamism of User-Centered Relevance Estimation , 1994, J. Am. Soc. Inf. Sci..

[14]  Ryen W. White,et al.  Policy capturing models for multi-faceted relevance judgments , 2005, ASIST.

[15]  C. A. Cuadra,et al.  OPENING THE BLACK BOX OF ‘RELEVANCE’ , 1967 .

[16]  Tefko Saracevic,et al.  RELEVANCE: A review of and a framework for the thinking on the notion in information science , 1997, J. Am. Soc. Inf. Sci..

[17]  Ellen M. Voorhees,et al.  TREC: Experiment and Evaluation in Information Retrieval (Digital Libraries and Electronic Publishing) , 2005 .

[18]  Linda Schamber Relevance and Information Behavior. , 1994 .

[19]  Paul B. Kantor,et al.  Using cross-evaluation to evaluate interactive QA systems , 2011, J. Assoc. Inf. Sci. Technol..

[20]  Carol A. Bean,et al.  Topical Relevance Relationships. II. An Exploratory Study and Preliminary Typology , 1995, J. Am. Soc. Inf. Sci..

[21]  C. J. van Rijsbergen,et al.  A New Theoretical Framework for Information Retrieval , 1986, SIGIR Forum.

[22]  Alexander T. Nicolai,et al.  A Note on the Concept of Relevance , 2007 .

[23]  M. E. Maron,et al.  An evaluation of retrieval effectiveness for a full-text document-retrieval system , 1985, CACM.

[24]  Don R. Swanson,et al.  Subjective versus Objective Relevance in Bibliographic Retrieval Systems , 1986, The Library Quarterly.

[25]  Gary Marchionini,et al.  Search histories for user support in user interfaces , 2006, J. Assoc. Inf. Sci. Technol..

[26]  Mortimer Taube Associates Storage and retrieval of information by means of the association of ideas , 1955 .

[27]  Cyril W. Cleverdon,et al.  Report on the first stage of an investigation into the comparative efficiency of indexing systems , 1960 .

[28]  C. J. van Rijsbergen,et al.  (invited paper) A new theoretical framework for information retrieval , 1986, SIGIR '86.

[29]  Birger Hjørland The foundation of the concept of relevance , 2009 .

[30]  R. A. Fairthorne,et al.  The patterns of retrieval , 1956 .

[31]  Nicholas J. Belkin,et al.  Characteristics of Texts Affecting Relevance Judgments , 1993 .

[32]  Douglas G. Schultz,et al.  A Field Experimental Approach to the Study of Relevance Assessments in Relation to Document Searching. Final Report to the National Science Foundation. Volume II, Appendices. , 1967 .

[33]  D. A. Kemp Relevance, pertinence and information system development , 1974, Inf. Storage Retr..

[34]  Dagobert Soergel,et al.  Relevance criteria used by teachers in selecting oral history materials , 2005, ASIST.

[35]  Joseph W. Janes,et al.  Other People's Judgments: A Comparison of Users' and Others' Judgments of Document Relevance, Topicality, and Utility , 1994, J. Am. Soc. Inf. Sci..

[36]  Robert A. Fairthorne,et al.  Empirical hyperbolic distributions (Bradford-Zipf-Mandelbrot) for bibliometric description and prediction , 1969, J. Documentation.

[37]  Nils Pharo,et al.  The effect of task type on preferred element types in an XML-based retrieval system , 2011, J. Assoc. Inf. Sci. Technol..

[38]  Patrick Wilson,et al.  Some Fundamental Concepts of Information Retrieval. , 1978 .

[39]  Mounia Lalmas,et al.  Logical Models in Information Retrieval: Introduction and Overview , 1998, Inf. Process. Manag..

[40]  Lauren B Doyle IS RELEVANCE AN ADEQUATE CRITERION IN RETRIEVAL SYSTEM EVALUATION , 1963 .

[41]  Yunjie Calvin Xu,et al.  Relevance judgment: What do information users consider beyond topicality? , 2006, J. Assoc. Inf. Sci. Technol..

[42]  C. J. van Rijsbergen,et al.  Towards an information logic , 1989, SIGIR '89.

[43]  Shyama Balakrishnan,et al.  Principles of information retrieval , 2000 .

[44]  Xiaoli Huang Relevance, Rhetoric, and Argumentation: A Cross-Disciplinary Inquiry into Patterns of Thinking and Information Structuring , 2009 .

[45]  Ole Harbo,et al.  Theory and application of information research : proceedings of the second International Research Forum on Information Science, 3-6 August 1977, Royal School of Librarianship, Copenhagen , 1980 .

[46]  Pia Borlund,et al.  The concept of relevance in IR , 2003, J. Assoc. Inf. Sci. Technol..

[47]  R. G. Thorne THE EFFICIENCY OF SUBJECT CATALOGUES AND THE COST OF INFORMATION SEARCHES , 1955 .

[48]  Dagobert Soergel,et al.  Relevance judges' understanding of topical relevance types: An explication of an enriched concept of topical relevance , 2004, ASIST.

[49]  Michael B. Eisenberg,et al.  A re-examination of relevance: toward a dynamic, situational definition , 1990, Inf. Process. Manag..

[50]  William S. Cooper,et al.  A definition of relevance for information retrieval , 1971, Inf. Storage Retr..

[51]  Pia Borlund,et al.  The IIR evaluation model: a framework for evaluation of interactive information retrieval systems , 2003, Inf. Res..

[52]  Thomas J. Froehlich,et al.  Relevance Reconsidered - Towards an Agenda for the 21st Century: Introduction to Special Topic Issue on Relevance Research , 1994, J. Am. Soc. Inf. Sci..

[53]  Peiling Wang A cognitive model of document selection of real users of information retrieval systems , 1994 .

[54]  Carl S. Wise Multiple word coding vs. random coding for the rapid selector. A reply to calvin N. mooers , 1952 .

[55]  T. Saracevic,et al.  Relevance: A review of the literature and a framework for thinking on the notion in information science. Part II: nature and manifestations of relevance , 2007, J. Assoc. Inf. Sci. Technol..

[56]  Carol L. Barry User-Defined Relevance Criteria: An Exploratory Study , 1994, J. Am. Soc. Inf. Sci..

[57]  Alfred J. Lotka,et al.  The frequency distribution of scientific productivity , 1926 .

[58]  Carol L. Barry,et al.  Order Effects: A Study of the Possible Influence of Presentation Order on User Judgments of Document Relevance. , 1988 .

[59]  S. Bradford "Sources of information on specific subjects" by S.C. Bradford , 1985 .

[60]  Edie M. Rasmussen,et al.  Searching for images: The analysis of users' queries for image retrieval in American history , 2003, J. Assoc. Inf. Sci. Technol..

[61]  Linda Schamber,et al.  User Criteria in Relevance Evaluation: Toward Development of a Measurement Scale. , 1996 .

[62]  M. E. Maron,et al.  On Relevance, Probabilistic Indexing and Information Retrieval , 1960, JACM.

[63]  G. Weiler IV.—ON RELEVANCE , 1962 .

[64]  William R. Hersh,et al.  Relevance and Retrieval Evaluation: Perspectives from Medicine , 1994, J. Am. Soc. Inf. Sci..

[65]  Birger Hjørland,et al.  The foundation of the concept of relevance , 2010, J. Assoc. Inf. Sci. Technol..

[66]  Dong Wang,et al.  Order effect in relevance judgment , 2008, J. Assoc. Inf. Sci. Technol..

[67]  Peter Ingwersen,et al.  Cognitive Perspectives of Information Retrieval Interaction: Elements of a Cognitive IR Theory , 1996, J. Documentation.

[68]  M. E. Maron,et al.  On indexing, retrieval and the meaning of about , 1977, J. Am. Soc. Inf. Sci..

[69]  Dagobert Soergel,et al.  Organizing information - principles of data base and retrieval systems , 1985 .

[70]  William S. Cooper,et al.  On selecting a measure of retrieval effectiveness , 1973, J. Am. Soc. Inf. Sci..

[71]  Birger Hjørland,et al.  Epistemology and the socio-cognitive perspective in information science , 2002, J. Assoc. Inf. Sci. Technol..

[72]  D. Hillman The notion of relevance (I) , 1964 .

[73]  Cyril W. Cleverdon,et al.  Factors determining the performance of indexing systems , 1966 .

[74]  T. Park The Nature of Relevance in Information Retrieval: An Empirical Study , 1993, The Library Quarterly.

[75]  Pertti Vakkari,et al.  Task-based information searching , 2005, Annu. Rev. Inf. Sci. Technol..

[76]  Peiling Wang,et al.  A Cognitive Model of Document Use During a Research Project. Study II. Decisions at the Reading and Citing Stages , 1999, Journal of the American Society for Information Science.

[77]  Linda Schamber Users' criteria for evaluation in multimedia information seeking and use situations , 1991 .

[78]  Carol L. Barry,et al.  Users' Criteria for Relevance Evaluation: A Cross-situational Comparison , 1998, Inf. Process. Manag..

[79]  William Goffman,et al.  A methodology for test and evaluation of information retrieval systems , 1966, Inf. Storage Retr..

[80]  Peter Ingwersen,et al.  Dimensions of relevance , 2000, Inf. Process. Manag..

[81]  Kam-Fai Wong,et al.  Aboutness from a commonsense perspective , 2000, J. Am. Soc. Inf. Sci..

[82]  Sandra G. Hirsh Children's Relevance Criteria and Information Seeking on Electronic Resources , 1999, J. Am. Soc. Inf. Sci..

[83]  David Bodoff,et al.  Relevance for browsing, relevance for searching , 2006, J. Assoc. Inf. Sci. Technol..

[84]  Documentation , 2006 .

[85]  T. Saracevic Relevance: A Review of the Literature and a Framework for Thinking on the Notion in Information Science. Part II , 2006 .

[86]  James Allan,et al.  A survey in indexing and searching XML documents , 2002, J. Assoc. Inf. Sci. Technol..

[87]  Deirdre Wilson,et al.  Relevance theory: A tutorial , 2002 .

[88]  Peiling Wang,et al.  A Cognitive Model of Document Use During a Research Project. Study I. Document Selection , 1998, J. Am. Soc. Inf. Sci..

[89]  Michael Eisenberg,et al.  Order effects: A study of the possible influence of presentation order on user judgments of document relevance , 1988, J. Am. Soc. Inf. Sci..

[90]  Stuart A. Sutton The Role of Attorney Mental Models of Law in Case Relevance Determinations: An Exploratory Analysis , 1994, J. Am. Soc. Inf. Sci..

[91]  Cyril W. Cleverdon,et al.  Aslib Cranfield research project - Factors determining the performance of indexing systems; Volume 1, Design; Part 2, Appendices , 1966 .

[92]  Mary Anne Warren The Relevance of Relationships , 2000 .

[93]  Rebecca Green,et al.  Topical Relevance Relationships. I. Why Topic Matching Fails , 1995, J. Am. Soc. Inf. Sci..

[94]  Marcia J. Bates,et al.  The design of browsing and berrypicking techniques for the online search interface , 1989 .

[95]  C. D. Gull Seven years of work on the organization of materials in the special library , 1956 .

[96]  Tefko Saracevic Relevance: A review of the literature and a framework for thinking on the notion in information science. Part III: Behavior and effects of relevance , 2007 .

[97]  Carol L. Barry The Identification of User Criteria of Relevance and Document Characteristics: Beyond the Topical Ap , 1993 .

[98]  George Kingsley Zipf,et al.  Human behavior and the principle of least effort , 1949 .

[99]  Dagobert Soergel,et al.  An evidence perspective on topical relevance types and its implications for exploratory and task-based retrieval , 2006, Inf. Res..