User-Oriented Relevance Judgment: A Conceptual Model

The concept of relevance has been heatedly debated in last decade. Not satisfied with the narrow and technical definition of system relevance, researchers turn to the subjective and situational aspect of this concept. How does a user perceive a document as relevant? The literature on relevance has identified numerous factors affecting such judgment. Taking a cognitive approach, this study focuses on the criteria users employ in making relevance judgment. Based on Grice's theory of communication, this paper proposes a five-factor model of relevance: topicality, novelty, reliability, understandability, and scope. Data are collected from a semi-controlled survey study and analyzed following a psychometric procedure. The result supports topicality and novelty as the key relevance criteria. Theoretical and practical implications of this study are discussed.

[1]  Robert D. Brown,et al.  Jargon and Data Do Make a Difference: , 1981 .

[2]  Izak Benbasat,et al.  Development of an Instrument to Measure the Perceptions of Adopting an Information Technology Innovation , 1991, Inf. Syst. Res..

[3]  Peiling Wang,et al.  A Cognitive Model of Document Use During a Research Project. Study I. Document Selection , 1998, J. Am. Soc. Inf. Sci..

[4]  Peiling Wang,et al.  A Cognitive Model of Document Use During a Research Project. Study II. Decisions at the Reading and Citing Stages , 1999, J. Am. Soc. Inf. Sci..

[5]  Chad Galloway,et al.  Relevance judging, evaluation, and decision making in virtual libraries: A descriptive study , 2001, J. Assoc. Inf. Sci. Technol..

[6]  D. Over,et al.  Studies in the Way of Words. , 1989 .

[7]  Michael B. Eisenberg,et al.  A re-examination of relevance: toward a dynamic, situational definition , 1990, Inf. Process. Manag..

[8]  Siobhan Chapman.  Logic and Conversation , 2005 .

[9]  Paul Solomon,et al.  Toward an Understanding of the Dynamics of Relevance Judgment: An Analysis of One Person's Search Behavior , 1998, Inf. Process. Manag..

[10]  C. Fornell,et al.  Evaluating structural equation models with unobservable variables and measurement error. , 1981 .

[11]  Thomas J. Froehlich,et al.  Relevance reconsidered—towards an agenda for the 21st century: introduction to special topic issue on relevance research , 1994 .

[12]  Rebecca Green,et al.  Topical Relevance Relationships. I. Why Topic Matching Fails , 1995, J. Am. Soc. Inf. Sci..

[13]  D. Sperber,et al.  Relevance: Communication and Cognition , 1989 .

[14]  Rolph E. Anderson,et al.  Multivariate Data Analysis with Readings , 1979 .

[15]  J. Atlas,et al.  It-clefts, informativeness and logical form: Radical pragmatics (revised standard version) , 1981 .

[16]  Bert R. Boyce,et al.  Beyond topicality : A two stage view of relevance and the retrieval process , 1982, Inf. Process. Manag..

[17]  William R. Hersh,et al.  Relevance and Retrieval Evaluation: Perspectives from Medicine , 1994, J. Am. Soc. Inf. Sci..

[18]  Judy Bateman.  Changes in Relevance Criteria: A Longitudinal Study. , 1998 .

[19]  Anany Levitin,et al.  Quality dimensions of a conceptual view , 1995 .

[20]  Parke Godfrey,et al.  An overview of cooperative answering , 1992, Journal of Intelligent Information Systems.

[21]  K. Bach,et al.  Linguistic Communication and Speech Acts , 1983 .

[22]  Linda Schamber,et al.  Users' Criteria for Evaluation in a Multimedia Environment. , 1991 .

[23]  M. E. Maron,et al.  On indexing, retrieval and the meaning of about , 1977, J. Am. Soc. Inf. Sci..

[24]  Diane M. Strong,et al.  Beyond Accuracy: What Data Quality Means to Data Consumers , 1996, J. Manag. Inf. Syst..

[25]  Nicholas J. Belkin,et al.  Characteristics of Texts Affecting Relevance Judgments , 1993 .

[26]  E. Hirschman.  Innovativeness, Novelty Seeking, and Consumer Creativity , 1980 .

[27]  F. W. Lancaster,et al.  Information retrieval systems; characteristics, testing, and evaluation , 1968 .

[28]  J A Swets,et al.  Information Retrieval Systems. , 1963, Science.

[29]  Pia Borlund,et al.  The concept of relevance in IR , 2003, J. Assoc. Inf. Sci. Technol..

[30]  Robert van Rooy.  Utility, Informativity and Protocols , 2004, J. Philos. Log..

[31]  M. Lynn.  Hawaii International Conference on System Sciences , 1996 .

[32]  S. Hirsh.  Children's Relevance Criteria and Information Seeking on Electronic Resources , 1999, J. Am. Soc. Inf. Sci..

[33]  Paul McReynolds,et al.  The “Need for Novelty”: A Comparison of Six Instruments , 1967 .

[34]  Amanda Spink,et al.  From Highly Relevant to Not Relevant: Examining Different Regions of Relevance , 1998, Inf. Process. Manag..

[35]  J. Greenstone.  Relevance , 2007 .

[36]  Douglas G. Schultz,et al.  A Field Experimental Approach to the Study of Relevance Assessments in Relation to Document Searching. Final Report to the National Science Foundation. Volume II, Appendices. , 1967 .

[37]  H. Park,et al.  Relevance of Science Information: Origins and Dimensions of Relevance and Their Implications to Information Retrieval , 1997, Inf. Process. Manag..

[38]  Louise T. Su.  Is Relevance an Adequate Criterion for Retrieval System Evaluation: An Empirical Inquiry into the User's Evaluation. , 1993 .

[39]  Patrick Wilson,et al.  Situational relevance , 1973, Inf. Storage Retr..

[40]  Stefano Mizzaro.  Relevance: the whole history , 1997 .

[41]  Duane T. Wegener,et al.  Cognitive processes in attitude change , 1994 .

[42]  Pertti Vakkari,et al.  Changes in relevance criteria and problem stages in task performance , 2000, J. Documentation.

[43]  C. A. Cuadra,et al.  OPENING THE BLACK BOX OF ‘RELEVANCE’ , 1967 .

[44]  James C. Anderson,et al.  STRUCTURAL EQUATION MODELING IN PRACTICE: A REVIEW AND RECOMMENDED TWO-STEP APPROACH , 1988 .

[45]  Linda Schamber.  Relevance and Information Behavior. , 1994 .

[46]  John T. Cacioppo,et al.  The Elaboration Likelihood Model of Persuasion , 1986, Advances in Experimental Social Psychology.

[47]  Niels Ole Bernsen,et al.  A methodology for diagnostic evaluation of spoken human -- machine dialogue , 1998, Int. J. Hum. Comput. Stud..

[48]  A. H. Adelberg.  METHODOLOGY FOR MEASURING THE UNDERSTANDABILITY OF FINANCIAL REPORT MESSAGES , 1979 .

[49]  Carol L. Barry.  User-defined relevance criteria: an exploratory study , 1994 .

[50]  Carol Stoak Saunders,et al.  The Social Construction of Meaning: An Alternative Perspective on Information Sharing , 2003, Inf. Syst. Res..

[51]  Tefko Saracevic,et al.  RELEVANCE: A review of and a framework for the thinking on the notion in information science , 1997, J. Am. Soc. Inf. Sci..

[52]  Diane H. Sonnenwald,et al.  User perspectives on relevance criteria: A comparison among relevant, partially relevant, and not-relevant judgments , 2002, J. Assoc. Inf. Sci. Technol..

[53]  Birger Hjørland,et al.  Work tasks and socio-cognitive relevance: A specific example , 2002, J. Assoc. Inf. Sci. Technol..

[54]  Howard Greisdorf,et al.  Relevance thresholds: a multi-stage predictive model of how users evaluate information , 2003, Inf. Process. Manag..

[55]  Richard Saul Wurman,et al.  Information Anxiety 2 , 1989 .

[56]  A. Avramides.  Studies in the Way of Words , 1992 .

[57]  Susan Dunman,et al.  Seeking meaning: A process approach to library and information services , 1996 .

[58]  Peiling Wang,et al.  A Cognitive Model of Document Use During a Research Project. Study II. Decisions at the Reading and Citing Stages , 1999, Journal of the American Society for Information Science.

[59]  Carol L. Barry,et al.  Users' Criteria for Relevance Evaluation: A Cross-situational Comparison , 1998, Inf. Process. Manag..

[60]  Greg Elofson,et al.  How the Packaging of Decision Explanations Affects Perceptions of Trustworthiness , 2000 .

[61]  Birger Hjørland,et al.  Information Seeking and Subject Representation: An Activity-Theoretical Approach to Information Science , 1997 .

[62]  William R. Hersh.  Relevance and retrieval evaluation: perspectives from medicine , 1994 .

[63]  S. P. Harter.  Psychological relevance and information science , 1992 .

[64]  Wynne W. Chin.  The partial least squares approach for structural equation modeling. , 1998 .

[65]  GreenRebecca.  Topical relevance relationships. I , 1995 .

[66]  Edie M. Rasmussen,et al.  Users' relevance criteria in image retrieval in American history , 2002, Inf. Process. Manag..