A decision-theoretic approach to the evaluation of information retrieval systems

Evaluation research on information retrieval (IR) systems has thus far been narrowly focused and disjointed. This research attempts to narrow the gap by providing a comprehensive and integrated multiple criteria decision-theoretic approach for the evaluation of IR systems. The approach, which is based on the Analytic Hierarchy Process (AHP), is illustrated in the context of a domain-specific IR system. The novelty of this approach lies in the focus on the user aspect and the application of decision-making theories in the IR field.

[1]  Leonard Adelman,et al.  Experiments, quasi-experiments, and case studies: A review of empirical methods for evaluating decision support systems , 1991, IEEE Trans. Syst. Man Cybern..

[2]  Narasimha Bolloju,et al.  THE EFFECT OF IMPRECISION IN SPECIFICATION OF PAIR-WISE COMPARISONS ON RANKING OF ALTERNATIVES USING FUZZY ANALYTIC HIERARCHY PROCESS , 2001 .

[3]  Carol Collier Kuhlthau,et al.  Seeking Meaning: a process approach to library and information services" Ablex Publishing , 2003 .

[4]  Peter Ingwersen,et al.  Cognitive Perspectives of Information Retrieval Interaction: Elements of a Cognitive IR Theory , 1996, J. Documentation.

[5]  Peter Ingwersen,et al.  Information seeking research needs extension toward tasks and technology , 2004, Inf. Res..

[6]  Stephen E. Robertson,et al.  On the Evaluation of IR Systems , 1992, Inf. Process. Manag..

[7]  R. Lipshitz,et al.  How Problems are Solved: Reconsidering the Phase Theorem , 1996 .

[8]  David Ellis,et al.  The Dilemma of Measurement in Information Retrieval Research , 1996, J. Am. Soc. Inf. Sci..

[9]  Gloria E. Phillips-Wren,et al.  E-delivery of personalised healthcare information to intermediaries for suicide prevention , 2005, Int. J. Electron. Heal..

[10]  Donna K. Harman,et al.  Overview of the Second Text REtrieval Conference (TREC-2) , 1994, HLT.

[11]  Mark P. Sharfman,et al.  Does Decision Process Matter? A Study Of Strategic Decision-making Effectiveness , 1996 .

[12]  Guisseppi A. Forgionne,et al.  A multiple-criteria framework for evaluation of decision support systems , 2004 .

[13]  M. E. Maron,et al.  An evaluation of retrieval effectiveness for a full-text document-retrieval system , 1985, CACM.

[14]  Donna Harman,et al.  The Second Text Retrieval Conference (TREC-2) , 1995, Inf. Process. Manag..

[15]  Pia Borlund,et al.  The concept of relevance in IR , 2003, J. Assoc. Inf. Sci. Technol..

[16]  Pia Borlund,et al.  Experimental components for the evaluation of interactive information retrieval systems , 2000, J. Documentation.

[17]  William S. Cooper,et al.  On selecting a measure of retrieval effectiveness , 1973, J. Am. Soc. Inf. Sci..

[18]  Stephen P. Harter,et al.  Evaluation of information retrieval systems : Approaches, issues, and methods , 1997 .

[19]  Guisseppi A. Forgionne,et al.  Decision-Making Support System Effectiveness: the Process to Outcome Link , 2000, Inf. Knowl. Syst. Manag..

[20]  Carol Collier Kuhlthau.  Inside the Search Process: Information Seeking from the User's Perspective. , 1991 .

[21]  Linda Schamber.  Relevance and Information Behavior. , 1994 .

[22]  Dagobert Soergel,et al.  Is user satisfaction a hobgoblin? , 1976, J. Am. Soc. Inf. Sci..

[23]  R. W. Saaty,et al.  The analytic hierarchy process—what it is and how it is used , 1987 .

[24]  Patrick T. Harker,et al.  The Art and Science of Decision Making: The Analytic Hierarchy Process , 1989 .

[25]  Gerard Salton,et al.  Automatic Text Processing: The Transformation, Analysis, and Retrieval of Information by Computer , 1989 .

[26]  Regine Freitag,et al.  Making Use of Scenarios for Validating Analysis and Design , 1998, IEEE Trans. Software Eng..

[27]  G. A. Forgionne.  An AHP model of DSS effectiveness , 1999 .

[28]  H. Raghav Rao,et al.  E-health and assurance: curing hospital websites , 2004, Int. J. Electron. Heal..

[29]  Tefko Saracevic,et al.  Evaluation of evaluation in information retrieval , 1995, SIGIR '95.

[30]  William S. Cooper,et al.  On selecting a measure of retrieval effectiveness part II. Implementation of the philosophy , 1973, J. Am. Soc. Inf. Sci..

[31]  Gerard Salton,et al.  The State of Retrieval System Evaluation , 1992, Inf. Process. Manag..

[32]  Rong Tang,et al.  Towards the Identification of the Optimal Number of Relevance Categories , 1999, J. Am. Soc. Inf. Sci..

[33]  George Demiris,et al.  Electronic home healthcare: concepts and challenges , 2004, Int. J. Electron. Heal..

[34]  M. E. Maron,et al.  Full-text information retrieval: Further analysis and clarification , 1990, Inf. Process. Manag..