Assessment of Measures for Information Retrieval System Evaluation: A Usercentered Approach

ever increasing need for Information globally is a primary reason for the scores of daily usage of IR system. Therefore there is a need to evaluate the system from a more holistic perspective - of both the system and the user. At the moment system-centered measures are not usable for the user-centered approach. Therefore, this paper attempts to determine and also suggest measures as well as methods to meet this need. The factor analytic technique was experimented for this purpose, and the structural equation modeling technique was used to estimate the resultant model. Results show that the study demonstrated high significance. Hence, the statistics presented is capable of inspiring further work in IR systems' evaluation from user's perspective.

[1]  Sanda M. Harabagiu,et al.  High performance question/answering , 2001, SIGIR '01.

[2]  Harris Wu,et al.  Evaluating Web-based Question Answering Systems , 2002, LREC.

[3]  Mark Sanderson,et al.  A review of factors influencing user satisfaction in information retrieval , 2010, J. Assoc. Inf. Sci. Technol..

[4]  Judit Bar-Ilan.  Criteria for Evaluating Information Retrieval Systems in Highly Dynamic Environments , 2002, WebDyn@WWW.

[5]  Zheng Zhou,et al.  Development and validation of an instrument to measure user perceived service quality of information presenting Web portals , 2005, Inf. Manag..

[6]  J. Beneke.  Towards a conceptual model: a path analysis of fundamental relationships affecting mobile advertising effectiveness , 2011 .

[7]  Alexander Dekhtyar,et al.  Information Retrieval , 2018, Lecture Notes in Computer Science.

[8]  Allen Kent,et al.  Machine literature searching VIII. Operational criteria for designing information retrieval systems , 1955 .

[9]  Ellen M. Voorhees,et al.  Overview of TREC 2001 , 2001, TREC.

[10]  B. Muthén,et al.  Exploratory Structural Equation Modeling , 2009 .

[11]  K. Järvelin,et al.  EVALUATING INFORMATION RETRIEVAL SYSTEMS UNDER THE CHALLENGES OF INTERACTION AND MULTIDIMENSIONAL DYNAMIC RELEVANCE , 2002 .

[12]  Leandre R. Fabrigar,et al.  Exploratory Factor Analysis , 2011 .

[13]  B. Dervin,et al.  Information needs and uses. , 1986 .

[14]  T. Perneger.  International journal for quality in health care. , 1998, Nursing management.

[15]  Marco Furini,et al.  International Journal of Computer and Applications , 2010 .

[16]  Jason W. Osborne,et al.  Best practices in exploratory factor analysis: four recommendations for getting the most from your analysis. , 2005 .

[17]  Harris Wu,et al.  Probabilistic question answering on the web , 2002, WWW '02.

[18]  Yiming Yang,et al.  Utility-based information distillation over temporally sequenced documents , 2007, SIGIR.

[19]  Falk Scholer,et al.  User performance versus precision measures for simple search tasks , 2006, SIGIR.

[20]  O. Lee,et al.  User acceptance of second life: An extended TAM including hedonic consumption behaviours , 2009 .

[21]  Thomas Mandl,et al.  Recent Developments in the Evaluation of Information Retrieval Systems: Moving Towards Diversity and Practical Relevance , 2008, Informatica.

[22]  Gilbert A. Churchill.  A Paradigm for Developing Better Measures of Marketing Constructs , 1979 .

[23]  R. MacCallum,et al.  Applications of structural equation modeling in psychological research. , 2000, Annual review of psychology.

[24]  Karen Sparck Jones.  Meta-reflections on TREC , 2006 .

[25]  José Luis Vicedo González,et al.  TREC: Experiment and evaluation in information retrieval , 2007, J. Assoc. Inf. Sci. Technol..

[26]  Gary Marchionini,et al.  Information-Seeking Support Systems [Guest Editors' Introduction] , 2009, Computer.

[27]  Nicholas J. Belkin,et al.  Some(what) grand challenges for information retrieval , 2008, SIGF.

[28]  Mei-Mei Wu,et al.  Reflections on Information Retrieval Evaluation , 1999 .

[29]  Dae-Ho Byun,et al.  Evaluating usability, user satisfaction and intention to revisit for successful e-government websites , 2011, Electron. Gov. an Int. J..

[30]  Phil Wood.  Confirmatory Factor Analysis for Applied Research , 2008 .

[31]  Yun Yang,et al.  User acceptance of second life: An extended TAM including hedonic consumption behaviours , 2009, ECIS.

[32]  Martin Schmettow.  User Interaction Design Patterns for Information Retrieval Systems , 2006, EuroPLoP.

[33]  Tefko Saracevic,et al.  Evaluation of evaluation in information retrieval , 1995, SIGIR '95.

[34]  Dirk Lewandowski,et al.  Web Searching: A Quality Measurement Perspective , 2008 .

[35]  Andrew Turpin,et al.  Why batch and user evaluations do not give the same results , 2001, SIGIR '01.

[36]  Mehdi Khosrow-Pour.  Managing Worldwide Operations and Communications With Information Technology , 2007 .

[37]  Xianggui Qu,et al.  Multivariate Data Analysis , 2007, Technometrics.

[38]  Wen-Lian Hsu,et al.  journal homepage: www.elsevier.com/locate/im , 2022 .

[39]  San Murugesan,et al.  Multi-Dimensional Evaluation of Information Retrieval Results , 2004, IEEE/WIC/ACM International Conference on Web Intelligence (WI'04).