An online framework for supporting the evaluation of personalised information retrieval systems

Scope - Personalised Information Retrieval (PIR) has been gaining attention because it investigates intelligent ways for enhancing content delivery. Web users can have personalised services and more accurate information. Problem - Several PIR systems have been proposed in the literature; however, they have not been properly tested or evaluated. Proposal - The authors propose a generally applicable web-based interface, which provides PIR developers and evaluators with: i) implicit recommendations on how to evaluate a specific PIR system; ii) a repository containing studies on user-centred and layered evaluation studies; iii) recommendations on how to best combine different evaluation methods, metrics and measurement criteria in order to most effectively evaluate their system; iv) a UCE methodology which details how to apply existing UCE techniques; v) a taxonomy of evaluations of adaptive systems; and vi) interface translation support (49 languages supported).

[1]  Cyril W. Cleverdon,et al.  Factors determining the performance of indexing systems , 1966 .

[2]  Yannis Avrithis,et al.  Self-tuning Personalized Information Retrieval in an Ontology-Based Framework , 2005, OTM Workshops.

[3]  Barry Smyth,et al.  Anonymous personalization in collaborative web search , 2006, Information Retrieval.

[4]  Thea van der Geest,et al.  User-centered evaluation of adaptive and adaptable systems: a literature review , 2008, The Knowledge Engineering Review.

[5]  Sofia Stamou,et al.  Search personalization through query and page topical analysis , 2009, User Modeling and User-Adapted Interaction.

[6]  Susan T. Dumais,et al.  Personalizing Search via Automated Analysis of Interests and Activities , 2005, SIGIR.

[7]  Vincent P. Wade,et al.  The evaluation of adaptive and personalised information retrieval systems: a review , 2011, Int. J. Knowl. Web Intell..

[8]  Susan T. Dumais,et al.  Improving Web Search Ranking by Incorporating User Behavior Information , 2019, SIGIR Forum.

[9]  Joachim Meyer,et al.  The Evaluation of In-Vehicle Adaptive Systems , 2005 .

[10]  Masatoshi Yoshikawa,et al.  Adaptive web search based on user profile constructed without any effort from users , 2004, WWW '04.

[11]  Vincent P. Wade,et al.  Adaptive educational hypermedia systems in technology enhanced learning: a literature review , 2010, SIGITE '10.

[12]  Wei Gao,et al.  Cross-lingual query suggestion using query logs of different languages , 2007, SIGIR.

[13]  Séamus Lawless,et al.  A Proposal for the Evaluation of Adaptive Personalised Information Retrieval , 2010 .

[14]  Cristina Gena,et al.  Usability Engineering for the Adaptive Web , 2007, The Adaptive Web.

[15]  Peter Brusilovsky,et al.  Adaptive Hypermedia , 2001, User Modeling and User-Adapted Interaction.

[16]  Leon Sterling,et al.  Applying Agent Technology to Evaluation Tasks in e-Learning Environments , 2003 .

[17]  Stephan Weibelzahl,et al.  Advantages, Opportunities and Limits of Empirical Evaluations: Evaluating Adaptive Systems , 2002, Künstliche Intell..

[18]  Francesco Ricci,et al.  Understanding Recommender Systems : Experimental Evaluation Challenges , 2003 .

[19]  Hinrich Schütze,et al.  Personalized search , 2002, CACM.

[20]  Paul-Alexandru Chirita,et al.  Personalized query expansion for the web , 2007, SIGIR.

[21]  Vincent P. Wade,et al.  A Framework for the Evaluation of Adaptive IR Systems through Implicit Recommendation , 2011, ICCS.

[22]  Alessandro Micarelli,et al.  Anatomy and Empirical Evaluation of an Adaptive Web-Based Information Filtering System , 2004, User Modeling and User-Adapted Interaction.

[23]  Judith Masthoff,et al.  Evaluating recommender explanations: Problems experienced and lessons learned for the evaluation of adaptive systems , 2009 .

[24]  Georgia Koutrika,et al.  Rule-based query personalization in digital libraries , 2004, International Journal on Digital Libraries.

[25]  Fabio Gasparetti,et al.  Personalized Search on the World Wide Web , 2007, The Adaptive Web.

[26]  Cyril W. Cleverdon,et al.  Aslib Cranfield research project - Factors determining the performance of indexing systems; Volume 1, Design; Part 2, Appendices , 1966 .

[27]  Alessandro Micarelli,et al.  User Profiles for Personalized Information Access , 2007, The Adaptive Web.

[28]  Alexander Pretschner,et al.  Ontology based personalized search , 1999, Proceedings 11th International Conference on Tools with Artificial Intelligence.

[29]  Séamus Lawless,et al.  The Evaluation of Adaptive and User-Adaptive Systems: A Review , 2011 .

[30]  Guisseppi A. Forgionne,et al.  A decision-theoretic approach to the evaluation of information retrieval systems , 2006 .

[31]  Peter Brusilovsky,et al.  Methods and techniques of adaptive hypermedia , 1996, User Modeling and User-Adapted Interaction.

[32]  Guisseppi A. Forgionne,et al.  A decision-theoretic approach to the evaluation of information retrieval systems , 2006, Inf. Process. Manag..

[33]  Susan Gauch,et al.  Personalizing Search Based on User Search Histories , 2004 .