Predicting the Performance of Recommender Systems: An Information Theoretic Approach

Performance prediction is an appealing problem in Recommender Systems, as it enables an array of strategies for deciding when to deliver or hold back recommendations based on their foreseen accuracy. The problem, however, has been barely addressed explicitly in the area. In this paper, we propose adaptations of query clarity techniques from ad-hoc Information Retrieval to define performance predictors in the context of Recommender Systems, which we refer to as user clarity. Our experiments show positive results with different user clarity models in terms of the correlation with single recommender's performance. Empiric results show significant dependency between this correlation and the recommendation method at hand, as well as competitive results in terms of average correlation.

[1]  Alejandro Bellogín,et al.  A Performance Prediction Approach to Enhance Collaborative Filtering Performance , 2010, ECIR.

[2]  Gediminas Adomavicius,et al.  Toward the next generation of recommender systems: a survey of the state-of-the-art and possible extensions , 2005, IEEE Transactions on Knowledge and Data Engineering.

[3]  Claudia Hauff,et al.  Predicting the effectiveness of queries and retrieval systems , 2010, SIGF.

[4]  Iadh Ounis,et al.  University of Glasgow at TREC 2004: Experiments in Web, Robust, and Terabyte Tracks with Terrier , 2004, TREC.

[5]  W. Bruce Croft,et al.  Predicting query performance , 2002, SIGIR '02.

[6]  Lourdes Araujo,et al.  Ranking List Dispersion as a Query Performance Predictor , 2009, ICTIR.

[7]  Iadh Ounis,et al.  Inferring Query Performance Using Pre-retrieval Predictors , 2004, SPIRE.

[8]  W. Bruce Croft,et al.  Query performance prediction in web search environments , 2007, SIGIR.

[9]  Thomas Hofmann,et al.  Latent semantic models for collaborative filtering , 2004, TOIS.

[10]  Josiane Mothe,et al.  Linguistic features to predict query difficulty - a case study on previous TREC campaigns , 2005 .

[11]  Claudio Carpineto,et al.  Query Difficulty, Robustness, and Selective Application of Query Expansion , 2004, ECIR.

[12]  John Riedl,et al.  An algorithmic framework for performing collaborative filtering , 1999, SIGIR '99.

[13]  Wojciech Rytter,et al.  Extracting Powers and Periods in a String from Its Runs Structure , 2010, SPIRE.

[14]  Jun Wang,et al.  Text Retrieval Methods for Item Ranking in Collaborative Filtering , 2011, ECIR.

[15]  Milad Shokouhi,et al.  Advances in Information Retrieval Theory, Second International Conference on the Theory of Information Retrieval, ICTIR 2009, Cambridge, UK, September 10-12, 2009, Proceedings , 2009, ICTIR.

[16]  Jun Wang,et al.  Unified relevance models for rating prediction in collaborative filtering , 2008, TOIS.

[17]  Elad Yom-Tov,et al.  Learning to estimate query difficulty: including applications to missing content detection and distributed information retrieval , 2005, SIGIR '05.

[18]  Djoerd Hiemstra,et al.  A survey of pre-retrieval query performance predictors , 2008, CIKM '08.

[19]  Ophir Frieder,et al.  Repeatable evaluation of search services in dynamic environments , 2007, TOIS.

[20]  Josiane Mothe,et al.  Linguistic features to predict query difficulty , 2005, SIGIR 2005.

[21]  W. Bruce Croft,et al.  Learning to rank query reformulations , 2010, SIGIR '10.

[22]  Peter Ingwersen,et al.  Developing a Test Collection for the Evaluation of Integrated Search , 2010, ECIR.

[23]  W. Bruce Croft,et al.  A framework for selective query expansion , 2004, CIKM '04.

[24]  Chris Buckley,et al.  Topic prediction based on comparative retrieval rankings , 2004, SIGIR '04.

[25]  Oren Kurland,et al.  Predicting Query Performance by Query-Drift Estimation , 2009, ICTIR.