FAiR: A Framework for Analyses and Evaluations on Recommender Systems

Recommender systems (RSs) have become essential tools in e-commerce applications, helping users in the decision-making process. Evaluation on these tools is, however, a major divergence point nowadays, since there is no consensus regarding which metrics are necessary to consolidate new RSs. For this reason, distinct frameworks have been developed to ease the deployment of RSs in research and/or production environments. In the present work, we perform an extensive study of the most popular evaluation metrics, organizing them into three groups: Effectiveness-based, Complementary Dimensions of Quality and Domain Profiling. Further, we consolidate a framework named FAiR to help researchers in evaluating their RSs using these metrics, besides identifying the characteristics of data collections that may intrinsically affect RSs performance. FAiR is compatible with the output format of the main existing RSs libraries (i.e., MyMediaLite and LensKit).

[1]  Saul Vargas,et al.  Rank and relevance in novelty and diversity metrics for recommender systems , 2011, RecSys '11.

[2]  Gerhard Friedrich,et al.  Recommender Systems - An Introduction , 2010 .

[3]  Martin P. Robillard,et al.  Recommendation Systems in Software Engineering , 2014, Springer Berlin Heidelberg.

[4]  Kourosh Modarresi,et al.  Recommendation System Based on Complete Personalization , 2016, ICCS.

[5]  Valentina Franzoni,et al.  Leveraging Zero Tail in Neighbourhood for Link Prediction , 2015, 2015 IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology (WI-IAT).

[6]  Roberto Turrin,et al.  Performance of recommender algorithms on top-n recommendation tasks , 2010, RecSys '10.

[7]  Joseph A. Konstan,et al.  Research resources for recommender systems , 1999 .

[8]  Daniele Quercia,et al.  Auralist: introducing serendipity into music recommendation , 2012, WSDM '12.

[9]  A. Ant Ozok,et al.  Design guidelines for effective recommender system interfaces based on a usability criteria conceptual model: results from a college student population , 2010, Behav. Inf. Technol..

[10]  Hayder Radha,et al.  Cold-Start Item and User Recommendation with Decoupled Completion and Transduction , 2015, RecSys.

[11]  James R. Foulds,et al.  HyPER: A Flexible and Extensible Probabilistic Framework for Hybrid Recommender Systems , 2015, RecSys.

[12]  Rajdeep Niyogi,et al.  Improving Link Ranking Quality by Quasi-Common Neighbourhood , 2015, 2015 15th International Conference on Computational Science and Its Applications.

[13]  Lars Schmidt-Thieme,et al.  MyMediaLite: a free recommender system library , 2011, RecSys '11.

[14]  Bela Gipp,et al.  Research-paper recommender systems: a literature survey , 2015, International Journal on Digital Libraries.

[15]  John Riedl,et al.  LensKit: a modular recommender framework , 2011, RecSys '11.

[16]  J. Bobadilla,et al.  Recommender systems survey , 2013, Knowl. Based Syst..

[17]  Shinya Nogami,et al.  A hybrid recommender system of collaborative and content based filtering , 2016 .

[18]  Ben Weeks.  The Paradox of Choice: Why More Is Less , 2004 .

[19]  Jonathan L. Herlocker,et al.  Evaluating collaborative filtering recommender systems , 2004, TOIS.

[20]  Le Hoang Son.  Dealing with the new user cold-start problem in recommender systems: A comparative review , 2016, Inf. Syst..

[21]  Yves Grandvalet,et al.  A Coverage-Based Approach to Recommendation Diversity On Similarity Graph , 2016, RecSys.

[22]  Sean M. McNee,et al.  Being accurate is not enough: how accuracy metrics have hurt recommender systems , 2006, CHI Extended Abstracts.

[23]  Wei Wang,et al.  Recommender system application developments: A survey , 2015, Decis. Support Syst..