A framework for unexpectedness evaluation in recommendation

Assessment of usefulness in Recommender Systems (RSs) is a major research challenge nowadays. Due to its close relation to the notion of usefulness, unexpectedness has become the focus of several works. However, there is no consensus in the literature about how to measure it. In this context, this work implements the most referenced metrics, consolidating a framework of unexpectedness assessments in recommendation, allowing us to characterize, compare and combine all those metrics. Empirical evaluations on real data and different RSs demonstrate the applicability of our framework. Besides evincing that the existing metrics diverge about which RS provides more unexpected recommendations, the framework allowed us to combine all metrics into a single one able to capture different perspectives. We expect to help researchers and professionals on RSs to understand the actual impact of distinct metrics w.r.t. unexpectedness as well as to select a proper metric to highlight gains or loses.

[1]  Wagner Meira,et al.  Exploiting non-content preference attributes through hybrid recommendation method , 2013, RecSys.

[2]  Ryohei Orihara,et al.  Metrics for Evaluating the Serendipity of Recommendation Lists , 2007, JSAI.

[3]  Mehrbakhsh Nilashi,et al.  Collaborative filtering recommender systems , 2013 .

[4]  Panagiotis Adamopoulos,et al.  On Unexpectedness in Recommender Systems: Or How to Expect the Unexpected , 2011, DiveRS@RecSys.

[5]  M. Kaminskas.  Measuring Surprise in Recommender Systems , 2014 .

[6]  Daniele Quercia,et al.  Auralist: introducing serendipity into music recommendation , 2012, WSDM '12.

[7]  Jonathan L. Herlocker,et al.  Evaluating collaborative filtering recommender systems , 2004, TOIS.

[8]  Bernhard Schölkopf,et al.  Nonlinear Component Analysis as a Kernel Eigenvalue Problem , 1998, Neural Computation.

[9]  John Riedl,et al.  Recommender systems: from algorithms to user experience , 2012, User Modeling and User-Adapted Interaction.

[10]  Takayuki Akiyama,et al.  Proposal and Evaluation of Serendipitous Recommendation Method Using General Unexpectedness , 2010, PRSAT@RecSys.

[11]  Nello Cristianini,et al.  Kernel Methods for Pattern Analysis , 2003, ICTAI.

[12]  Éric Gaussier,et al.  Toward a New Protocol to Evaluate Recommender Systems , 2012, RUE@RecSys.

[13]  Lars Schmidt-Thieme,et al.  MyMediaLite: a free recommender system library , 2011, RecSys '11.

[14]  Mouzhi Ge,et al.  Beyond accuracy: evaluating recommender systems by coverage and serendipity , 2010, RecSys '10.