Are All Rejected Recommendations Equally Bad?: Towards Analysing Rejected Recommendations

When evaluating algorithms that recommend a list of relevant items to a user, it is common to use metrics such as precision to measure the system accuracy. When computing precision, one computes the number of items that were selected by the user among the recommended items. As such, recommended items that were not selected by the user, which we call \em rejected recommendations, are all considered to be bad recommendations, resulting in no increase to the system accuracy metric. Our ultimate goal is to develop a new recommendation accuracy evaluation metric, which may assign some value to the rejected recommendations. In this paper, as a first step, we claim that some rejected recommendations are better than others. Specifically, we consider items that are similar to the item that was finally selected, as better recommendations than items that bear little similarity. We conduct a user study, showing that rejected recommendations that have high content or collaborative similarity to the selected item are perceived by users as better recommendations than items with low similarity. In addition, we study the correlations between the recommended items shown to a user and the un-recommended items that the user has selected in a real-life job posting dataset. We show that when considering item similarity rather than simple precision, the correlations are much higher. This may be attributed to the influence of the recommended items on the decisions of the user.

[1]  Sean M. McNee,et al.  Being accurate is not enough: how accuracy metrics have hurt recommender systems , 2006, CHI Extended Abstracts.

[2]  B. Mobasher,et al.  USING SEMANTIC SIMILARITY TO ENHANCE ITEM-BASED COLLABORATIVE FILTERING , 2003 .

[3]  Makbule Gulcin Ozsoy,et al.  From Word Embeddings to Item Recommendation , 2016, ArXiv.

[4]  George Karypis,et al.  Item-based top-N recommendation algorithms , 2004, TOIS.

[5]  Guy Shani,et al.  Evaluating Recommendation Systems , 2011, Recommender Systems Handbook.

[6]  Loren G. Terveen,et al.  Exploring the filter bubble: the effect of using recommender systems on content diversity , 2014, WWW.

[7]  Mouzhi Ge,et al.  Beyond accuracy: evaluating recommender systems by coverage and serendipity , 2010, RecSys '10.

[8]  David Heckerman,et al.  Empirical Analysis of Predictive Algorithms for Collaborative Filtering , 1998, UAI.

[9]  Paulo J. G. Lisboa,et al.  The value of personalised recommender systems to e-business: a case study , 2008, RecSys '08.

[10]  Ram D. Gopal,et al.  Empirical Analysis of the Impact of Recommender Systems on Sales , 2010, J. Manag. Inf. Syst..

[11]  Ellen M. Voorhees,et al.  Retrieval evaluation with incomplete information , 2004, SIGIR '04.

[12]  Ulrich Paquet,et al.  Beyond Collaborative Filtering: The List Recommendation Problem , 2016, WWW.

[13]  Patrick Gallinari,et al.  Ranking with non-random missing ratings: influence of popularity and positivity on evaluation metrics , 2012, RecSys.

[14]  Tetsuya Sakai,et al.  Alternatives to Bpref , 2007, SIGIR.

[15]  Ido Guy,et al.  Personalized recommendation of social software items based on social relations , 2009, RecSys '09.

[16]  Jonathan L. Herlocker,et al.  Evaluating collaborative filtering recommender systems , 2004, TOIS.

[17]  Dokyun Lee,et al.  Impact of Recommender Systems on Sales Volume and Diversity , 2014, ICIS.

[18]  Derek Bridge,et al.  Diversity, Serendipity, Novelty, and Coverage , 2016, ACM Trans. Interact. Intell. Syst..

[19]  Judith Masthoff,et al.  Designing and Evaluating Explanations for Recommender Systems , 2011, Recommender Systems Handbook.

[20]  Pasquale Lops,et al.  Content-based Recommender Systems: State of the Art and Trends , 2011, Recommender Systems Handbook.

[21]  Thorsten Joachims,et al.  Recommendations as Treatments: Debiasing Learning and Evaluation , 2016, ICML.

[22]  Yehuda Koren,et al.  Towards scalable and accurate item-oriented recommendations , 2013, RecSys.

[23]  Roberto Turrin,et al.  Performance of recommender algorithms on top-n recommendation tasks , 2010, RecSys '10.

[24]  Michael J. Pazzani,et al.  Content-Based Recommendation Systems , 2007, The Adaptive Web.

[25]  John Hannon,et al.  Recommending twitter users to follow using content and collaborative filtering approaches , 2010, RecSys '10.

[26]  Panagiotis Symeonidis,et al.  Providing Justifications in Recommender Systems , 2008, IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans.

[27]  Harald Steck,et al.  Training and testing of recommender systems on data missing not at random , 2010, KDD.

[28]  Pasquale Lops,et al.  Semantics-aware Content-based Recommender Systems , 2014, CBRecSys@RecSys.

[29]  Mao Ye,et al.  Exploiting geographical influence for collaborative point-of-interest recommendation , 2011, SIGIR.

[30]  Yiyu Yao,et al.  Measuring Retrieval Effectiveness Based on User Preference of Documents , 1995, J. Am. Soc. Inf. Sci..

[31]  Edward Y. Chang,et al.  Combinational collaborative filtering for personalized community recommendation , 2008, KDD.

[32]  Raymond J. Mooney,et al.  Content-boosted collaborative filtering for improved recommendations , 2002, AAAI/IAAI.

[33]  Amihood Amir,et al.  Data Quality Matters in Recommender Systems , 2015, RecSys.

[34]  Duncan J. Watts,et al.  Estimating the Causal Impact of Recommendation Systems from Observational Data , 2015, EC.

[35]  Lior Rokach,et al.  Recommender Systems: Introduction and Challenges , 2015, Recommender Systems Handbook.

[36]  Liang He,et al.  Evaluating recommender systems , 2012, Seventh International Conference on Digital Information Management (ICDIM 2012).

[37]  George Karypis,et al.  Evaluation of Item-Based Top-N Recommendation Algorithms , 2001, CIKM '01.

[38]  Jason J. Jung,et al.  Item-Based Collaborative Filtering with Attribute Correlation: A Case Study on Movie Recommendation , 2014, ACIIDS.

[39]  Fabian Abel,et al.  RecSys Challenge 2017: Offline and Online Evaluation , 2017, RecSys.

[40]  Jonathan L. Herlocker,et al.  A collaborative filtering algorithm and evaluation metric that accurately model the user experience , 2004, SIGIR '04.

[41]  John Riedl,et al.  Item-based collaborative filtering recommendation algorithms , 2001, WWW '01.