A Novel Classification Framework for Evaluating Individual and Aggregate Diversity in Top-N Recommendations

The primary goal of a recommender system is to generate high quality user-centred recommendations. However, the traditional evaluation methods and metrics were developed before researchers understood all the factors that increase user satisfaction. This study is an introduction to a novel user and item classification framework. It is proposed that this framework should be used during user-centred evaluation of recommender systems and the need for this framework is justified through experiments. User profiles are constructed and matched against other users’ profiles to formulate neighbourhoods and generate top-N recommendations. The recommendations are evaluated to measure the success of the process. In conjunction with the framework, a new diversity metric is presented and explained. The accuracy, coverage, and diversity of top-N recommendations is illustrated and discussed for groups of users. It is found that in contradiction to common assumptions, not all users suffer as expected from the data sparsity problem. In fact, the group of users that receive the most accurate recommendations do not belong to the least sparse area of the dataset.

[1]  Gediminas Adomavicius,et al.  Toward the next generation of recommender systems: a survey of the state-of-the-art and possible extensions , 2005, IEEE Transactions on Knowledge and Data Engineering.

[2]  Sean M. McNee,et al.  Being accurate is not enough: how accuracy metrics have hurt recommender systems , 2006, CHI Extended Abstracts.

[3]  Georg Lausen,et al.  Making Product Recommendations More Diverse , 2009, IEEE Data Eng. Bull..

[4]  Sean M. McNee,et al.  Improving recommendation lists through topic diversification , 2005, WWW '05.

[5]  Greg Linden,et al.  Amazon . com Recommendations Item-to-Item Collaborative Filtering , 2001 .

[6]  Saul Vargas,et al.  Improving sales diversity by recommending users to items , 2014, RecSys '14.

[7]  Georg Lausen,et al.  On exploiting classification taxonomies in recommender systems , 2008, AI Commun..

[8]  I K Fodor,et al.  A Survey of Dimension Reduction Techniques , 2002 .

[9]  Sally McClean,et al.  A Study of Evaluation Metrics for Recommender Algorithms , 2008 .

[10]  Barry Smyth,et al.  On the Role of Diversity in Conversational Recommender Systems , 2003, ICCBR.

[11]  George Karypis,et al.  Evaluation of Item-Based Top-N Recommendation Algorithms , 2001, CIKM '01.

[12]  Remco M. Dijkman,et al.  Similarity Search of Business Process Models , 2009, IEEE Data Eng. Bull..

[13]  Sally I. McClean,et al.  Collaborative Filtering: The Aim of Recommender Systems and the Significance of User Ratings , 2010, ECIR.

[14]  Sean M. McNee,et al.  Getting to know you: learning new user preferences in recommender systems , 2002, IUI '02.

[15]  Mukkai S. Krishnamoorthy,et al.  A random walk method for alleviating the sparsity problem in collaborative filtering , 2008, RecSys '08.

[16]  Ian Witten,et al.  Data Mining , 2000 .

[17]  Richard S. Zemel,et al.  Collaborative prediction and ranking with non-random missing data , 2009, RecSys '09.

[18]  Kenneth Y. Goldberg,et al.  Eigentaste: A Constant Time Collaborative Filtering Algorithm , 2001, Information Retrieval.

[19]  Sophie Ahrens,et al.  Recommender Systems , 2012 .

[20]  Sally I. McClean,et al.  User-based Collaborative Filtering: Sparsity and Performance , 2010, STAIRS.

[21]  David McSherry,et al.  Diversity-Conscious Retrieval , 2002, ECCBR.

[22]  John Riedl,et al.  GroupLens: an open architecture for collaborative filtering of netnews , 1994, CSCW '94.

[23]  John Riedl,et al.  Applying Knowledge from KDD to Recommender Systems , 1999 .

[24]  Kirsten Swearingen,et al.  Interaction Design for Recommender Systems , 2002 .

[25]  Pattie Maes,et al.  Social information filtering: algorithms for automating “word of mouth” , 1995, CHI '95.

[26]  Joseph A. Konstan,et al.  Research resources for recommender systems , 1999 .

[27]  Pablo Castells,et al.  A study of heterogeneity in recommendations for a social music service , 2010, HetRec '10.

[28]  Taghi M. Khoshgoftaar,et al.  A Survey of Collaborative Filtering Techniques , 2009, Adv. Artif. Intell..

[29]  Philip S. Yu,et al.  Horting hatches an egg: a new graph-theoretic approach to collaborative filtering , 1999, KDD '99.

[30]  อนิรุธ สืบสิงห์,et al.  Data Mining Practical Machine Learning Tools and Techniques , 2014 .

[31]  John Riedl,et al.  Analysis of recommendation algorithms for e-commerce , 2000, EC '00.

[32]  Hai Yang,et al.  ACM Transactions on Intelligent Systems and Technology - Special Section on Urban Computing , 2014 .

[33]  Saul Vargas,et al.  Rank and relevance in novelty and diversity metrics for recommender systems , 2011, RecSys '11.

[34]  Tiejian Luo,et al.  A novel approach to solve the sparsity problem in collaborative filtering , 2010, 2010 International Conference on Networking, Sensing and Control (ICNSC).

[35]  Jonathan L. Herlocker,et al.  Evaluating collaborative filtering recommender systems , 2004, TOIS.

[36]  Gediminas Adomavicius,et al.  Improving Aggregate Recommendation Diversity Using Ranking-Based Techniques , 2012, IEEE Transactions on Knowledge and Data Engineering.

[37]  David Heckerman,et al.  Empirical Analysis of Predictive Algorithms for Collaborative Filtering , 1998, UAI.

[38]  John Riedl,et al.  Item-based collaborative filtering recommendation algorithms , 2001, WWW '01.

[39]  Yehuda Koren,et al.  Matrix Factorization Techniques for Recommender Systems , 2009, Computer.

[40]  Robin D. Burke,et al.  Hybrid Recommender Systems: Survey and Experiments , 2002, User Modeling and User-Adapted Interaction.

[41]  Félix Hernández-del-Olmo,et al.  Evaluation of recommender systems: A new approach , 2008, Expert Syst. Appl..

[42]  Guy Shani,et al.  Evaluating Recommendation Systems , 2011, Recommender Systems Handbook.

[43]  Barry Smyth,et al.  Similarity vs. Diversity , 2001, ICCBR.

[44]  Adam Prügel-Bennett,et al.  Leveraging clustering approaches to solve the gray-sheep users problem in recommender systems , 2014, Expert Syst. Appl..

[45]  Hsinchun Chen,et al.  A Comparison of Collaborative-Filtering Recommendation Algorithms for E-commerce , 2007, IEEE Intelligent Systems.

[46]  A. Chris,et al.  Long Tail: How Endless Choice is Creating Unlimited Demand, London: Random House. , 1996 .

[47]  Jonathan L. Herlocker,et al.  A collaborative filtering algorithm and evaluation metric that accurately model the user experience , 2004, SIGIR '04.

[48]  Li Chen,et al.  A user-centric evaluation framework for recommender systems , 2011, RecSys '11.

[49]  Panagiotis Symeonidis,et al.  Collaborative Filtering Process in a Whole New Light , 2006, 2006 10th International Database Engineering and Applications Symposium (IDEAS'06).

[50]  Douglas B. Terry,et al.  Using collaborative filtering to weave an information tapestry , 1992, CACM.

[51]  John Riedl,et al.  Explaining collaborative filtering recommendations , 2000, CSCW '00.

[52]  Dean P. Foster,et al.  Clustering Methods for Collaborative Filtering , 1998, AAAI 1998.

[53]  Michael R. Lyu,et al.  Effective missing data prediction for collaborative filtering , 2007, SIGIR.

[54]  Alejandro Bellogín,et al.  Precision-oriented evaluation of recommender systems: an algorithmic comparison , 2011, RecSys '11.

[55]  John Riedl,et al.  Application of Dimensionality Reduction in Recommender System - A Case Study , 2000 .

[56]  Bamshad Mobasher,et al.  Intelligent Techniques for Web Personalization , 2005, Lecture Notes in Computer Science.

[57]  P. W. Foltz,et al.  Using latent semantic indexing for information filtering , 1990, COCS '90.

[58]  Gerhard Friedrich,et al.  Recommender Systems - An Introduction , 2010 .

[59]  M. Matteucci,et al.  An Evaluation Methodology for Collaborative Recommender Systems , 2008, 2008 International Conference on Automated Solutions for Cross Media Content and Multi-Channel Distribution.

[60]  Raymond J. Mooney,et al.  Content-boosted collaborative filtering for improved recommendations , 2002, AAAI/IAAI.