Do Metrics Make Recommender Algorithms?

Recommender systems are used to suggest customized products to users. Most recommender algorithms create collaborative models by taking advantage of web user profiles. In the last years, in the area of recommender systems, the Netflix contest has been very attractive for the researchers. However, many recent papers on recommender systems present results evaluated with the methodology used in the Netflix contest in domains where the objectives are different from the contest (e.g., top-N recommendation task). In this paper we do not propose new recommender algorithms but, rather, we compare different aspects of the official Netflix contest methodology based on RMSE and holdout with methodologies based on k-fold and classification accuracy metrics.We show, with case studies, that different evaluation methodologies lead to totally contrasting conclusions about the quality of recommendations.

[1]  Yehuda Koren,et al.  Scalable Collaborative Filtering with Jointly Derived Neighborhood Interpolation Weights , 2007, Seventh IEEE International Conference on Data Mining (ICDM 2007).

[2]  AdomaviciusGediminas,et al.  Toward the Next Generation of Recommender Systems , 2005 .

[3]  D. Sorensen.  Numerical methods for large eigenvalue problems , 2002, Acta Numerica.

[4]  James Bennett,et al.  The Netflix Prize , 2007 .

[5]  Dimitris Plexousakis,et al.  Qualitative analysis of user-based and item-based prediction algorithms for recommendation agents , 2004, Eng. Appl. Artif. Intell..

[6]  John Riedl,et al.  Analysis of recommendation algorithms for e-commerce , 2000, EC '00.

[7]  David G. Stork,et al.  Pattern classification, 2nd Edition , 2000 .

[8]  John Riedl,et al.  Item-based collaborative filtering recommendation algorithms , 2001, WWW '01.

[9]  George Karypis,et al.  Item-based top-N recommendation algorithms , 2004, TOIS.

[10]  David G. Stork,et al.  Pattern Classification , 1973 .

[11]  John Riedl,et al.  An algorithmic framework for performing collaborative filtering , 1999, SIGIR '99.

[12]  Jonathan L. Herlocker,et al.  Evaluating collaborative filtering recommender systems , 2004, TOIS.

[13]  Gediminas Adomavicius,et al.  Toward the next generation of recommender systems: a survey of the state-of-the-art and possible extensions , 2005, IEEE Transactions on Knowledge and Data Engineering.

[14]  M. Matteucci,et al.  An Evaluation Methodology for Collaborative Recommender Systems , 2008, 2008 International Conference on Automated Solutions for Cross Media Content and Multi-Channel Distribution.

[15]  John Riedl,et al.  Application of Dimensionality Reduction in Recommender System - A Case Study , 2000 .