Evaluating Strategies for Selecting Test Datasets in Recommender Systems

Recommender systems based on collaborative filtering are widely used to predict users’ behaviour in large databases, where users rate items. The prediction model is built from a training dataset according to matrix factorization method and validated using a test dataset in order to measure the prediction error. Random selection is the most simple and instinctive way to build test datasets. Nevertheless, we could think about other deterministic methods to select test ratings uniformly along the database, in order to obtain a balanced contribution from all the users and items. In this paper, we perform several experiments of validating recommender systems using random and deterministic strategies to select test datasets. We considered a zigzag deterministic strategy that selects ratings uniformly across the rows and columns of the ratings matrix, following a diagonal path. After analysing the statistical results, we conclude that there are no particular advantages in considering the deterministic strategy.

[1]  Fernando Ortega,et al.  Recommending items to group of users using Matrix Factorization based Collaborative Filtering , 2016, Inf. Sci..

[2]  Fernando Ortega,et al.  A non negative matrix factorization for collaborative filtering recommender systems based on a Bayesian probabilistic model , 2016, Knowl. Based Syst..

[3]  Gediminas Adomavicius,et al.  Toward the next generation of recommender systems: a survey of the state-of-the-art and possible extensions , 2005, IEEE Transactions on Knowledge and Data Engineering.

[4]  Gerhard Friedrich,et al.  Recommender Systems - An Introduction , 2010 .

[5]  F. Maxwell Harper,et al.  The MovieLens Datasets: History and Context , 2016, TIIS.

[6]  Yehuda Koren,et al.  Matrix Factorization Techniques for Recommender Systems , 2009, Computer.

[7]  Steffen Bickel,et al.  Discriminative learning for differing training and test distributions , 2007, ICML '07.

[8]  Francesco Ricci,et al.  Context-Aware Recommender Systems , 2011, AI Mag..

[9]  Ruslan Salakhutdinov,et al.  Probabilistic Matrix Factorization , 2007, NIPS.

[10]  Hyung Jun Ahn,et al.  A new similarity measure for collaborative filtering to alleviate the new user cold-starting problem , 2008, Inf. Sci..

[11]  Jesús Bobadilla,et al.  A new collaborative filtering metric that improves the behavior of recommender systems , 2010, Knowl. Based Syst..

[12]  P. Paatero,et al.  Positive matrix factorization: A non-negative factor model with optimal utilization of error estimates of data values† , 1994 .

[13]  Amos Storkey,et al.  When Training and Test Sets are Different: Characterising Learning Transfer , 2013 .

[14]  Jonathan L. Herlocker,et al.  Evaluating collaborative filtering recommender systems , 2004, TOIS.

[15]  Lior Rokach,et al.  Introduction to Recommender Systems Handbook , 2011, Recommender Systems Handbook.

[16]  Yehuda Koren,et al.  Scalable Collaborative Filtering with Jointly Derived Neighborhood Interpolation Weights , 2007, Seventh IEEE International Conference on Data Mining (ICDM 2007).

[17]  Léon Bottou,et al.  Large-Scale Machine Learning with Stochastic Gradient Descent , 2010, COMPSTAT.

[18]  Lars Schmidt-Thieme,et al.  Online-updating regularized kernel matrix factorization models for large-scale recommender systems , 2008, RecSys '08.