Introduction to the Special Issue on Recommender System Benchmarking

Recommender systems addvalue to vast content resources by matching users with items of interest. In recent years, immense progress has been made in recommendation techniques. The evaluation of these systems is still based on traditional information retrieval and statistics metrics (e.g., precision, recall, RMSE), often not taking the use case and situation of the system into consideration. However, the rapid evolution of recommender systems in both their goals and their application domains fosters the need for new evaluation methodologies and environments. This special issue serves as a venue for work on novel, recommendation-centric benchmarking approaches taking the users’ utility, the business values, and the technical constraints into consideration. Building on the success of the Recommendation Utility Evaluation Workshop [Amatriain et al. 2012] held at Recsys 2012, the Workshop on Benchmarking Adaptive Retrieval and Recommender Systems [Castells et al. 2013a, 2013b] held at SIGIR 2013, the Workshop on Reproducibility and Replication in Recommender System Evaluation [Bellogı́n et al. 2013, 2014], the various Recommender System Challenges [Adomavicius et al. 2010; Said et al. 2011; Manouselis et al. 2012; Blomo et al. 2013; Said et al. 2014; Ben-Shimon et al. 2015], and other similar events, this special issue collects articles focusing on various aspects of challenges in benchmarking and evaluation or recommender systems. In the decade since the inception of the ACM RecSys conference and the decades since the first papers on this topic started to appear [Resnick et al. 1994], the field has