CLEF 2017 NewsREEL Overview: Offline and Online Evaluation of Stream-based News Recommender Systems

The CLEF NewsREEL challenge allows researchers to evaluate news recommendation algorithms both online (NewsREEL Live) and offline (News- REEL Replay). Compared with the previous year NewsREEL challenged participants with a higher volume of messages and new news portals. In the 2017 edition of the CLEF NewsREEL challenge a wide variety of new approaches have been implemented ranging from the use of existing machine learning frameworks, to ensemble methods to the use of deep neural networks. This paper gives an overview over the implemented approaches and discusses the evaluation results. In addition, the main results of Living Lab and the Replay task are explained.

[1]  Frank Hopfgartner,et al.  CLEF 2017 NewsREEL Overview: A Stream-Based Recommender Task for Evaluation and Education , 2017, CLEF.

[2]  Martha Larson,et al.  Idomaar: A Framework for Multi-dimensional Benchmarking of Recommender Algorithms , 2016, RecSys Posters.

[3]  Andreas Lommatzsch,et al.  A System for Online News Recommendations in Real-Time with Apache Mahout , 2017, CLEF.

[4]  Cornelius A. Ludmann.  Recommending News Articles in the CLEF News Recommendation Evaluation Lab with the Data Stream Management System Odysseus , 2017, CLEF.

[5]  Martha Larson,et al.  CLEF NewsREEL 2016: Comparing Multi-dimensional Offline and Online Evaluation of News Recommender Systems , 2016, CLEF.

[6]  Martha Larson,et al.  Stream-Based Recommendations: Online and Offline Evaluation as a Service , 2015, CLEF.

[7]  Martha Larson,et al.  A Stream-based Resource for Multi-Dimensional Evaluation of Recommender Algorithms , 2017, SIGIR.

[8]  Frank Hopfgartner,et al.  Benchmarking News Recommendations in a Living Lab , 2014, CLEF.

[9]  Martha Larson,et al.  Overview of NewsREEL'16: Multi-dimensional Evaluation of Real-Time Stream-Recommendation Algorithms , 2016, CLEF.

[10]  Frank Hopfgartner,et al.  The plista dataset , 2013, NRS '13.

[11]  Jaroslav Kuchar,et al.  News Recommender System based on Association Rules @ CLEF NewsREEL 2017 , 2017, CLEF.

[12]  Frank Hopfgartner,et al.  Overview of CLEF NewsREEL 2015: News Recommendation Evaluation Lab , 2015, CLEF.

[13]  Frank Hopfgartner,et al.  Shedding light on a living lab: the CLEF NEWSREEL open recommendation platform , 2014, IIiX.

[14]  Vasudeva Varma,et al.  Deep Neural Architecture for News Recommendation , 2017, CLEF.

[15]  Peter Kampstra,et al.  A News Recommender Engine with a Killer Sequence , 2017, CLEF.

[16]  Martha Larson,et al.  CLEF NewsREEL 2017: Contextual Bandit News Recommendation , 2017, CLEF.

[17]  Martha Larson,et al.  Benchmarking News Recommendations: The CLEF NewsREEL Use Case , 2016, SIGF.