Shedding light on a living lab: the CLEF NEWSREEL open recommendation platform

In the CLEF NEWSREEL lab, participants are invited to evaluate news recommendation techniques in real-time by providing news recommendations to actual users that visit commercial news portals to satisfy their information needs. A central role within this lab is the communication between participants and the users. This is enabled by The Open Recommendation Platform (ORP), a web-based platform which distributes users' impressions of news articles to the participants and returns their recommendations to the readers. In this demo, we illustrate the platform and show how requests are handled to provide relevant news articles in real-time.

[1]  Mark D. Smucker,et al.  Report on the CIKM workshop on living labs for information retrieval evaluation , 2014, SIGF.

[2]  Kenneth Y. Goldberg,et al.  Eigentaste: A Constant Time Collaborative Filtering Algorithm , 2001, Information Retrieval.

[3]  Xavier Amatriain,et al.  Mining large streams of user data for personalized recommendations , 2013, SKDD.

[4]  Krisztian Balog,et al.  Towards a Living Lab for Information Retrieval Research and Development - A Proposal for a Living Lab for Product Search Tasks , 2011, CLEF.

[5]  José Luis Vicedo González,et al.  TREC: Experiment and evaluation in information retrieval , 2007, J. Assoc. Inf. Sci. Technol..

[6]  Yehuda Koren,et al.  All Together Now: A Perspective on the Netflix Prize , 2010 .

[7]  Carol Peters,et al.  Report on the SIGIR 2009 workshop on the future of IR evaluation , 2009, SIGF.

[8]  James Bennett,et al.  The Netflix Prize , 2007 .

[9]  Yehuda Koren,et al.  The Yahoo! Music Dataset and KDD-Cup '11 , 2012, KDD Cup.

[10]  Susan T. Dumais,et al.  Evaluation Challenges and Directions for Information-Seeking Support Systems , 2009, Computer.

[11]  Guy Shani,et al.  A Survey of Accuracy Evaluation Metrics of Recommendation Tasks , 2009, J. Mach. Learn. Res..

[12]  Frank Hopfgartner,et al.  The plista dataset , 2013, NRS '13.

[13]  Frank Hopfgartner,et al.  Users' reading habits in online news portals , 2014, IIiX.

[14]  Sean M. McNee,et al.  Improving recommendation lists through topic diversification , 2005, WWW '05.

[15]  Bart P. Knijnenburg,et al.  Explaining the user experience of recommender systems , 2012, User Modeling and User-Adapted Interaction.

[16]  Frank Hopfgartner,et al.  Benchmarking News Recommendations in a Living Lab , 2014, CLEF.

[17]  Peter Pirolli.  Powers of 10: Modeling Complex Information-Seeking Systems at Multiple Scales , 2009, Computer.

[18]  Cyril W. Cleverdon,et al.  Aslib Cranfield research project - Factors determining the performance of indexing systems; Volume 1, Design; Part 2, Appendices , 1966 .

[19]  Pertti Vakkari,et al.  Evaluation Methodologies in Information Retrieval (Dagstuhl Seminar 13441) , 2013, Dagstuhl Reports.

[20]  Kevin C. Almeroth,et al.  Workshop and challenge on news recommender systems , 2013, RecSys.

[21]  Guy Shani,et al.  Evaluating Recommendation Systems , 2011, Recommender Systems Handbook.

[22]  Stephen E. Robertson,et al.  On the Evaluation of IR Systems , 1992, Inf. Process. Manag..

[23]  Cyril W. Cleverdon,et al.  Factors determining the performance of indexing systems , 1966 .

[24]  Ellen M. Voorhees,et al.  TREC: Experiment and Evaluation in Information Retrieval (Digital Libraries and Electronic Publishing) , 2005 .