Leveraging Multi-Method Evaluation for Multi-Stakeholder Settings

In this paper, we focus on recommendation settings with multiple stakeholders with possibly varying goals and interests, and argue that a single evaluation method or measure is not able to evaluate all relevant aspects in such a complex setting. We reason that employing a multi-method evaluation, where multiple evaluation methods or measures are combined and integrated, allows for getting a richer picture and prevents blind spots in the evaluation outcome.

[1]  Li Chen,et al.  A user-centric evaluation framework for recommender systems , 2011, RecSys '11.

[2]  Derek Bridge,et al.  Diversity, Serendipity, Novelty, and Coverage , 2016, ACM Trans. Interact. Intell. Syst..

[3]  Audrey Laplante,et al.  Improving Music Recommender Systems: What Can We Learn from Research on Music Tastes? , 2014, ISMIR.

[4]  Ron Kohavi,et al.  Online controlled experiments at large scale , 2013, KDD.

[5]  Dietmar Jannach,et al.  Price and Profit Awareness in Recommender Systems , 2017, ArXiv.

[6]  Ilknur Celik,et al.  UMAP 2018 Intelligent User-Adapted Interfaces: Design and Multi-Modal Evaluation (IUadaptMe) Workshop Chairs' Welcome &Organization , 2018, UMAP.

[7]  Bart P. Knijnenburg,et al.  Evaluating Recommender Systems with User Experiments , 2015, Recommender Systems Handbook.

[8]  Martha Larson,et al.  Recommender Systems Evaluation: A 3D Benchmark , 2012, RUE@RecSys.

[9]  Dietmar Jannach,et al.  Measuring the impact of online personalisation: Past, present and future , 2019, Int. J. Hum. Comput. Stud..

[10]  Fabian Abel,et al.  RecSys Challenge 2017: Offline and Online Evaluation , 2017, RecSys.

[11]  L. Giddings.  Research Design: Qualitative, Quantitative, and Mixed Methods Approaches, 2d ed , 2005 .

[12]  Abbas Tashakkori,et al.  Foundations of Mixed Methods Research: Integrating Quantitative and Qualitative Approaches in the Social and Behavioral Sciences , 2008 .

[13]  Himan Abdollahpouri,et al.  Multiple Stakeholders in Music Recommender Systems , 2017, ArXiv.

[14]  Sean M. McNee,et al.  Being accurate is not enough: how accuracy metrics have hurt recommender systems , 2006, CHI Extended Abstracts.

[15]  Miguel P Caldas,et al.  Research design: qualitative, quantitative, and mixed methods approaches , 2003 .

[16]  Pär J. Ågerfalk.  Embracing diversity through mixed methods research , 2013, Eur. J. Inf. Syst..

[17]  Jöran Beel,et al.  A comparative analysis of offline and online evaluations and discussion of research paper recommender system evaluation , 2013, RepSys '13.

[18]  Amos Azaria,et al.  Movie recommender system for profit maximization , 2013, AAAI.

[19]  Guy Shani,et al.  Evaluating Recommender Systems , 2015, Recommender Systems Handbook.

[20]  Jonathan L. Herlocker,et al.  Evaluating collaborative filtering recommender systems , 2004, TOIS.

[21]  Martijn C. Willemsen,et al.  Behaviorism is Not Enough: Better Recommendations through Listening to Users , 2016, RecSys.

[22]  Viswanath Venkatesh,et al.  Bridging the Qualitative-Quantitative Divide: Guidelines for Conducting Mixed Methods Research in Information Systems , 2013, MIS Q..

[23]  Dietmar Jannach,et al.  User Perception of Next-Track Music Recommendations , 2017, UMAP.

[24]  Robin D. Burke,et al.  Multisided Fairness for Recommendation , 2017, ArXiv.

[25]  John Riedl,et al.  Recommender systems: from algorithms to user experience , 2012, User Modeling and User-Adapted Interaction.