Understanding and Evaluating User Satisfaction with Music Discovery

We study the use and evaluation of a system for supporting music discovery, the experience of finding and listening to content previously unknown to the user. We adopt a mixed methods approach, including interviews, unsupervised learning, survey research, and statistical modeling, to understand and evaluate user satisfaction in the context of discovery. User interviews and survey data show that users' behaviors change according to their goals, such as listening to recommended tracks in the moment, or using recommendations as a starting point for exploration. We use these findings to develop a statistical model of user satisfaction at scale from interactions with a music streaming platform. We show that capturing users' goals, their deviations from their usual behavior, and their peak interactions on individual tracks are informative for estimating user satisfaction. Finally, we present and validate heuristic metrics that are grounded in user experience for online evaluation of recommendation performance. Our findings, supported with evidence from both qualitative and quantitative studies, reveal new insights about user expectations with discovery and their behavioral responses to satisfying and dissatisfying systems.

[1]  Tianqi Chen,et al.  XGBoost: A Scalable Tree Boosting System , 2016, KDD.

[2]  Jin Ha Lee,et al.  Understanding User Requirements for Music Information Services , 2012, ISMIR.

[3]  D. Kahneman,et al.  Back to Bentham? Explorations of experience utility , 1997 .

[4]  Audrey Laplante Users' Relevance Criteria in Music Retrieval in Everyday Life: An Exploratory Study , 2010, ISMIR.

[5]  Mouzhi Ge,et al.  Beyond accuracy: evaluating recommender systems by coverage and serendipity , 2010, RecSys '10.

[6]  Wei Chu,et al.  Enhancing personalized search by mining and modeling task behavior , 2013, WWW.

[7]  Jin Ha Lee,et al.  User experience with commercial music services: An empirical exploration , 2016, J. Assoc. Inf. Sci. Technol..

[8]  Aaron Sedley,et al.  HaTS: large-scale in-product measurement of user attitudes & experiences with happiness tracking surveys , 2014, OZCHI.

[9]  Òscar Celma,et al.  A new approach to evaluating novel recommendations , 2008, RecSys '08.

[10]  Matti Mäntymäki,et al.  Gratifications from using freemium music streaming services: Differences between basic and premium users , 2015, ICIS.

[11]  Saul Vargas,et al.  Rank and relevance in novelty and diversity metrics for recommender systems , 2011, RecSys '11.

[12]  Gleb Gusev,et al.  Future User Engagement Prediction and Its Application to Improve the Sensitivity of Online Experiments , 2015, WWW.

[13]  J. S. Downie,et al.  The utilitarian and hedonic outcomes of music information-seeking in everyday life , 2011 .

[14]  Tuck Wah Leong,et al.  Revisiting social practices surrounding music , 2013, CHI.

[15]  Neil J. Hurley,et al.  Novelty and Diversity in Top-N Recommendation -- Analysis and Evaluation , 2011, TOIT.

[16]  Steve Fox,et al.  Evaluating implicit measures to improve web search , 2005, TOIS.

[17]  Eugene Agichtein,et al.  Understanding Music Listening Intents During Daily Activities with Implications for Contextual Music Recommendation , 2018, CHIIR.

[18]  Yuchen Zhang,et al.  User-click modeling for understanding and predicting search-behavior , 2011, KDD.

[19]  Thomas Hess,et al.  Escaping from the Filter Bubble? The Effects of Novelty and Serendipity on Users' Evaluations of Online Recommendations , 2014, ICIS.

[20]  Markus Schedl,et al.  The neglected user in music information retrieval research , 2013, Journal of Intelligent Information Systems.

[21]  Jonathan L. Herlocker,et al.  Evaluating collaborative filtering recommender systems , 2004, TOIS.

[23]  Jin Ha Lee,et al.  Understanding Users of Commercial Music Services through Personas: Design Implications , 2015, ISMIR.

[24]  Alex Deng,et al.  Data-Driven Metric Development for Online Controlled Experiments: Seven Lessons Learned , 2016, KDD.

[25]  F. Maxwell Harper,et al.  User perception of differences in recommender algorithms , 2014, RecSys '14.

[26]  Hyerim Cho,et al.  Users' music information needs and behaviors: Design implications for music information retrieval systems , 2016, J. Assoc. Inf. Sci. Technol..

[27]  Panagiotis Adamopoulos,et al.  On Unexpectedness in Recommender Systems: Or How to Expect the Unexpected , 2011, DiveRS@RecSys.

[28]  Dietmar Jannach,et al.  User Perception of Next-Track Music Recommendations , 2017, UMAP.