Using Shortlists to Support Decision Making and Improve Recommender System Performance

In this paper, we study shortlists as an interface component for recommender systems with the dual goal of supporting the user's decision process, as well as improving implicit feedback elicitation for increased recommendation quality. A shortlist is a temporary list of candidates that the user is currently considering, e.g., a list of a few movies the user is currently considering for viewing. From a cognitive perspective, shortlists serve as digital short-term memory where users can off-load the items under consideration -- thereby decreasing their cognitive load. From a machine learning perspective, adding items to the shortlist generates a new implicit feedback signal as a by-product of exploration and decision making which can improve recommendation quality. Shortlisting therefore provides additional data for training recommendation systems without the increases in cognitive load that requesting explicit feedback would incur. We perform an user study with a movie recommendation setup to compare interfaces that offer shortlist support with those that do not. From the user studies we conclude: (i) users make better decisions with a shortlist; (ii) users prefer an interface with shortlist support; and (iii) the additional implicit feedback from sessions with a shortlist improves the quality of recommendations by nearly a factor of two.

[1]  R. G. Crowder.  Principles of learning and memory , 1977 .

[2]  Tao Luo,et al.  Using sequential and non-sequential patterns in predictive Web usage mining tasks , 2002, 2002 IEEE International Conference on Data Mining, 2002. Proceedings..

[3]  Juho Hamari,et al.  Does Gamification Work? -- A Literature Review of Empirical Studies on Gamification , 2014, 2014 47th Hawaii International Conference on System Sciences.

[4]  J. Wyatt.  Decision support systems. , 2000, Journal of the Royal Society of Medicine.

[5]  Yuan Jia,et al.  Should i stay or should i go: two features to help people stop an exploratory search wisely , 2014, CHI Extended Abstracts.

[6]  Jaime Teevan,et al.  Implicit feedback for inferring user preference: a bibliography , 2003, SIGF.

[7]  Francesco Ricci,et al.  Product Recommendation with Interactive Query Management and Twofold Similarity , 2003, ICCBR.

[8]  Russell Revlin,et al.  Cognition - Theory and Practice , 2012 .

[9]  Franca Garzotto,et al.  Decision-Making in Recommender Systems: The Role of User's Goals and Bounded Resources , 2012, Decisions@RecSys.

[10]  Anthony Jameson.  Recommender Systems as Part of a Choice Architecture for HCI , 2014, DMRS.

[11]  References , 1971 .

[12]  Dietmar Jannach,et al.  Adaptation and Evaluation of Recommendations for Short-term Shopping Goals , 2015, RecSys.

[13]  Li Chen,et al.  Human Decision Making and Recommender Systems , 2015, Recommender Systems Handbook.

[14]  H. Simon,et al.  A Behavioral Model of Rational Choice , 1955 .

[15]  Filip Radlinski,et al.  How does clickthrough data reflect retrieval quality? , 2008, CIKM '08.

[16]  William Lidwell,et al.  Universal principles of design : 100 ways to enhance usability,influence perception, increase appeal, make better, designdecisions, and teach through design , 2003 .

[17]  Martha Larson,et al.  xCLiMF: optimizing expected reciprocal rank for data with multiple levels of relevance , 2013, RecSys.

[18]  Thorsten Joachims,et al.  Optimizing search engines using clickthrough data , 2002, KDD.

[19]  Burr Settles,et al.  Active Learning Literature Survey , 2009 .

[20]  Dietmar Jannach,et al.  Using graded implicit feedback for bayesian personalized ranking , 2014, RecSys '14.

[21]  Murdock,et al.  The serial position effect of free recall , 1962 .

[22]  Thorsten Joachims,et al.  A support vector method for multivariate performance measures , 2005, ICML.

[23]  Ramesh Sharda,et al.  Reflections on the Past and Future of Decision Support Systems: Perspective of Eleven Pioneers , 2011, Decision Support - An Examination of the DSS Discipline.

[24]  Lars-Göran Nilsson,et al.  Aging, Memory, and Decision Making , 2015 .

[25]  Lars Schmidt-Thieme,et al.  Factorizing personalized Markov chains for next-basket recommendation , 2010, WWW '10.

[26]  Gerard Salton,et al.  The SMART Retrieval System—Experiments in Automatic Document Processing , 1971 .

[27]  G. A. Miller.  THE PSYCHOLOGICAL REVIEW THE MAGICAL NUMBER SEVEN, PLUS OR MINUS TWO: SOME LIMITS ON OUR CAPACITY FOR PROCESSING INFORMATION 1 , 1956 .

[28]  Steven M. Drucker,et al.  The Visual Decision Maker: a recommendation system for collocated users , 2005, DUX '05.

[29]  Eric J. Johnson,et al.  Adaptive Strategy Selection in Decision Making. , 1988 .

[30]  H. Simon,et al.  Rational choice and the structure of the environment. , 1956, Psychological review.

[31]  Fabio Del Missier,et al.  Memory and Decision Making: From Basic Cognitive Research to Design Issues , 2014, DMRS.

[32]  Toon De Pessemier,et al.  An Online Evaluation of Explicit Feedback Mechanisms for Recommender Systems , 2011, WEBIST.

[33]  William Lidwell,et al.  Universal Principles of Design , 2003 .

[34]  A. Baddeley.  Essentials of Human Memory , 1999 .

[35]  Gerard Salton,et al.  Improving retrieval performance by relevance feedback , 1997, J. Am. Soc. Inf. Sci..

[36]  Dorota Glowacka,et al.  Supporting exploratory search tasks with interactive user modeling , 2013, ASIST.

[37]  Pasquale Lops,et al.  Human Decision Making and Recommender Systems , 2013, TIIS.