Shaping Feedback Data in Recommender Systems with Interventions Based on Information Foraging Theory

Recommender systems rely heavily on the predictive accuracy of the learning algorithm. Most work on improving accuracy has focused on the learning algorithm itself. We argue that this algorithmic focus is myopic. In particular, since learning algorithms generally improve with more and better data, we propose shaping the feedback generation process as an alternate and complementary route to improving accuracy. To this effect, we explore how changes to the user interface can impact the quality and quantity of feedback data -- and therefore the learning accuracy. Motivated by information foraging theory, we study how feedback quality and quantity are influenced by interface design choices along two axes: information scent and information access cost. We present a user study of these interface factors for the common task of picking a movie to watch, showing that these factors can effectively shape and improve the implicit feedback data that is generated while maintaining the user experience.

[1]  E. Charnov.  Optimal foraging, the marginal value theorem. , 1976, Theoretical population biology.

[2]  S. Hart,et al.  Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research , 1988 .

[3]  Mary Czerwinski,et al.  Web page design: implications of memory, structure and scent for information retrieval , 1998, CHI.

[4]  Pattie Maes,et al.  Footprints: history-rich tools for information foraging , 1999, CHI '99.

[5]  Ed H. Chi,et al.  Using information scent to model user information needs and actions and the Web , 2001, CHI.

[6]  Mark Claypool,et al.  Implicit interest indicators , 2001, IUI '01.

[7]  Christopher Olston,et al.  ScentTrails: Integrating browsing and searching on the Web , 2003, TCHI.

[8]  Thomas Hofmann,et al.  Unifying collaborative and content-based filtering , 2004, ICML.

[9]  Jonathan L. Herlocker,et al.  Evaluating collaborative filtering recommender systems , 2004, TOIS.

[10]  D. Angluin,et al.  Learning From Noisy Examples , 1988, Machine Learning.

[11]  Wayne D. Gray,et al.  The soft constraints hypothesis: a rational analysis approach to resource allocation for interactive behavior. , 2006, Psychological review.

[12]  John Zimmerman,et al.  Vio: a mixed-initiative approach to learning and automating procedural update tasks , 2007, CHI.

[13]  Wai-Tat Fu,et al.  SNIF-ACT: A Cognitive Model of User Navigation on the World Wide Web , 2007, Hum. Comput. Interact..

[14]  Peter Pirolli,et al.  Information foraging theory , 2007 .

[15]  P. Pirolli.  Information Foraging Theory: Adaptive Interaction with Information , 2007 .

[16]  Yoshua. Bengio,et al.  Learning Deep Architectures for AI , 2007, Found. Trends Mach. Learn..

[17]  Yifan Hu,et al.  Collaborative Filtering for Implicit Feedback Datasets , 2008, 2008 Eighth IEEE International Conference on Data Mining.

[18]  Aniket Kittur,et al.  Crowdsourcing user studies with Mechanical Turk , 2008, CHI.

[19]  Filip Radlinski,et al.  How does clickthrough data reflect retrieval quality? , 2008, CIKM '08.

[20]  Peter Pirolli,et al.  Information Foraging , 2009, Encyclopedia of Database Systems.

[21]  Ryen W. White,et al.  Understanding web browsing behaviors through Weibull analysis of dwell time , 2010, SIGIR.

[22]  Wai-Tat Fu,et al.  Facilitating exploratory search by model-based navigational cues , 2010, IUI '10.

[23]  John Riedl,et al.  Recommender systems: from algorithms to user experience , 2012, User Modeling and User-Adapted Interaction.

[24]  Wendy W. Moe,et al.  Measuring the Value of Social Dynamics in Online Product Ratings Forums , 2010 .

[25]  Ryen W. White,et al.  No clicks, no problem: using cursor movements to understand and improve search , 2011, CHI.

[26]  Ryen W. White,et al.  User see, user point: gaze and cursor alignment in web search , 2012, CHI.

[27]  Bonnie E. John,et al.  CogTool-Explorer: a model of goal-directed user exploration that considers information layout , 2012, CHI.

[28]  Eugene Agichtein,et al.  Beyond dwell time: estimating document relevance from cursor movements and other post-click searcher behavior , 2012, WWW.

[29]  Katharina Reinecke,et al.  Crowdsourcing performance evaluations of user interfaces , 2013, CHI.

[30]  Bipin Indurkhya,et al.  Cognitively inspired task design to improve user performance on crowdsourcing platforms , 2014, CHI.

[31]  Todd Kulesza,et al.  Structured labeling for facilitating concept evolution in machine learning , 2014, CHI.

[32]  Shai Ben-David,et al.  Understanding Machine Learning: From Theory to Algorithms , 2014 .

[33]  Ryen W. White,et al.  Modeling dwell time to predict click-level satisfaction , 2014, WSDM.

[34]  Imed Zitouni,et al.  Automatic Online Evaluation of Intelligent Assistants , 2015, WWW.

[35]  Guido Zuccon,et al.  An Analysis of the Cost and Benefit of Search Interactions , 2016, ICTIR.

[36]  Susan T. Dumais,et al.  Using Shortlists to Support Decision Making and Improve Recommender System Performance , 2015, WWW.

[37]  Shuai Li,et al.  Collaborative Filtering Bandits , 2015, SIGIR.

[38]  Yale Song,et al.  Mouse Activity as an Indicator of Interestingness in Video , 2016, ICMR.

[39]  Qian Yang,et al.  The Role of Design in Creating Machine-Learning-Enhanced User Experience , 2017, AAAI Spring Symposia.

[40]  Thorsten Joachims,et al.  Unbiased Learning-to-Rank with Biased Feedback , 2016, WSDM.

[41]  Kim Halskov,et al.  UX Design Innovation: Challenges for Working with Machine Learning as a Design Material , 2017, CHI.

[42]  Chang Liu,et al.  Scroll up or down?: Using Wheel Activity as an Indicator of Browsing Strategy across Different Contextual Factors , 2017, CHIIR.

[43]  Thorsten Joachims,et al.  Ranking with Social Cues: Integrating Online Review Scores and Popularity Information , 2017, ICWSM.