Effects of Personalized and Aggregate Top-N Recommendation Lists on User Preference Ratings

Prior research has shown a robust effect of personalized product recommendations on user preference judgments for items. Specifically, the display of system-predicted preference ratings as item recommendations has been shown in multiple studies to bias users’ preference ratings after item consumption in the direction of the predicted rating. Top-N lists represent another common approach for presenting item recommendations in recommender systems. Through three controlled laboratory experiments, we show that top-N lists do not induce a discernible bias in user preference judgments. This result is robust, holding for both lists of personalized item recommendations and lists of items that are top-rated based on averages of aggregate user ratings. Adding numerical ratings to the list items does generate a bias, consistent with earlier studies. Thus, in contexts where preference biases are of concern to an online retailer or platform, top-N lists, without numerical predicted ratings, would be a promising format for displaying item recommendations.

[1]  Thomas Mussweiler,et al.  Hypothesis-consistent testing and semantic priming in the anchoring paradigm: A selective accessibility model. , 1999 .

[2]  Tong Zhang,et al.  Recommender Systems Using Linear Classifiers , 2002 .

[3]  John Riedl,et al.  GroupLens: an open architecture for collaborative filtering of netnews , 1994, CSCW '94.

[4]  George Karypis,et al.  Item-based top-N recommendation algorithms , 2004, TOIS.

[5]  J. Lygeros,et al.  Decision Making I , 2014 .

[6]  Thorsten Joachims,et al.  Unbiased Learning-to-Rank with Biased Feedback , 2016, WSDM.

[7]  Bamshad Mobasher,et al.  Controlling Popularity Bias in Learning-to-Rank Recommendation , 2017, RecSys.

[8]  Greg Linden,et al.  Amazon . com Recommendations Item-to-Item Collaborative Filtering , 2001 .

[9]  S. Sénécal,et al.  The influence of online product recommendations on consumers' online choices , 2004 .

[10]  John C. S. Lui,et al.  Modeling the Assimilation-Contrast Effects in Online Product Rating Systems: Debiasing and Recommendations , 2017, ACM Conference on Recommender Systems.

[11]  Fuyuki Ishikawa,et al.  Improving Accuracy of Recommender System by Item Clustering , 2007, IEICE Trans. Inf. Syst..

[12]  Korris Fu-Lai Chung,et al.  A Deep Bayesian Tensor-Based System for Video Recommendation , 2018, ACM Trans. Inf. Syst..

[13]  Loren G. Terveen,et al.  Exploring the filter bubble: the effect of using recommender systems on content diversity , 2014, WWW.

[14]  Sean A. Munson,et al.  Bursting your (filter) bubble: strategies for promoting diverse exposure , 2013, CSCW '13.

[15]  Michael D. Smith,et al.  Location, Location, Location: An Analysis of Profitability of Position in Online Advertising Markets , 2008 .

[16]  S. Curley,et al.  Reducing Recommender Systems Biases: An Investigation of Rating Display Designs , 2019, MIS Q..

[17]  Ke Wang,et al.  Trip Recommendation Meets Real-World Constraints , 2016, ACM Trans. Inf. Syst..

[18]  Kannan Srinivasan,et al.  A "Position Paradox" in Sponsored Search Auctions , 2011, Mark. Sci..

[19]  K. Markman,et al.  Activating a mental simulation mind-set through generation of alternatives: Implications for debiasing in related and unrelated domains , 2004 .

[20]  Kenneth Y. Goldberg,et al.  Eigentaste: A Constant Time Collaborative Filtering Algorithm , 2001, Information Retrieval.

[21]  Eli Pariser.  FILTER BUBBLE: Wie wir im Internet entmündigt werden , 2012 .

[22]  Greg Linden,et al.  Two Decades of Recommender Systems at Amazon.com , 2017, IEEE Internet Computing.

[23]  Raluca M. Ursu.  The Power of Rankings: Quantifying the Effect of Rankings on Online Consumer Search and Purchase Decisions , 2018, Mark. Sci..

[24]  Roberto Turrin,et al.  Performance of recommender algorithms on top-n recommendation tasks , 2010, RecSys '10.

[25]  A. Tversky,et al.  Judgment under Uncertainty: Heuristics and Biases , 1974, Science.

[26]  David Heckerman,et al.  Empirical Analysis of Predictive Algorithms for Collaborative Filtering , 1998, UAI.

[27]  Thorsten Joachims,et al.  Recommendations as Treatments: Debiasing Learning and Evaluation , 2016, ICML.

[28]  Sanjay Krishnan,et al.  A methodology for learning, analyzing, and mitigating social influence bias in recommender systems , 2014, RecSys '14.

[29]  Yehuda Koren,et al.  Improved Neighborhood-based Collaborative Filtering , 2007 .

[30]  Mengting Wan,et al.  Addressing Marketing Bias in Product Recommendations , 2019, WSDM.

[31]  Beibei Li,et al.  Designing Ranking Systems for Hotels on Travel Search Engines by Mining User-Generated and Crowd-Sourced Content , 2011, Mark. Sci..

[32]  Pasquale Lops,et al.  Human Decision Making and Recommender Systems , 2013, TIIS.

[33]  Li Chen,et al.  Human Decision Making and Recommender Systems , 2015, Recommender Systems Handbook.

[34]  Shawn P. Curley,et al.  Do Recommender Systems Manipulate Consumer Preferences? A Study of Anchoring Effects , 2013, Inf. Syst. Res..

[35]  Alan T. Sorensen.  Bestseller Lists and Product Variety , 2007 .

[36]  Balaji Padmanabhan,et al.  The "Most Popular News" Recommender: Count Amplification and Manipulation Resistance , 2014, Inf. Syst. Res..

[37]  George Karypis,et al.  SLIM: Sparse Linear Methods for Top-N Recommender Systems , 2011, 2011 IEEE 11th International Conference on Data Mining.

[38]  Bracha Shapira,et al.  Recommender Systems Handbook , 2015, Springer US.

[39]  Joseph A. Konstan,et al.  Introduction to recommender systems: Algorithms and Evaluation , 2004, TOIS.

[40]  Martha Larson,et al.  Top-N Recommendation with Multi-Channel Positive Feedback using Factorization Machines , 2019, ACM Trans. Inf. Syst..

[41]  Saul Vargas,et al.  Improving sales diversity by recommending users to items , 2014, RecSys '14.

[42]  John Riedl,et al.  Item-based collaborative filtering recommendation algorithms , 2001, WWW '01.

[43]  Shawn P. Curley,et al.  Effects of Online Recommendations on Consumers’ Willingness to Pay , 2012, Decisions@RecSys.

[44]  Yehuda Koren,et al.  Advances in Collaborative Filtering , 2011, Recommender Systems Handbook.

[45]  Dietmar Jannach,et al.  What recommenders recommend: an analysis of recommendation biases and possible countermeasures , 2015, User Modeling and User-Adapted Interaction.

[46]  Gediminas Adomavicius,et al.  Improving Aggregate Recommendation Diversity Using Ranking-Based Techniques , 2012, IEEE Transactions on Knowledge and Data Engineering.

[47]  M. B. Blake,et al.  Two Decades of Recommender Systems at Amazon.com , 2017 .

[48]  George Karypis,et al.  Sparse linear methods with side information for top-n recommendations , 2012, WWW.

[49]  Rachel Pottinger,et al.  A Generic Top-N Recommendation Framework for Trading-Off Accuracy, Novelty, and Coverage , 2018, 2018 IEEE 34th International Conference on Data Engineering (ICDE).

[50]  WangWei,et al.  Recommender system application developments , 2015 .

[51]  Tat-Seng Chua,et al.  Fast Matrix Factorization for Online Recommendation with Implicit Feedback , 2016, SIGIR.

[52]  Alexander Tuzhilin,et al.  On over-specialization and concentration bias of recommendations: probabilistic neighborhood selection in collaborative filtering systems , 2014, RecSys '14.

[53]  Jingjing Zhang,et al.  Consumption and Performance: Understanding Longitudinal Dynamics of Recommender Systems via an Agent-Based Simulation Framework , 2019, Inf. Syst. Res..

[54]  Gediminas Adomavicius,et al.  Toward the next generation of recommender systems: a survey of the state-of-the-art and possible extensions , 2005, IEEE Transactions on Knowledge and Data Engineering.

[55]  Katherine L. Milkman,et al.  A User's Guide to Debiasing , 2014 .

[56]  George Karypis,et al.  FISM: factored item similarity models for top-N recommender systems , 2013, KDD.

[57]  Kartik Hosanagar,et al.  Blockbuster Culture's Next Rise or Fall: The Impact of Recommender Systems on Sales Diversity , 2007, Manag. Sci..

[58]  Deborah Estrin,et al.  Unbiased offline recommender evaluation for missing-not-at-random implicit feedback , 2018, RecSys.

[59]  Kai Liu,et al.  Deep Item-based Collaborative Filtering for Top-N Recommendation , 2018, ACM Trans. Inf. Syst..

[60]  Eli Pariser,et al.  The Filter Bubble: How the New Personalized Web Is Changing What We Read and How We Think , 2012 .

[61]  Zhao Kang,et al.  Top-N Recommender System via Matrix Completion , 2016, AAAI.

[62]  David B. Dunson,et al.  Uncovering Systematic Bias in Ratings across Categories: a Bayesian Approach , 2015, RecSys.

[63]  Ricardo Giglio,et al.  Case study on the business value impact of personalized recommendations on a large online retailer , 2012, RecSys.

[64]  Nuria Oliver,et al.  I Like It... I Like It Not: Evaluating User Ratings Noise in Recommender Systems , 2009, UMAP.

[65]  Robert M. Schindler,et al.  The Top-Ten Effect: Consumers' Subjective Categorization of Ranked Lists , 2014 .