Recommender Systems in Industry: A Netflix Case Study

The Netflix Prize put a spotlight on the importance and use of recommender systems in real-world applications. Many the competition provided many lessons about how to approach recommendation and many more have been learned since the Grand Prize was awarded in 2009. The evolution of industrial applications of recommender systems has been driven by the availability of different kinds of user data and the level of interest for the area within the research community. The goal of this chapter is to give an up-to-date overview of recommender systems techniques used in an industrial setting. We will give a high-level description the practical use of recommendation and personalization techniques. We will highlight some of the main lessons learned from the Netflix Prize. We will then use Netflix personalization as a case study to describe several approaches and techniques used in a real-world recommendation system. Finally, we will pinpoint what we see as some promising current research avenues and unsolved problems that deserve attention in this domain from an industry perspective.

[1]  Yehuda Koren,et al.  Lessons from the Netflix prize challenge , 2007, SKDD.

[2]  Òscar Celma,et al.  Music Recommendation and Discovery - The Long Tail, Long Fail, and Long Play in the Digital Music Space , 2010 .

[3]  Bart P. Knijnenburg.  Conducting user experiments in recommender systems , 2012, RecSys.

[4]  Martha Larson,et al.  TFMAP: optimizing MAP for top-n context-aware recommendation , 2012, SIGIR '12.

[5]  Xi Chen,et al.  Temporal Collaborative Filtering with Bayesian Probabilistic Tensor Factorization , 2010, SDM.

[6]  Nuria Oliver,et al.  Multiverse recommendation: n-dimensional tensor factorization for context-aware collaborative filtering , 2010, RecSys '10.

[7]  Robin D. Burke,et al.  Hybrid Web Recommender Systems , 2007, The Adaptive Web.

[8]  Hang Li,et al.  AdaRank: a boosting algorithm for information retrieval , 2007, SIGIR.

[9]  Harald Steck,et al.  Item popularity and recommendation accuracy , 2011, RecSys '11.

[10]  Lars Schmidt-Thieme,et al.  BPR: Bayesian Personalized Ranking from Implicit Feedback , 2009, UAI.

[11]  Yehuda Koren,et al.  OrdRec: an ordinal model for predicting personalized item rating distributions , 2011, RecSys '11.

[12]  Mihai Georgescu,et al.  Swarming to rank for recommender systems , 2012, RecSys.

[13]  Tian Xia,et al.  Direct optimization of ranking measures for learning to rank models , 2013, KDD.

[14]  Gregory N. Hullender,et al.  Learning to rank using gradient descent , 2005, ICML.

[15]  Yu He,et al.  The YouTube video recommendation system , 2010, RecSys '10.

[16]  Yoram Singer,et al.  An Efficient Boosting Algorithm for Combining Preferences by , 2013 .

[17]  Shuang-Hong Yang,et al.  Collaborative competitive filtering: learning recommender using context of user choice , 2011, SIGIR.

[18]  Tie-Yan Liu,et al.  Learning to rank: from pairwise approach to listwise approach , 2007, ICML '07.

[19]  Alexander J. Smola,et al.  Fair and balanced: learning to present news stories , 2012, WSDM '12.

[20]  Yehuda Koren,et al.  Matrix Factorization Techniques for Recommender Systems , 2009, Computer.

[21]  Harald Steck,et al.  Evaluation of recommendations: rating-prediction and ranking , 2013, RecSys.

[22]  Michael I. Jordan,et al.  Latent Dirichlet Allocation , 2001, J. Mach. Learn. Res..

[23]  Yang Guo,et al.  On top-k recommendation using social networks , 2012, RecSys.

[24]  Rong Jin,et al.  Learning to Rank by Optimizing NDCG Measure , 2009, NIPS.

[25]  Christian Posse,et al.  Metaphor: a system for related search recommendations , 2012, CIKM.

[26]  Steffen Rendle,et al.  Factorization Machines , 2010, 2010 IEEE International Conference on Data Mining.

[27]  Paul Lamere.  I've got 10 million songs in my pocket: now what? , 2012, RecSys '12.

[28]  Nuria Oliver,et al.  I Like It... I Like It Not: Evaluating User Ratings Noise in Recommender Systems , 2009, UMAP.

[29]  Delbert Dueck,et al.  Clustering by Passing Messages Between Data Points , 2007, Science.

[30]  Randy H. Katz,et al.  Chukwa: A System for Reliable Large-Scale Log Collection , 2010, LISA.

[31]  Adriano Veloso,et al.  Pareto-efficient hybridization for multi-objective recommender systems , 2012, RecSys.

[32]  Jiahui Liu,et al.  Personalized news recommendation based on click behavior , 2010, IUI '10.

[33]  Daniele Quercia,et al.  Spotting trends: the wisdom of the few , 2012, RecSys '12.

[34]  Edward Y. Chang,et al.  Collaborative filtering for orkut communities: discovery of user latent behavior , 2009, WWW '09.

[35]  Abhinandan Das,et al.  Google news personalization: scalable online collaborative filtering , 2007, WWW '07.

[36]  Saul Vargas,et al.  Rank and relevance in novelty and diversity metrics for recommender systems , 2011, RecSys '11.

[37]  Denis Parra,et al.  Implicit Feedback Recommendation via Implicit-to-Explicit Ordinal Logistic Regression Mapping , 2011 .

[38]  Kenta Oku,et al.  Context-Aware SVM for Context-Dependent Information Recommendation , 2006, 7th International Conference on Mobile Data Management (MDM'06).

[39]  Christian Posse,et al.  Multiple objective optimization in recommender systems , 2012, RecSys.

[40]  Lars Schmidt-Thieme,et al.  Factorizing personalized Markov chains for next-basket recommendation , 2010, WWW '10.

[41]  Domonkos Tikk,et al.  Major components of the gravity recommendation system , 2007, SKDD.

[42]  Tie-Yan Liu,et al.  Listwise approach to learning to rank: theory and algorithm , 2008, ICML '08.

[43]  Alfred Kobsa,et al.  The Adaptive Web, Methods and Strategies of Web Personalization , 2007, The Adaptive Web.

[44]  Charles Elkan,et al.  Learning classifiers from only positive and unlabeled data , 2008, KDD.

[45]  Jimmy J. Lin,et al.  WTF: the who to follow service at Twitter , 2013, WWW.

[46]  Yehuda Koren,et al.  Factorization meets the neighborhood: a multifaceted collaborative filtering model , 2008, KDD.

[47]  Chao Liu,et al.  Wisdom of the better few: cold start recommendation via representative based rating elicitation , 2011, RecSys '11.

[48]  Yehuda Koren,et al.  Collaborative filtering with temporal dynamics , 2009, KDD.

[49]  Greg Linden,et al.  Amazon . com Recommendations Item-to-Item Collaborative Filtering , 2001 .

[50]  Alexander Tuzhilin,et al.  The effect of context-aware recommendations on customer purchasing behavior and trust , 2011, RecSys '11.

[51]  Jie Li,et al.  Predictive model performance: offline and online evaluations , 2013, KDD.

[52]  Chih-Hung Hsieh,et al.  Towards better measurement of attention and satisfaction in mobile search , 2014, SIGIR.

[53]  Scott Sanner,et al.  New objective functions for social collaborative filtering , 2012, WWW.

[54]  J. Friedman.  Greedy function approximation: A gradient boosting machine. , 2001 .

[55]  Domonkos Tikk,et al.  Alternating least squares for personalized ranking , 2012, RecSys.

[56]  Judy Kay,et al.  RECON: a reciprocal recommender for online dating , 2010, RecSys '10.

[57]  Denis Parra,et al.  Walk the talk: analyzing the relation between implicit and explicit feedback for preference elicitation , 2011, UMAP'11.

[58]  Deepak Agarwal,et al.  Personalized Recommendation of User Comments via Factor Models , 2011, EMNLP.

[59]  Alexander J. Smola,et al.  Measurement and modeling of eye-mouse behavior in the presence of nonlinear page layouts , 2013, WWW.

[60]  Yi Zhang,et al.  Is it time for a career switch? , 2013, WWW.

[61]  Jason Weston,et al.  Learning to rank recommendations with the k-order statistic loss , 2013, RecSys.

[62]  Michael I. Jordan,et al.  Hierarchical Dirichlet Processes , 2006 .

[63]  William W. Cohen,et al.  Recommendation as Classification: Using Social and Content-Based Information in Recommendation , 1998, AAAI/IAAI.

[64]  George Karypis,et al.  Evaluation of Item-Based Top-N Recommendation Algorithms , 2001, CIKM '01.

[65]  Xavier Amatriain,et al.  The wisdom of the few: a collaborative filtering approach based on expert opinions from the web , 2009, SIGIR.

[66]  Leo Breiman,et al.  Random Forests , 2001, Machine Learning.

[67]  Ron Kohavi,et al.  Trustworthy online controlled experiments: five puzzling outcomes explained , 2012, KDD.

[68]  Wei Li,et al.  A stochastic learning-to-rank algorithm and its application to contextual advertising , 2011, WWW.

[69]  Adam Tauman Kalai,et al.  Trust-based recommendation systems: an axiomatic approach , 2008, WWW.

[70]  Ulrich Paquet,et al.  The Xbox recommender system , 2012, RecSys '12.

[71]  Jonathan L. Herlocker,et al.  A collaborative filtering algorithm and evaluation metric that accurately model the user experience , 2004, SIGIR '04.

[72]  George Karypis,et al.  Sparse linear methods with side information for Top-N recommendations , 2012, WWW.

[73]  Geoffrey E. Hinton,et al.  Restricted Boltzmann machines for collaborative filtering , 2007, ICML '07.

[74]  Pattie Maes,et al.  Social information filtering: algorithms for automating “word of mouth” , 1995, CHI '95.

[75]  Ron Kohavi,et al.  Practical guide to controlled experiments on the web: listen to your customers not to the hippo , 2007, KDD '07.

[76]  Barry Smyth,et al.  Trust in recommender systems , 2005, IUI.

[77]  Jonathan L. Herlocker,et al.  Evaluating collaborative filtering recommender systems , 2004, TOIS.

[78]  I. Berlin.  I like it. , 1921 .

[79]  Barry Smyth,et al.  Power to the people: exploring neighbourhood formations in social recommender system , 2011, RecSys '11.

[80]  Jennifer Widom,et al.  SimRank: a measure of structural-context similarity , 2002, KDD.

[81]  Wei Chu,et al.  A contextual-bandit approach to personalized news article recommendation , 2010, WWW '10.

[82]  Jose L. Marzo,et al.  User Modeling, Adaption and Personalization - 19th International Conference, UMAP 2011, Girona, Spain, July 11-15, 2011. Proceedings , 2011, UMAP.

[83]  Harald Steck,et al.  Training and testing of recommender systems on data missing not at random , 2010, KDD.

[84]  Martin Ester,et al.  TrustWalker: a random walk model for combining trust-based and item-based recommendation , 2009, KDD.

[85]  Thore Graepel,et al.  Matchbox: large scale online bayesian recommendations , 2009, WWW '09.

[86]  S. Sathiya Keerthi,et al.  Efficient algorithms for ranking with SVMs , 2010, Information Retrieval.

[87]  Yasuhiro Fujiwara,et al.  Efficient personalized pagerank with accuracy assurance , 2012, KDD.

[88]  Sean M. McNee,et al.  Being accurate is not enough: how accuracy metrics have hurt recommender systems , 2006, CHI Extended Abstracts.

[89]  Martha Larson,et al.  CLiMF: learning to maximize reciprocal rank with collaborative less-is-more filtering , 2012, RecSys.

[90]  Yifan Hu,et al.  Collaborative Filtering for Implicit Feedback Datasets , 2008, 2008 Eighth IEEE International Conference on Data Mining.

[91]  Deepak Agarwal,et al.  Content recommendation on web portals , 2013, CACM.

[92]  Jian Wang,et al.  Utilizing related products for post-purchase recommendation in e-commerce , 2011, RecSys '11.

[93]  Lars Schmidt-Thieme,et al.  Fast context-aware recommendations with factorization machines , 2011, SIGIR.

[94]  Filip Radlinski,et al.  How does clickthrough data reflect retrieval quality? , 2008, CIKM '08.

[95]  Tie-Yan Liu,et al.  Directly optimizing evaluation measures in learning to rank , 2008, SIGIR.

[96]  Roelof van Zwol,et al.  Flickr tag recommendation based on collective knowledge , 2008, WWW.