Related Pins at Pinterest: The Evolution of a Real-World Recommender System

Related Pins is the Web-scale recommender system that powers over 40% of user engagement on Pinterest. This paper is a longitudinal study of three years of its development, exploring the evolution of the system and its components from prototypes to present state. Each component was originally built with many constraints on engineering effort and computational resources, so we prioritized the simplest and highest-leverage solutions. We show how organic growth led to a complex system and how we managed this complexity. Many challenges arose while building this system, such as avoiding feedback loops, evaluating performance, activating content, and eliminating legacy heuristics. Finally, we offer suggestions for tackling these challenges when engineering Web-scale recommender systems.

[1]  Paul Covington,et al.  Deep Neural Networks for YouTube Recommendations , 2016, RecSys.

[2]  Eric Tzeng,et al.  Human Curation and Convnets: Powering Item-to-Item Recommendations on Pinterest , 2015, ArXiv.

[3]  Gregory N. Hullender,et al.  Learning to rank using gradient descent , 2005, ICML.

[4]  Heng-Tze Cheng,et al.  Wide & Deep Learning for Recommender Systems , 2016, DLRS@RecSys.

[5]  Trevor Darrell,et al.  Visual Discovery at Pinterest , 2017, WWW.

[6]  Jeffrey Dean,et al.  Efficient Estimation of Word Representations in Vector Space , 2013, ICLR.

[7]  Wei Vivian Zhang,et al.  Comparing Click Logs and Editorial Labels for Training Query Rewriting , 2007 .

[8]  Jimmy J. Lin,et al.  WTF: the who to follow service at Twitter , 2013, WWW.

[9]  CARLOS A. GOMEZ-URIBE,et al.  The Netflix Recommender System , 2015, ACM Trans. Manag. Inf. Syst..

[10]  D. Sculley,et al.  Hidden Technical Debt in Machine Learning Systems , 2015, NIPS.

[11]  Yifan Hu,et al.  Collaborative Filtering for Implicit Feedback Datasets , 2008, 2008 Eighth IEEE International Conference on Data Mining.

[12]  Thorsten Joachims,et al.  Optimizing search engines using clickthrough data , 2002, KDD.

[13]  Tie-Yan Liu,et al.  Learning to Rank for Information Retrieval , 2011 .

[14]  Alan L. Yuille,et al.  Training and Evaluating Multimodal Word Embeddings with Large-scale Web Annotated Images , 2016, NIPS.

[15]  Wei-Ying Ma,et al.  Scalable music recommendation by search , 2007, ACM Multimedia.

[16]  Yu He,et al.  The YouTube video recommendation system , 2010, RecSys '10.

[17]  Jeff Donahue,et al.  Visual Search at Pinterest , 2015, KDD.

[18]  References , 1971 .

[19]  Junghoo Cho,et al.  Power of Human Curation in Recommendation System , 2016, WWW.

[20]  Martín Abadi,et al.  TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems , 2016, ArXiv.

[21]  Shankar Kumar,et al.  Video suggestion and discovery for youtube: taking random walks through the view graph , 2008, WWW.

[22]  Shumeet Baluja,et al.  Pagerank for product image search , 2008, WWW.

[23]  Jure Leskovec,et al.  Pixie: A System for Recommending 3+ Billion Items to 200+ Million Users in Real-Time , 2017, WWW.