Improving Recommendation Quality in Google Drive

Quick Access is a machine-learned system in Google Drive that predicts which files a user wants to open. Adding Quick Access recommendations to the Drive homepage cut the amount of time that users spend locating their files in half. Aggregated over the ~1 billion users of Drive, the time saved up adds up to ~1000 work weeks every day. In this paper, we discuss both the challenges of iteratively improving the quality of a personal recommendation system as well as the variety of approaches that we took in order to improve this feature. We explored different deep network architectures, novel modeling techniques, additional data sources, and the effects of latency and biases in the UX. We share both pitfalls as well as successes in our attempts to improve this product, and also discuss how we scaled and managed the complexity of the system. We believe that these insights will be especially useful to those who are working with private corpora as well as those who are building a large-scale production recommendation system.

[1]  Stephanie Rogers,et al.  Related Pins at Pinterest: The Evolution of a Real-World Recommender System , 2017, WWW.

[2]  Sandeep Tata,et al.  Quick Access: Building a Smart Experience for Google Drive , 2017, KDD.

[3]  Sergey Ioffe,et al.  Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift , 2015, ICML.

[4]  Ning Zhou,et al.  Deep neural network marketplace recommenders in online experiments , 2018, RecSys.

[5]  Jia Li,et al.  Latent Cross: Making Use of Context in Recurrent Recommender Systems , 2018, WSDM.

[6]  Barbara E. Engelhardt,et al.  How algorithmic confounding in recommendation systems increases homogeneity and decreases utility , 2017, RecSys.

[7]  Xuanhui Wang,et al.  Combining Decision Trees and Neural Networks for Learning-to-Rank in Personal Search , 2019, KDD.

[8]  Xin Zhang,et al.  TFX: A TensorFlow-Based Production-Scale Machine Learning Platform , 2017, KDD.

[9]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[10]  D. Sculley,et al.  Google Vizier: A Service for Black-Box Optimization , 2017, KDD.

[11]  Sandeep Tata,et al.  ItemSuggest: A Data Management Platform for Machine Learned Ranking Services , 2019, CIDR.

[12]  Paul Covington,et al.  Deep Neural Networks for YouTube Recommendations , 2016, RecSys.

[13]  Tie-Yan Liu,et al.  Learning to rank: from pairwise approach to listwise approach , 2007, ICML '07.

[14]  Shawn P. Curley,et al.  De-biasing user preference ratings in recommender systems completed research paper , 2014, RecSys 2014.

[15]  Gang Fu,et al.  Deep & Cross Network for Ad Click Predictions , 2017, ADKDD@KDD.

[16]  Zhen Qin,et al.  Parameter Tuning in Personal Search Systems , 2020, WSDM.

[17]  Zhen Qin,et al.  Attribute-based Propensity for Unbiased Learning in Recommender Systems: Algorithm and Case Studies , 2020, KDD.

[18]  Thorsten Joachims,et al.  Recommendations as Treatments: Debiasing Learning and Evaluation , 2016, ICML.

[19]  Hongbo Deng,et al.  Ranking Relevance in Yahoo Search , 2016, KDD.

[20]  Sebastian Bruch,et al.  TF-Ranking: Scalable TensorFlow Library for Learning-to-Rank , 2018, KDD.

[21]  Elad Eban,et al.  Scalable Learning of Non-Decomposable Objectives , 2016, AISTATS.

[22]  Tao Xu,et al.  Applying Deep Learning to Airbnb Search , 2018, KDD.

[23]  Le Song,et al.  Recurrent Coevolutionary Latent Feature Processes for Continuous-Time Recommendation , 2016, DLRS@RecSys.

[24]  Li Wei,et al.  Recommending what video to watch next: a multitask ranking system , 2019, RecSys.

[25]  Tie-Yan Liu,et al.  Learning to rank for information retrieval , 2009, SIGIR.

[26]  Jürgen Ziegler,et al.  Sequential User-based Recurrent Neural Network Recommendations , 2017, RecSys.

[27]  Xing Xie,et al.  xDeepFM: Combining Explicit and Implicit Feature Interactions for Recommender Systems , 2018, KDD.

[28]  Yi Tay,et al.  Deep Learning based Recommender System: A Survey and New Perspectives , 2018 .

[29]  Gediminas Adomavicius,et al.  Context-aware recommender systems , 2008, RecSys '08.

[30]  Heng-Tze Cheng,et al.  Wide & Deep Learning for Recommender Systems , 2016, DLRS@RecSys.