A Large Scale Prediction Engine for App Install Clicks and Conversions

Predicting the probability of users clicking on app install ads and installing those apps comes with its own specific challenges. In this paper, we describe (a) how we built a scalable machine learning pipeline from scratch to predict the probability of users clicking and installing apps in response to ad impressions, (b) the novel features we developed to improve our model performance, (c) the training and scoring pipelines that were put into production, (d) our A/B testing process along with the metrics used to determine significant improvements, and (e) the results of our experiments. Our algorithmic improvements resulted in a 3X improvement in satisfaction for app install advertisers on our ad platform. In addition, we dive into how sequential model training, deep learning, and transfer learning resulted in a further 7% lift in conversion rate and 11% lift in revenue. Finally, we share the scientific, data-related, and product-related challenges that we encountered -- we expect others across the industry would greatly benefit from these considerations and our experiences when they kick-start similar efforts.

[1]  Andrei Z. Broder.  Computational advertising , 2008, SODA '08.

[2]  Jürgen Schmidhuber,et al.  Deep learning in neural networks: An overview , 2014, Neural Networks.

[3]  Joaquin Quiñonero Candela,et al.  Practical Lessons from Predicting Clicks on Ads at Facebook , 2014, ADKDD'14.

[4]  Steffen Rendle,et al.  Factorization Machines , 2010, 2010 IEEE International Conference on Data Mining.

[5]  Martin Wattenberg,et al.  Ad click prediction: a view from the trenches , 2013, KDD.

[6]  Joaquin Quiñonero Candela,et al.  Web-Scale Bayesian Click-Through rate Prediction for Sponsored Search Advertising in Microsoft's Bing Search Engine , 2010, ICML.

[7]  Rajiv Khanna,et al.  Estimating rates of rare events with multiple hierarchies through scalable log-linear models , 2010, KDD '10.

[8]  Matthew Richardson,et al.  Predicting clicks: estimating the click-through rate for new ads , 2007, WWW '07.

[9]  S. Hyakin,et al.  Neural Networks: A Comprehensive Foundation , 1994 .

[10]  Rómer Rosales,et al.  Post-click conversion modeling and analysis for non-guaranteed delivery display advertising , 2012, WSDM '12.

[11]  Yoshua Bengio,et al.  How transferable are features in deep neural networks? , 2014, NIPS.

[12]  Dong Yu,et al.  Deep Crossing: Web-Scale Modeling without Manually Crafted Combinatorial Features , 2016, KDD.

[13]  Martín Abadi,et al.  TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems , 2016, ArXiv.

[14]  Andrei Z. Broder,et al.  Estimating rates of rare events at multiple resolutions , 2007, KDD '07.

[15]  Deepak Agarwal,et al.  Temporal multi-hierarchy smoothing for estimating rates of rare events , 2011, KDD.

[16]  Yee Whye Teh,et al.  A Fast Learning Algorithm for Deep Belief Nets , 2006, Neural Computation.