Open Benchmarking for Click-Through Rate Prediction

Click-through rate (CTR) prediction is a critical task for many applications, as its accuracy has a direct impact on user experience and platform revenue. In recent years, CTR prediction has been widely studied in both academia and industry, resulting in a wide variety of CTR prediction models. Unfortunately, there is still a lack of standardized benchmarks and uniform evaluation protocols for CTR prediction research. This leads to non-reproducible or even inconsistent experimental results among existing studies, which largely limit the practical value and potential impact of their research. In this work, we aim to perform open benchmarking for CTR prediction and present a rigorous comparison of different models in a reproducible manner. To this end, we ran over 7,000 experiments for more than 12,000 GPU hours in total to re-evaluate 24 existing models on multiple dataset settings. Surprisingly, our experiments show that with sufficient hyper-parameter search and model tuning, many deep models have smaller differences than expected. The results also reveal that making real progress on the modeling of CTR prediction is indeed a very challenging research task. We believe that our benchmarking work could not only allow researchers to gauge the effectiveness of new models conveniently but also make them fairly compare with the state of the arts. We have publicly released the benchmarking tools, evaluation protocols, and experimental settings of our work to promote reproducible research in this field.

[1]  Paul Covington,et al.  Deep Neural Networks for YouTube Recommendations , 2016, RecSys.

[2]  Yu Wang,et al.  Telepath: Understanding Users from a Human Vision Perspective in Large-Scale Recommender Systems , 2017, AAAI.

[3]  Chih-Jen Lin,et al.  Field-aware Factorization Machines for CTR Prediction , 2016, RecSys.

[4]  Gang Fu,et al.  Deep & Cross Network for Ad Click Predictions , 2017, ADKDD@KDD.

[5]  Guorui Zhou,et al.  Deep Interest Network for Click-Through Rate Prediction , 2017, KDD.

[6]  Bin Liu,et al.  Feature Generation by Convolutional Neural Network for Click-Through Rate Prediction , 2019, WWW.

[7]  Zhe Zhao,et al.  Modeling Task Relationships in Multi-task Learning with Multi-gate Mixture-of-Experts , 2018, KDD.

[8]  Zhenguo Li,et al.  An Embedding Learning Framework for Numerical Features in CTR Prediction , 2020, KDD.

[9]  Martin Wattenberg,et al.  Ad click prediction: a view from the trenches , 2013, KDD.

[10]  Chang Zhou,et al.  Deep Interest Evolution Network for Click-Through Rate Prediction , 2018, AAAI.

[11]  Priyanka Bhatt,et al.  Robust Factorization Machines for User Response Prediction , 2018, WWW.

[12]  Tianqi Chen,et al.  XGBoost: A Scalable Tree Boosting System , 2016, KDD.

[13]  Yunming Ye,et al.  DeepFM: A Factorization-Machine based Neural Network for CTR Prediction , 2017, IJCAI.

[14]  Joaquin Quiñonero Candela,et al.  Practical Lessons from Predicting Clicks on Ads at Facebook , 2014, ADKDD'14.

[15]  Naonori Ueda,et al.  Higher-Order Factorization Machines , 2016, NIPS.

[16]  Chen Lin,et al.  FLEN: Leveraging Field for Scalable CTR Prediction , 2019, ArXiv.

[17]  Dong Yu,et al.  Deep Crossing: Web-Scale Modeling without Manually Crafted Combinatorial Features , 2016, KDD.

[18]  Lina Yao,et al.  Holographic Factorization Machines for Recommendation , 2019, AAAI.

[19]  Steffen Rendle,et al.  Factorization Machines , 2010, 2010 IEEE International Conference on Data Mining.

[20]  Heng-Tze Cheng,et al.  Wide & Deep Learning for Recommender Systems , 2016, DLRS@RecSys.

[21]  Omer Levy,et al.  GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding , 2018, BlackboxNLP@EMNLP.

[22]  Jian Zhao,et al.  Operation-aware Neural Networks for User Response Prediction , 2019, Neural Networks.

[23]  Lei Zheng,et al.  Lifelong Sequential Modeling with Personalized Memorization for User Response Prediction , 2019, SIGIR.

[24]  Jun Wang,et al.  Product-Based Neural Networks for User Response Prediction , 2016, 2016 IEEE 16th International Conference on Data Mining (ICDM).

[25]  Li Fei-Fei,et al.  ImageNet: A large-scale hierarchical image database , 2009, CVPR.

[26]  Yu Geng,et al.  Accurate and Interpretable Factorization Machines , 2019, AAAI.

[27]  Ming Zhao,et al.  Progressive Layered Extraction (PLE): A Novel Multi-Task Learning (MTL) Model for Personalized Recommendations , 2020, RecSys.

[28]  Ge Chen,et al.  Interaction-aware Factorization Machines for Recommender Systems , 2019, AAAI.

[29]  Liang Wang,et al.  Fi-GNN: Modeling Feature Interactions via Graph Neural Networks for CTR Prediction , 2019, CIKM.

[30]  Linpeng Huang,et al.  Adaptive Factorization Network: Learning Adaptive-Order Feature Interactions , 2019, AAAI.

[31]  Feng Yu,et al.  A Convolutional Click Prediction Model , 2015, CIKM.

[32]  Xiuqiang He,et al.  UNBERT: User-News Matching BERT for News Recommendation , 2021, IJCAI.

[33]  Jie Yang,et al.  Are We Evaluating Rigorously? Benchmarking Recommendation for Reproducible Evaluation and Fair Comparison , 2020, RecSys.

[34]  Yu Zhang,et al.  Image Matters: Visually Modeling User Behaviors Using Advanced Model Server , 2017, CIKM.

[35]  Yi Tay,et al.  Deep Learning based Recommender System: A Survey and New Perspectives , 2018 .

[36]  Matthew Richardson,et al.  Predicting clicks: estimating the click-through rate for new ads , 2007, WWW '07.

[37]  Yu Sun,et al.  Field-weighted Factorization Machines for Click-Through Rate Prediction in Display Advertising , 2018, WWW.

[38]  Tat-Seng Chua,et al.  Why Do We Click: Visual Impression-aware News Recommendation , 2021, ACM Multimedia.

[39]  Jian Tang,et al.  AutoInt: Automatic Feature Interaction Learning via Self-Attentive Neural Networks , 2018, CIKM.

[40]  Tat-Seng Chua,et al.  Neural Factorization Machines for Sparse Predictive Analytics , 2017, SIGIR.

[41]  Li Li,et al.  Deep Spatio-Temporal Neural Networks for Click-Through Rate Prediction , 2019, KDD.

[42]  Chao Wang,et al.  Adversarial Multimodal Representation Learning for Click-Through Rate Prediction , 2020, WWW.

[43]  Tat-Seng Chua,et al.  Attentional Factorization Machines: Learning the Weight of Feature Interactions via Attention Networks , 2017, IJCAI.

[44]  Yang Chen,et al.  Interpretable Click-Through Rate Prediction through Hierarchical Attention , 2020, WSDM.

[45]  Xiao Ma,et al.  Entire Space Multi-Task Model: An Effective Approach for Estimating Post-Click Conversion Rate , 2018, SIGIR.

[46]  Jieming Zhu,et al.  Ensembled CTR Prediction via Knowledge Distillation , 2020, CIKM.

[47]  Junlin Zhang,et al.  FiBiNET: combining feature importance and bilinear feature interaction for click-through rate prediction , 2019, RecSys.

[48]  Lina Yao,et al.  Deep Learning Based Recommender System , 2017, ACM Comput. Surv..

[49]  Dietmar Jannach,et al.  Are we really making much progress? A worrying analysis of recent neural recommendation approaches , 2019, RecSys.

[50]  Weiwei Deng,et al.  Model Ensemble for Click Prediction in Bing Search Ads , 2017, WWW.

[51]  Keping Yang,et al.  Deep Session Interest Network for Click-Through Rate Prediction , 2019, IJCAI.

[52]  Xing Xie,et al.  xDeepFM: Combining Explicit and Implicit Feature Interactions for Recommender Systems , 2018, KDD.

[53]  Ming Wu,et al.  Learning Feature Interactions with Lorentzian Factorization Machine , 2019, AAAI.