JointCTR: a joint CTR prediction framework combining feature interaction and sequential behavior learning

Click-through rate (CTR) is a positive feedback of user preferences or product purchases, and its small increase can bring huge benefits. Therefore, CTR prediction plays a key role in computing advertising and recommendation systems. Research shows that the accuracy of CTR prediction models is closely related to the input features. Existing related models usually focus on certain aspects of features, such as second-order interactions or temporal changes, and ignore the diversity of features. In this paper, a modular click-through rate prediction framework JointCTR is proposed. The framework integrates four types of prediction models, each of which learns different types of features, including original features, embedded features, interactive features, and sequential features. According to actual application scenarios, these models can be assembled or removed flexibly, and models of the same type can be replaced by each other. In order to avoid the impact of data diversity, sparsity, and high-dimensional features, an embedding layer is added to the framework to achieve unified embedding processing for different data types. To learn sequential behaviors, we propose a model SeqCTR based on the attention mechanism to capture the dynamics of user interests in the framework. To better learn high-order features, we apply HorderCTR proposed in our previous work to the framework to automatically identify high-value feature combinations. Extensive experiments on four public datasets show the effectiveness of the proposed framework, which yields competitive performance compared to state-of-the-art models (the AUC increases by + 0.24% on MovieLens-1 M and 0.85% on Criteo). Almost all different types of popular models can be assembled into it, that shows the flexibility and scalability of the framework.

[1]  Chang Zhou,et al.  Deep Interest Evolution Network for Click-Through Rate Prediction , 2018, AAAI.

[2]  Lucas Theis,et al.  Addressing delayed feedback for continuous training with neural networks in CTR prediction , 2019, RecSys.

[3]  Xing Xie,et al.  xDeepFM: Combining Explicit and Implicit Feature Interactions for Recommender Systems , 2018, KDD.

[4]  Jianfeng Gao,et al.  Scalable training of L1-regularized log-linear models , 2007, ICML '07.

[5]  Ed H. Chi,et al.  DCN V2: Improved Deep & Cross Network and Practical Lessons for Web-scale Learning to Rank Systems , 2021, WWW.

[6]  Kai Liu,et al.  Learning Piece-wise Linear Models from Large Scale Data for Ad Click Prediction , 2017, ArXiv.

[7]  Xue Zhao,et al.  An Intelligent Field-Aware Factorization Machine Model , 2017, DASFAA.

[8]  Qing He,et al.  Warm Up Cold-start Advertisements: Improving CTR Predictions via Learning to Learn ID Embeddings , 2019, SIGIR.

[9]  Yang Song,et al.  Multi-Rate Deep Learning for Temporal Recommendation , 2016, SIGIR.

[10]  Jian Zhao,et al.  Operation-aware Neural Networks for User Response Prediction , 2019, Neural Networks.

[11]  Junlin Zhang,et al.  FiBiNET: combining feature importance and bilinear feature interaction for click-through rate prediction , 2019, RecSys.

[12]  R. Polikar,et al.  Ensemble based systems in decision making , 2006, IEEE Circuits and Systems Magazine.

[13]  Richard Socher,et al.  Dynamic Memory Networks for Visual and Textual Question Answering , 2016, ICML.

[14]  Yunming Ye,et al.  DeepFM: A Factorization-Machine based Neural Network for CTR Prediction , 2017, IJCAI.

[15]  Chang Zhou,et al.  ATRank: An Attention-Based User Behavior Modeling Framework for Recommendation , 2017, AAAI.

[16]  Alexandros Karatzoglou,et al.  Session-based Recommendations with Recurrent Neural Networks , 2015, ICLR.

[17]  Joaquin Quiñonero Candela,et al.  Practical Lessons from Predicting Clicks on Ads at Facebook , 2014, ADKDD'14.

[18]  Gang Fu,et al.  Deep & Cross Network for Ad Click Predictions , 2017, ADKDD@KDD.

[19]  Guorui Zhou,et al.  Deep Interest Network for Click-Through Rate Prediction , 2017, KDD.

[20]  Tao Deng,et al.  Learning Compositional, Visual and Relational Representations for CTR Prediction in Sponsored Search , 2019, CIKM.

[21]  Steffen Rendle,et al.  Factorization Machines with libFM , 2012, TIST.

[22]  Liang Wang,et al.  Fi-GNN: Modeling Feature Interactions via Graph Neural Networks for CTR Prediction , 2019, CIKM.

[23]  Wei Guo,et al.  Order-aware Embedding Neural Network for CTR Prediction , 2019, SIGIR.

[24]  Feng Yu,et al.  A Dynamic Recurrent Model for Next Basket Recommendation , 2016, SIGIR.

[25]  Huichuan Duan,et al.  A CTR prediction model based on user interest via attention mechanism , 2020, Applied Intelligence.

[26]  Jun Wang,et al.  Product-Based Neural Networks for User Response Prediction , 2016, 2016 IEEE 16th International Conference on Data Mining (ICDM).

[27]  Xiao Bai,et al.  Position-Aware Deep Character-Level CTR Prediction for Sponsored Search , 2019 .

[28]  Chih-Jen Lin,et al.  Field-aware Factorization Machines for CTR Prediction , 2016, RecSys.

[29]  Lukasz Kaiser,et al.  Attention is All you Need , 2017, NIPS.

[30]  Heng-Tze Cheng,et al.  Wide & Deep Learning for Recommender Systems , 2016, DLRS@RecSys.

[31]  Tat-Seng Chua,et al.  Neural Collaborative Filtering , 2017, WWW.

[32]  Tat-Seng Chua,et al.  Attentional Factorization Machines: Learning the Weight of Feature Interactions via Attention Networks , 2017, IJCAI.

[33]  Xiaohui Zhao,et al.  A Hierarchical Attention Model for CTR Prediction Based on User Interest , 2020, IEEE Systems Journal.

[34]  Cairong Yan,et al.  Modeling low- and high-order feature interactions with FM and self-attention network , 2020, Applied Intelligence.

[35]  Bin Liu,et al.  Feature Generation by Convolutional Neural Network for Click-Through Rate Prediction , 2019, WWW.

[36]  Jun Wang,et al.  Deep Learning over Multi-field Categorical Data - - A Case Study on User Response Prediction , 2016, ECIR.

[37]  Jian Tang,et al.  AutoInt: Automatic Feature Interaction Learning via Self-Attentive Neural Networks , 2018, CIKM.

[38]  Tat-Seng Chua,et al.  Neural Factorization Machines for Sparse Predictive Analytics , 2017, SIGIR.