Boosting Factorization Machines via Saliency-Guided Mixup

—Factorization machines (FMs) are widely used in recommender systems due to their adaptability and ability to learn from sparse data. However, for the ubiquitous non-interactive features in sparse data, existing FMs can only estimate the parameters corresponding to these features via the inner product of their embeddings. Undeniably, they cannot learn the direct interactions of these features, which limits the model’s expressive power. To this end, we first present MixFM, inspired by Mixup, to generate auxiliary training data to boost FMs. Unlike existing augmentation strategies that require labor costs and expertise to collect additional information such as position and fields, these extra data generated by MixFM only by the convex combination of the raw ones without any professional knowledge support. More importantly, if the parent samples to be mixed have non-interactive features, MixFM will establish their direct interactions. Second, considering that MixFM may generate redundant or even detrimental instances, we further put forward a novel Factorization Machine powered by Saliency-guided Mixup (denoted as SMFM). Guided by the customized saliency, SMFM can generate more informative neighbor data. Through theoretical analysis, we prove that the proposed methods minimize the upper bound of the generalization error, which hold a beneficial effect on enhancing FMs. Significantly, we give the first generalization bound of FM, implying the generalization requires more data and a smaller embedding size under the sufficient representation capability. Finally, extensive experiments on five datasets confirm that our approaches are superior to baselines. Besides, the results show that ”poisoning” mixed data is likewise beneficial to the FM variants.

[1]  Enhong Chen,et al.  Fight Fire with Fire: Towards Robust Recommender Systems via Adversarial Poisoning Training , 2021, SIGIR.

[2]  Enhong Chen,et al.  xLightFM: Extremely Memory-Efficient Factorization Machine , 2021, SIGIR.

[3]  Qing-Long Zhang,et al.  Group-CAM: Group Score-Weighted Visual Explanations for Deep Convolutional Networks , 2021, ArXiv.

[4]  Defu Lian,et al.  Automated Creative Optimization for E-Commerce Advertising , 2021, WWW.

[5]  Shuiwang Ji,et al.  Towards Improved and Interpretable Deep Metric Learning via Attentive Grouping , 2020, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[6]  James Y. Zou,et al.  How Does Mixup Help With Robustness and Generalization? , 2020, ICLR.

[7]  Guang Cheng,et al.  On the Generalization Properties of Adversarial Training , 2020, AISTATS.

[8]  Sabine E. Susstrunk,et al.  On the Loss Landscape of Adversarial Training: Identifying Challenges and How to Overcome Them , 2020, NeurIPS.

[9]  Yun Fu,et al.  Heterogeneous Recommendation via Deep Low-Rank Sparse Collective Factorization , 2020, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[10]  Xing Xie,et al.  LightRec: A Memory and Search-Efficient Recommender System , 2020, WWW.

[11]  Nathan Srebro,et al.  Dropout: Explicit Forms and Capacity Control , 2020, ICML.

[12]  Mohan S. Kankanhalli,et al.  Attacks Which Do Not Kill Training Make Adversarial Learning Stronger , 2020, ICML.

[13]  Mao Ye,et al.  MaxUp: A Simple Way to Improve Generalization of Neural Network Training , 2020, ArXiv.

[14]  Xiangnan He,et al.  LightGCN: Simplifying and Powering Graph Convolution Network for Recommendation , 2020, SIGIR.

[15]  Linpeng Huang,et al.  Adaptive Factorization Network: Learning Adaptive-Order Feature Interactions , 2019, AAAI.

[16]  Junlin Zhang,et al.  FiBiNET: combining feature importance and bilinear feature interaction for click-through rate prediction , 2019, RecSys.

[17]  Kun He,et al.  Improving the Generalization of Adversarial Training with Domain Adaptation , 2018, ICLR.

[18]  Zibin Zheng,et al.  Factorization Machine Based Service Recommendation on Heterogeneous Information Networks , 2018, 2018 IEEE International Conference on Web Services (ICWS).

[19]  Xiaoyu Du,et al.  Adversarial Personalized Ranking for Recommendation , 2018, SIGIR.

[20]  Chenglin Li,et al.  Android Malware Detection Based on Factorization Machine , 2018, IEEE Access.

[21]  Rui Liu,et al.  Discrete Factorization Machines for Fast Feature-based Recommendation , 2018, IJCAI.

[22]  Priyanka Bhatt,et al.  Robust Factorization Machines for User Response Prediction , 2018, WWW.

[23]  Xing Xie,et al.  xDeepFM: Combining Explicit and Implicit Feature Interactions for Recommender Systems , 2018, KDD.

[24]  Matthew D. Hoffman,et al.  Variational Autoencoders for Collaborative Filtering , 2018, WWW.

[25]  Shuai Wang,et al.  Contextual and Position-Aware Factorization Machines for Sentiment Classification , 2018, ArXiv.

[26]  Bolei Zhou,et al.  Interpreting Deep Visual Representations via Network Dissection , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[27]  Hongyi Zhang,et al.  mixup: Beyond Empirical Risk Minimization , 2017, ICLR.

[28]  Gang Fu,et al.  Deep & Cross Network for Ad Click Predictions , 2017, ADKDD@KDD.

[29]  Tat-Seng Chua,et al.  Neural Factorization Machines for Sparse Predictive Analytics , 2017, SIGIR.

[30]  Jianhui Chen,et al.  Convex Factorization Machine for Toxicogenomics Prediction , 2017, KDD.

[31]  Tat-Seng Chua,et al.  Attentional Factorization Machines: Learning the Weight of Feature Interactions via Attention Networks , 2017, IJCAI.

[32]  Philip S. Yu,et al.  Learning from Multi-View Multi-Way Data via Structural Factorization Machines , 2017, WWW.

[33]  Yunming Ye,et al.  DeepFM: A Factorization-Machine based Neural Network for CTR Prediction , 2017, IJCAI.

[34]  Philip S. Yu,et al.  Multilinear Factorization Machines for Multi-Task Multi-View Learning , 2017, WSDM.

[35]  Nicholas Jing Yuan,et al.  Regularized Content-Aware Tensor Factorization Meets Temporal-Aware Location Recommendation , 2016, 2016 IEEE 16th International Conference on Data Mining (ICDM).

[36]  Chih-Jen Lin,et al.  Field-aware Factorization Machines for CTR Prediction , 2016, RecSys.

[37]  Dong Yu,et al.  Deep Crossing: Web-Scale Modeling without Manually Crafted Combinatorial Features , 2016, KDD.

[38]  Naonori Ueda,et al.  Higher-Order Factorization Machines , 2016, NIPS.

[39]  F. Maxwell Harper,et al.  The MovieLens Datasets: History and Context , 2016, TIIS.

[40]  Ananthram Swami,et al.  The Limitations of Deep Learning in Adversarial Settings , 2015, 2016 IEEE European Symposium on Security and Privacy (EuroS&P).

[41]  Philip S. Yu,et al.  Multi-view Machines , 2015, WSDM.

[42]  Nuria Oliver,et al.  Frappe: Understanding the Usage and Perception of Mobile App Recommendations In-The-Wild , 2015, ArXiv.

[43]  Tong Zhang,et al.  Gradient boosting factorization machines , 2014, RecSys '14.

[44]  Alexandros Karatzoglou,et al.  Gaussian process factorization machines for context-aware recommendations , 2014, SIGIR.

[45]  David Lo,et al.  Predicting response in mobile advertising with hierarchical importance-aware factorization machine , 2014, WSDM.

[46]  Andrew Zisserman,et al.  Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps , 2013, ICLR.

[47]  Jiming Liu,et al.  Proceedings of the Twenty-Third International Joint Conference on Artificial Intelligence Social Collaborative Filtering by Trust , 2022 .

[48]  Ameet Talwalkar,et al.  Foundations of Machine Learning , 2012, Adaptive computation and machine learning.

[49]  Lars Schmidt-Thieme,et al.  Fast context-aware recommendations with factorization machines , 2011, SIGIR.

[50]  Steffen Rendle,et al.  Factorization Machines , 2010, 2010 IEEE International Conference on Data Mining.

[51]  Peter L. Bartlett,et al.  Rademacher and Gaussian Complexities: Risk Bounds and Structural Results , 2003, J. Mach. Learn. Res..

[52]  Ding-Xuan Zhou,et al.  The covering number in learning theory , 2002, J. Complex..

[53]  Anselm Blumer,et al.  Learnability Models and Vapnik-Chervonenkis Combinatorics , 1992 .