Graph Neural Pre-training for Enhancing Recommendations using Side Information

Leveraging the side information associated with entities (i.e. users and items) to enhance the performance of recommendation systems has been widely recognized as an important modelling dimension. While many existing approaches focus on the integration scheme to incorporate entity side information – by combining the recommendation loss function with an extra side information-aware loss – in this paper, we propose instead a novel pre-training scheme for leveraging the side information. In particular, we first pre-train a representation model using the side information of the entities, and then fine-tune it using an existing general representation-based recommendation model. Specifically, we propose two pre-training models, named GCN-P and COM-P, by considering the entities and their relations constructed from side information as two different types of graphs respectively, to pre-train entity embeddings. For the GCN-P model, two single-relational graphs are constructed from all the users’ and items’ side information respectively, to pre-train entity representations by using the Graph Convolutional Networks. For the COM-P model, two multi-relational graphs are constructed to pre-train the entity representations by using the Composition-based Graph Convolutional Networks. An extensive evaluation of our pre-training models fine-tuned under four general representation-based recommender models, i.e. MF, NCF, NGCF and LightGCN, shows that effectively pre-training embeddings with both the user’s and item’s side information can significantly improve these original models in terms of both effectiveness and stability. ACM Reference Format: Zaiqiao Meng, Siwei Liu, Craig Macdonald, and Iadh Ounis. 2021. Graph Neural Pre-training for Enhancing Recommendations using Side Information. In Proceedings of ACM Conference (Conference’17). ACM, New York, NY, USA, 13 pages. https://doi.org/10.1145/nnnnnnn.nnnnnnn ∗ Both authors contributed equally to this work. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from permissions@acm.org. Conference’17, July 2017, Washington, DC, USA © 2021 Association for Computing Machinery. ACM ISBN 978-x-xxxx-xxxx-x/YY/MM. . . $15.00 https://doi.org/10.1145/nnnnnnn.nnnnnnn

[1]  Tat-Seng Chua,et al.  Neural Collaborative Filtering , 2017, WWW.

[2]  Hans-Peter Kriegel,et al.  A Three-Way Model for Collective Learning on Multi-Relational Data , 2011, ICML.

[3]  Xiao Huang,et al.  Towards Deeper Graph Neural Networks with Differentiable Group Normalization , 2020, NeurIPS.

[4]  Philip S. Yu,et al.  BasConv: Aggregating Heterogeneous Interactions for Basket Recommendation with Graph Convolutional Neural Network , 2020, SDM.

[5]  A. Tordai,et al.  Modeling Relational Data with Graph Convolutional Networks , 2017 .

[6]  Yiqun Liu,et al.  Jointly Learning Explainable Rules for Recommendation with Knowledge Graph , 2019, WWW.

[7]  Joemon M. Jose,et al.  A Simple Convolutional Generative Network for Next Item Recommendation , 2018, WSDM.

[8]  Tat-Seng Chua,et al.  Neural Graph Collaborative Filtering , 2019, SIGIR.

[9]  Elena Smirnova,et al.  Meta-Prod2Vec: Product Embeddings Using Side-Information for Recommendation , 2016, RecSys.

[10]  Xu Chen,et al.  Joint Representation Learning for Top-N Recommendation with Heterogeneous Information Sources , 2017, CIKM.

[11]  Cao Xiao,et al.  FastGCN: Fast Learning with Graph Convolutional Networks via Importance Sampling , 2018, ICLR.

[12]  Kenta Oono,et al.  Optimization and Generalization Analysis of Transduction through Gradient Boosting and Application to Multi-scale Graph Neural Networks , 2020, NeurIPS.

[13]  Max Welling,et al.  Modeling Relational Data with Graph Convolutional Networks , 2017, ESWC.

[14]  Lina Yao,et al.  Quaternion Knowledge Graph Embeddings , 2019, NeurIPS.

[15]  Iadh Ounis,et al.  A Heterogeneous Graph Neural Model for Cold-start Recommendation , 2020, SIGIR.

[16]  Jian-Yun Nie,et al.  RotatE: Knowledge Graph Embedding by Relational Rotation in Complex Space , 2018, ICLR.

[17]  Yixin Cao,et al.  KGAT: Knowledge Graph Attention Network for Recommendation , 2019, KDD.

[18]  Yuan He,et al.  Graph Neural Networks for Social Recommendation , 2019, WWW.

[19]  Zaiqiao Meng,et al.  Bayesian Deep Collaborative Matrix Factorization , 2019, AAAI.

[20]  Hengrui Zhang,et al.  Stacked Mixed-Order Graph Convolutional Networks for Collaborative Filtering , 2020, SDM.

[21]  Jure Leskovec,et al.  Inductive Representation Learning on Large Graphs , 2017, NIPS.

[22]  Shangsong Liang,et al.  Semi-supervisedly Co-embedding Attributed Networks , 2019, NeurIPS.

[23]  Yuhong Guo,et al.  Learning Discriminative Recommendation Systems with Side Information , 2017, IJCAI.

[24]  Carl Allen,et al.  Benchmark and Best Practices for Biomedical Knowledge Graph Embeddings , 2020, BIONLP.

[25]  Yongdong Zhang,et al.  Graph Convolution Machine for Context-aware Recommender System , 2020, ArXiv.

[26]  Lars Schmidt-Thieme,et al.  BPR: Bayesian Personalized Ranking from Implicit Feedback , 2009, UAI.

[27]  Vikram Nitin,et al.  Composition-based Multi-Relational Graph Convolutional Networks , 2020, ICLR.

[28]  Zhiyuan Liu,et al.  Graph Neural Networks with Generated Parameters for Relation Extraction , 2019, ACL.

[29]  Walid Krichene,et al.  Neural Collaborative Filtering vs. Matrix Factorization Revisited , 2020, RecSys.

[30]  Meng Wang,et al.  Revisiting Graph based Collaborative Filtering: A Linear Residual Graph Convolutional Network Approach , 2020, AAAI.

[31]  Hong Chen,et al.  Pre-Training Graph Neural Networks for Cold-Start Users and Items Representation , 2020, WSDM.

[32]  Yi Tay,et al.  Deep Learning based Recommender System: A Survey and New Perspectives , 2018 .

[33]  Julian J. McAuley,et al.  Self-Attentive Sequential Recommendation , 2018, 2018 IEEE International Conference on Data Mining (ICDM).

[34]  Max Welling,et al.  Semi-Supervised Classification with Graph Convolutional Networks , 2016, ICLR.

[35]  Craig MacDonald,et al.  A Deep Recurrent Collaborative Filtering Framework for Venue Recommendation , 2017, CIKM.

[36]  Xiangliang Zhang,et al.  Co-Embedding Attributed Networks , 2019, WSDM.

[37]  Linmei Hu,et al.  Graph Neural News Recommendation with Long-term and Short-term Interest Modeling , 2020, Inf. Process. Manag..

[38]  Xiangnan He,et al.  MMGCN: Multi-modal Graph Convolution Network for Personalized Recommendation of Micro-video , 2019, ACM Multimedia.

[39]  Zhiwei Wang,et al.  Recommender Systems with Heterogeneous Side Information , 2019, WWW.

[40]  Jie Liu,et al.  Representing and Recommending Shopping Baskets with Complementarity, Compatibility and Loyalty , 2018, CIKM.

[41]  George Karypis,et al.  Sparse linear methods with side information for top-n recommendations , 2012, RecSys.

[42]  Qing Guo,et al.  Exploiting Side Information for Recommendation , 2019, ICWE.

[43]  M. de Rijke,et al.  A Collective Variational Autoencoder for Top-N Recommendation with Side Information , 2018, DLRS@RecSys.

[44]  Min Yang,et al.  A Novel Top-N Recommendation Approach Based on Conditional Variational Auto-Encoder , 2019, PAKDD.

[45]  Jure Leskovec,et al.  Graph Convolutional Policy Network for Goal-Directed Molecular Graph Generation , 2018, NeurIPS.

[46]  Yongdong Zhang,et al.  LightGCN: Simplifying and Powering Graph Convolution Network for Recommendation , 2020, SIGIR.

[47]  Jure Leskovec,et al.  How Powerful are Graph Neural Networks? , 2018, ICLR.

[48]  Mark Coates,et al.  Multi-graph Convolution Collaborative Filtering , 2019, 2019 IEEE International Conference on Data Mining (ICDM).

[49]  Dit-Yan Yeung,et al.  Relational Stacked Denoising Autoencoder for Tag Recommendation , 2015, AAAI.

[50]  Bin Shen,et al.  Collaborative Memory Network for Recommendation Systems , 2018, SIGIR.

[51]  Chen Gao,et al.  Price-aware Recommendation with Graph Convolutional Networks , 2020, 2020 IEEE 36th International Conference on Data Engineering (ICDE).

[52]  Fan Liu,et al.  An Attribute-Aware Attentive GCN Model for Attribute Missing in Recommendation , 2020, IEEE Transactions on Knowledge and Data Engineering.

[53]  Sunho Park,et al.  Hierarchical Bayesian Matrix Factorization with Side Information , 2013, IJCAI.

[54]  Iadh Ounis,et al.  A Hybrid Conditional Variational Autoencoder Model for Personalised Top-n Recommendation , 2020, ICTIR.

[55]  Samy Bengio,et al.  Cluster-GCN: An Efficient Algorithm for Training Deep and Large Graph Convolutional Networks , 2019, KDD.

[56]  Sheng Li,et al.  Deep Collaborative Filtering via Marginalized Denoising Auto-encoder , 2015, CIKM.

[57]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[58]  Jason Weston,et al.  Translating Embeddings for Modeling Multi-relational Data , 2013, NIPS.