Semi-deterministic and Contrastive Variational Graph Autoencoder for Recommendation

Variational AutoEncoder (VAE) is a popular deep generative framework with a solid theoretical basis. There are many research efforts on improving VAE. Among the existing works, a recently proposed deterministic Regularized AutoEncoder (RAE) provides a new scheme for generative modeling. RAE fixes the variance of the inferred Gaussian approximate posterior distribution as a hyperparameter, and substitutes the stochastic encoder by injecting noise into the input of a deterministic decoder. However, the deterministic RAE has three limitations: 1) RAE needs to fit the variance; 2) RAE requires ex-post density estimation to ensure sample quality; 3) RAE employs an additional gradient regularization to ensure training smoothness. Thus, it raises an interesting research question: Can we maintain the flexibility of variational inference while simplifying VAE, and at the same time ensuring a smooth training process to obtain good generative performance? Based on the above motivation, in this paper, we propose a novel Semi-deterministic and Contrastive Variational Graph autoencoder (SCVG) for item recommendation. The core design of SCVG is to learn the variance of the approximate Gaussian posterior distribution in a semi-deterministic manner by aggregating inferred mean vectors from other connected nodes via graph convolution operation. We analyze the expressive power of SCVG for the Weisfeiler-Lehman graph isomorphism test, and we deduce the simplified form of the evidence lower bound of SCVG. Besides, we introduce an efficient contrastive regularization instead of gradient regularization. We empirically show that the contrastive regularization makes learned user/item latent representation more personalized and helps to smooth the training process. We conduct extensive experiments on three real-world datasets to show the superiority of our model over state-of-the-art methods for the item recommendation task. Codes are available at https://github.com/syxkason/SCVG.

[1]  Hongxia Yang,et al.  Learning Disentangled Representations for Recommendation , 2019, NeurIPS.

[2]  Nikos Komodakis,et al.  GraphVAE: Towards Generation of Small Graphs Using Variational Autoencoders , 2018, ICANN.

[3]  Max Welling,et al.  Auto-Encoding Variational Bayes , 2013, ICLR.

[4]  Kilian Q. Weinberger,et al.  Simplifying Graph Convolutional Networks , 2019, ICML.

[5]  Alexander A. Alemi,et al.  Deep Variational Information Bottleneck , 2017, ICLR.

[6]  Yisong Yue,et al.  Iterative Amortized Inference , 2018, ICML.

[7]  Stephen Roberts,et al.  Towards a Theoretical Understanding of the Robustness of Variational Autoencoders , 2020, ArXiv.

[8]  M. de Rijke,et al.  A Collective Variational Autoencoder for Top-N Recommendation with Side Information , 2018, DLRS@RecSys.

[9]  Max Welling,et al.  Variational Graph Auto-Encoders , 2016, ArXiv.

[10]  Simon King,et al.  A Vector Quantized Variational Autoencoder (VQ-VAE) Autoregressive Neural $F_0$ Model for Statistical Parametric Speech Synthesis , 2020, IEEE/ACM Transactions on Audio, Speech, and Language Processing.

[11]  Minlie Huang,et al.  Long and Diverse Text Generation with Planning-based Hierarchical Variational Model , 2019, EMNLP.

[12]  David Duvenaud,et al.  Inference Suboptimality in Variational Autoencoders , 2018, ICML.

[13]  Naftali Tishby,et al.  Deep learning and the information bottleneck principle , 2015, 2015 IEEE Information Theory Workshop (ITW).

[14]  Christopher Burgess,et al.  beta-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework , 2016, ICLR 2016.

[15]  James She,et al.  Collaborative Variational Autoencoder for Recommender Systems , 2017, KDD.

[16]  Tat-Seng Chua,et al.  Neural Graph Collaborative Filtering , 2019, SIGIR.

[17]  Oriol Vinyals,et al.  Neural Discrete Representation Learning , 2017, NIPS.

[18]  Shiyu Chang,et al.  Coupled Variational Recurrent Collaborative Filtering , 2019, KDD.

[19]  Jan Kautz,et al.  NVAE: A Deep Hierarchical Variational Autoencoder , 2020, NeurIPS.

[20]  Yongdong Zhang,et al.  LightGCN: Simplifying and Powering Graph Convolution Network for Recommendation , 2020, SIGIR.

[21]  Jure Leskovec,et al.  How Powerful are Graph Neural Networks? , 2018, ICLR.

[22]  Sergey I. Nikolenko,et al.  RecVAE: A New Variational Autoencoder for Top-N Recommendations with Implicit Feedback , 2019, WSDM.

[23]  David M. Blei,et al.  Variational Inference: A Review for Statisticians , 2016, ArXiv.

[24]  Ali Razavi,et al.  Preventing Posterior Collapse with delta-VAEs , 2019, ICLR.

[25]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[26]  Yoshua Bengio,et al.  Learning deep representations by mutual information estimation and maximization , 2018, ICLR.

[27]  Frank D. Wood,et al.  The Thermodynamic Variational Objective , 2019, NeurIPS.

[28]  Bernhard Schölkopf,et al.  From Variational to Deterministic Autoencoders , 2019, ICLR.

[29]  Matthew D. Hoffman,et al.  Variational Autoencoders for Collaborative Filtering , 2018, WWW.

[30]  Pietro Liò,et al.  Deep Graph Infomax , 2018, ICLR.

[31]  Zaiqiao Meng,et al.  Hierarchical Neural Variational Model for Personalized Sequential Recommendation , 2019, WWW.

[32]  Carl Doersch,et al.  Tutorial on Variational Autoencoders , 2016, ArXiv.

[33]  Vikram Pudi,et al.  Sequential Variational Autoencoders for Collaborative Filtering , 2018, WSDM.

[34]  David Lopez-Paz,et al.  Optimizing the Latent Space of Generative Networks , 2017, ICML.

[35]  Xiangliang Zhang,et al.  Dataset Recommendation via Variational Graph Autoencoder , 2019, 2019 IEEE International Conference on Data Mining (ICDM).

[36]  Yue Ding,et al.  Adversarial and Contrastive Variational Autoencoder for Sequential Recommendation , 2021, WWW.

[37]  LightGCN , 2020, Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval.

[38]  Kyungwoo Song,et al.  Augmented Variational Autoencoders for Collaborative Filtering with Auxiliary Information , 2017, CIKM.

[39]  Yehuda Koren,et al.  Matrix Factorization Techniques for Recommender Systems , 2009, Computer.

[40]  Yoshua Bengio,et al.  Understanding the difficulty of training deep feedforward neural networks , 2010, AISTATS.

[41]  Yoshua Bengio,et al.  Bidirectional Helmholtz Machines , 2015, ICML.

[42]  Hong Cheng,et al.  Dirichlet Graph Variational Autoencoder , 2020, NeurIPS.

[43]  Mohammad Norouzi,et al.  Don't Blame the ELBO! A Linear VAE Perspective on Posterior Collapse , 2019, NeurIPS.

[44]  Tat-Seng Chua,et al.  Neural Collaborative Filtering , 2017, WWW.

[45]  Arindam Sarkar,et al.  Graph Representation Learning via Ladder Gamma Variational Autoencoders , 2020, AAAI.