Towards Bayesian Learning of the Architecture, Graph and Parameters for Graph Neural Networks

Real life data often arises from relational structures that are best modeled by graphs. Bayesian learning on graphs has emerged as a framework which allows us to model prior beliefs about network data in a mathematically principled way. The approach provides uncertainty estimates and can perform very well on a small sample size when provided with an informative prior. Much of the work on Bayesian graph neural networks (GNNs) has focused on inferring the structure of the underlying graph and the model weights. Although research effort has been directed towards network architecture search for GNNs, existing strategies are not Bayesian and return a point estimate of the optimal architecture. In this work, we propose a method for principled Bayesian modelling for GNNs that allows for inference of a posterior over the architecture (number of layers, number of active neurons, aggregators, pooling), the graph, and the model parameters. We evaluate our proposed method on three mainstream datasets.

[1]  J. Leskovec,et al.  Efficient Automatic Machine Learning via Design Graphs , 2022, ArXiv.

[2]  Florence Regol,et al.  Bag Graph: Multiple Instance Learning using Bayesian Graph Neural Networks , 2022, AAAI.

[3]  J. Leskovec,et al.  Design Space for Graph Neural Networks , 2020, NeurIPS.

[4]  Dengcheng Zhang,et al.  A Framework for Recommending Accurate and Diverse Items Using Bayesian Graph Convolutional Neural Networks , 2020, KDD.

[5]  Longhui Wei,et al.  Fitting the Search Space of Weight-sharing NAS with Graph Convolutional Networks , 2020, AAAI.

[6]  Bo Zhang,et al.  Fast, Accurate and Lightweight Super-Resolution with Neural Architecture Search , 2019, 2020 25th International Conference on Pattern Recognition (ICPR).

[7]  Justin Bayer,et al.  Bayesian Learning of Neural Network Architectures , 2019, AISTATS.

[8]  Liang Lin,et al.  SNAS: Stochastic Neural Architecture Search , 2018, ICLR.

[9]  Yiming Yang,et al.  DARTS: Differentiable Architecture Search , 2018, ICLR.

[10]  Razvan Pascanu,et al.  Relational inductive biases, deep learning, and graph networks , 2018, ArXiv.

[11]  David Barber,et al.  A Scalable Laplace Approximation for Neural Networks , 2018, ICLR.

[12]  Yong Yu,et al.  Efficient Architecture Search by Network Transformation , 2017, AAAI.

[13]  Andrei A. Rusu,et al.  Overcoming catastrophic forgetting in neural networks , 2016, Proceedings of the National Academy of Sciences.

[14]  Quoc V. Le,et al.  Neural Architecture Search with Reinforcement Learning , 2016, ICLR.

[15]  Max Welling,et al.  Semi-Supervised Classification with Graph Convolutional Networks , 2016, ICLR.

[16]  Ruslan Salakhutdinov,et al.  Revisiting Semi-Supervised Learning with Graph Embeddings , 2016, ICML.

[17]  Jasper Snoek,et al.  Practical Bayesian Optimization of Machine Learning Algorithms , 2012, NIPS.

[18]  David J. C. MacKay,et al.  A Practical Bayesian Framework for Backpropagation Networks , 1992, Neural Computation.

[19]  Yann LeCun,et al.  Optimal Brain Damage , 1989, NIPS.