Graph Posterior Network: Bayesian Predictive Uncertainty for Node Classification

The interdependence between nodes in graphs is key to improve class predictions on nodes and utilized in approaches like Label Propagation (LP) or in Graph Neural Networks (GNNs). Nonetheless, uncertainty estimation for non-independent node-level predictions is under-explored. In this work, we explore uncertainty quantification for node classification in three ways: (1) We derive three axioms explicitly characterizing the expected predictive uncertainty behavior in homophilic attributed graphs. (2) We propose a new model Graph Posterior Network (GPN) which explicitly performs Bayesian posterior updates for predictions on interdependent nodes. GPN provably obeys the proposed axioms. (3) We extensively evaluate GPN and a strong set of baselines on semi-supervised node classification including detection of anomalous features, and detection of left-out classes. GPN outperforms existing approaches for uncertainty estimation in the experiments.

[1]  Max Welling,et al.  Semi-Supervised Classification with Graph Convolutional Networks , 2016, ICLR.

[2]  Yarin Gal,et al.  Liberty or Depth: Deep Bayesian Neural Nets Do Not Need Complex Weight Posterior Approximations , 2020, NeurIPS.

[3]  Yee Whye Teh,et al.  Do Deep Generative Models Know What They Don't Know? , 2018, ICLR.

[4]  Alexander A. Alemi,et al.  Density of States Estimation for Out-of-Distribution Detection , 2020, ArXiv.

[5]  Julien Cornebise,et al.  Weight Uncertainty in Neural Network , 2015, ICML.

[6]  Murat Sensoy,et al.  Evidential Deep Learning to Quantify Classification Uncertainty , 2018, NeurIPS.

[7]  Raman Arora,et al.  Understanding Deep Neural Networks with Rectified Linear Units , 2016, Electron. Colloquium Comput. Complex..

[8]  Eric Jang,et al.  Generative Ensembles for Robust Anomaly Detection , 2018, ArXiv.

[9]  Stephan Günnemann,et al.  Diffusion Improves Graph Learning , 2019, NeurIPS.

[10]  Lise Getoor,et al.  Collective Classification in Network Data , 2008, AI Mag..

[11]  Yao Hu,et al.  Uncertainty Aware Graph Gaussian Process for Semi-Supervised Learning , 2020, AAAI.

[12]  J. Leskovec,et al.  Open Graph Benchmark: Datasets for Machine Learning on Graphs , 2020, NeurIPS.

[13]  Tingyang Xu,et al.  DropEdge: Towards Deep Graph Convolutional Networks on Node Classification , 2020, ICLR.

[14]  Taesup Moon,et al.  Uncertainty-based Continual Learning with Adaptive Regularization , 2019, NeurIPS.

[15]  Mark Coates,et al.  Bayesian Graph Convolutional Neural Networks Using Non-Parametric Graph Learning , 2019, ArXiv.

[16]  Andrey Malinin,et al.  Reverse KL-Divergence Training of Prior Networks: Improved Uncertainty and Adversarial Robustness , 2019, NeurIPS.

[17]  Brandon M. Greenwell,et al.  Interpretable Machine Learning , 2019, Hands-On Machine Learning with R.

[18]  C. Lee Giles,et al.  CiteSeer: an automatic citation indexing system , 1998, DL '98.

[19]  Laurent Dinh,et al.  Perfect Density Models Cannot Guarantee Anomaly Detection , 2020, Entropy.

[20]  Jasper Snoek,et al.  Efficient and Scalable Bayesian Neural Nets with Rank-1 Factors , 2020, ICML.

[21]  Stephan Günnemann,et al.  Uncertainty on Asynchronous Time Event Prediction , 2019, NeurIPS.

[22]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[23]  Pietro Liò,et al.  Graph Attention Networks , 2017, ICLR.

[24]  Zoubin Ghahramani,et al.  Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning , 2015, ICML.

[25]  Christos Faloutsos,et al.  The Power of Certainty: A Dirichlet-Multinomial Model for Belief Propagation , 2017, SDM.

[26]  Yarin Gal,et al.  Sufficient Conditions for Idealised Models to Have No Adversarial Examples: a Theoretical and Empirical Study with Bayesian Neural Networks , 2018, 1806.00667.

[27]  Jan Eric Lenssen,et al.  Fast Graph Representation Learning with PyTorch Geometric , 2019, ArXiv.

[28]  Stephan Günnemann,et al.  Deep Gaussian Embedding of Graphs: Unsupervised Inductive Learning via Ranking , 2017, ICLR.

[29]  Mark Coates,et al.  Bayesian Graph Convolutional Neural Networks using Node Copying , 2019, ArXiv.

[30]  Cao Xiao,et al.  FastGCN: Fast Learning with Graph Convolutional Networks via Importance Sampling , 2018, ICLR.

[31]  Weitang Liu,et al.  Energy-based Out-of-distribution Detection , 2020, NeurIPS.

[32]  Andrew McCallum,et al.  Automating the Construction of Internet Portals with Machine Learning , 2000, Information Retrieval.

[33]  Stephan Günnemann,et al.  Pitfalls of Graph Neural Network Evaluation , 2018, ArXiv.

[34]  Louis C. Tiao,et al.  Variational Inference for Graph Convolutional Networks in the Absence of Graph Data and Adversarial Settings , 2019, Neural Information Processing Systems.

[35]  Mark J. F. Gales,et al.  Predictive Uncertainty Estimation via Prior Networks , 2018, NeurIPS.

[36]  Pushmeet Kohli,et al.  Contrastive Training for Improved Out-of-Distribution Detection , 2020, ArXiv.

[37]  Yuxiao Dong,et al.  Microsoft Academic Graph: When experts are not enough , 2020, Quantitative Science Studies.

[38]  Sebastian Nowozin,et al.  Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift , 2019, NeurIPS.

[39]  Regina Barzilay,et al.  Uncertainty Quantification Using Neural Networks for Molecular Property Prediction , 2020, J. Chem. Inf. Model..

[40]  Stephan Günnemann,et al.  Predict then Propagate: Graph Neural Networks meet Personalized PageRank , 2018, ICLR.

[41]  Seongok Ryu,et al.  Uncertainty quantification of molecular property prediction with Bayesian neural networks , 2019, ArXiv.

[42]  Stephan Günnemann,et al.  Bayesian Robust Attributed Graph Clustering: Joint Learning of Partial Anomalies and Group Structure , 2018, AAAI.

[43]  Reynold Cheng,et al.  On Embedding Uncertain Graphs , 2017, CIKM.

[44]  Natalia Gimelshein,et al.  PyTorch: An Imperative Style, High-Performance Deep Learning Library , 2019, NeurIPS.

[45]  Zheng Wang,et al.  Uncertainty-aware Attention Graph Neural Network for Defending Adversarial Attacks , 2020, AAAI.

[46]  Anton van den Hengel,et al.  Image-Based Recommendations on Styles and Substitutes , 2015, SIGIR.

[47]  Uncertainty Estimation Using a Single Deep Deterministic Neural Network-ML Reproducibility Challenge 2020 , 2021 .

[48]  Dustin Tran,et al.  Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awareness , 2020, NeurIPS.

[49]  Yarin Gal,et al.  Uncertainty in Deep Learning , 2016 .

[50]  Lise Getoor,et al.  Query-driven Active Surveying for Collective Classification , 2012 .

[51]  Shu Hu,et al.  Uncertainty Aware Semi-Supervised Learning on Graph Data , 2020, NeurIPS.

[52]  Richard E. Turner,et al.  On the Expressiveness of Approximate Inference in Bayesian Neural Networks , 2019, NeurIPS.

[53]  Bernt Schiele,et al.  Confidence-Calibrated Adversarial Training: Generalizing to Unseen Attacks , 2019, ICML.

[54]  Eric T. Nalisnick,et al.  Detecting Out-of-Distribution Inputs to Deep Generative Models Using Typicality , 2019 .

[55]  Li Liu,et al.  A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges , 2020, Inf. Fusion.

[56]  Wenwu Zhu,et al.  Robust Graph Convolutional Networks Against Adversarial Attacks , 2019, KDD.

[57]  Jasper Snoek,et al.  Training independent subnetworks for robust prediction , 2020, ICLR.

[58]  L. Getoor,et al.  Link-Based Classification , 2003, Encyclopedia of Machine Learning and Data Mining.

[59]  Milos Hauskrecht,et al.  Obtaining Well Calibrated Probabilities Using Bayesian Binning , 2015, AAAI.

[60]  Andrey Malinin,et al.  Ensemble Distribution Distillation , 2019, ICLR.

[61]  Ian Osband,et al.  Risk versus Uncertainty in Deep Learning: Bayes, Bootstrap and the Dangers of Dropout , 2016 .

[62]  Xiaowen Dong,et al.  Gaussian Processes on Graphs Via Spectral Kernel Learning , 2020, IEEE Transactions on Signal and Information Processing over Networks.

[63]  Charu C. Aggarwal,et al.  Node classification in uncertain graphs , 2014, SSDBM '14.

[64]  Alex Graves,et al.  Practical Variational Inference for Neural Networks , 2011, NIPS.

[65]  Shakir Mohamed,et al.  Variational Inference with Normalizing Flows , 2015, ICML.

[66]  L. Akoglu,et al.  Beyond Homophily in Graph Neural Networks: Current Limitations and Effective Designs , 2020, NeurIPS.

[67]  Nitish Srivastava,et al.  Dropout: a simple way to prevent neural networks from overfitting , 2014, J. Mach. Learn. Res..

[68]  Stephan Gunnemann,et al.  Natural Posterior Network: Deep Bayesian Predictive Uncertainty for Exponential Family Distributions , 2022, ICLR.

[69]  Haoxi Zhan,et al.  I-GCN: Robust Graph Convolutional Network via Influence Mechanism , 2020, ArXiv.

[70]  Koby Crammer,et al.  Graph-Based Transduction with Confidence , 2012, ECML/PKDD.

[71]  Agustinus Kristiadi,et al.  Being Bayesian, Even Just a Bit, Fixes Overconfidence in ReLU Networks , 2020, ICML.

[72]  David Wagner,et al.  Adversarial Examples Are Not Easily Detected: Bypassing Ten Detection Methods , 2017, AISec@CCS.

[73]  Marc Peter Deisenroth,et al.  Matern Gaussian Processes on Graphs , 2021, AISTATS.

[74]  Gustavo Carneiro,et al.  Uncertainty in Model-Agnostic Meta-Learning using Variational Inference , 2019, 2020 IEEE Winter Conference on Applications of Computer Vision (WACV).

[75]  Kilian Q. Weinberger,et al.  On Calibration of Modern Neural Networks , 2017, ICML.

[76]  William R. Clements,et al.  Estimating Risk and Uncertainty in Deep Reinforcement Learning , 2019, ArXiv.

[77]  Andrew Gordon Wilson,et al.  A Simple Baseline for Bayesian Uncertainty in Deep Learning , 2019, NeurIPS.

[78]  Mark J. F. Gales,et al.  Incorporating Uncertainty into Deep Learning for Spoken Language Assessment , 2017, ACL.

[79]  Xiaoning Qian,et al.  Bayesian Graph Neural Networks with Adaptive Connection Sampling , 2020, ICML.

[80]  Yin Cheng Ng,et al.  Bayesian Semi-supervised Learning with Graph Gaussian Processes , 2018, NeurIPS.

[81]  Eric Xing,et al.  Methods for comparing uncertainty quantifications for material property predictions. , 2019 .

[82]  Matthias Hein,et al.  Why ReLU Networks Yield High-Confidence Predictions Far Away From the Training Data and How to Mitigate the Problem , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[83]  Jasper Snoek,et al.  Hyperparameter Ensembles for Robustness and Uncertainty Quantification , 2020, NeurIPS.

[84]  Dustin Tran,et al.  BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learning , 2020, ICLR.

[85]  Alpha A. Lee,et al.  Bayesian semi-supervised learning for uncertainty-calibrated prediction of molecular properties and active learning , 2019, Chemical science.

[86]  Mark Coates,et al.  Bayesian graph convolutional neural networks for semi-supervised classification , 2018, AAAI.

[87]  Mohammad Norouzi,et al.  Your Classifier is Secretly an Energy Based Model and You Should Treat it Like One , 2019, ICLR.

[88]  D. Rus,et al.  Deep Evidential Regression , 2019, NeurIPS.

[89]  Stephan Günnemann,et al.  Posterior Network: Uncertainty Estimation without OOD Samples via Density-Based Pseudo-Counts , 2020, NeurIPS.

[90]  Andrew Gordon Wilson,et al.  Why Normalizing Flows Fail to Detect Out-of-Distribution Data , 2020, NeurIPS.

[91]  Qian Huang,et al.  Combining Label Propagation and Simple Models Out-performs Graph Neural Networks , 2020, ICLR.

[92]  Stephan Günnemann,et al.  Evaluating Robustness of Predictive Uncertainty Estimation: Are Dirichlet-based Models Reliable? , 2020, ICML.

[93]  Zoubin Ghahramani,et al.  Variational Bayesian dropout: pitfalls and fixes , 2018, ICML.

[94]  Finale Doshi-Velez,et al.  Decomposition of Uncertainty in Bayesian Deep Learning for Efficient and Risk-sensitive Learning , 2017, ICML.

[95]  Matthias Hein,et al.  Towards neural networks that provably know when they don't know , 2020, ICLR.

[96]  Jure Leskovec,et al.  Unifying Graph Convolutional Neural Networks and Label Propagation , 2020, ArXiv.

[97]  Hisashi Kashima,et al.  BayesGrad: Explaining Predictions of Graph Convolutional Networks , 2018, ICONIP.

[98]  Talal Rahwan,et al.  Hiding individuals and communities in a social network , 2016, Nature Human Behaviour.

[99]  Charles Blundell,et al.  Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles , 2016, NIPS.

[100]  Eunho Yang,et al.  Learning to Balance: Bayesian Meta-Learning for Imbalanced and Out-of-distribution Tasks , 2019, ICLR.

[101]  R. Srikant,et al.  Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks , 2017, ICLR.