Recognizing Predictive Substructures With Subgraph Information Bottleneck

The emergence of Graph Convolutional Network (GCN) has greatly boosted the progress of graph learning. However, two disturbing factors, noise and redundancy in graph data, and lack of interpretation for prediction results, impede further development of GCN. One solution is to recognize a predictive yet compressed subgraph to get rid of the noise and redundancy and obtain the interpretable part of the graph. This setting of subgraph is similar to the information bottleneck (IB) principle, which is less studied on graph-structured data and GCN. Inspired by the IB principle, we propose a novel subgraph information bottleneck (SIB) framework to recognize such subgraphs, named IB-subgraph. However, the intractability of mutual information and the discrete nature of graph data makes the objective of SIB notoriously hard to optimize. To this end, we introduce a bilevel optimization scheme coupled with a mutual information estimator for irregular graphs. Moreover, we propose a continuous relaxation for subgraph selection with a connectivity loss for stabilization. We further theoretically prove the error bound of our estimation scheme for mutual information and the noise-invariant nature of IB-subgraph. Extensive experiments on graph learning and large-scale point cloud tasks demonstrate the superior property of IB-subgraph.

[1]  Mark Hasegawa-Johnson,et al.  Unsupervised Speech Decomposition via Triple Information Bottleneck , 2020, ICML.

[2]  Silvio Savarese,et al.  3D Semantic Parsing of Large-Scale Indoor Spaces , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[3]  Stephan Günnemann,et al.  Adversarial Attacks on Graph Neural Networks via Meta Learning , 2019, ICLR.

[4]  Bernard Ghanem,et al.  DeepGCNs: Can GCNs Go As Deep As CNNs? , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[5]  Sam Devlin,et al.  Generalization in Reinforcement Learning with Selective Noise Injection and Information Bottleneck , 2019, NeurIPS.

[6]  Ajmal Mian,et al.  SegGCN: Efficient 3D Point Cloud Segmentation With Fuzzy Spherical Kernel , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[7]  Yu Rong,et al.  Graph Information Bottleneck for Subgraph Recognition , 2020, ICLR.

[8]  Bo Zong,et al.  Robust Graph Representation Learning via Neural Sparsification , 2020, ICML.

[9]  Yu-Chiang Frank Wang,et al.  Convolution in the Cloud: Learning Deformable Kernels in 3D Graph Convolution Networks for Point Cloud Analysis , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[10]  Stephan Gunnemann,et al.  Adversarial Attacks on Graph Neural Networks via Meta Learning , 2019, ICLR.

[11]  W. Hager,et al.  and s , 2019, Shallow Water Hydraulics.

[12]  Jure Leskovec,et al.  Graph Information Bottleneck , 2020, NeurIPS.

[13]  S. Varadhan,et al.  Asymptotic evaluation of certain Markov process expectations for large time , 1975 .

[14]  Junzhou Huang,et al.  Adaptive Sampling Towards Fast Graph Representation Learning , 2018, NeurIPS.

[15]  Yoshua Bengio,et al.  Mutual Information Neural Estimation , 2018, ICML.

[16]  Ben Poole,et al.  Categorical Reparameterization with Gumbel-Softmax , 2016, ICLR.

[17]  Bo An,et al.  Learning Efficient Multi-agent Communication: An Information Bottleneck Approach , 2019, ICML.

[18]  Mohammed J. Zaki.  Efficiently Mining Frequent Embedded Unordered Trees , 2004, Fundam. Informaticae.

[19]  Reynold Cheng,et al.  Efficient Algorithms for Densest Subgraph Discovery , 2019, Proc. VLDB Endow..

[20]  Yixin Chen,et al.  An End-to-End Deep Learning Architecture for Graph Classification , 2018, AAAI.

[21]  Suhang Wang,et al.  Graph Structure Learning for Robust Graph Neural Networks , 2020, KDD.

[22]  Partha Pratim Talukdar,et al.  ASAP: Adaptive Structure Aware Pooling for Learning Hierarchical Graph Representations , 2020, AAAI.

[23]  Le Song,et al.  Adversarial Attack on Graph Structured Data , 2018, ICML.

[24]  Danny Z. Chen,et al.  A Hierarchical Graph Network for 3D Object Detection on Point Clouds , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[25]  Sergey Levine,et al.  Variational Discriminator Bottleneck: Improving Imitation Learning, Inverse RL, and GANs by Constraining Information Flow , 2018, ICLR.

[26]  W. Marsden.  I and J , 2012 .

[27]  Jiawei Han,et al.  gSpan: graph-based substructure pattern mining , 2002, 2002 IEEE International Conference on Data Mining, 2002. Proceedings..

[28]  Aaas News,et al.  Book Reviews , 1893, Buffalo Medical and Surgical Journal.

[29]  Frederik Diehl,et al.  Edge Contraction Pooling for Graph Neural Networks , 2019, ArXiv.

[30]  Jure Leskovec,et al.  GNNExplainer: Generating Explanations for Graph Neural Networks , 2019, NeurIPS.

[31]  Gim Hee Lee,et al.  Weakly Supervised Semantic Point Cloud Segmentation: Towards 10× Fewer Labels , 2022 .

[32]  Jaewoo Kang,et al.  Self-Attention Graph Pooling , 2019, ICML.

[33]  Alexander A. Alemi,et al.  Deep Variational Information Bottleneck , 2017, ICLR.

[34]  Jure Leskovec,et al.  How Powerful are Graph Neural Networks? , 2018, ICLR.

[35]  Hans-Peter Kriegel,et al.  Protein function prediction via graph kernels , 2005, ISMB.

[36]  Wenbing Huang,et al.  A Restricted Black-Box Adversarial Framework Towards Attacking Graph Embedding Models , 2019, AAAI.

[37]  K. Müller,et al.  Fast and accurate modeling of molecular atomization energies with machine learning. , 2011, Physical review letters.

[38]  Jure Leskovec,et al.  Hierarchical Graph Representation Learning with Differentiable Pooling , 2018, NeurIPS.

[39]  Cao Xiao,et al.  FastGCN: Fast Learning with Graph Convolutional Networks via Importance Sampling , 2018, ICLR.

[40]  Junqing Yu,et al.  Significance-Aware Information Bottleneck for Domain Adaptive Semantic Segmentation , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[41]  Ryan A. Rossi,et al.  The Network Data Repository with Interactive Graph Analytics and Visualization , 2015, AAAI.

[42]  Samuel S. Schoenholz,et al.  Neural Message Passing for Quantum Chemistry , 2017, ICML.

[43]  Hong Cheng,et al.  Semi-Supervised Graph Classification: A Hierarchical Graph Perspective , 2019, WWW.

[44]  Jian Tang,et al.  InfoGraph: Unsupervised and Semi-supervised Graph-Level Representation Learning via Mutual Information Maximization , 2019, ICLR.

[45]  Brian K. Shoichet,et al.  ZINC - A Free Database of Commercially Available Compounds for Virtual Screening , 2005, J. Chem. Inf. Model..

[46]  P. Alam.  ‘A’ , 2021, Composites Engineering: An A–Z Guide.

[47]  Stefano Soatto,et al.  Emergence of Invariance and Disentanglement in Deep Representations , 2017, 2018 Information Theory and Applications Workshop (ITA).

[48]  Regina Barzilay,et al.  Multi-Objective Molecule Generation using Interpretable Substructures , 2020, ICML.

[49]  Sergey Levine,et al.  InfoBot: Transfer and Exploration via the Information Bottleneck , 2019, ICLR.

[50]  Ke Chen,et al.  Label-Efficient Point Cloud Semantic Segmentation: An Active Learning Approach , 2021, ArXiv.

[51]  Lawrence B. Holder,et al.  Subdue: compression-based frequent pattern discovery in graph data , 2005 .

[52]  Jure Leskovec,et al.  Inductive Representation Learning on Large Graphs , 2017, NIPS.

[53]  Pietro Liò,et al.  Graph Attention Networks , 2017, ICLR.

[54]  Max Welling,et al.  Semi-Supervised Classification with Graph Convolutional Networks , 2016, ICLR.

[55]  Michelle M. Li,et al.  Subgraph Neural Networks , 2020, NeurIPS.

[56]  Charalampos E. Tsourakakis,et al.  Dense Subgraph Discovery: KDD 2015 tutorial , 2015, KDD.

[57]  Shuiwang Ji,et al.  Graph U-Nets , 2019, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[58]  Weijing Shi,et al.  Point-GNN: Graph Neural Network for 3D Object Detection in a Point Cloud , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[59]  Massimiliano Pontil,et al.  Learning Discrete Structures for Graph Neural Networks , 2019, ICML.

[60]  Tingyang Xu,et al.  DropEdge: Towards Deep Graph Convolutional Networks on Node Classification , 2020, ICLR.

[61]  Naftali Tishby,et al.  The information bottleneck method , 2000, ArXiv.

[62]  Cesare Alippi,et al.  Spectral Clustering with Graph Neural Networks for Graph Pooling , 2019, ICML.

[63]  Leonidas J. Guibas,et al.  PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[64]  Mohamed R. Amer,et al.  Understanding Attention and Generalization in Graph Neural Networks , 2019, NeurIPS.

[65]  Ulrich Neumann,et al.  Grid-GCN for Fast and Scalable Point Cloud Learning , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[66]  Bernhard Schölkopf,et al.  Learning with Local and Global Consistency , 2003, NIPS.