Graph Condensation for Graph Neural Networks

Given the prevalence of large-scale graphs in real-world applications, the storage and time for training neural models have raised increasing concerns. To alleviate the concerns, we propose and study the problem of graph condensation for graph neural networks (GNNs). Specifically, we aim to condense the large, original graph into a small, synthetic and highly-informative graph, such that GNNs trained on the small graph and large graph have comparable performance. We approach the condensation problem by imitating the GNN training trajectory on the original graph through the optimization of a gradient matching loss and design a strategy to condense node futures and structural information simultaneously. Extensive experiments have demonstrated the effectiveness of the proposed framework in condensing different graph datasets into informative smaller graphs. In particular, we are able to approximate the original test accuracy by 95.3% on Reddit, 99.8% on Flickr and 99.0% on Citeseer, while reducing their graph size by more than 99.9%, and the condensed graphs can be used to train various GNN architectures.

[1]  Stephan Günnemann,et al.  Predict then Propagate: Graph Neural Networks meet Personalized PageRank , 2018, ICLR.

[2]  Suhang Wang,et al.  Graph Structure Learning for Robust Graph Neural Networks , 2020, KDD.

[3]  Christoph H. Lampert,et al.  iCaRL: Incremental Classifier and Representation Learning , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[4]  Pierre Vandergheynst,et al.  Spectrally approximating large graphs with smaller graphs , 2018, ICML.

[5]  Kilian Q. Weinberger,et al.  Simplifying Graph Convolutional Networks , 2019, ICML.

[6]  Jiliang Tang,et al.  A Unified View on Graph Neural Networks as Graph Signal Denoising , 2020, CIKM.

[7]  Massimiliano Pontil,et al.  Learning Discrete Structures for Graph Neural Networks , 2019, ICML.

[8]  Pascal Frossard,et al.  Learning Laplacian Matrix in Smooth Graph Signal Representations , 2014, IEEE Transactions on Signal Processing.

[9]  Jure Leskovec,et al.  Inductive Representation Learning on Large Graphs , 2017, NIPS.

[10]  Pietro Liò,et al.  Graph Attention Networks , 2017, ICLR.

[11]  Zhiru Zhang,et al.  GraphZoom: A multi-level spectral approach for accurate and scalable graph embedding , 2019, ICLR.

[12]  Jan Eric Lenssen,et al.  Fast Graph Representation Learning with PyTorch Geometric , 2019, ArXiv.

[13]  Razvan Pascanu,et al.  Relational inductive biases, deep learning, and graph networks , 2018, ArXiv.

[14]  Jaehoon Lee,et al.  Dataset Meta-Learning from Kernel Ridge-Regression , 2020, ICLR.

[15]  Xavier Bresson,et al.  Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering , 2016, NIPS.

[16]  J. Leskovec,et al.  Open Graph Benchmark: Datasets for Machine Learning on Graphs , 2020, NeurIPS.

[17]  Yu Chen,et al.  Iterative Deep Graph Learning for Graph Neural Networks: Better and Robust Node Embeddings , 2019, NeurIPS.

[18]  Max Welling,et al.  Semi-Supervised Classification with Graph Convolutional Networks , 2016, ICLR.

[19]  Silvio Savarese,et al.  Active Learning for Convolutional Neural Networks: A Core-Set Approach , 2017, ICLR.

[20]  Philip S. Yu,et al.  A Comprehensive Survey on Graph Neural Networks , 2019, IEEE Transactions on Neural Networks and Learning Systems.

[21]  Alán Aspuru-Guzik,et al.  Convolutional Networks on Graphs for Learning Molecular Fingerprints , 2015, NIPS.

[22]  Rajgopal Kannan,et al.  GraphSAINT: Graph Sampling Based Inductive Learning Method , 2019, ICLR.

[23]  Jose Augusto Ramos Soares,et al.  Graph Spanners: a Survey , 1992 .

[24]  Antonio Ortega,et al.  Graph Learning From Data Under Laplacian and Structural Constraints , 2016, IEEE Journal of Selected Topics in Signal Processing.

[25]  Max Welling,et al.  Herding dynamical weights to learn , 2009, ICML '09.

[26]  Song Han,et al.  Deep Leakage from Gradients , 2019, NeurIPS.

[27]  Jennifer Neville,et al.  Attributed graph models: modeling network structure with correlated attributes , 2014, WWW.

[28]  Hakan Bilen,et al.  Dataset Condensation with Differentiable Siamese Augmentation , 2021, ICML.

[29]  Shuiwang Ji,et al.  Graph U-Nets , 2019, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[30]  David R. Karger,et al.  Random sampling in cut, flow, and network design problems , 1994, STOC '94.

[31]  Yixin Chen,et al.  An End-to-End Deep Learning Architecture for Graph Classification , 2018, AAAI.

[32]  Jure Leskovec,et al.  Hierarchical Graph Representation Learning with Differentiable Pooling , 2018, NeurIPS.

[33]  Cosma Rohilla Shalizi,et al.  Homophily and Contagion Are Generically Confounded in Observational Social Network Studies , 2010, Sociological methods & research.

[34]  Yusu Wang,et al.  Graph Coarsening with Neural Networks , 2021, ICLR.

[35]  Cordelia Schmid,et al.  End-to-End Incremental Learning , 2018, ECCV.

[36]  Bernard Ghanem,et al.  DeepGCNs: Can GCNs Go As Deep As CNNs? , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[37]  Zhiyuan Liu,et al.  Graph Neural Networks: A Review of Methods and Applications , 2018, AI Open.

[38]  Jure Leskovec,et al.  Graph Convolutional Neural Networks for Web-Scale Recommender Systems , 2018, KDD.

[39]  Hakan Bilen,et al.  Dataset Condensation with Gradient Matching , 2020, ICLR.

[40]  Franziska Abend,et al.  Facility Location Concepts Models Algorithms And Case Studies , 2016 .

[41]  L. Akoglu,et al.  Beyond Homophily in Graph Neural Networks: Current Limitations and Effective Designs , 2020, NeurIPS.

[42]  Peng Cui,et al.  Interpreting and Unifying Graph Neural Networks with An Optimization Framework , 2021, WWW.

[43]  Shang-Hua Teng,et al.  Spectral Sparsification of Graphs , 2008, SIAM J. Comput..

[44]  Andreas Loukas,et al.  Graph Reduction with Spectral and Cut Guarantees , 2018, J. Mach. Learn. Res..

[45]  Qian Huang,et al.  Combining Label Propagation and Simple Models Out-performs Graph Neural Networks , 2020, ICLR.

[46]  Yuan He,et al.  Graph Neural Networks for Social Recommendation , 2019, WWW.

[47]  Yongxin Yang,et al.  Flexible Dataset Distillation: Learn Labels Instead of Images , 2020, ArXiv.

[48]  Min Zhou,et al.  Scaling Up Graph Neural Networks Via Graph Coarsening , 2021, KDD.

[49]  Geoffrey E. Hinton,et al.  Visualizing Data using t-SNE , 2008 .