Elastic Graph Neural Networks

While many existing graph neural networks (GNNs) have been proven to perform `2-based graph smoothing that enforces smoothness globally, in this work we aim to further enhance the local smoothness adaptivity of GNNs via `1-based graph smoothing. As a result, we introduce a family of GNNs (Elastic GNNs) based on `1 and `2-based graph smoothing. In particular, we propose a novel and general message passing scheme into GNNs. This message passing algorithm is not only friendly to back-propagation training but also achieves the desired smoothing properties with a theoretical convergence guarantee. Experiments on semi-supervised learning tasks demonstrate that the proposed Elastic GNNs obtain better adaptivity on benchmark datasets and are significantly robust to graph adversarial attacks. The implementation of Elastic GNNs is available at https: //github.com/lxiaorui/ElasticGNN.

[1]  Yonina C. Eldar,et al.  Semi-Supervised Learning in Network-Structured Data via Total Variation Minimization , 2019, IEEE Transactions on Signal Processing.

[2]  Xavier Bresson,et al.  An Adaptive Total Variation Algorithm for Computing the Balanced Cut of a Graph , 2013, 1302.2717.

[3]  Saba A. Al-Sayouri,et al.  All You Need Is Low (Rank): Defending Against Adversarial Attacks on Graphs , 2020, WSDM.

[4]  Stephan Günnemann,et al.  Predict then Propagate: Graph Neural Networks meet Personalized PageRank , 2018, ICLR.

[5]  Xavier Bresson,et al.  Total Variation, Cheeger Cuts , 2010, ICML.

[6]  Suhang Wang,et al.  Graph Structure Learning for Robust Graph Neural Networks , 2020, KDD.

[7]  Jelena Kovacevic,et al.  Vector-Valued Graph Trend Filtering With Non-Convex Penalties , 2019, IEEE Transactions on Signal and Information Processing over Networks.

[8]  Matthias Hein,et al.  Spectral clustering based on the graph p-Laplacian , 2009, ICML '09.

[9]  Michael Elad,et al.  Sparse and Redundant Representations - From Theory to Applications in Signal and Image Processing , 2010 .

[10]  Ah Chung Tsoi,et al.  The Graph Neural Network Model , 2009, IEEE Transactions on Neural Networks.

[11]  Xavier Bresson,et al.  Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering , 2016, NIPS.

[12]  Leman Akoglu,et al.  PairNorm: Tackling Oversmoothing in GNNs , 2020, ICLR.

[13]  R. Tibshirani.  Adaptive piecewise polynomial estimation via trend filtering , 2013, 1304.2986.

[14]  L. Rudin,et al.  Nonlinear total variation based noise removal algorithms , 1992 .

[15]  Xiaojin Zhu,et al.  --1 CONTENTS , 2006 .

[16]  R. Tibshirani,et al.  Sparsity and smoothness via the fused lasso , 2005 .

[17]  Stephan Gunnemann,et al.  Adversarial Attacks on Graph Neural Networks via Meta Learning , 2019, ICLR.

[18]  Jiliang Tang,et al.  A Unified View on Graph Neural Networks as Graph Signal Denoising , 2020, CIKM.

[19]  Angelica I. Avilés-Rivero,et al.  When Labelled Data Hurts: Deep Semi-Supervised Classification with the Graph 1-Laplacian , 2019 .

[20]  Heinz H. Bauschke,et al.  Convex Analysis and Monotone Operator Theory in Hilbert Spaces , 2011, CMS Books in Mathematics.

[21]  Kilian Q. Weinberger,et al.  Simplifying Graph Convolutional Networks , 2019, ICML.

[22]  U. Feige,et al.  Spectral Graph Theory , 2015 .

[23]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[24]  Stephan Günnemann,et al.  Adversarial Attacks on Neural Networks for Graph Data , 2018, KDD.

[25]  Zoubin Ghahramani,et al.  Learning from labeled and unlabeled data with label propagation , 2002 .

[26]  Feiping Nie,et al.  Unsupervised and semi-supervised learning via ℓ1-norm graph , 2011, 2011 International Conference on Computer Vision.

[27]  Ming Yan,et al.  A primal-dual algorithm with optimal stepsizes and its application in decentralized consensus optimization , 2017, ArXiv.

[28]  Defferrard Michaël,et al.  Deep Learning on Graphs , 2016 .

[29]  Yonina C. Eldar,et al.  Graph Unrolling Networks: Interpretable Neural Networks for Graph Signal Denoising , 2020, IEEE Transactions on Signal Processing.

[30]  Xavier Bresson,et al.  Multiclass Total Variation Clustering , 2013, NIPS.

[31]  Lada A. Adamic,et al.  The political blogosphere and the 2004 U.S. election: divided they blog , 2005, LinkKDD '05.

[32]  I. Loris,et al.  On a generalization of the iterative soft-thresholding algorithm for the case of non-separable penalty , 2011, 1104.1087.

[33]  Stephan Günnemann,et al.  Pitfalls of Graph Neural Network Evaluation , 2018, ArXiv.

[34]  Alfred O. Hero,et al.  Semi-Supervised Learning via Sparse Label Propagation , 2016 .

[35]  Samuel S. Schoenholz,et al.  Neural Message Passing for Quantum Chemistry , 2017, ICML.

[36]  Max Welling,et al.  Semi-Supervised Classification with Graph Convolutional Networks , 2016, ICLR.

[37]  Xiaoqun Zhang,et al.  A primal–dual fixed point algorithm for convex separable minimization with applications to image restoration , 2013 .

[38]  Alessandro Rinaldo,et al.  Sparsistency of the Edge Lasso over Graphs , 2012, AISTATS.

[39]  Peng Cui,et al.  Interpreting and Unifying Graph Neural Networks with An Optimization Framework , 2021, WWW.

[40]  Takanori Maehara,et al.  Revisiting Graph Neural Networks: All We Have is Low-Pass Filters , 2019, ArXiv.

[41]  Lise Getoor,et al.  Collective Classification in Network Data , 2008, AI Mag..

[42]  Trevor Hastie,et al.  Statistical Learning with Sparsity: The Lasso and Generalizations , 2015 .

[43]  Alexander J. Smola,et al.  Trend Filtering on Graphs , 2014, J. Mach. Learn. Res..

[44]  Bernhard Schölkopf,et al.  Learning with Local and Global Consistency , 2003, NIPS.

[45]  Jure Leskovec,et al.  Inductive Representation Learning on Large Graphs , 2017, NIPS.

[46]  Jiliang Tang,et al.  DeepRobust: A PyTorch Library for Adversarial Attacks and Defenses , 2020, ArXiv.