What graph neural networks cannot learn: depth vs width

This paper studies the expressive power of graph neural networks falling within the message-passing framework (GNNmp). Two results are presented. First, GNNmp are shown to be Turing universal under sufficient conditions on their depth, width, node attributes, and layer expressiveness. Second, it is discovered that GNNmp can lose a significant portion of their power when their depth and width is restricted. The proposed impossibility statements stem from a new technique that enables the repurposing of seminal results from distributed computing and leads to lower bounds for an array of decision, optimization, and estimation problems involving graphs. Strikingly, several of these problems are deemed impossible unless the product of a GNNmp's depth and width exceeds a polynomial of the graph size; this dependence remains significant even for tasks that appear simple or when considering approximation.

[1]  Hisashi Kashima,et al.  Approximation Ratios of Graph Neural Networks for Combinatorial Problems , 2019, NeurIPS.

[2]  Jochen Seidel,et al.  Anonymous distributed computing: computability, randomization and checkability , 2015 .

[3]  Pablo Barceló,et al.  On the Turing Completeness of Modern Neural Network Architectures , 2019, ICLR.

[4]  Ami Paz,et al.  Quadratic and Near-Quadratic Lower Bounds for the CONGEST Model , 2017, DISC.

[5]  Fabian Kuhn,et al.  Distributed Minimum Cut Approximation , 2013, DISC.

[6]  Richard S. Zemel,et al.  Gated Graph Sequence Neural Networks , 2015, ICLR.

[7]  Keren Censor-Hillel,et al.  Near-Linear Lower Bounds for Distributed Distance Computations, Even in Sparse Networks , 2016, DISC.

[8]  G. Lewicki,et al.  Approximation by Superpositions of a Sigmoidal Function , 2003 .

[9]  Doina Precup,et al.  Break the Ceiling: Stronger Multi-scale Deep Graph Convolutional Networks , 2019, NeurIPS.

[10]  Liwei Wang,et al.  The Expressive Power of Neural Networks: A View from the Width , 2017, NIPS.

[11]  Samuel S. Schoenholz,et al.  Neural Message Passing for Quantum Chemistry , 2017, ICML.

[12]  Roger Wattenhofer,et al.  Networks cannot compute their diameter in sublinear time , 2012, SODA.

[13]  Xavier Bresson,et al.  An Efficient Graph Convolutional Network Technique for the Travelling Salesman Problem , 2019, ArXiv.

[14]  Le Song,et al.  2 Common Formulation for Greedy Algorithms on Graphs , 2018 .

[15]  Vijay S. Pande,et al.  Molecular graph convolutions: moving beyond fingerprints , 2016, Journal of Computer-Aided Molecular Design.

[16]  Joan Bruna,et al.  On the equivalence between graph isomorphism testing and function approximation with GNNs , 2019, NeurIPS.

[17]  M. Newman,et al.  The structure of scientific collaboration networks. , 2000, Proceedings of the National Academy of Sciences of the United States of America.

[18]  Martin Grohe,et al.  Weisfeiler and Leman Go Neural: Higher-order Graph Neural Networks , 2018, AAAI.

[19]  Alexander J. Smola,et al.  Deep Sets , 2017, 1703.06114.

[20]  Yaron Lipman,et al.  Provably Powerful Graph Networks , 2019, NeurIPS.

[21]  Max Welling,et al.  Semi-Supervised Classification with Graph Convolutional Networks , 2016, ICLR.

[22]  Pierre Fraigniaud,et al.  Towards a complexity theory for local distributed computing , 2013, JACM.

[23]  Kurt Hornik,et al.  Multilayer feedforward networks are universal approximators , 1989, Neural Networks.

[24]  Yaron Lipman,et al.  On the Universality of Invariant Networks , 2019, ICML.

[25]  Pierre Fraigniaud,et al.  Survey of Distributed Decision , 2016, Bull. EATCS.

[26]  Gabriel Peyré,et al.  Universal Invariant and Equivariant Graph Neural Networks , 2019, NeurIPS.

[27]  Fernando Gama,et al.  EdgeNets: Edge Varying Graph Neural Networks , 2020, ArXiv.

[28]  Younjoo Seo,et al.  Discriminative structural graph classification , 2019, ArXiv.

[29]  Bala Kalyanasundaram,et al.  The Probabilistic Communication Complexity of Set Intersection , 1992, SIAM J. Discret. Math..

[30]  Oleg Verbitsky,et al.  On the Power of Color Refinement , 2015, FCT.

[31]  Jukka Suomela,et al.  Survey of local algorithms , 2013, CSUR.

[32]  P. Erdos,et al.  On the evolution of random graphs , 1984 .

[33]  Moti Medina,et al.  Three Notes on Distributed Property Testing , 2017, DISC.

[34]  Hava T. Siegelmann,et al.  Turing Universality of Neural Nets (Revisited) , 1997, EUROCAST.

[35]  Oded Goldreich,et al.  Unbiased Bits from Sources of Weak Randomness and Probabilistic Communication Complexity , 1988, SIAM J. Comput..

[36]  David Peleg,et al.  Distributed Computing: A Locality-Sensitive Approach , 1987 .

[37]  Cesare Alippi,et al.  Mincut pooling in Graph Neural Networks , 2019, ArXiv.

[38]  Moni Naor,et al.  What can be computed locally? , 1993, STOC.

[39]  Jure Leskovec,et al.  How Powerful are Graph Neural Networks? , 2018, ICLR.

[40]  Razvan Pascanu,et al.  Interaction Networks for Learning about Objects, Relations and Physics , 2016, NIPS.

[41]  Max Welling,et al.  Graph Convolutional Matrix Completion , 2017, ArXiv.

[42]  Janne H. Korhonen,et al.  Deterministic Subgraph Detection in Broadcast CONGEST , 2017, OPODIS.

[43]  Albert,et al.  Emergence of scaling in random networks , 1999, Science.

[44]  Razvan Pascanu,et al.  Relational inductive biases, deep learning, and graph networks , 2018, ArXiv.

[45]  Zhuwen Li,et al.  Combinatorial Optimization with Graph Convolutional Networks and Guided Tree Search , 2018, NeurIPS.

[46]  Nathan Linial,et al.  Locality in Distributed Graph Algorithms , 1992, SIAM J. Comput..

[47]  Christos Faloutsos,et al.  Graph mining: Laws, generators, and algorithms , 2006, CSUR.

[48]  Fabian Kuhn,et al.  On the power of the congested clique model , 2014, PODC.

[49]  Zhizhen Zhao,et al.  LanczosNet: Multi-Scale Deep Graph Convolutional Networks , 2019, ICLR.

[50]  Pascal Schweitzer,et al.  Graphs Identified by Logics with Counting , 2015, MFCS.

[51]  Xavier Bresson,et al.  Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering , 2016, NIPS.

[52]  B. Bollobás.  The evolution of random graphs , 1984 .

[53]  Rose Yu,et al.  Understanding the Representation Power of Graph Neural Networks in Learning Graph Topology , 2019, NeurIPS.

[54]  Alán Aspuru-Guzik,et al.  Convolutional Networks on Graphs for Learning Molecular Fingerprints , 2015, NIPS.

[55]  Dana Angluin,et al.  Local and global properties in networks of processors (Extended Abstract) , 1980, STOC '80.