Exponential Separations in Symmetric Neural Networks

In this work we demonstrate a novel separation between symmetric neural network architectures. Specifically, we consider the Relational Network [19] architecture as a natural generalization of the DeepSets [30] architecture, and study their representational gap. Under the restriction to analytic activation functions, we construct a symmetric function acting on sets of size N with elements in dimension D, which can be efficiently approximated by the former architecture, but provably requires width exponential in N and D for the latter.

[1]  O. Shamir,et al.  Width is Less Important than Depth in ReLU Neural Networks , 2022, COLT.

[2]  Michael A. Osborne,et al.  Universal Approximation of Functions on Sets , 2021, J. Mach. Learn. Res..

[3]  Joan Bruna,et al.  Depth separation beyond radial functions , 2021, J. Mach. Learn. Res..

[4]  O. Shamir,et al.  Size and Depth Separation in Approximating Benign Functions with Neural Networks , 2021, COLT.

[5]  Lijun Qian,et al.  A Survey of Complex-Valued Neural Networks , 2021, ArXiv.

[6]  Joan Bruna,et al.  Can graph neural networks count substructures? , 2020, NeurIPS.

[7]  On Universal Equivariant Set Networks , 2019, ICLR.

[8]  Yaron Lipman,et al.  Provably Powerful Graph Networks , 2019, NeurIPS.

[9]  Michael A. Osborne,et al.  On the Limitations of Representing Functions on Sets , 2019, ICML.

[10]  Jure Leskovec,et al.  How Powerful are Graph Neural Networks? , 2018, ICLR.

[11]  Yee Whye Teh,et al.  Set Transformer , 2018, ICML.

[12]  Ryan L. Murphy,et al.  Janossy Pooling: Learning Deep Permutation-Invariant Functions for Variable-Size Inputs , 2018, ICLR.

[13]  Asim Kadav,et al.  Attend and Interact: Higher-Order Object Interactions for Video Understanding , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[14]  Lukasz Kaiser,et al.  Attention is All you Need , 2017, NIPS.

[15]  Razvan Pascanu,et al.  A simple neural network module for relational reasoning , 2017, NIPS.

[16]  Samuel S. Schoenholz,et al.  Neural Message Passing for Quantum Chemistry , 2017, ICML.

[17]  Alexander J. Smola,et al.  Deep Sets , 2017, 1703.06114.

[18]  Amit Daniely,et al.  Depth Separation for Neural Networks , 2017, COLT.

[19]  Leonidas J. Guibas,et al.  PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[20]  Ohad Shamir,et al.  Depth-Width Tradeoffs in Approximating Natural Functions with Neural Networks , 2016, ICML.

[21]  Max Welling,et al.  Semi-Supervised Classification with Graph Convolutional Networks , 2016, ICLR.

[22]  Matus Telgarsky,et al.  Benefits of Depth in Neural Networks , 2016, COLT.

[23]  Ohad Shamir,et al.  The Power of Depth for Feedforward Neural Networks , 2015, COLT.

[24]  Ah Chung Tsoi,et al.  The Graph Neural Network Model , 2009, IEEE Transactions on Neural Networks.

[25]  David Rydh,et al.  A minimal Set of Generators for the Ring of multisymmetric Functions , 2007, 0710.0470.

[26]  M. Domokos.  Vector invariants of a class of pseudo-reflection groups and multisymmetric syzygies , 2007, 0706.2154.

[27]  E. Langmann.  A method to derive explicit formulas for an elliptic generalization of the Jack polynomials , 2005, math-ph/0511015.

[28]  P. Diaconis,et al.  On the eigenvalues of random matrices , 1994, Journal of Applied Probability.

[29]  I. G. MacDonald,et al.  Symmetric functions and Hall polynomials , 1979 .

[30]  Z. Nehari.  Bounded analytic functions , 1950 .