LieGG: Studying Learned Lie Group Generators

Symmetries built into a neural network have appeared to be very beneficial for a wide range of tasks as it saves the data to learn them. We depart from the position that when symmetries are not built into a model a priori, it is advantageous for robust networks to learn symmetries directly from the data to fit a task function. In this paper, we present a method to extract symmetries learned by a neural network and to evaluate the degree to which a network is invariant to them. With our method, we are able to explicitly retrieve learned invariances in a form of the generators of corresponding Lie-groups without prior knowledge of symmetries in the data. We use the proposed method to study how symmetrical properties depend on a neural network’s parameterization and configuration. We found that the ability of a network to learn symmetries generalizes over a range of architectures. However, the quality of learned symmetries depends on the depth and the number of parameters.

[1]  Rui Wang,et al.  Approximately Equivariant Networks for Imperfectly Symmetric Dynamics , 2022, ICML.

[2]  Maurice Weiler,et al.  Steerable Partial Differential Operators for Equivariant Neural Networks , 2021, ICLR.

[3]  Andrew Gordon Wilson,et al.  Residual Pathway Priors for Soft Equivariance Constraints , 2021, NeurIPS.

[4]  Samee Ur Rehman,et al.  Implicit Equivariance in Convolutional Networks , 2021, ArXiv.

[5]  Nima Dehmamy,et al.  Automatic Symmetry Discovery with Lie Algebra Convolutional Network , 2021, NeurIPS.

[6]  A. Bernacchia,et al.  Uniform Generalization Bounds for Overparameterized Neural Networks , 2021, ArXiv.

[7]  Ivan Sosnovik,et al.  DISCO: accurate Discrete Scale Convolutions , 2021, BMVC.

[8]  A. Dosovitskiy,et al.  MLP-Mixer: An all-MLP Architecture for Vision , 2021, NeurIPS.

[9]  Andrew Gordon Wilson,et al.  A Practical Method for Constructing Equivariant Multilayer Perceptrons for Arbitrary Matrix Groups , 2021, ICML.

[10]  Samy Bengio,et al.  Understanding deep learning (still) requires rethinking generalization , 2021, Commun. ACM.

[11]  Thao Nguyen,et al.  Do Wide and Deep Networks Learn the Same Things? Uncovering How Neural Network Representations Vary with Width and Depth , 2020, ICLR.

[12]  S. Gelly,et al.  An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale , 2020, ICLR.

[13]  Chelsea Finn,et al.  Meta-Learning Symmetries by Reparameterization , 2020, ICLR.

[14]  E. Weinan,et al.  On the emergence of tetrahedral symmetry in the final and penultimate layers of neural network classifiers , 2020, ArXiv.

[15]  C. Olah,et al.  Naturally Occurring Equivariance in Neural Networks , 2020 .

[16]  Quoc V. Le,et al.  Rethinking Pre-training and Self-training , 2020, NeurIPS.

[17]  Marissa Connor,et al.  Representing Closed Transformation Paths in Encoded Network Latent Space , 2019, AAAI.

[18]  Martin Jaggi,et al.  On the Relationship between Self-Attention and Convolutional Layers , 2019, ICLR.

[19]  A. Smeulders,et al.  Scale-Equivariant Steerable Networks , 2019, ICLR.

[20]  E. Bekkers.  B-Spline CNNs on Lie Groups , 2019, ICLR.

[21]  Natalia Gimelshein,et al.  PyTorch: An Imperative Style, High-Performance Deep Learning Library , 2019, NeurIPS.

[22]  Maurice Weiler,et al.  General E(2)-Equivariant Steerable CNNs , 2019, NeurIPS.

[23]  U. Alon.  An introduction to systems biology : design principles of biological circuits , 2019 .

[24]  Daniel E. Worrall,et al.  Deep Scale-spaces: Equivariance Over Scale , 2019, NeurIPS.

[25]  Patrick Kidger,et al.  Universal Approximation with Deep Narrow Networks , 2019, COLT 2019.

[26]  Kimin Lee,et al.  Using Pre-Training Can Improve Model Robustness and Uncertainty , 2019, ICML.

[27]  Kaiming He,et al.  Rethinking ImageNet Pre-Training , 2018, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[28]  Yuanzhi Li,et al.  Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layers , 2018, NeurIPS.

[29]  Quoc V. Le,et al.  Searching for Activation Functions , 2018, arXiv.

[30]  Liwei Wang,et al.  The Expressive Power of Neural Networks: A View from the Width , 2017, NIPS.

[31]  Arnold W. M. Smeulders,et al.  Dynamic Steerable Blocks in Deep Residual Networks , 2017, BMVC.

[32]  Stephan J. Garbin,et al.  Harmonic Networks: Deep Translation and Rotation Equivariance , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[33]  Dmitry Yarotsky,et al.  Error bounds for approximations with deep ReLU networks , 2016, Neural Networks.

[34]  Surya Ganguli,et al.  On the Expressive Power of Deep Neural Networks , 2016, ICML.

[35]  Max Welling,et al.  Group Equivariant Convolutional Networks , 2016, ICML.

[36]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[37]  Andrea Vedaldi,et al.  Understanding Image Representations by Measuring Their Equivariance and Equivalence , 2014, International Journal of Computer Vision.

[38]  Yoshua Bengio,et al.  Why Does Unsupervised Pre-training Help Deep Learning? , 2010, AISTATS.

[39]  Bruno A. Olshausen,et al.  An Unsupervised Algorithm For Learning Lie Group Transformations , 2010, ArXiv.

[40]  Bruno A. Olshausen,et al.  Learning transport operators for image manifolds , 2009, NIPS.

[41]  Quoc V. Le,et al.  Measuring Invariances in Deep Networks , 2009, NIPS.

[42]  Yoshua Bengio,et al.  An empirical evaluation of deep architectures on problems with many factors of variation , 2007, ICML '07.

[43]  Yoshua Bengio,et al.  Greedy Layer-Wise Training of Deep Networks , 2006, NIPS.

[44]  Yoshua Bengio,et al.  Object Recognition with Gradient-Based Learning , 1999, Shape, Contour and Grouping in Computer Vision.

[45]  David A. Forsyth,et al.  Shape, Contour and Grouping in Computer Vision , 1999, Lecture Notes in Computer Science.

[46]  Rajesh P. N. Rao,et al.  Learning Lie Groups for Invariant Visual Perception , 1998, NIPS.

[47]  P. Olver.  Applications of Lie Groups to Differential Equations , 1986 .