Group Invariance and Stability to Deformations of Deep Convolutional Representations

In this paper, we study deep signal representations that are invariant to groups of transformations and stable to the action of diffeomorphisms without losing signal information. This is achieved by generalizing the multilayer kernel introduced in the context of convolutional kernel networks and by studying the geometry of the corresponding reproducing kernel Hilbert space. We show that the signal representation is stable, and that models from this functional space, such as a large class of convolutional neural networks with homogeneous activation functions, may enjoy the same stability.

[1]  Bernhard Schölkopf,et al.  Local Group Invariant Representations via Orbit Embeddings , 2016, AISTATS.

[2]  齋藤 三郎.  Integral transforms, reproducing kernels and their applications , 1997 .

[3]  Cordelia Schmid,et al.  Convolutional Kernel Networks , 2014, NIPS.

[4]  Bernhard Schölkopf,et al.  Support vector learning , 1997 .

[5]  J. Diestel,et al.  On vector measures , 1974 .

[6]  Lawrence D. Jackel,et al.  Backpropagation Applied to Handwritten Zip Code Recognition , 1989, Neural Computation.

[7]  Katya Scheinberg,et al.  Efficient SVM Training Using Low-Rank Kernel Representations , 2002, J. Mach. Learn. Res..

[8]  Samy Bengio,et al.  Adversarial examples in the physical world , 2016, ICLR.

[9]  Yoshua Bengio,et al.  Gradient-based learning applied to document recognition , 1998, Proc. IEEE.

[10]  Timothy S. Murphy,et al.  Harmonic Analysis: Real-Variable Methods, Orthogonality, and Oscillatory Integrals , 1993 .

[11]  Klaus-Robert Müller,et al.  Kernel Analysis of Deep Networks , 2011, J. Mach. Learn. Res..

[12]  Yoram Singer,et al.  Toward Deeper Understanding of Neural Networks: The Power of Initialization and a Dual View on Expressivity , 2016, NIPS.

[13]  Martin J. Wainwright,et al.  Convexified Convolutional Neural Networks , 2016, ICML.

[14]  Antonio Torralba,et al.  Statistics of natural image categories , 2003, Network.

[15]  Tomaso A. Poggio,et al.  Learning with Group Invariant Features: A Kernel Perspective , 2015, NIPS.

[16]  Benjamin Recht,et al.  Random Features for Large-Scale Kernel Machines , 2007, NIPS.

[17]  Deborah Silver,et al.  Feature Visualization , 1994, Scientific Visualization.

[18]  David A. McAllester,et al.  A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networks , 2017, ICLR.

[19]  Bernhard Schölkopf,et al.  Sparse Greedy Matrix Approximation for Machine Learning , 2000, International Conference on Machine Learning.

[20]  Alexander J. Smola,et al.  Regularization with Dot-Product Kernels , 2000, NIPS.

[21]  Lawrence K. Saul,et al.  Kernel Methods for Deep Learning , 2009, NIPS.

[22]  Moustapha Cissé,et al.  Parseval Networks: Improving Robustness to Adversarial Examples , 2017, ICML.

[23]  Max Welling,et al.  Group Equivariant Convolutional Networks , 2016, ICML.

[24]  Yuchen Zhang,et al.  L1-regularized Neural Networks are Improperly Learnable in Polynomial Time , 2015, ICML.

[25]  Yoram Singer,et al.  Random Features for Compositional Kernels , 2017, ArXiv.

[26]  Lorenzo Rosasco,et al.  Deep Convolutional Networks are Hierarchical Kernel Machines , 2015, ArXiv.

[27]  Shai Ben-David,et al.  Understanding Machine Learning: From Theory to Algorithms , 2014 .

[28]  Thomas Wiatowski,et al.  A Mathematical Theory of Deep Convolutional Neural Networks for Feature Extraction , 2015, IEEE Transactions on Information Theory.

[29]  Lorenzo Rosasco,et al.  On Invariance and Selectivity in Representation Learning , 2015, ArXiv.

[30]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[31]  Y. Amit,et al.  Towards a coherent statistical framework for dense deformable template estimation , 2007 .

[32]  Stéphane Mallat,et al.  Group Invariant Scattering , 2011, ArXiv.

[33]  Stéphane Mallat,et al.  Invariant Scattering Convolution Networks , 2012, IEEE transactions on pattern analysis and machine intelligence.

[34]  Anthony Widjaja,et al.  Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond , 2003, IEEE Transactions on Neural Networks.

[35]  Nico Schmid,et al.  Learning with Hierarchical Gaussian Kernels , 2016, ArXiv.

[36]  Joan Bruna,et al.  Learning Stable Group Invariant Representations with Convolutional Networks , 2013, ICLR.

[37]  I. J. Schoenberg.  Positive definite functions on spheres , 1942 .

[38]  Nello Cristianini,et al.  An Introduction to Support Vector Machines and Other Kernel-based Learning Methods , 2000 .

[39]  Ivor W. Tsang,et al.  Improved Nyström low-rank approximation and error analysis , 2008, ICML '08.

[40]  Matthias W. Seeger,et al.  Using the Nyström Method to Speed Up Kernel Machines , 2000, NIPS.

[41]  Francis R. Bach,et al.  On the Equivalence between Kernel Quadrature Rules and Random Feature Expansions , 2015, J. Mach. Learn. Res..

[42]  Bernhard Schölkopf,et al.  Nonlinear Component Analysis as a Kernel Eigenvalue Problem , 1998, Neural Computation.

[43]  Bernhard Schölkopf,et al.  Kernel Mean Embedding of Distributions: A Review and Beyonds , 2016, Found. Trends Mach. Learn..

[44]  Dieter Fox,et al.  Object recognition with hierarchical kernel descriptors , 2011, CVPR 2011.

[45]  Alain Trouvé,et al.  Local Geometry of Deformable Templates , 2005, SIAM J. Math. Anal..

[46]  Julien Mairal,et al.  End-to-End Kernel Learning with Supervised Convolutional Kernel Networks , 2016, NIPS.

[47]  Alexander J. Smola,et al.  Fastfood: Approximate Kernel Expansions in Loglinear Time , 2014, ArXiv.

[48]  S. Boucheron,et al.  Theory of classification : a survey of some recent advances , 2005 .

[49]  Stéphane Mallat,et al.  Rotation, Scaling and Deformation Invariant Scattering for Texture Discrimination , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[50]  Stéphane Mallat,et al.  Deep roto-translation scattering for object classification , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[51]  Lawrence K. Saul,et al.  Large-Margin Classification in Infinite Neural Networks , 2010, Neural Computation.