Deep Learning for 2D and 3D Rotatable Data: An Overview of Methods

One of the reasons for the success of convolutional networks is their equivariance/invariance under translations. However, rotatable data such as molecules, living cells, everyday objects, or galaxies require processing with equivariance/invariance under rotations in cases where the rotation of the coordinate system does not affect the meaning of the data (e.g. object classification). On the other hand, estimation/processing of rotations is necessary in cases where rotations are important (e.g. motion estimation). There has been recent progress in methods and theory in all these regards. Here we provide an overview of existing methods, both for 2D and 3D rotations (and translations), and identify commonalities and links between them, in the hope that our insights will be useful for choosing and perfecting the methods.

[1]  Max Welling,et al.  Group Equivariant Convolutional Networks , 2016, ICML.

[2]  R. Vidal,et al.  3 D Pose Regression using Convolutional Neural Networks , .

[3]  Alexander S. Ecker,et al.  A rotation-equivariant convolutional neural network model of primary visual cortex , 2018, ICLR.

[4]  Noah Snavely,et al.  Unsupervised Learning of Depth and Ego-Motion from Video , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[5]  Gabriel J. Brostow,et al.  CubeNet: Equivariance to 3D Rotation and Translation , 2018, ECCV.

[6]  Zhichao Yin,et al.  GeoNet: Unsupervised Learning of Dense Depth, Optical Flow and Camera Pose , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[7]  Roberto Cipolla,et al.  Convolutional networks for real-time 6-DOF camera relocalization , 2015, ArXiv.

[8]  Dejan V. VraniC.  An improvement of rotation invariant 3D-shape based on functions on concentric spheres , 2003, Proceedings 2003 International Conference on Image Processing (Cat. No.03CH37429).

[9]  Mitko Veta,et al.  Roto-Translation Covariant Convolutional Networks for Medical Image Analysis , 2018, MICCAI.

[10]  Risi Kondor,et al.  On the Generalization of Equivariance and Convolution in Neural Networks to the Action of Compact Groups , 2018, ICML.

[11]  René Vidal,et al.  3D Pose Regression Using Convolutional Neural Networks , 2017, 2017 IEEE International Conference on Computer Vision Workshops (ICCVW).

[12]  Stephan J. Garbin,et al.  Harmonic Networks: Deep Translation and Rotation Equivariance , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[13]  Esa Rahtu,et al.  Relative Camera Pose Estimation Using Convolutional Neural Networks , 2017, ACIVS.

[14]  Max Welling,et al.  Spherical CNNs , 2018, ICLR.

[15]  Koray Kavukcuoglu,et al.  Exploiting Cyclic Symmetry in Convolutional Neural Networks , 2016, ICML.

[16]  Pascal Fua,et al.  Real-Time Seamless Single Shot 6D Object Pose Prediction , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[17]  Edward H. Adelson,et al.  The Design and Use of Steerable Filters , 1991, IEEE Trans. Pattern Anal. Mach. Intell..

[18]  Luc Van Gool,et al.  Deep Learning on Lie Groups for Skeleton-Based Action Recognition , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[19]  Alfred Mertins,et al.  Learning Transformation Invariant Representations with Weak Supervision , 2018, VISIGRAPP.

[20]  András Lörincz,et al.  3D Human Pose Estimation with Siamese Equivariant Embedding , 2018, Neurocomputing.

[21]  Maurice Weiler,et al.  A General Theory of Equivariant CNNs on Homogeneous Spaces , 2018, NeurIPS.

[22]  Xiaowei Zhou,et al.  Polar Transformer Networks , 2017, ICLR.

[23]  Maurice Weiler,et al.  Learning Steerable Filters for Rotation Equivariant CNNs , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[24]  Ziyan Wu,et al.  Learning Local RGB-to-CAD Correspondences for Object Pose Estimation , 2018, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[25]  Slobodan Ilic,et al.  PPF-FoldNet: Unsupervised Learning of Rotation Invariant 3D Local Descriptors , 2018, ECCV.

[26]  Yinlong Liu,et al.  Efficient Global Point Cloud Registration by Matching Rotation Invariant Features Through Translation Search , 2018, ECCV.

[27]  Max Welling,et al.  3D Steerable CNNs: Learning Rotationally Equivariant Features in Volumetric Data , 2018, NeurIPS.

[28]  Szymon Rusinkiewicz,et al.  Rotation Invariant Spherical Harmonic Representation of 3D Shape Descriptors , 2003, Symposium on Geometry Processing.

[29]  Nikos Komodakis,et al.  Rotation Equivariant Vector Field Networks , 2016, 2017 IEEE International Conference on Computer Vision (ICCV).

[30]  Zoltan-Csaba Marton,et al.  Learning Implicit Representations of 3D Object Orientations from RGB , 2018 .

[31]  Li Li,et al.  Tensor Field Networks: Rotation- and Translation-Equivariant Neural Networks for 3D Point Clouds , 2018, ArXiv.

[32]  A. Robert Calderbank,et al.  RotDCF: Decomposition of Convolutional Filters for Rotation-Equivariant Deep Networks , 2018, ICLR.

[33]  Qiang Qiu,et al.  Oriented Response Networks , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[34]  A. Makadia,et al.  Learning SO(3) Equivariant Representations with Spherical CNNs , 2019, International Journal of Computer Vision.

[35]  Andrew Zisserman,et al.  Spatial Transformer Networks , 2015, NIPS.

[36]  Geoffrey E. Hinton,et al.  Dynamic Routing Between Capsules , 2017, NIPS.

[37]  Max Welling,et al.  Steerable CNNs , 2016, ICLR.

[38]  Christopher K. I. Williams,et al.  Transformation Equivariant Boltzmann Machines , 2011, ICANN.

[39]  Risi Kondor,et al.  N-body Networks: a Covariant Hierarchical Neural Network Architecture for Learning Atomic Potentials , 2018, ArXiv.

[40]  Rohit Pandey,et al.  Efficient 6-DoF Tracking of Handheld Objects from an Egocentric Viewpoint , 2018, ECCV.

[41]  Deniz Erdogmus,et al.  Real-Time Deep Pose Estimation With Geodesic Loss for Image-to-Template Rigid Registration , 2018, IEEE Transactions on Medical Imaging.

[42]  Qing Wang,et al.  2D/3D rotation-invariant detection using equivariant filters and kernel weighted mapping , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[43]  Risi Kondor,et al.  Cormorant: Covariant Molecular Neural Networks , 2019, NeurIPS.

[44]  Yi Li,et al.  Deformable Convolutional Networks , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).