McTorch, a manifold optimization library for deep learning

In this paper, we introduce McTorch, a manifold optimization library for deep learning that extends PyTorch. It aims to lower the barrier for users wishing to use manifold constraints in deep learning applications, i.e., when the parameters are constrained to lie on a manifold. Such constraints include the popular orthogonality and rank constraints, and have been recently used in a number of applications in deep learning. McTorch follows PyTorch's architecture and decouples manifold definitions and optimizers, i.e., once a new manifold is added it can be used with any existing optimizer and vice-versa. McTorch is available at this https URL .

[1]  Alan Edelman,et al.  The Geometry of Algorithms with Orthogonality Constraints , 1998, SIAM J. Matrix Anal. Appl..

[2]  Yoshua Bengio,et al.  Unitary Evolution Recurrent Neural Networks , 2015, ICML.

[3]  Suvrit Sra,et al.  Geometric Mean Metric Learning , 2016, ICML.

[4]  P. Absil,et al.  Erratum to: ``Global rates of convergence for nonconvex optimization on manifolds'' , 2016, IMA Journal of Numerical Analysis.

[5]  Bamdev Mishra,et al.  Manopt, a matlab toolbox for optimization on manifolds , 2013, J. Mach. Learn. Res..

[6]  Bamdev Mishra,et al.  Fixed-rank matrix factorizations and Riemannian low-rank optimization , 2012, Comput. Stat..

[7]  Soumava Kumar Roy,et al.  Geometry Aware Constrained Optimization Techniques for Deep Learning , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[8]  Nicolas Boumal,et al.  Near-Optimal Bounds for Phase Synchronization , 2017, SIAM J. Optim..

[9]  Suvrit Sra,et al.  Matrix Manifold Optimization for Gaussian Mixtures , 2015, NIPS.

[10]  Luca Antiga,et al.  Automatic differentiation in PyTorch , 2017 .

[11]  Hiroyuki Sato,et al.  A new, globally convergent Riemannian conjugate gradient method , 2013, 1302.0125.

[12]  Douwe Kiela,et al.  Learning Continuous Hierarchies in the Lorentz Model of Hyperbolic Geometry , 2018, ICML.

[13]  Roberto Cipolla,et al.  Symmetry-invariant optimization in deep networks , 2015, ArXiv.

[14]  Bamdev Mishra,et al.  A Dual Framework for Low-rank Tensor Completion , 2017, NeurIPS.

[15]  Robert E. Mahony,et al.  Optimization Algorithms on Matrix Manifolds , 2007 .

[16]  Silvere Bonnabel,et al.  Linear Regression under Fixed-Rank Constraints: A Riemannian Approach , 2011, ICML.

[17]  Xianglong Liu,et al.  Orthogonal Weight Normalization: Solution to Optimization over Multiple Dependent Stiefel Manifolds in Deep Neural Networks , 2017, AAAI.

[18]  Silvere Bonnabel,et al.  Stochastic Gradient Descent on Riemannian Manifolds , 2011, IEEE Transactions on Automatic Control.

[19]  Bart Vandereycken,et al.  Low-Rank Matrix Completion by Riemannian Optimization , 2013, SIAM J. Optim..

[20]  R. Bhatia.  Positive Definite Matrices , 2007 .

[21]  Niklas Koep,et al.  Pymanopt: A Python Toolbox for Optimization on Manifolds using Automatic Differentiation , 2016, J. Mach. Learn. Res..

[22]  Takayuki Okatani,et al.  Training CNNs With Normalized Kernels , 2018, AAAI.

[23]  Bamdev Mishra,et al.  Learning Multilingual Word Embeddings in Latent Metric Space: A Geometric Approach , 2018, TACL.

[24]  Hiroyuki Kasai,et al.  Riemannian stochastic variance reduced gradient , 2016, SIAM J. Optim..

[25]  Luc Van Gool,et al.  Building Deep Networks on Grassmann Manifolds , 2016, AAAI.

[26]  Suvrit Sra,et al.  Fast stochastic optimization on Riemannian manifolds , 2016, ArXiv.

[27]  Kostas Daniilidis,et al.  The Space of Essential Matrices as a Riemannian Quotient Manifold , 2017, SIAM J. Imaging Sci..

[28]  Michael M. Bronstein,et al.  MADMM: A Generic Algorithm for Non-smooth Optimization on Manifolds , 2015, ECCV.

[29]  Nicolas Boumal,et al.  Nonconvex Phase Synchronization , 2016, SIAM J. Optim..

[30]  Hiroyuki Kasai,et al.  Low-rank tensor completion: a Riemannian manifold preconditioning approach , 2016, ICML.

[31]  Wen Huang,et al.  ROPTLIB , 2018, ACM Trans. Math. Softw..

[32]  Bart Vandereycken,et al.  Low-rank tensor completion by Riemannian optimization , 2014 .

[33]  Bamdev Mishra,et al.  Riemannian Stochastic Recursive Gradient Algorithm , 2018, ICML 2018.

[34]  Bamdev Mishra,et al.  A Unified Framework for Structured Low-rank Matrix Learning , 2017, ICML.