Accelerated Optimization with Orthogonality Constraints

We develop a generalization of Nesterov's accelerated gradient descent method which is designed to deal with orthogonality constraints. To demonstrate the effectiveness of our method, we perform numerical experiments which demonstrate that the number of iterations scales with the square root of the condition number, and also compare with existing state-of-the-art Riemannian optimization methods. Our experiments show that our method outperforms existing state-of-the-art methods on some large, ill-conditioned problems.

[1]  J. Sherman,et al.  Adjustment of an Inverse Matrix Corresponding to a Change in One Element of a Given Matrix , 1950 .

[2]  Emmanuel J. Candès,et al.  Adaptive Restart for Accelerated Gradient Schemes , 2012, Foundations of Computational Mathematics.

[3]  Xiaojing Zhu,et al.  A Riemannian conjugate gradient method for optimization on the Stiefel manifold , 2016, Computational Optimization and Applications.

[4]  Lin Xiao,et al.  An adaptive accelerated proximal gradient method and its homotopy continuation for sparse optimization , 2014, Computational Optimization and Applications.

[5]  M. Spivak.  A comprehensive introduction to differential geometry , 1979 .

[6]  Ralf Zimmermann,et al.  A Matrix-Algebraic Algorithm for the Riemannian Logarithm on the Stiefel Manifold under the Canonical Metric , 2016, SIAM J. Matrix Anal. Appl..

[7]  S. Osher,et al.  Compressed modes for variational problems in mathematics and physics , 2013, Proceedings of the National Academy of Sciences.

[8]  Alan Edelman,et al.  The Geometry of Algorithms with Orthogonality Constraints , 1998, SIAM J. Matrix Anal. Appl..

[9]  S. Lang.  Fundamentals of differential geometry , 1998 .

[10]  Suvrit Sra,et al.  Towards Riemannian Accelerated Gradient Methods , 2018, ArXiv.

[11]  Wotao Yin,et al.  A feasible method for optimization with orthogonality constraints , 2013, Math. Program..

[12]  R. Martin,et al.  Electronic Structure: Basic Theory and Practical Methods , 2004 .

[13]  Levent Tunçel,et al.  Optimization algorithms on matrix manifolds , 2009, Math. Comput..

[14]  Wen Huang,et al.  A Broyden Class of Quasi-Newton Methods for Riemannian Optimization , 2015, SIAM J. Optim..

[15]  Y. Nesterov.  A method for solving the convex programming problem with convergence rate O(1/k^2) , 1983 .

[16]  Sébastien Bubeck,et al.  Convex Optimization: Algorithms and Complexity , 2014, Found. Trends Mach. Learn..

[17]  Y. Nesterov.  Gradient methods for minimizing composite objective function , 2007 .

[18]  Stephen P. Boyd,et al.  A Differential Equation for Modeling Nesterov's Accelerated Gradient Method: Theory and Insights , 2014, J. Mach. Learn. Res..

[19]  Xin Zhang,et al.  Gradient Type Optimization Methods For Electronic Structure Calculations , 2013, SIAM J. Sci. Comput..

[20]  S. Yau.  Non-existence of continuous convex functions on certain Riemannian manifolds , 1974 .

[21]  Hong Cheng,et al.  Accelerated First-order Methods for Geodesically Convex Optimization on Riemannian Manifolds , 2017, NIPS.