A Brief Introduction to Manifold Optimization

Manifold optimization is ubiquitous in computational and applied mathematics, statistics, engineering, machine learning, physics, chemistry, etc. One of the main challenges usually is the non-convexity of the manifold constraints. By utilizing the geometry of manifold, a large class of constrained optimization problems can be viewed as unconstrained optimization problems on manifold. From this perspective, intrinsic structures, optimality conditions and numerical algorithms for manifold optimization are investigated. Some recent progress on the theoretical results of manifold optimization is also presented.

[1]  P. Pulay.  Convergence acceleration of iterative sequences. the case of scf iteration , 1980 .

[2]  P. Pulay.  Improved SCF convergence acceleration , 1982 .

[3]  D. Gabay.  Minimizing a differentiable function over a differential manifold , 1982 .

[4]  E. Oja,et al.  On stochastic approximation of the eigenvectors and eigenvalues of the expectation of a random matrix , 1985 .

[5]  R. Kass.  Nonlinear Regression Analysis and its Applications , 1990 .

[6]  C. Udriste,et al.  Convex Functions and Optimization Methods on Riemannian Manifolds , 1994 .

[7]  Alexander I. Barvinok,et al.  Problems of distance geometry and convex properties of quadratic maps , 1995, Discret. Comput. Geom..

[8]  S. Yau,et al.  Lectures on Harmonic Maps , 1997 .

[9]  Alan Edelman,et al.  The Geometry of Algorithms with Orthogonality Constraints , 1998, SIAM J. Matrix Anal. Appl..

[10]  Gábor Pataki,et al.  On the Rank of Extreme Matrices in Semidefinite Programs and the Multiplicity of Optimal Eigenvalues , 1998, Math. Oper. Res..

[11]  Stephen J. Wright,et al.  Numerical Optimization , 2018, Fundamental Statistical Inference.

[12]  Stephen J. Wright,et al.  Numerical Optimization (Springer Series in Operations Research and Financial Engineering) , 2000 .

[13]  장윤희,et al.  Y. , 2003, Industrial and Labor Relations Terms.

[14]  Renato D. C. Monteiro,et al.  A nonlinear programming algorithm for solving semidefinite programs via low-rank factorization , 2003, Math. Program..

[15]  I. Jolliffe,et al.  A Modified Principal Component Technique Based on the LASSO , 2003 .

[16]  William W. Hager,et al.  A Nonmonotone Line Search Technique and Its Application to Unconstrained Optimization , 2004, SIAM J. Optim..

[17]  Renato D. C. Monteiro,et al.  Digital Object Identifier (DOI) 10.1007/s10107-004-0564-1 , 2004 .

[18]  Shotaro Akaho,et al.  Learning algorithms utilizing quasi-geodesic flows on the Stiefel manifold , 2005, Neurocomputing.

[19]  Amnon Shashua,et al.  Nonnegative Sparse PCA , 2006, NIPS.

[20]  Pierre-Antoine Absil,et al.  Joint Diagonalization on the Oblique Manifold for Independent Component Analysis , 2006, 2006 IEEE International Conference on Acoustics Speech and Signal Processing Proceedings.

[21]  Robert E. Mahony,et al.  Optimization Algorithms on Matrix Manifolds , 2007 .

[22]  R. Bhatia.  Positive Definite Matrices , 2007 .

[23]  U. Helmke,et al.  Nonsmooth Riemannian Optimization with Applications to Sphere Packing and Grasping , 2007 .

[24]  Pierre-Antoine Absil,et al.  Trust-Region Methods on Riemannian Manifolds , 2007, Found. Comput. Math..

[25]  F. Bach,et al.  Low-rank optimization for semidefinite convex problems , 2008, 0807.4423.

[26]  D. Simon,et al.  Author's Personal Copy Linear Algebra and Its Applications a Majorization Algorithm for Constrained Correlation Matrix Approximation , 2022 .

[27]  Yoram Singer,et al.  Adaptive Subgradient Methods for Online Learning and Stochastic Optimization , 2011, J. Mach. Learn. Res..

[28]  Francis R. Bach,et al.  Low-Rank Optimization on the Cone of Positive Semidefinite Matrices , 2008, SIAM J. Optim..

[29]  Ya-Xiang Yuan,et al.  Optimization Theory and Methods: Nonlinear Programming , 2010 .

[30]  Defeng Sun,et al.  A Majorized Penalty Approach for Calibrating Rank Constrained Correlation Matrix Problems , 2010 .

[31]  G. C. Bento,et al.  Convergence of inexact descent methods for nonconvex optimization on Riemannian manifolds , 2011 .

[32]  Chunhong Qi.  Numerical Optimization Methods on Riemannian Manifolds , 2011 .

[33]  Yoel Shkolnisky,et al.  Three-Dimensional Structure Determination from Common Lines in Cryo-EM by Eigenvectors and Semidefinite Programming , 2011, SIAM J. Imaging Sci..

[34]  K. Hüper,et al.  Properties of the BFGS method on Riemannian manifolds , 2012 .

[35]  Benedikt Wirth,et al.  Optimization Methods on Riemannian Manifolds and Their Application to Shape Space , 2012, SIAM J. Optim..

[36]  Yin Zhang,et al.  Solving a low-rank factorization model for matrix completion by a nonlinear successive over-relaxation algorithm , 2012, Mathematical Programming Computation.

[37]  Jérôme Malick,et al.  Projection-like Retractions on Matrix Manifolds , 2012, SIAM J. Optim..

[38]  W. Yang,et al.  Optimality conditions for the nonlinear programming problems on Riemannian manifolds , 2012 .

[39]  Bart Vandereycken,et al.  Low-Rank Matrix Completion by Riemannian Optimization , 2013, SIAM J. Optim..

[40]  Silvere Bonnabel,et al.  Stochastic Gradient Descent on Riemannian Manifolds , 2011, IEEE Transactions on Automatic Control.

[41]  Michael Ulbrich,et al.  Adaptive Regularized Self-Consistent Field Iteration with Exact Hessian for Electronic Structure Calculation , 2013, SIAM J. Sci. Comput..

[42]  Yin Zhang,et al.  Limited Memory Block Krylov Subspace Optimization for Computing Dominant Singular Value Decompositions , 2013, SIAM J. Sci. Comput..

[43]  Wen Huang,et al.  Optimization algorithms on Riemannian manifolds with applications , 2013 .

[44]  Wotao Yin,et al.  A feasible method for optimization with orthogonality constraints , 2013, Math. Program..

[45]  Ya-Xiang Yuan,et al.  On the Convergence of the Self-Consistent Field Iteration in Kohn-Sham Density Functional Theory , 2013, SIAM J. Matrix Anal. Appl..

[46]  Bart Vandereycken,et al.  Low-rank tensor completion by Riemannian optimization , 2014 .

[47]  Xin Zhang,et al.  Gradient Type Optimization Methods For Electronic Structure Calculations , 2013, SIAM J. Sci. Comput..

[48]  Lok Ming Lui,et al.  Folding-Free Global Conformal Mapping for Genus-0 Surfaces by Harmonic Energy Minimization , 2013, Journal of Scientific Computing.

[49]  Steven Thomas Smith,et al.  Optimization Techniques on Riemannian Manifolds , 2014, ArXiv.

[50]  Pierre-Antoine Absil,et al.  A Riemannian subgradient algorithm for economic dispatch with valve-point effect , 2014, J. Comput. Appl. Math..

[51]  Rongjie Lai,et al.  A Splitting Method for Orthogonality Constrained Problems , 2014, J. Sci. Comput..

[52]  Ren-Cang Li,et al.  Maximization of the sum of the trace ratio on the Stiefel manifold, I: Theory , 2014 .

[53]  Yin Zhang,et al.  An Efficient Gauss-Newton Algorithm for Symmetric Low-Rank Product Matrix Approximations , 2015, SIAM J. Optim..

[54]  Ohad Shamir,et al.  A Stochastic PCA and SVD Algorithm with an Exponential Convergence Rate , 2014, ICML.

[55]  Wen Huang,et al.  A Broyden Class of Quasi-Newton Methods for Riemannian Optimization , 2015, SIAM J. Optim..

[56]  Ren-Cang Li,et al.  Maximization of the sum of the trace ratio on the Stiefel manifold, II: Computation , 2015 .

[57]  Alexandre d'Aspremont,et al.  Phase recovery, MaxCut and complex semidefinite programming , 2012, Math. Program..

[58]  Michael Ulbrich,et al.  On the Analysis of the Discretized Kohn-Sham Density Functional Theory , 2014, SIAM J. Numer. Anal..

[59]  Xiao-Qing Jin,et al.  A Riemannian Newton Algorithm for Nonlinear Eigenvalue Problems , 2015, SIAM J. Matrix Anal. Appl..

[60]  Geoffrey E. Hinton,et al.  Deep Learning , 2015, Nature.

[61]  Bo Jiang,et al.  A framework of constraint preserving update schemes for optimization on Stiefel manifold , 2013, Math. Program..

[62]  Chao Yang,et al.  A Proximal Gradient Method for Ensemble Density Functional Theory , 2015, SIAM J. Sci. Comput..

[63]  Wen Huang,et al.  A Riemannian symmetric rank-one trust-region method , 2014, Mathematical Programming.

[64]  Chao Yang,et al.  Trace-Penalty Minimization for Large-Scale Eigenspace Computation , 2016, J. Sci. Comput..

[65]  Nicolas Boumal,et al.  The non-convex Burer-Monteiro approach works on smooth semidefinite programs , 2016, NIPS.

[66]  Andrea Montanari,et al.  Non-Negative Principal Component Analysis: Message Passing Algorithms and Sharp Asymptotics , 2014, IEEE Transactions on Information Theory.

[67]  Michael M. Bronstein,et al.  MADMM: A Generic Algorithm for Non-smooth Optimization on Manifolds , 2015, ECCV.

[68]  Nicolas Boumal,et al.  On the low-rank approach for semidefinite programs arising in synchronization and community detection , 2016, COLT.

[69]  Pierre-Antoine Absil,et al.  Robust Low-Rank Matrix Completion by Riemannian Optimization , 2016, SIAM J. Sci. Comput..

[70]  Z. Wen,et al.  A note on semidefinite programming relaxations for polynomial optimization over a single sphere , 2016 .

[71]  Rongjie Lai,et al.  Localized density matrix minimization and linear-scaling algorithms , 2016, J. Comput. Phys..

[72]  Suvrit Sra,et al.  First-order Methods for Geodesically Convex Optimization , 2016, COLT.

[73]  Suvrit Sra,et al.  Fast stochastic optimization on Riemannian manifolds , 2016, ArXiv.

[74]  P. Grohs,et al.  Nonsmooth trust region algorithms for locally Lipschitz functions on Riemannian manifolds , 2016 .

[75]  Amit Singer,et al.  Approximating the little Grothendieck problem over the orthogonal and unitary groups , 2013, Mathematical Programming.

[76]  Ya-Feng Liu,et al.  Lp-norm Regularization Algorithms for Optimization Over Permutation Matrices , 2016, SIAM J. Optim..

[77]  João X. da Cruz Neto,et al.  A New Approach to the Proximal Point Method: Convergence on General Riemannian Manifolds , 2016, J. Optim. Theory Appl..

[78]  Tony F. Chan,et al.  Guarantees of Riemannian Optimization for Low Rank Matrix Recovery , 2015, SIAM J. Matrix Anal. Appl..

[79]  Gabriele Steidl,et al.  A Second Order Nonsmooth Variational Model for Restoring Manifold-Valued Images , 2015, SIAM J. Sci. Comput..

[80]  S. Hosseini.  Convergence of nonsmooth descent methods via Kurdyka-Lojasiewicz inequality on Riemannian manifolds , 2017 .

[81]  Aihui Zhou,et al.  A Conjugate Gradient Method for Electronic Structure Calculations , 2016, SIAM J. Sci. Comput..

[82]  Hong Cheng,et al.  Accelerated First-order Methods for Geodesically Convex Optimization on Riemannian Manifolds , 2017, NIPS.

[83]  Z. Wen,et al.  Adaptive Regularized Newton Method for Riemannian Optimization , 2017, 1708.02016.

[84]  John Wright,et al.  On the Global Geometry of Sphere-Constrained Sparse Blind Deconvolution , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[85]  Minhyung Cho,et al.  Riemannian approach to batch normalization , 2017, NIPS.

[86]  Yin Zhang,et al.  Subspace Methods with Local Refinements for Eigenvalue Computation Using Low-Rank Tensor-Train Format , 2017, J. Sci. Comput..

[87]  Roger P. Pawlowski,et al.  Local Improvement Results for Anderson Acceleration with Inaccurate Function Evaluations , 2017, SIAM J. Sci. Comput..

[88]  Anthony Man-Cho So,et al.  Vector Transport-Free SVRG with General Retraction for Riemannian Optimization: Complexity Analysis and Practical Implementation , 2017, 1705.09059.

[89]  Jefferson G. Melo,et al.  Iteration-Complexity of Gradient, Subgradient and Proximal Point Methods on Riemannian Manifolds , 2016, Journal of Optimization Theory and Applications.

[90]  Wen Huang,et al.  Intrinsic representation of tangent vectors and vector transports on matrix manifolds , 2017, Numerische Mathematik.

[91]  Yin Zhang,et al.  Accelerating Convergence by Augmented Rayleigh-Ritz Projections For Large-Scale Eigenpair Computation , 2017, SIAM J. Matrix Anal. Appl..

[92]  Jian-Feng Cai,et al.  Subspace clustering by ( k , k )-sparse matrix factorization , 2017 .

[93]  Andrea Montanari,et al.  Solving SDPs for synchronization and MaxCut problems via the Grothendieck inequality , 2017, COLT.

[94]  Weizhu Bao,et al.  A Regularized Newton Method for Computing Ground States of Bose–Einstein Condensates , 2015, Journal of Scientific Computing.

[95]  Xiaojing Zhu,et al.  A Riemannian conjugate gradient method for optimization on the Stiefel manifold , 2016, Computational Optimization and Applications.

[96]  André Uschmajew,et al.  A Riemannian Gradient Sampling Algorithm for Nonsmooth Optimization on Manifolds , 2017, SIAM J. Optim..

[97]  Dustin G. Mixon,et al.  Manifold optimization for k-means clustering , 2017, 2017 International Conference on Sampling Theory and Applications (SampTA).

[98]  Shuzhong Zhang,et al.  A Cubic Regularized Newton's Method over Riemannian Manifolds , 2018, 1805.05565.

[99]  Carl Ollivier-Gooch,et al.  Adjoint-Based Functional Correction for Unstructured Mesh Finite Volume Methods , 2018, J. Sci. Comput..

[100]  Nisheeth K. Vishnoi.  Geodesic Convex Optimization: Differentiation on Manifolds, Geodesics, and Convexity , 2018, ArXiv.

[101]  Zhaojun Bai,et al.  On an Eigenvector-Dependent Nonlinear Eigenvalue Problem , 2018, SIAM J. Matrix Anal. Appl..

[102]  Gabriel Haeser,et al.  Augmented Lagrangians with constrained subproblems and convergence to second-order stationary points , 2018, Comput. Optim. Appl..

[103]  Bruno Iannazzo,et al.  The Riemannian Barzilai–Borwein method with nonmonotone line search and the matrix geometric mean computation , 2018 .

[104]  Nicolas Boumal,et al.  Adaptive regularization with cubics on manifolds with a first-order analysis , 2018 .

[105]  Feiyu Chen,et al.  Non-convex clustering via proximal alternating linearized minimization method , 2018, Int. J. Wavelets Multiresolution Inf. Process..

[106]  Wen Huang,et al.  A Riemannian BFGS Method Without Differentiated Retraction for Nonconvex Optimization Problems , 2018, SIAM J. Optim..

[107]  Anthony Man-Cho So,et al.  Proximal Gradient Method for Manifold Optimization , 2018, 1811.00980.

[108]  Yongfeng Li,et al.  A Regularized Semi-Smooth Newton Method with Projection Steps for Composite Convex Programs , 2016, J. Sci. Comput..

[109]  Zhaojun Bai,et al.  Robust Rayleigh Quotient Minimization and Nonlinear Eigenvalue Problems , 2018, SIAM J. Sci. Comput..

[110]  Tong Zhang,et al.  Near-optimal stochastic approximation for online principal component estimation , 2016, Math. Program..

[111]  Ya-Xiang Yuan,et al.  Adaptive Quadratically Regularized Newton Method for Riemannian Optimization , 2018, SIAM J. Matrix Anal. Appl..

[112]  Shuzhong Zhang,et al.  A Sparse Completely Positive Relaxation of the Modularity Maximization for Community Detection , 2017, SIAM J. Sci. Comput..

[113]  Xiaojun Chen,et al.  A New First-Order Algorithmic Framework for Optimization Problems with Orthogonality Constraints , 2018, SIAM J. Optim..

[114]  Bo Jiang,et al.  Structured Quasi-Newton Methods for Optimization with Orthogonality Constraints , 2018, SIAM J. Sci. Comput..

[115]  Gary Bécigneul,et al.  Riemannian Adaptive Optimization Methods , 2018, ICLR.

[116]  Rongjie Lai,et al.  Global Optimization with Orthogonality Constraints via Stochastic Diffusion on Manifold , 2017, Journal of Scientific Computing.

[117]  Ke Wei,et al.  Riemannian proximal gradient methods , 2019, Mathematical Programming.

[118]  Wotao Yin,et al.  Global Convergence of ADMM in Nonconvex Nonsmooth Optimization , 2015, Journal of Scientific Computing.

[119]  Shiqian Ma,et al.  An Alternating Manifold Proximal Gradient Method for Sparse PCA and Sparse CCA , 2019, ArXiv.

[120]  Ke Wei,et al.  Extending FISTA to Riemannian Optimization for Sparse PCA. , 2019 .

[121]  Jonathan W. Siegel.  Accelerated Optimization with Orthogonality Constraints , 2019, 1903.05204.

[122]  Yang Wang,et al.  Fast Rank-One Alternating Minimization Algorithm for Phase Retrieval , 2017, Journal of Scientific Computing.

[123]  Nicolas Boumal,et al.  Simple Algorithms for Optimization on Riemannian Manifolds with Constraints , 2019, Applied Mathematics & Optimization.

[124]  Hiroyuki Kasai,et al.  Riemannian stochastic variance reduced gradient , 2016, SIAM J. Optim..

[125]  P. Absil,et al.  Erratum to: ``Global rates of convergence for nonconvex optimization on manifolds'' , 2016, IMA Journal of Numerical Analysis.

[126]  Mario Lezcano Casado,et al.  Cheap Orthogonal Constraints in Neural Networks: A Simple Parametrization of the Orthogonal and Unitary Group , 2019, ICML.

[127]  A. Bandeira,et al.  Deterministic Guarantees for Burer‐Monteiro Factorizations of Smooth Semidefinite Programs , 2018, Communications on Pure and Applied Mathematics.

[128]  Shiqian Ma,et al.  Proximal Gradient Method for Nonsmooth Optimization over the Stiefel Manifold , 2018, SIAM J. Optim..

[129]  Shiqian Ma,et al.  Primal-dual optimization algorithms over Riemannian manifolds: an iteration complexity analysis , 2017, Mathematical Programming.

[130]  Jun Li,et al.  Efficient Riemannian Optimization on the Stiefel Manifold via the Cayley Transform , 2020, ICLR.

[131]  Pierre-Antoine Absil,et al.  Quotient Geometry with Simple Geodesics for the Manifold of Fixed-Rank Positive-Semidefinite Matrices , 2020, SIAM J. Matrix Anal. Appl..

[132]  P. Alam.  ‘Z’ , 2021, Composites Engineering: An A–Z Guide.

[133]  P. Alam.  ‘A’ , 2021, Composites Engineering: An A–Z Guide.