Using negative curvature in solving nonlinear programs

Minimization methods that search along a curvilinear path composed of a non-ascent negative curvature direction in addition to the direction of steepest descent, dating back to the late 1970s, have been an effective approach to finding a stationary point of a function at which its Hessian is positive semidefinite. For constrained nonlinear programs arising from recent applications, the primary goal is to find a stationary point that satisfies the second-order necessary optimality conditions. Motivated by this, we generalize the approach of using negative curvature directions from unconstrained optimization to equality constrained problems and prove that our proposed negative curvature method is guaranteed to converge to a stationary point satisfying second-order necessary conditions.

[1]  Stefano Lucidi,et al.  Curvilinear Stabilization Techniques for Truncated Newton Methods in Large Scale Unconstrained Optimization , 1998, SIAM J. Optim..

[2]  Donald Goldfarb,et al.  Successive Rank-One Approximations for Nearly Orthogonally Decomposable Symmetric Tensors , 2015, SIAM J. Matrix Anal. Appl..

[3]  Domenico Conforti,et al.  A curvilinear search algorithm for unconstrained optimization by automatic differentiation , 2001 .

[4]  Michael C. Ferris,et al.  Nonmonotone curvilinear line search methods for unconstrained optimization , 1996, Comput. Optim. Appl..

[5]  Nicolas Boumal,et al.  Nonconvex Phase Synchronization , 2016, SIAM J. Optim..

[6]  S. Lucidi,et al.  Exploiting negative curvature directions in linesearch methods for unconstrained optimization , 2000 .

[7]  Stephen J. Wright,et al.  Numerical Optimization , 2018, Fundamental Statistical Inference.

[8]  Gábor Pataki,et al.  On the Rank of Extreme Matrices in Semidefinite Programs and the Multiplicity of Optimal Eigenvalues , 1998, Math. Oper. Res..

[9]  Yun S. Song,et al.  Orthogonal Tensor Decompositions via Two-Mode Higher-Order SVD (HOSVD) , 2016, ArXiv.

[10]  Tengyu Ma,et al.  Matrix Completion has No Spurious Local Minimum , 2016, NIPS.

[11]  Furong Huang,et al.  Escaping From Saddle Points - Online Stochastic Gradient for Tensor Decomposition , 2015, COLT.

[12]  Anima Anandkumar,et al.  Tensor decompositions for learning latent variable models , 2012, J. Mach. Learn. Res..

[13]  Robert E. Mahony,et al.  Optimization Algorithms on Matrix Manifolds , 2007 .

[14]  Pierre Comon,et al.  Handbook of Blind Source Separation: Independent Component Analysis and Applications , 2010 .

[15]  John Wright,et al.  A Geometric Analysis of Phase Retrieval , 2016, International Symposium on Information Theory.

[16]  D K Smith,et al.  Numerical Optimization , 2001, J. Oper. Res. Soc..

[17]  Andreas Stathopoulos,et al.  PRIMME: preconditioned iterative multimethod eigensolver—methods and software description , 2010, TOMS.

[18]  Donald Goldfarb,et al.  Curvilinear path steplength algorithms for minimization which use directions of negative curvature , 1980, Math. Program..

[19]  David P. Williamson,et al.  Improved approximation algorithms for maximum cut and satisfiability problems using semidefinite programming , 1995, JACM.

[20]  R. Mahony,et al.  All roads lead to Newton: Feasible second-order methods for equality-constrained optimization ∗ , 2009 .

[21]  P. McCullagh.  Tensor Methods in Statistics , 1987 .

[22]  Robert E. Mahony,et al.  An Extrinsic Look at the Riemannian Hessian , 2013, GSI.

[23]  L. Grandinetti.  Nonlinear optimization by a curvilinear path strategy , 1984 .

[24]  Alexander I. Barvinok,et al.  Problems of distance geometry and convex properties of quadratic maps , 1995, Discret. Comput. Geom..

[25]  Danny C. Sorensen,et al.  On the use of directions of negative curvature in a modified newton method , 1979, Math. Program..

[26]  José Mario Martínez,et al.  An algorithm for solving nonlinear least-squares problems with a new curvilinear search , 2005, Computing.

[27]  Wotao Yin,et al.  A feasible method for optimization with orthogonality constraints , 2013, Math. Program..

[28]  Donald Goldfarb,et al.  Greedy Approaches to Symmetric Orthogonal Tensor Decomposition , 2017, SIAM J. Matrix Anal. Appl..

[29]  Garth P. McCormick,et al.  A modification of Armijo's step-size rule for negative curvature , 1977, Math. Program..

[30]  Renato D. C. Monteiro,et al.  Digital Object Identifier (DOI) 10.1007/s10107-004-0564-1 , 2004 .

[31]  Pierre Comon,et al.  Independent component analysis, A new concept? , 1994, Signal Process..

[32]  Renato D. C. Monteiro,et al.  A nonlinear programming algorithm for solving semidefinite programs via low-rank factorization , 2003, Math. Program..

[33]  John Wright,et al.  Complete dictionary recovery over the sphere , 2015, 2015 International Conference on Sampling Theory and Applications (SampTA).

[34]  Nicolas Boumal,et al.  On the low-rank approach for semidefinite programs arising in synchronization and community detection , 2016, COLT.

[35]  D. G. Sotiropoulos,et al.  A curvilinear method based on minimal-memory BFGS updates , 2010, Appl. Math. Comput..

[36]  Nicolas Boumal,et al.  The non-convex Burer-Monteiro approach works on smooth semidefinite programs , 2016, NIPS.

[37]  I. Holopainen.  Riemannian Geometry , 1927, Nature.