Convergence of the Iterates of Descent Methods for Analytic Cost Functions

In the early eighties Lojasiewicz [in Seminari di Geometria 1982-1983, Universita di Bologna, Istituto di Geometria, Dipartimento di Matematica, 1984, pp. 115--117] proved that a bounded solution of a gradient flow for an analytic cost function converges to a well-defined limit point. In this paper, we show that the iterates of numerical descent algorithms, for an analytic cost function, share this convergence property if they satisfy certain natural descent conditions. The results obtained are applicable to a broad class of optimization schemes and strengthen classical "weak convergence" results for descent methods to "strong limit-point convergence" for a large class of cost functions of practical interest. The result does not require that the cost has isolated critical points and requires no assumptions on the convexity of the cost nor any nondegeneracy conditions on the Hessian of the cost at critical points.

[1]  H. B. Curry.  The method of steepest descent for non-linear minimization problems , 1944 .

[2]  S. Łojasiewicz.  Sur le problème de la division , 1959 .

[3]  S. Łojasiewicz.  Ensembles semi-analytiques , 1965 .

[4]  P. Wolfe.  Convergence Conditions for Ascent Methods. II , 1969 .

[5]  M. Powell.  A New Algorithm for Unconstrained Optimization , 1970 .

[6]  G. Zoutendijk,et al.  Mathematical Programming Methods , 1976 .

[7]  Katta G. Murty,et al.  Mathematical Programming Methods , 1977 .

[8]  J. Dennis,et al.  Two new unconstrained optimization algorithms which use function and gradient values , 1979 .

[9]  J. Dunn.  Global and Asymptotic Convergence Rate Estimates for a Class of Projected Gradient Processes , 1981 .

[10]  Philippe L. Toint,et al.  Towards an efficient sparsity exploiting newton method for minimization , 1981 .

[11]  J. Palis,et al.  Geometric theory of dynamical systems , 1982 .

[12]  T. Steihaug.  The Conjugate Gradient Method and Trust Regions in Large Scale Optimization , 1983 .

[13]  Jorge J. Moré,et al.  Computing a Trust Region Step , 1983 .

[14]  John E. Dennis,et al.  Numerical methods for unconstrained optimization and nonlinear equations , 1983, Prentice Hall series in computational mathematics.

[15]  J. Palis,et al.  Geometric theory of dynamical systems : an introduction , 1984 .

[16]  Roger Fletcher,et al.  Practical methods of optimization; (2nd ed.) , 1987 .

[17]  J. Dunn.  On the convergence of projected gradient processes to singular critical points , 1987 .

[18]  R. Fletcher.  Practical Methods of Optimization , 1988 .

[19]  E. Bierstone,et al.  Semianalytic and subanalytic sets , 1988 .

[20]  Richard H. Byrd,et al.  Approximate solution of the trust region problem by minimization over two-dimensional subspaces , 1988, Math. Program..

[21]  J. Nocedal,et al.  A tool for the analysis of Quasi-Newton methods with application to unconstrained minimization , 1989 .

[22]  S. Wiggins.  Introduction to Applied Nonlinear Dynamical Systems and Chaos , 1989 .

[23]  Shirley Dex,et al.  JR 旅客販売総合システム（マルス）における運用及び管理について , 1991 .

[24]  Nicholas I. M. Gould,et al.  Global Convergence of a Class of Trust Region Algorithms for Optimization Using Inexact Projections on Convex Constraints , 1993, SIAM J. Optim..

[25]  S. Łojasiewicz.  Sur la géométrie semi- et sous- analytique , 1993 .

[26]  Zhi-Quan Luo,et al.  Error bounds for analytic systems and their applications , 1994, Math. Program..

[27]  K. Kurdyka,et al.  Wf-stratification of subanalytic functions and the Lojasiewicz inequality , 1994 .

[28]  K. G. Murty,et al.  Convergence of the steepest descent method for minimizing quasiconvex functions , 1996 .

[29]  U. Helmke,et al.  Optimization and Dynamical Systems , 1994, Proceedings of the IEEE.

[30]  K. Kurdyka.  On gradients of functions definable in o-minimal structures , 1998 .

[31]  K. Kurdyka,et al.  Proof of the gradient conjecture of R. Thom , 1999, math/9906212.

[32]  T. Mostowski,et al.  Proof of the gradient conjecture of , 1999 .

[33]  Jean-Pierre Dedieu,et al.  Approximate Solutions of Analytic Inequality Systems , 2000, SIAM J. Optim..

[34]  Francisco Facchinei,et al.  On the Identification of Zero Variables in an Interior-Point Framework , 1999, SIAM J. Optim..

[35]  Nicholas I. M. Gould,et al.  Trust Region Methods , 2000, MOS-SIAM Series on Optimization.

[36]  D K Smith,et al.  Numerical Optimization , 2001, J. Oper. Res. Soc..

[37]  Nobuo Yamashita,et al.  On the identification of degenerate indices in the nonlinear complementarity problem with the proximal point algorithm , 2004, Math. Program..

[38]  K. Schittkowski,et al.  NONLINEAR PROGRAMMING , 2022 .