Convergence Properties of the BFGS Algoritm

The BFGS method is one of the most famous quasi-Newton algorithms for unconstrained optimization. In 1984, Powell presented an example of a function of two variables that shows that the Polak--Ribiere--Polyak (PRP) conjugate gradient method and the BFGS quasi-Newton method may cycle around eight nonstationary points if each line search picks a local minimum that provides a reduction in the objective function. In this paper, a new technique of choosing parameters is introduced, and an example with only six cyclic points is provided. It is also noted through the examples that the BFGS method with Wolfe line searches need not converge for nonconvex objective functions.

[1]  Boris Polyak.  The conjugate gradient method in extremal problems , 1969 .

[2]  D. Goldfarb.  A family of variable-metric methods derived by variational means , 1970 .

[3]  C. G. Broyden.  The Convergence of a Class of Double-rank Minimization Algorithms 2. The New Algorithm , 1970 .

[4]  J. D. Pearson.  ON VARIABLE METRIC METHODS OF MINIMIZATION , 1968 .

[5]  J. Nocedal,et al.  Global Convergence of a Class of Quasi-newton Methods on Convex Problems, Siam Some Global Convergence Properties of a Variable Metric Algorithm for Minimization without Exact Line Searches, Nonlinear Programming, Edited , 1996 .

[6]  D. Shanno.  Conditioning of Quasi-Newton Methods for Function Minimization , 1970 .

[7]  M. Hestenes,et al.  Methods of conjugate gradients for solving linear systems , 1952 .

[8]  C. G. Broyden.  The Convergence of a Class of Double-rank Minimization Algorithms 1. General Considerations , 1970 .

[9]  M. Powell.  Nonconvex minimization calculations and the conjugate gradient method , 1984 .

[10]  E. Polak,et al.  Note sur la convergence de méthodes de directions conjuguées , 1969 .

[11]  M. J. D. Powell.  Quadratic Termination Properties of Minimization Algorithms II. Proofs of Theorems , 1972 .

[12]  R. Fletcher,et al.  A New Approach to Variable Metric Algorithms , 1970, Comput. J..

[13]  Roger Fletcher,et al.  A Rapidly Convergent Descent Method for Minimization , 1963, Comput. J..

[14]  M. Powell.  On the Convergence of the Variable Metric Algorithm , 1971 .

[15]  M. J. D. Powell,et al.  On the convergence of the DFP algorithm for unconstrained optimization when there are only two variables , 2000, Math. Program..

[16]  Jorge Nocedal,et al.  Theory of algorithms for unconstrained optimization , 1992, Acta Numerica.

[17]  D. Pu,et al.  On the convergence property of the DFP algorithm , 1990 .

[18]  L. Dixon.  Variable metric algorithms: Necessary and sufficient conditions for identical behavior of nonquadratic functions , 1972 .

[19]  N. S. Barnett,et al.  Private communication , 1969 .

[20]  P. Wolfe.  Convergence Conditions for Ascent Methods. II , 1969 .