Many techniques for solving general nonlinear unconstrained optimization problems involve iteratively minimizing a model function that satisfies certain interpolation conditions. These conditions provide a model that behaves like the objective function in the neighborhood of the current iterate. The model functions often involve second-order derivatives of the objective function, which can be expensive to calculate. The fundamental idea behind quasi-Newton methods is to maintain an approximation to the Hessian matrix. The practical success of quasi-Newton methods has spurred a great deal of interest and research that has resulted in a considerable number of variations of this idea. The analytical difficulties associated with characterizing the performance of these algorithms means there is a real need for practical testing to support theoretical claims. The goal of this project is to describe, implement, and test these methods in a way that is uniform, systematic, and consistent. In the first part of the paper, we derive several classical quasi-Newton methods, discuss their relative benefits, and show how to implement them. In the second part, we investigate more recent variations, explain their motivation and theory, and analyze their performance.
[1]
Roger Fletcher,et al.
A Rapidly Convergent Descent Method for Minimization
,
1963,
Comput. J..
[2]
L. Armijo.
Minimization of functions having Lipschitz continuous first partial derivatives.
,
1966
.
[3]
J. D. Pearson.
ON VARIABLE METRIC METHODS OF MINIMIZATION
,
1968
.
[4]
James M. Ortega,et al.
Iterative solution of nonlinear equations in several variables
,
2014,
Computer science and applied mathematics.
[5]
P. Wolfe.
On the convergence of gradient methods under constraint
,
1972
.
[6]
Oren.
Self-scaling variable metric algorithms without line search for unconstrained minimization
,
1973
.
[7]
Shmuel S. Oren,et al.
On the selection of parameters in Self Scaling Variable Metric Algorithms
,
1974,
Math. Program..
[8]
S. Oren.
SELF-SCALING VARIABLE METRIC (SSVM) ALGORITHMS Part II: Implementation and Experiments*t
,
1974
.