A Levenberg–Marquardt method for large nonlinear least-squares problems with dynamic accuracy in functions and gradients

In this paper we consider large scale nonlinear least-squares problems for which function and gradient are evaluated with dynamic accuracy and propose a Levenberg–Marquardt method for solving such problems. More precisely, we consider the case in which the exact function to optimize is not available or its evaluation is computationally demanding, but approximations of it are available at any prescribed accuracy level. The proposed method relies on a control of the accuracy level, and imposes an improvement of function approximations when the accuracy is detected to be too low to proceed with the optimization process. We prove global and local convergence and complexity of our procedure and show encouraging numerical results on test problems arising in data assimilation and machine learning.

[1]  Mark W. Schmidt,et al.  Hybrid Deterministic-Stochastic Methods for Data Fitting , 2011, SIAM J. Sci. Comput..

[2]  Serge Gratton,et al.  Preconditioning and globalizing conjugate gradients in dual space for quadratically penalized nonlinear-least squares problems , 2013, Comput. Optim. Appl..

[3]  Jérôme Vialard,et al.  Three- and Four-Dimensional Variational Assimilation with a General Circulation Model of the Tropical Pacific Ocean. Part I: Formulation, Internal Diagnostics, and Consistency Checks , 2003 .

[4]  Yurii Nesterov,et al.  Introductory Lectures on Convex Optimization - A Basic Course , 2014, Applied Optimization.

[5]  Katya Scheinberg,et al.  Stochastic optimization using a trust-region method and random models , 2015, Mathematical Programming.

[6]  Barbara Kaltenbacher,et al.  Iterative Regularization Methods for Nonlinear Ill-Posed Problems , 2008, Radon Series on Computational and Applied Mathematics.

[7]  José Mario Martínez,et al.  Inexact Restoration approach for minimization with inexact evaluation of the objective function , 2015, Math. Comput..

[8]  Stephen J. Wright,et al.  Numerical Optimization , 2018, Fundamental Statistical Inference.

[9]  Serge Gratton,et al.  Levenberg-Marquardt Methods Based on Probabilistic Gradient Models and Inexact Subproblem Solution, with Application to Data Assimilation , 2016, SIAM/ASA J. Uncertain. Quantification.

[10]  E. RICCIETTI.  ON NON-STATIONARY TIKHONOV PROCEDURES FOR ILL-POSED NONLINEAR LEAST SQUARES PROBLEMS , 2017 .

[11]  Nicholas I. M. Gould,et al.  Trust Region Methods , 2000, MOS-SIAM Series on Optimization.

[12]  D. Darling,et al.  A Test of Goodness of Fit , 1954 .

[13]  D K Smith,et al.  Numerical Optimization , 2001, J. Oper. Res. Soc..

[14]  Jinyan Fan,et al.  Global complexity bound of the Levenberg–Marquardt method , 2016, Optim. Methods Softw..

[15]  Michael W. Mahoney,et al.  Sub-Sampled Newton Methods I: Globally Convergent Algorithms , 2016, ArXiv.

[16]  Katya Scheinberg,et al.  Convergence of Trust-Region Methods Based on Probabilistic Models , 2013, SIAM J. Optim..

[17]  Natasa Krejic,et al.  Nonmonotone line search methods with variable sample size , 2015, Numerical Algorithms.

[18]  J. Blanchet,et al.  Convergence Rate Analysis of a Stochastic Trust Region Method for Nonconvex Optimization , 2016 .

[19]  Stefania Bellavia,et al.  On an Elliptical Trust-Region Procedure for Ill-Posed Nonlinear Least-Squares Problems , 2018, J. Optim. Theory Appl..

[20]  Y. Trémolet.  Model‐error estimation in 4D‐Var , 2007 .

[21]  Stefania Bellavia,et al.  On an adaptive regularization for ill-posed nonlinear systems and its trust-region implementation , 2015, Comput. Optim. Appl..

[22]  P. Courtier,et al.  A strategy for operational implementation of 4D‐Var, using an incremental approach , 1994 .

[23]  Katya Scheinberg,et al.  Introduction to derivative-free optimization , 2010, Math. Comput..

[24]  M. Stephens.  EDF Statistics for Goodness of Fit and Some Comparisons , 1974 .

[25]  M. Hanke.  A regularizing Levenberg - Marquardt scheme, with applications to inverse groundwater filtration problems , 1997 .

[26]  Jorge Nocedal,et al.  Optimization Methods for Large-Scale Machine Learning , 2016, SIAM Rev..

[27]  Yannick Trémolet,et al.  520 Model error estimation in 4 D-Var Yannick Trémolet , 2007 .

[28]  Serge Gratton,et al.  Observation thinning in data assimilation computations , 2015, EURO J. Comput. Optim..

[29]  Peter Richtárik,et al.  Iteration complexity of randomized block-coordinate descent methods for minimizing a composite function , 2011, Mathematical Programming.

[30]  J. Nocedal,et al.  Exact and Inexact Subsampled Newton Methods for Optimization , 2016, 1609.08502.