A Levenberg-Marquardt method for large-scale bound-constrained nonlinear least-squares

The well known Levenberg-Marquardt method is used extensively for solving nonlinear least-squares problems. We describe an extension of the LevenbergMarquardt method to problems with bound constraints on the variables. Each iteration of our algorithm approximately solves a linear least-squares problem subject to the original bound constraints. Our approach is especially suited to large-scale problems whose functions are expensive to compute; only matrixvector products with the Jacobian are required. We present the results of numerical experiments that illustrate the effectiveness of the approach. Moreover, we describe its application to a practical curve fitting problem in fluorescence optical imaging.

[1]  Jinyan Fan,et al.  Convergence Properties of a Self-adaptive Levenberg-Marquardt Algorithm Under Local Error Bound Condition , 2006, Comput. Optim. Appl..

[2]  Nicholas I. M. Gould,et al.  Lancelot: A FORTRAN Package for Large-Scale Nonlinear Optimization (Release A) , 1992 .

[3]  H. B. Nielsen.  DAMPING PARAMETER IN MARQUARDT ’ S METHOD , 1999 .

[4]  Jorge J. Moré,et al.  Digital Object Identifier (DOI) 10.1007/s101070100263 , 2001 .

[5]  D. Marquardt.  An Algorithm for Least-Squares Estimation of Nonlinear Parameters , 1963 .

[6]  Michael P. Friedlander,et al.  Computing non-negative tensor factorizations , 2008, Optim. Methods Softw..

[7]  Jorge J. Moré,et al.  Recent Developments in Algorithms and Software for Trust Region Methods , 1982, ISMP.

[8]  William W. Hager,et al.  A New Active Set Algorithm for Box Constrained Optimization , 2006, SIAM J. Optim..

[9]  Chih-Jen Lin,et al.  Newton's Method for Large Bound-Constrained Optimization Problems , 1999, SIAM J. Optim..

[10]  Jorge J. Moré,et al.  Testing Unconstrained Optimization Software , 1981, TOMS.

[11]  J. Navarro-Pedreño.  Numerical Methods for Least Squares Problems , 1996 .

[12]  Jin-yanFan.  A MODIFIED LEVENBERG-MARQUARDT ALGORITHM FOR SINGULAR SYSTEM OF NONLINEAR EQUATIONS , 2003 .

[13]  Jorge J. Moré,et al.  Algorithm 566: FORTRAN Subroutines for Testing Unconstrained Optimization Software [C5], [E4] , 1981, TOMS.

[14]  Gene H. Golub,et al.  Matrix computations , 1983 .

[15]  A. N. Tikhonov,et al.  Solutions of ill-posed problems , 1977 .

[16]  John E. Dennis,et al.  An Adaptive Nonlinear Least-Squares Algorithm , 1977, TOMS.

[17]  David A Boas,et al.  Comparison of simplified Monte Carlo simulation and diffusion approximation for the fluorescence signal from phantoms with typical mouse tissue optical properties. , 2007, Applied optics.

[18]  Kenneth Levenberg.  A METHOD FOR THE SOLUTION OF CERTAIN NON – LINEAR PROBLEMS IN LEAST SQUARES , 1944 .

[19]  Jorge J. Moré,et al.  The Levenberg-Marquardt algo-rithm: Implementation and theory , 1977 .

[20]  Frederic Leblond,et al.  Fluorescence lifetime estimation of multiple near-infrared dyes in mice , 2008, SPIE BiOS.

[21]  M. Powell.  CONVERGENCE PROPERTIES OF A CLASS OF MINIMIZATION ALGORITHMS , 1975 .

[22]  M. Friedlander,et al.  Computing non-negative tensor factorizations , 2008, Optim. Methods Softw..

[23]  Jorge J. Moré,et al.  User Guide for Minpack-1 , 1980 .

[24]  Jorge Nocedal,et al.  Algorithm 778: L-BFGS-B: Fortran subroutines for large-scale bound-constrained optimization , 1997, TOMS.

[25]  P. Toint,et al.  Global convergence of a class of trust region algorithms for optimization with simple bounds , 1988 .

[26]  Qingying Sun,et al.  A self-adaptive trust region method with line search based on a simple subproblem model , 2009, J. Comput. Appl. Math..

[27]  Ya-Xiang Yuan,et al.  Trust Region Algorithms for Nonlinear Equations , 1994 .

[28]  D. D. Morrison.  Methods for nonlinear least squares problems and convergence proofs , 1960 .

[29]  L. Trefethen,et al.  Numerical linear algebra , 1997 .

[30]  H. Zhangl.  RECENT ADVANCES IN BOUND CONSTRAINED OPTIMIZATION , 2006 .

[31]  R. Dembo,et al.  INEXACT NEWTON METHODS , 1982 .

[32]  C. Fraley.  Solution of nonlinear least-squares problems , 1987 .

[33]  P. Toint,et al.  Testing a class of methods for solving minimization problems with simple bounds on the variables , 1988 .

[34]  Michael A. Saunders,et al.  LSQR: An Algorithm for Sparse Linear Equations and Sparse Least Squares , 1982, TOMS.

[35]  M. Fukushima,et al.  On the Rate of Convergence of the Levenberg-Marquardt Method , 2001 .

[36]  Gerardo Toraldo,et al.  Convergence properties of trust region methods for linear and convex constraints , 1990, Math. Program..

[37]  Yong Wang,et al.  Simple time-domain optical method for estimating the depth and concentration of a fluorescent inclusion in a turbid medium. , 2004, Optics letters.

[38]  Michael P. Friedlander,et al.  Diffuse optical fluorescence tomography using time-resolved data acquired in transmission , 2007, SPIE BiOS.

[39]  Jorge Nocedal,et al.  A Limited Memory Algorithm for Bound Constrained Optimization , 1995, SIAM J. Sci. Comput..

[40]  D K Smith,et al.  Numerical Optimization , 2001, J. Oper. Res. Soc..

[41]  J. Lakowicz.  Principles of fluorescence spectroscopy , 1983 .

[42]  Nicholas I. M. Gould,et al.  CUTEr and SifDec: A constrained and unconstrained testing environment, revisited , 2003, TOMS.

[43]  Jorge Nocedal,et al.  On the limited memory BFGS method for large scale optimization , 1989, Math. Program..

[44]  Tamar Schlick,et al.  Remark on Algorithm 566 , 1994, TOMS.

[45]  Charles L. Lawson,et al.  Solving least squares problems , 1976, Classics in applied mathematics.

[46]  John E. Dennis,et al.  Algorithm 573: NL2SOL—An Adaptive Nonlinear Least-Squares Algorithm [E4] , 1981, TOMS.

[47]  Philip E. Gill,et al.  Practical optimization , 1981 .

[48]  Dimitri P. Bertsekas,et al.  Constrained Optimization and Lagrange Multiplier Methods , 1982 .

[49]  Stephen J. Wright,et al.  Optimization Software Guide , 1987 .

[50]  Prof. Dr. K. Schittkowski.  DFNLP: A Fortran Implementation of an SQP-Gauss-Newton Algorithm - User’s Guide, Version 2.0 - , 2007 .

[51]  J. Nocedal,et al.  A Limited Memory Algorithm for Bound Constrained Optimization , 1995, SIAM J. Sci. Comput..

[52]  Masao Fukushima,et al.  Convergence Properties of the Inexact Levenberg-Marquardt Method under Local Error Bound Conditions , 2002, Optim. Methods Softw..