Secant variable projection method for solving nonnegative separable least squares problems

The variable projection method is a classical and efficient method for solving separable nonlinear least squares (SNLLS) problems. However, it is hard to handle the constrained SNLLS problems since the explicit form of the Jacobian matrix is required in each iteration. In this paper, we propose a secant variable projection (SVP) method, which employs a rank-one update to estimate the Jacobian matrices. The main advantages of our method are efficiency and ease of applicability to constrained SNLLS problems. The local convergence of our SVP method is also analyzed. Finally, some data fitting and image processing problems are solved to compare the performance of our proposed method with the classical variable projection method. Numerical results illustrate the efficiency and stability of our proposed SVP method in solving the SNLLS problems arising from the blind deconvolution problems.

[1]  D. D. Morrison.  Methods for nonlinear least squares problems and convergence proofs , 1960 .

[2]  E. A. Sylvestre,et al.  Elimination of Linear Parameters in Nonlinear Regression , 1971 .

[3]  L. Kaufman.  A variable projection method for solving separable nonlinear least squares problems , 1974 .

[4]  L. Kaufman,et al.  A Method for Separable Nonlinear Least Squares Problems with Separable Nonlinear Equality Constraints , 1978 .

[5]  Axel Ruhe,et al.  Algorithms for separable nonlinear least squares problems , 1980 .

[6]  John E. Dennis,et al.  Numerical methods for unconstrained optimization and nonlinear equations , 1983, Prentice Hall series in computational mathematics.

[7]  Stephen J. Wright,et al.  Numerical Optimization , 2018, Fundamental Statistical Inference.

[8]  D K Smith,et al.  Numerical Optimization , 2001, J. Oper. Res. Soc..

[9]  G. Golub,et al.  Separable nonlinear least squares: the variable projection method and its applications , 2003 .

[10]  James G. Nagy,et al.  Iterative Methods for Image Deblurring: A Matlab Object-Oriented Approach , 2004, Numerical Algorithms.

[11]  J. Nagy,et al.  Numerical methods for coupled super-resolution , 2006 .

[12]  J. Nagy,et al.  A weighted-GCV method for Lanczos-hybrid regularization. , 2007 .

[13]  S. Huffel,et al.  Separable nonlinear least squares fitting with linear bound constraints and its application in magnetic resonance spectroscopy data quantification , 2007 .

[14]  Gene H. Golub,et al.  The differentiation of pseudo-inverses and non-linear least squares problems whose variables separate , 1972, Milestones in Matrix Computation.

[15]  Dianne P. O'Leary,et al.  Deblurring Images: Matrices, Spectra and Filtering , 2006, J. Electronic Imaging.

[16]  Ju-Jang Lee,et al.  Training Two-Layered Feedforward Networks With Variable Projection Method , 2008, IEEE Transactions on Neural Networks.

[17]  Katharine M. Mullen,et al.  The variable projection algorithm in time-resolved spectroscopy, microscopy and mass spectrometry applications , 2009, Numerical Algorithms.

[18]  James G. Nagy,et al.  An Efficient Iterative Approach for Large-Scale Separable Nonlinear Inverse Problems , 2009, SIAM J. Sci. Comput..

[19]  P. Hansen.  Discrete Inverse Problems: Insight and Algorithms , 2010 .

[20]  Per Christian Hansen,et al.  Least Squares Data Fitting with Applications , 2012 .

[21]  Wei Xu,et al.  A secant method for nonlinear least-squares minimization , 2012, Comput. Optim. Appl..

[22]  James G. Nagy,et al.  Constrained numerical optimization methods for blind deconvolution , 2013, Numerical Algorithms.

[23]  Dianne P. O'Leary,et al.  Variable projection for nonlinear least squares problems , 2012, Computational Optimization and Applications.

[24]  J. Nagy,et al.  Iterative Methods for Image Restoration , 2014 .

[25]  Wei Xu,et al.  Automatic differentiation in MATLAB using ADMAT with applications , 2016, Software, environments, and tools.

[26]  Ning Zheng,et al.  Modulus-Type Inner Outer Iteration Methods for Nonnegative Constrained Least Squares Problems , 2016, SIAM J. Matrix Anal. Appl..

[27]  Lars Ruthotto,et al.  LAP: a Linearize and Project Method for Solving Inverse Problems with Coupled Variables , 2017, ArXiv.

[28]  T. Ypma,et al.  Deblurring Images , 2020 .