Separable nonlinear least squares: the variable projection method and its applications

In this paper we review 30 years of developments and applications of the variable projection method for solving separable nonlinear least-squares problems. These are problems for which the model function is a linear combination of nonlinear functions. Taking advantage of this special structure, the method of variable projections eliminates the linear variables obtaining a somewhat more complicated function that involves only the nonlinear parameters. This procedure not only reduces the dimension of the parameter space but also results in a better-conditioned problem. The same optimization method applied to the original and reduced problems will always converge faster for the latter. We present first a historical account of the basic theoretical work and its various computer implementations, and then report on a variety of applications from electrical engineering, medical and biological imaging, chemistry, robotics, vision, and environmental sciences. An extensive bibliography is included. The method is particularly well suited for solving real and complex exponential model fitting problems, which are pervasive in their applications and are notoriously hard to solve.

[1]  Victor Pereyra,et al.  Accelerating the Convergence of Discretization Algorithms , 1967 .

[2]  Victor Pereyra,et al.  Iterative methods for solving nonlinear least squares problems , 1967 .

[3]  J. Capon.  High-resolution frequency-wavenumber spectrum analysis , 1969 .

[4]  R. Brent.  Algorithms for finding zeros and extrema of functions without calculating derivatives , 1971 .

[5]  Hugo D. Scolnik,et al.  On the Solution of Non-linear Least Squares Problems , 1971, IFIP Congress.

[6]  V. Pereyra,et al.  Efficient Computer Manipulation of Tensor Products with Applications to Multidimensional Approximation , 1973 .

[7]  V. Pereyra,et al.  Least Squares Estimation for a Class of Non-Linear Models , 1973 .

[8]  P. Wedin.  Perturbation theory for pseudo-inverses , 1973 .

[9]  Fred T. Krogh,et al.  Efficient implementation of a variable projection algorithm for nonlinear least squares problems , 1974, CACM.

[10]  Linda Kaufman,et al.  A Variable Projection Method for Solving Separable Nonlinear Least Squares Problems , 1974 .

[11]  J. Wheeler,et al.  Analysis of spectral densities using modified moments , 1974 .

[12]  M. R. Osborne.  Some Special Nonlinear Least Squares Problems , 1975 .

[13]  W. Stoeckenius,et al.  Bacteriorhodopsin: a light-driven proton pump in Halobacterium Halobium. , 1975, Biophysical journal.

[14]  B. Rust,et al.  Evidence for a radioactive decay hypothesis for supernova luminosity , 1976, Nature.

[15]  B. L. Kirk,et al.  Inductive Modelling of Population Time Series , 1977 .

[16]  G. Stewart.  On the Perturbation of Pseudo-Inverses, Projections and Linear Least Squares Problems , 1977 .

[17]  J. J. Moré,et al.  Levenberg--Marquardt algorithm: implementation and theory , 1977 .

[18]  D. Jupp.  Approximation to Data by Splines with Free Knots , 1978 .

[19]  Luciano Stefanini,et al.  Computational experience with algorithms for separable nonlinear least squares problems , 1978 .

[20]  L. Kaufman,et al.  A Method for Separable Nonlinear Least Squares Problems with Separable Nonlinear Equality Constraints , 1978 .

[21]  R. O. Schmidt,et al.  Multiple emitter location and signal Parameter estimation , 1986 .

[22]  Georges Bienvenu,et al.  Adaptivity to background noise spatial coherence for high resolution passive methods , 1980, ICASSP.

[23]  Axel Ruhe,et al.  Algorithms for separable nonlinear least squares problems , 1980 .

[24]  B. L. Kirk,et al.  Inductive modeling of time series: a detrending approach , 1981 .

[25]  S. F. Bowne,et al.  THE EFFECT OF VISCOSITY ON THE PHOTOCYCLE OF BACTERIORHODOPSIN , 1981 .

[26]  John E. Dennis,et al.  An Adaptive Nonlinear Least-Squares Algorithm , 1977, TOMS.

[27]  C. Corradi.  A Note on the Solution of Separable Nonlinear Least-Squares Problems with Separable Nonlinear Equality Constraints , 1981 .

[28]  B. L. Kirk,et al.  Modulation of fossil fuel production by global temperature variations , 1982 .

[29]  D. Bates.  The Derivative of |X′X| and Its Uses , 1983 .

[30]  B. Mccormac.  Weather and climate responses to solar variations , 1983 .

[31]  K. Arun,et al.  State-space and singular-value decomposition-based approximation methods for the harmonic retrieval problem , 1983 .

[32]  J. Marque,et al.  Pressure effects on the photocycle of purple membrane. , 1984, Biochemistry.

[33]  A Continuation Method for Solving Separable Nonlinear Least Squares Problems , 1984 .

[34]  Teresa Anne Parks.  Reducible Nonlinear Programming Problems , 1985 .

[35]  Basis Set Requirement for Small Components Besides Kinetic Balance in Relativistic Self-Consistent-Field Calculations of Many Electron Systems , 1986 .

[36]  Improved modified-moment-singularity method. , 1986, Physical review. A, General physics.

[37]  Neutrino-induced meson production in nucleon-decay detectors. , 1986, Physical review. D, Particles and fields.

[38]  Douglas M. Bates,et al.  A generalized Guass-Newton procedure for multi-response parameter estimation , 1987 .

[39]  P. Jones,et al.  Hemispheric surface air temperature variations: Recent trends and an update to 1987 , 1988 .

[40]  P. Luyten,et al.  Accurate quantification of in vivo 31P NMR signals using the variable projection method and prior knowledge , 1988, Magnetic resonance in medicine.

[41]  R. Kumaresan,et al.  Superresolution by structured matrix approximation , 1988 .

[43]  Thomas Kailath,et al.  ESPRIT-estimation of signal parameters via rotational invariance techniques , 1989, IEEE Trans. Acoust. Speech Signal Process..

[44]  James A. Cadzow,et al.  Multiple source location-the signal subspace approach , 1990, IEEE Trans. Acoust. Speech Signal Process..

[45]  Benjamin Friedlander,et al.  Sensitivity analysis of the maximum likelihood direction-finding algorithm , 1990 .

[46]  Carl E. Baum,et al.  The singularity expansion method and its application to target identification , 1991, Proc. IEEE.

[47]  Linda Kaufman,et al.  Separable Nonlinear Least Squares with Multiple Right-Hand Sides , 1992, SIAM J. Matrix Anal. Appl..

[48]  J.C. Mosher,et al.  Multiple dipole modeling and localization from spatio-temporal MEG data , 1992, IEEE Transactions on Biomedical Engineering.

[49]  C. Loan,et al.  Approximation with Kronecker Products , 1992 .

[50]  Bhaskar D. Mo.  Model Based Processing of Signals: A State Space Approach , 1992 .

[51]  Bhaskar D. Rao,et al.  Model based processing of signals: a state space approach , 1992, Proc. IEEE.

[52]  Mats Viberg,et al.  Blind estimation of multiple co-channel digital signals arriving at an antenna array , 1993, Proceedings of 27th Asilomar Conference on Signals, Systems and Computers.

[53]  Carlo Tomasi,et al.  Direction of heading from image deformations , 1993, Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.

[54]  Marc Berthod,et al.  Neural Networks as Dynamical Bases in Function Space , 1993 .

[55]  Heekuck Oh,et al.  Neural Networks for Pattern Recognition , 1993, Adv. Comput..

[56]  Gérard Giraudon,et al.  Application of Projection Learning to the Detection of Urban Areas in SPOT Satellite Images , 1993 .

[57]  Yagyensh C. Pati,et al.  Analysis and synthesis of feedforward neural networks using discrete affine wavelet transformations , 1993, IEEE Trans. Neural Networks.

[58]  Marc Berthod,et al.  Projection learning: alternative approaches to the computation of the projection , 1994, ESANN.

[59]  Linda Kaufman,et al.  Structured Linear Least-Squares Problems in System Identification and Separable Nonlinear Data Fitting , 1994, SIAM J. Optim..

[60]  Frank J. Crosby,et al.  Further studies on the modulation of fossil fuel production by global temperature variations , 1994 .

[61]  J. S. Abel.  A variable projection method for additive components with application to GPS , 1994 .

[62]  A. Paulraj,et al.  Blind estimation of multiple co-channel digital signals using an antenna array , 1994, IEEE Signal Processing Letters.

[63]  C. B. Ockmann.  A Modiication of the Trust-region Gauss-newton Method to Solve Separable Nonlinear Least Squares Problems , 1995 .

[64]  Jeng Yen,et al.  COMPUTATIONAL CHALLENGES IN THE SOLUTION OF NONLINEAR OSCILLATORY MULTIBODY DYNAMICS SYSTEMS , 1995 .

[65]  Hubert Schwetlick,et al.  Least squares approximation by splines with free knots , 1995 .

[66]  Anders Barrlund,et al.  Comparing stability properties of three methods in DAEs or ODEs with invariants , 1995 .

[67]  Gordon K. Smyth,et al.  A Modified Prony Algorithm for Exponential Function Fitting , 1995, SIAM J. Sci. Comput..

[68]  Shilpa Talwar,et al.  Blind Space-Time Algorithms For Wireless Communication Systems , 1996 .

[69]  Sabine Van Huffel,et al.  Fast and accurate parameter estimation of noisy complex exponentials with use of prior knowledge , 1996, 1996 8th European Signal Processing Conference (EUSIPCO 1996).

[70]  Gonzalo R. Arce,et al.  Piecewise linear system modeling based on a continuous threshold decomposition , 1996, IEEE Trans. Signal Process..

[71]  J. Navarro-Pedreño.  Numerical Methods for Least Squares Problems , 1996 .

[72]  M. Haugg,et al.  LAKE : A PROGRAM SYSTEM FOR EQUILIBRIUM ANALYTICAL TREATMENT OF MULTIMETHOD DATA, ESPECIALLY COMBINED POTENTIOMETRIC AND NUCLEAR MAGNETIC RESONANCE DA TA , 1996 .

[73]  J. Eriksson,et al.  Regularization Tools for Training Large-scale Neural Networks , 1996 .

[74]  Åke Björck,et al.  Numerical methods for least square problems , 1996 .

[75]  Randall Bramley,et al.  Solving Linear Inequalities in a Least Squares Sense , 1996, SIAM J. Sci. Comput..

[76]  Linda R. Petzold,et al.  Convergence of the Iterative Methods for Coordinate- Splitting Formulation in Multibody Dynamics. , 1996 .

[77]  Paul Dierckx,et al.  Curve and surface fitting with splines , 1994, Monographs on numerical analysis.

[78]  John W. Woods,et al.  Maximum-likelihood parameter estimation of the harmonic, evanescent, and purely indeterministic components of discrete homogeneous random fields , 1993, IEEE Trans. Inf. Theory.

[79]  Paul J. Lanzkron,et al.  An Analysis of Approximate Nonlinear Elimination , 1996, SIAM J. Sci. Comput..

[80]  J. Lilleberg,et al.  Blind iterative multiuser delay estimator for CDMA , 1996, Proceedings of PIMRC '96 - 7th International Symposium on Personal, Indoor, and Mobile Communications.

[81]  M. Trosset.  Computing Distances Between Convex Sets and Subsets of the Positive Semidefinite Matrices , 1997 .

[82]  Jiunn-Tsair Chen,et al.  A two-stage hybrid approach for CCI/ISI reduction with space-time processing , 1997, IEEE Communications Letters.

[83]  Kenneth Holmstrom,et al.  TOMLAB - An Environment for Solving Optimization Problems in MATLAB , 1997 .

[84]  Hubert Schwetlick,et al.  Constrained approximation by splines with free knots , 1997 .

[85]  Vanhamme,et al.  Improved method for accurate and efficient quantification of MRS data with use of prior knowledge , 1997, Journal of magnetic resonance.

[86]  Michel Verhaegen,et al.  Linear and Non-Linear System Identification Using Separable Least-Squares , 1997 .

[87]  M. Viberg,et al.  Separable non-linear least-squares minimization-possible improvements for neural net fitting , 1997, Neural Networks for Signal Processing VII. Proceedings of the 1997 IEEE Signal Processing Society Workshop.

[88]  Joris De Schutter,et al.  The analytical Jacobian and its derivative for a parallel manipulator , 1997, Proceedings of International Conference on Robotics and Automation.

[89]  Algorithms and Software for the Computation of Parameters Occurring in ODE-models , 1997 .

[90]  Gregory Beylkin,et al.  On Applications of Unequally Spaced Fast Fourier Transforms , 1998 .

[91]  C. B. Ockmann.  A Modi cation of the Trust-Region Gauss-Newton Method to Solve Separable Nonlinear Least Squares Problems , 1998 .

[92]  Linda R. Petzold,et al.  An Efficient Newton-Type Iteration for the Numerical Solution of Highly Oscillatory Constrained Multibody Dynamic Systems , 1998, SIAM J. Sci. Comput..

[93]  K W Langenberger,et al.  Absolute metabolite quantification by in vivo NMR spectroscopy: III. Multicentre 1H MRS of the human brain addressed by one and the same data-analysis protocol. , 1998, Magnetic resonance imaging.

[94]  Volker Mehrmann,et al.  Where is the nearest non-regular pencil? , 1998 .

[95]  Off- and online identification of discrete time LTI systems using separable least squares , 1998, Proceedings of the 37th IEEE Conference on Decision and Control (Cat. No.98CH36171).

[96]  J.C. Mosher,et al.  Recursive MUSIC: A framework for EEG and MEG source localization , 1998, IEEE Transactions on Biomedical Engineering.

[97]  P. Wedin,et al.  Regularization tools for training large feed-forward neural networks using automatic differentiation ∗ , 1998 .

[98]  Kenneth Holmstrom.  Constrained Separable NLLS Algorithms for Chemical Equilibrium Analysis , 1998 .

[99]  Henry Leung,et al.  Tracking the direction-of-arrival of multiple moving targets by passive arrays: algorithm , 1999, IEEE Trans. Signal Process..

[100]  J. Barberà,et al.  Effects of endurance training on skeletal muscle bioenergetics in chronic obstructive pulmonary disease. , 1999, American journal of respiratory and critical care medicine.

[101]  Piero Barone.  Fast Deconvolution by a Two-Step Method , 1999, SIAM J. Sci. Comput..

[102]  Andrew R. Conn,et al.  Two-Step Algorithms for Nonlinear Optimization with Structured Applications , 1999, SIAM J. Optim..

[103]  T Oostendorp,et al.  Interictal spike localization using a standard realistic head model: simulations and analysis of clinical data , 1999, Clinical Neurophysiology.

[104]  Separable Overdetermined Nonlinear.  APPLICATION OF THE SHEN-YPMA ALGORITHM FOR , 1999 .

[105]  Michel Verhaegen,et al.  Linear and Non-linear System Identification Using Separable Least-Squares , 1997, Eur. J. Control.

[106]  Uwe Prells,et al.  APPLICATION OF THE VARIABLE PROJECTION METHOD FOR UPDATING MODELS OF MECHANICAL SYSTEMS , 1999 .

[107]  Lester S.H. Ngia.  System Modeling Using Basis Functions and Application to Echo Cancelation , 2000 .

[108]  D. Westwick,et al.  Identification of a Hammerstein model of the stretch reflex EMG using separable least squares , 2000, Proceedings of the 22nd Annual International Conference of the IEEE Engineering in Medicine and Biology Society (Cat. No.00CH37143).

[109]  Mårten Gulliksson,et al.  The Use and Properties of Tikhonov Filter Matrices , 2000, SIAM J. Matrix Anal. Appl..

[110]  V. Pereyra.  Ray tracing methods for inverse problems , 2000 .

[111]  Peter J. Ramadge,et al.  Efficiently estimating projective transformations , 2000, Proceedings 2000 International Conference on Image Processing (Cat. No.00CH37101).

[112]  Continuity and differentiability of the Moore-Penrose inverse in $C^*$-algebras , 2001 .

[113]  S. Huffel,et al.  MR spectroscopy quantitation: a review of time‐domain methods , 2001, NMR in biomedicine.

[114]  B. Borden.  Radar Scattering Center Localization by Subspace Fitting , 2001 .

[115]  R. de Beer,et al.  Java-based graphical user interface for MRUI, a software package for quantitation of in vivo/medical magnetic resonance spectroscopy signals , 2001, Comput. Biol. Medicine.

[116]  Richard M. Leahy,et al.  Electromagnetic brain mapping , 2001, IEEE Signal Process. Mag..

[117]  Rafael Escovar,et al.  Transmission line design of clock trees , 2002, ICCAD 2002.

[118]  Robert McDermott,et al.  Liquid-State NMR and Scalar Couplings in Microtesla Magnetic Fields , 2002, Science.

[119]  Kenneth Holmström,et al.  A review of the parameter estimation problem of fitting positive exponential sums to empirical data , 2002, Appl. Math. Comput..

[120]  Spectroscopy scales new peaks , 2002 .

[121]  Peyman Milanfar,et al.  Reconstruction of Convex Bodies from Brightness Functions , 2003, Discret. Comput. Geom..

[122]  Stefano Soatto,et al.  Optimal Structure from Motion: Local Ambiguities and Global Estimates , 2004, International Journal of Computer Vision.

[123]  Kenneth Holmström,et al.  The TOMLAB Optimization Environment , 2004 .

[124]  Tjalling J. Ypma,et al.  SolvingN+m nonlinear equations with onlym nonlinear variables , 2005, Computing.