On Some Separated Algorithms for Separable Nonlinear Least Squares Problems

For a class of nonlinear least squares problems, it is usually very beneficial to separate the variables into a linear and a nonlinear part and take full advantage of reliable linear least squares techniques. Consequently, the original problem is turned into a reduced problem which involves only nonlinear parameters. We consider in this paper four separated algorithms for such problems. The first one is the variable projection (VP) algorithm with full Jacobian matrix of Golub and Pereyra. The second and third ones are VP algorithms with simplified Jacobian matrices proposed by Kaufman and Ruano et al. respectively. The fourth one only uses the gradient of the reduced problem. Monte Carlo experiments are conducted to compare the performance of these four algorithms. From the results of the experiments, we find that: 1) the simplified Jacobian proposed by Ruano et al. is not a good choice for the VP algorithm; moreover, it may render the algorithm hard to converge; 2) the fourth algorithm perform moderately among these four algorithms; 3) the VP algorithm with the full Jacobian matrix perform more stable than that of the VP algorithm with Kuafman’s simplified one; and 4) the combination of VP algorithm and Levenberg–Marquardt method is more effective than the combination of VP algorithm and Gauss–Newton method.

[1]  V. Pereyra,et al.  Least Squares Estimation for a Class of Non-Linear Models , 1973 .

[2]  F. Ding,et al.  Recasted models-based hierarchical extended stochastic gradient method for MIMO nonlinear systems , 2017 .

[3]  Feng Ding,et al.  Decomposition based least squares iterative identification algorithm for multivariate pseudo-linear ARMA systems using the data filtering , 2017, J. Frankl. Inst..

[4]  E. Mizutani,et al.  Neuro-Fuzzy and Soft Computing-A Computational Approach to Learning and Machine Intelligence [Book Review] , 1997, IEEE Transactions on Automatic Control.

[5]  C. L. Philip Chen,et al.  A rapid learning and dynamic stepwise updating algorithm for flat neural networks and the application to time-series prediction , 1999, IEEE Trans. Syst. Man Cybern. Part B.

[6]  Min Gan,et al.  Stability analysis of RBF network-based state-dependent autoregressive model for nonlinear time series , 2012, Appl. Soft Comput..

[7]  Sundaram Suresh,et al.  Metacognitive Learning in a Fully Complex-Valued Radial Basis Function Neural Network , 2012, Neural Computation.

[8]  Feng Ding,et al.  Recursive parameter and state estimation for an input nonlinear state space system using the hierarchical identification principle , 2015, Signal Process..

[9]  M. Viberg,et al.  Separable non-linear least-squares minimization-possible improvements for neural net fitting , 1997, Neural Networks for Signal Processing VII. Proceedings of the 1997 IEEE Signal Processing Society Workshop.

[10]  T. Hayat,et al.  Parameter estimation for pseudo-linear systems using the auxiliary model and the decomposition technique , 2017 .

[11]  Fred T. Krogh,et al.  Efficient implementation of a variable projection algorithm for nonlinear least squares problems , 1974, CACM.

[12]  Min Gan,et al.  A Variable Projection Approach for Efficient Estimation of RBF-ARX Model , 2015, IEEE Transactions on Cybernetics.

[13]  G. Golub,et al.  Separable nonlinear least squares: the variable projection method and its applications , 2003 .

[14]  Gene H. Golub,et al.  The differentiation of pseudo-inverses and non-linear least squares problems whose variables separate , 1972, Milestones in Matrix Computation.

[15]  Feng Ding,et al.  Highly Efficient Identification Methods for Dual-Rate Hammerstein Systems , 2015, IEEE Transactions on Control Systems Technology.

[16]  Feng Ding,et al.  The auxiliary model based hierarchical gradient algorithms and convergence analysis using the filtering technique , 2016, Signal Process..

[17]  C. L. Philip Chen,et al.  Exploiting the interpretability and forecasting ability of the RBF-AR model for nonlinear time series , 2016, Int. J. Syst. Sci..

[18]  Feng Ding,et al.  Convergence Analysis of the Hierarchical Least Squares Algorithm for Bilinear-in-Parameter Systems , 2016, Circuits Syst. Signal Process..

[19]  James G. Nagy,et al.  Constrained numerical optimization methods for blind deconvolution , 2013, Numerical Algorithms.

[20]  Peter J. Fleming,et al.  A new formulation of the learning problem of a neural network controller , 1991, [1991] Proceedings of the 30th IEEE Conference on Decision and Control.

[21]  Min Gan,et al.  An Efficient Variable Projection Formulation for Separable Nonlinear Least Squares Problems , 2014, IEEE Transactions on Cybernetics.

[22]  Feng Ding,et al.  Hierarchical Stochastic Gradient Algorithm and its Performance Analysis for a Class of Bilinear-in-Parameter Systems , 2017, Circuits Syst. Signal Process..

[23]  Feng Ding,et al.  Recursive Least Squares Parameter Estimation for a Class of Output Nonlinear Systems Based on the Model Decomposition , 2016, Circuits Syst. Signal Process..

[24]  Ju-Jang Lee,et al.  Training Two-Layered Feedforward Networks With Variable Projection Method , 2008, IEEE Transactions on Neural Networks.

[25]  T. Teräsvirta.  Specification, Estimation, and Evaluation of Smooth Transition Autoregressive Models , 1994 .

[26]  Gordon K. Smyth,et al.  A Modified Prony Algorithm for Exponential Function Fitting , 1995, SIAM J. Sci. Comput..

[27]  Dianne P. O'Leary,et al.  Variable projection for nonlinear least squares problems , 2012, Computational Optimization and Applications.

[28]  T. Ozaki,et al.  Modelling nonlinear random vibrations using an amplitude-dependent autoregressive time series model , 1981 .

[29]  Feng Ding,et al.  Parameter estimation algorithms for multivariable Hammerstein CARMA systems , 2016, Inf. Sci..

[30]  Feng Ding,et al.  Kalman state filtering based least squares iterative parameter estimation for observer canonical state space systems using decomposition , 2016, J. Comput. Appl. Math..

[31]  Kui Fu Chen.  Estimating Parameters of a Sine Wave by Separable Nonlinear Least Squares Fitting , 2010, IEEE Transactions on Instrumentation and Measurement.

[32]  Zhen Zhang,et al.  Maximum likelihood estimation method for dual-rate Hammerstein systems , 2017 .

[33]  George W. Irwin,et al.  A hybrid linear/nonlinear training algorithm for feedforward neural networks , 1998, IEEE Trans. Neural Networks.

[34]  Linda Kaufman,et al.  A Variable Projection Method for Solving Separable Nonlinear Least Squares Problems , 1974 .

[35]  Axel Ruhe,et al.  Algorithms for separable nonlinear least squares problems , 1980 .

[36]  A. Gilbert,et al.  A generalization of variable elimination for separable inverse problems beyond least squares , 2013, 1302.0441.

[37]  James G. Nagy,et al.  An Efficient Iterative Approach for Large-Scale Separable Nonlinear Inverse Problems , 2009, SIAM J. Sci. Comput..

[38]  Feng Ding,et al.  Novel data filtering based parameter identification for multiple-input multiple-output systems using the auxiliary model , 2016, Autom..

[39]  Zhi Liu,et al.  A Probabilistic Neural-Fuzzy Learning System for Stochastic Modeling , 2008, IEEE Transactions on Fuzzy Systems.