Modified Gram–Schmidt Method-Based Variable Projection Algorithm for Separable Nonlinear Models

Separable nonlinear models are very common in various research fields, such as machine learning and system identification. The variable projection (VP) approach is efficient for the optimization of such models. In this paper, we study various VP algorithms based on different matrix decompositions. Compared with the previous method, we use the analytical expression of the Jacobian matrix instead of finite differences. This improves the efficiency of the VP algorithms. In particular, based on the modified Gram–Schmidt (MGS) method, a more robust implementation of the VP algorithm is introduced for separable nonlinear least-squares problems. In numerical experiments, we compare the performance of five different implementations of the VP algorithm. Numerical results show the efficiency and robustness of the proposed MGS method-based VP algorithm.

[1]  M. Viberg,et al.  Separable non-linear least-squares minimization-possible improvements for neural net fitting , 1997, Neural Networks for Signal Processing VII. Proceedings of the 1997 IEEE Signal Processing Society Workshop.

[2]  Min Gan,et al.  A Variable Projection Approach for Efficient Estimation of RBF-ARX Model , 2015, IEEE Transactions on Cybernetics.

[3]  C. L. Philip Chen,et al.  A Regularized Variable Projection Algorithm for Separable Nonlinear Least-Squares Problems , 2019, IEEE Transactions on Automatic Control.

[4]  G. Golub,et al.  Numerical computations for univariate linear models , 1973 .

[5]  Takayuki Okatani,et al.  On the Wiberg Algorithm for Matrix Factorization in the Presence of Missing Components , 2007, International Journal of Computer Vision.

[6]  G. Golub,et al.  Separable nonlinear least squares: the variable projection method and its applications , 2003 .

[7]  Peter J. Fleming,et al.  A new formulation of the learning problem of a neural network controller , 1991, [1991] Proceedings of the 30th IEEE Conference on Decision and Control.

[8]  K. S. Banerjee.  Generalized Inverse of Matrices and Its Applications , 1973 .

[9]  Min Gan,et al.  An Efficient Variable Projection Formulation for Separable Nonlinear Least Squares Problems , 2014, IEEE Transactions on Cybernetics.

[10]  Gene H. Golub,et al.  The differentiation of pseudo-inverses and non-linear least squares problems whose variables separate , 1972, Milestones in Matrix Computation.

[11]  Yukihiro Toyoda,et al.  A parameter optimization method for radial basis function type models , 2003, IEEE Trans. Neural Networks.

[12]  Christopher Zach,et al.  Revisiting the Variable Projection Method for Separable Nonlinear Least Squares Problems , 2017, CVPR 2017.

[13]  Kazushi Nakano,et al.  RBF-ARX model based nonlinear system modeling and predictive control with application to a NOx decomposition process , 2004 .

[14]  Michael R. Osborne,et al.  SEPARABLE LEAST SQUARES, VARIABLE PROJECTION, AND THE GAUSS-NEWTON ALGORITHM ∗ , 2007 .

[15]  Yanjun Liu,et al.  Model recovery for Hammerstein systems using the hierarchical orthogonal matching pursuit method , 2019, J. Comput. Appl. Math..

[16]  C. L. Philip Chen,et al.  Exploiting the interpretability and forecasting ability of the RBF-AR model for nonlinear time series , 2016, Int. J. Syst. Sci..

[17]  James G. Nagy,et al.  Constrained numerical optimization methods for blind deconvolution , 2013, Numerical Algorithms.

[18]  Andrew W. Fitzgibbon,et al.  Secrets of Matrix Factorization: Approximations, Numerics, Manifold Optimization and Random Restarts , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[19]  Dianne P. O'Leary,et al.  Variable projection for nonlinear least squares problems , 2012, Computational Optimization and Applications.

[20]  Feng Ding,et al.  An efficient hierarchical identification method for general dual-rate sampled-data systems , 2014, Autom..

[21]  Fred T. Krogh,et al.  Efficient implementation of a variable projection algorithm for nonlinear least squares problems , 1974, CACM.

[22]  Min Gan,et al.  Generalized exponential autoregressive models for nonlinear time series: Stationarity, estimation and applications , 2018, Inf. Sci..

[23]  Feng Ding,et al.  Novel data filtering based parameter identification for multiple-input multiple-output systems using the auxiliary model , 2016, Autom..

[24]  Erfu Yang,et al.  State filtering‐based least squares parameter estimation for bilinear systems using the hierarchical identification principle , 2018, IET Control Theory & Applications.

[25]  Shuo Zhang,et al.  Separate block-based parameter estimation method for Hammerstein systems , 2018, Royal Society Open Science.

[26]  Hui Peng,et al.  A Regularized SNPOM for Stable Parameter Estimation of RBF-AR(X) Model , 2018, IEEE Transactions on Neural Networks and Learning Systems.

[27]  Long Chen,et al.  On Some Separated Algorithms for Separable Nonlinear Least Squares Problems , 2018, IEEE Transactions on Cybernetics.

[28]  Aleix M. Martínez,et al.  Computing Smooth Time Trajectories for Camera and Deformable Shape in Structure from Motion with Occlusion , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[29]  Xi-Lin Li,et al.  Preconditioned Stochastic Gradient Descent , 2015, IEEE Transactions on Neural Networks and Learning Systems.

[30]  Feng Ding,et al.  Hierarchical Least Squares Identification for Linear SISO Systems With Dual-Rate Sampled-Data , 2011, IEEE Transactions on Automatic Control.

[31]  C. R. Rao,et al.  Generalized Inverse of Matrices and its Applications , 1972 .

[32]  Jian Pan,et al.  Adaptive Gradient-Based Iterative Algorithm for Multivariable Controlled Autoregressive Moving Average Systems Using the Data Filtering Technique , 2018, Complex..

[33]  James G. Nagy,et al.  An Efficient Iterative Approach for Large-Scale Separable Nonlinear Inverse Problems , 2009, SIAM J. Sci. Comput..

[34]  Tianyou Chai,et al.  An Alternating Identification Algorithm for a Class of Nonlinear Dynamical Systems , 2017, IEEE Transactions on Neural Networks and Learning Systems.

[35]  Linda Kaufman,et al.  A Variable Projection Method for Solving Separable Nonlinear Least Squares Problems , 1974 .

[36]  Axel Ruhe,et al.  Algorithms for separable nonlinear least squares problems , 1980 .

[37]  Feng Ding,et al.  A hierarchical least squares identification algorithm for Hammerstein nonlinear systems using the key term separation , 2018, J. Frankl. Inst..

[38]  Dongqing Wang,et al.  Model recovery for Hammerstein systems using the auxiliary model based orthogonal matching pursuit method , 2018 .