On Tikhonov regularization, bias and variance in nonlinear system identification

Abstract Regularization is a general method for solving ill-posed and ill-conditioned problems. Traditionally, ill-conditioning in system identification problems is approached using regularization methods such as ridge regression and principal component regression. In this work it is argued that the Tikhonov regularization method is a powerful alternative for regularization of nonlinear system identification problems by introducing smoothness of the model as a prior. Its properties is discussed in terms of an analysis of bias and variance, and illustrated by a semirealistic simulation example.

[1]  L. Ljung,et al.  Overtraining, Regularization, and Searching for Minimum in Neural Networks , 1992 .

[2]  Kenneth Levenberg.  A METHOD FOR THE SOLUTION OF CERTAIN NON – LINEAR PROBLEMS IN LEAST SQUARES , 1944 .

[3]  Tomaso Poggio,et al.  Computational vision and regularization theory , 1985, Nature.

[4]  Petre Stoica,et al.  Decentralized Control , 2018, The Control Systems Handbook.

[5]  Tor Arne Johansen,et al.  Identification of non-linear systems using empirical data and prior knowledge - an optimization approach , 1996, Autom..

[6]  T. McKelveyDepartment.  On the Use of Regularization in System Identification , 1992 .

[7]  Lennart Ljung,et al.  System Identification: Theory for the User , 1987 .

[8]  D. Marquardt.  An Algorithm for Least-Squares Estimation of Nonlinear Parameters , 1963 .

[9]  E. Carlstein.  Resampling techniques for stationary time-series: Some recent developments , 1990 .

[10]  F. O’Sullivan.  A Statistical Perspective on Ill-posed Inverse Problems , 1986 .

[11]  A. N. Tikhonov,et al.  Solutions of ill-posed problems , 1977 .

[12]  Tor Arne Johansen,et al.  Constrained and regularized system identification , 1997 .

[13]  B. F. Swindel.  Good ridge estimators based on prior information , 1976 .

[14]  David R. Brillinger,et al.  New directions in time series analysis , 1993 .

[15]  Tor Arne Johansen,et al.  Non-linear predictive control using local models-applied to a batch fermentation process , 1995 .

[16]  Lennart Ljung,et al.  On the Use of Regularization in System Identification , 1993 .

[17]  David R. Brillinger,et al.  New Directions in Time Series Analysis : Part II , 1993 .

[18]  L. Ljung.  Convergence analysis of parametric identification methods , 1978 .

[19]  Torsten Söderström,et al.  Model-structure selection by cross-validation , 1986 .

[20]  C. Kravaris,et al.  Identification of parameters in distributed parameter systems by regularization , 1983, The 22nd IEEE Conference on Decision and Control.

[21]  L. Ljung,et al.  Asymptotic normality of prediction error estimators for approximate system models , 1978, 1978 IEEE Conference on Decision and Control including the 17th Symposium on Adaptive Processes.

[22]  Lars Kai Hansen,et al.  Generalization performance of regularized neural network models , 1994, Proceedings of IEEE Workshop on Neural Networks for Signal Processing.

[23]  A. Sano,et al.  Minimum MSE based regularization for system identification in the presence of input and output noise , 1995, Proceedings of 1995 34th IEEE Conference on Decision and Control.

[24]  G. Wahba.  Spline models for observational data , 1990 .