Nonlinear system identification: From multiple-model networks to Gaussian processes

Neural networks have been widely used to model nonlinear systems for control. The curse of dimensionality and lack of transparency of such neural network models has forced a shift towards local model networks and recently towards the nonparametric Gaussian processes approach. Assuming common validity functions, all of these models have a similar structure. This paper examines the evolution from the radial basis function network to the local model network and finally to the Gaussian process model. A simulated example is used to explain the advantages and disadvantages of each structure.

[1]  George W. Irwin,et al.  Nonlinear Optimization of Rbf Networks , 1998, Int. J. Syst. Sci..

[2]  T. A. Johansen,et al.  On the optimality of the Takagi-Sugeno-Kang fuzzy inference mechanism , 1995, Proceedings of 1995 IEEE International Conference on Fuzzy Systems..

[3]  J. Kocijan,et al.  Gaussian process model based predictive control , 2004, Proceedings of the 2004 American Control Conference.

[4]  Roderick Murray-Smith,et al.  Multiple Model Approaches to Modelling and Control , 1997 .

[5]  Carl E. Rasmussen,et al.  Gaussian processes for machine learning , 2005, Adaptive computation and machine learning.

[6]  J. Kocijan,et al.  Predictive control with Gaussian process models , 2003, The IEEE Region 8 EUROCON 2003. Computer as a Tool..

[7]  Tor Arne Johansen,et al.  Speed control design for an experimental vehicle using a generalized gain scheduling approach , 2000, IEEE Trans. Control. Syst. Technol..

[8]  J. Richalet,et al.  Industrial applications of model based predictive control , 1993, Autom..

[9]  Michael I. Jordan.  Learning in Graphical Models , 1999, NATO ASI Series.

[10]  Roderick Murray-Smith,et al.  Extending the functional equivalence of radial basis function networks and fuzzy inference systems , 1996, IEEE Trans. Neural Networks.

[11]  Roderick Murray-Smith,et al.  Gaussian process priors with ARMA noise models , 2001 .

[12]  Gordon Lightbody,et al.  Local Model Network Identification With Gaussian Processes , 2007, IEEE Transactions on Neural Networks.

[13]  K. Obermayer,et al.  Multiple-step ahead prediction for non linear dynamic systems: A Gaussian Process treatment with propagation of the uncertainty , 2003, NIPS 2003.

[14]  Chuen-Tsai Sun,et al.  Neuro-fuzzy modeling and control , 1995, Proc. IEEE.

[15]  M. Morari,et al.  Internal Model Control: extension to nonlinear system , 1986 .

[16]  Norman S. Nise,et al.  Control Systems Engineering , 1991 .

[17]  Kevin Warwick,et al.  Neural Network Engineering in Dynamic Control Systems: Advances in Industrial Control , 1995 .

[18]  A. O'Hagan,et al.  Curve Fitting and Optimal Design for Prediction , 1978 .

[19]  Tor Arne Johansen,et al.  Non-linear predictive control using local models-applied to a batch fermentation process , 1995 .

[20]  Geoffrey E. Hinton,et al.  Evaluation of Gaussian processes and other methods for non-linear regression , 1997 .

[21]  Kumpati S. Narendra,et al.  Identification and control of dynamical systems using neural networks , 1990, IEEE Trans. Neural Networks.

[22]  D. Mackay,et al.  Introduction to Gaussian processes , 1998 .

[23]  Roderick Murray-Smith,et al.  Nonlinear structure identification: a Gaussian process prior/velocity-based approach , 2000 .

[24]  Jonathan C. Jarvis,et al.  A nonlinear approach to modeling of electrically stimulated skeletal muscle , 2001, IEEE Transactions on Biomedical Engineering.

[25]  Henrik Gollee,et al.  A constructive learning algorithm for local model networks , 1994 .

[26]  Lennart Ljung,et al.  System Identification: Theory for the User , 1987 .

[27]  Daniel G. Sbarbaro-Hofer,et al.  Self-tuning Control of Non-linear Systems Using Gaussian Process Prior Models , 2003, European Summer School on Multi-AgentControl.

[28]  Jonas Sjöberg,et al.  Gaussian processes framework for validation of linear and nonlinear models , 2003 .

[29]  T. Johansen,et al.  On transient dynamics, off-equilibrium behaviour and identification in blended multiple model structures , 1999, 1999 European Control Conference (ECC).

[30]  Lyle H. Ungar,et al.  A comparison of two nonparametric estimation schemes: MARS and neural networks , 1993 .

[31]  David J. C. MacKay,et al.  Information Theory, Inference, and Learning Algorithms , 2004, IEEE Transactions on Information Theory.

[32]  J Kocijan,et al.  Application of Gaussian processes for black-box modelling of biosystems. , 2007, ISA transactions.

[33]  Christopher M. Bishop,et al.  Neural networks and machine learning , 1998 .

[34]  Tor Arne Johansen,et al.  Design and analysis of gain-scheduled control using local controller networks , 1997 .

[35]  T. Johansen,et al.  A NARMAX model representation for adaptive control based on local models , 1992 .

[36]  Tor Arne Johansen,et al.  ORBIT – operating-regime-based modeling and identification toolkit , 1997 .

[37]  Franck Tancret,et al.  Comparison of artificial neural networks with gaussian processes to model the yield strength of nickel-base superalloys , 1999 .

[38]  János Abonyi,et al.  Fuzzy Model Identification for Control , 2003 .

[39]  Dale E. Seborg,et al.  Nonlinear internal model control strategy for neural network models , 1992 .

[40]  Ahmad Lotfi,et al.  Comments on "Functional equivalence between radial basis function networks and fuzzy inference systems" [and reply] , 1998, IEEE Trans. Neural Networks.

[41]  George W. Irwin,et al.  Constructing networks of continuous-time velocity-based models , 2001 .

[42]  Johan A. K. Suykens,et al.  Advances in learning theory : methods, models and applications , 2003 .

[43]  Christopher M. Bishop,et al.  Bayesian Regression and Classification , 2003 .

[44]  George W. Irwin,et al.  On Gaussian Weighting Functions For Velocity-Based Local Model Networks , 2002 .

[45]  Matthias W. Seeger,et al.  Bayesian Gaussian process models : PAC-Bayesian generalisation error bounds and sparse approximations , 2003 .

[46]  Tony J. Dodd,et al.  Identification of non-linear time series via kernels , 2002, Int. J. Syst. Sci..

[47]  Oliver Nelles,et al.  Nonlinear system identification with local linear neuro-fuzzy models , 1999 .

[48]  Tor Arne Johansen,et al.  Local learning in local model networks , 1995 .

[49]  Richard Von Mises,et al.  Mathematical Theory of Probability and Statistics , 1966 .

[50]  J. Kocijan,et al.  Derivative observations used in predictive control , 2004, Proceedings of the 12th IEEE Mediterranean Electrotechnical Conference (IEEE Cat. No.04CH37521).

[51]  A. Morris,et al.  Artificial neural networks : studies in process modelling and control : Process operation and control , 1994 .

[52]  Chuen-Tsai Sun,et al.  Functional equivalence between radial basis function networks and fuzzy inference systems , 1993, IEEE Trans. Neural Networks.

[53]  Christopher K. I. Williams,et al.  Discovering Hidden Features with Gaussian Processes Regression , 1998, NIPS.

[54]  David J. C. MacKay,et al.  Variational Gaussian process classifiers , 2000, IEEE Trans. Neural Networks Learn. Syst..

[55]  Kenneth J. Hunt,et al.  Local controller network for autonomous vehicle steering , 1996 .

[56]  Gregor Gregorcic,et al.  Internal model control based on a Gaussian process prior model , 2003, Proceedings of the 2003 American Control Conference, 2003..

[57]  Rolf Isermann,et al.  Local Linear Model Trees (LOLIMOT) Toolbox for Nonlinear System Identification , 2000 .

[58]  Christopher K. I. Williams.  Prediction with Gaussian Processes: From Linear Regression to Linear Prediction and Beyond , 1999, Learning in Graphical Models.

[59]  W. Leithead,et al.  Gain-scheduled and nonlinear systems : dynamic analysis by velocity-based linearization families , 1998 .

[60]  Agathe Girard,et al.  Adaptive, Cautious, Predictive control with Gaussian Process Priors , 2003 .

[61]  G. Irwin,et al.  Nonlinear internal model control using local model networks , 1997 .

[62]  David J. C. Mackay,et al.  Introduction to Monte Carlo Methods , 1998, Learning in Graphical Models.

[63]  I. H. Öğüş,et al.  NATO ASI Series , 1997 .

[64]  Tor Arne Johansen,et al.  Constructive empirical modelling of longitudinal vehicle dynamics using local model networks , 1996 .

[65]  Russell Reed,et al.  Pruning algorithms-a survey , 1993, IEEE Trans. Neural Networks.

[66]  Tor Arne Johansen,et al.  Identification of non-linear system structure and parameters using regime decomposition , 1995, Autom..

[67]  T. Johansen,et al.  Constructing NARMAX models using ARMAX models , 1993 .

[68]  William P. Marnane,et al.  Gaussian Process Modeling of EEG for the Detection of Neonatal Seizures , 2007, IEEE Transactions on Biomedical Engineering.

[69]  Venkat Venkatasubramanian,et al.  A fuzzy neural-network approach for nonlinear process control , 1995 .

[70]  David Barber,et al.  Bayesian Classification With Gaussian Processes , 1998, IEEE Trans. Pattern Anal. Mach. Intell..

[71]  William P. Marnane,et al.  Gaussian Process Modelling as an Indicator of Neonatal Seizure , 2006, SPPRA.

[72]  George W. Irwin,et al.  Nonlinear control structures based on embedded neural system models , 1997, IEEE Trans. Neural Networks.

[73]  John C. Platt,et al.  Learning a Gaussian Process Prior for Automatically Generating Music Playlists , 2001, NIPS.

[74]  Radford M. Neal.  Monte Carlo Implementation of Gaussian Process Models for Bayesian Regression and Classification , 1997, physics/9701026.

[75]  Stanko Strmčnik,et al.  Fuzzy gain-scheduling control of a gas-liquid separation plant implemented on a PLC , 2002 .

[76]  William Leithead,et al.  Gain-scheduled controller design: An analytic framework directly incorporating non-equilibrium plant dynamics , 1998 .

[77]  R. Murray-Smith,et al.  On normalising radial basis function networks , 1994 .

[78]  W. Leithead,et al.  Analytic framework for blended multiple model systems using linear local models , 1999 .

[79]  Gordon Lightbody,et al.  Gaussian process approaches to nonlinear modelling for control , 2005 .

[80]  K. R. Godfrey,et al.  Introduction to binary signals used in system identification , 1991 .

[81]  Carl E. Rasmussen,et al.  In Advances in Neural Information Processing Systems , 2011 .

[82]  Wallace E. Larimore,et al.  Optimal Reduced Rank Modeling, Prediction, Monitoring and Control using Canonical Variate Analysis , 1997 .

[83]  Agathe Girard,et al.  Dynamic systems identification with Gaussian processes , 2005 .

[84]  Robert Shorten,et al.  On the interpretation and identification of dynamic Takagi-Sugeno fuzzy models , 2000, IEEE Trans. Fuzzy Syst..

[85]  Shang-Liang Chen,et al.  Orthogonal least squares learning algorithm for radial basis function networks , 1991, IEEE Trans. Neural Networks.

[86]  Dale E. Seborg,et al.  Experience with nonlinear control and identification strategies , 1994 .

[87]  Robert Shorten,et al.  On the interpretation of local models in blended multiple model structures. , 1999 .

[88]  Ka-Yiu San,et al.  Process identification using neural networks , 1992 .

[89]  Kenneth J. Hunt,et al.  Local Model Architectures for Nonlinear Modelling and Control , 1995 .

[90]  David J. C. MacKay,et al.  Comparison of Approximate Methods for Handling Hyperparameters , 1999, Neural Computation.

[91]  Stephen A. Billings,et al.  Identi cation of nonlinear systems-A survey , 1980 .

[92]  Michio Sugeno,et al.  Fuzzy identification of systems and its applications to modeling and control , 1985, IEEE Transactions on Systems, Man, and Cybernetics.

[93]  Jianliang Wang,et al.  Neural-fuzzy scheduling of H robust controllers for a high performance fighter aircraft under a Herbst-like manoeuvre , 1999 .

[94]  José Ragot,et al.  Non-linear dynamic system identification: A multi-model approach , 1999 .

[95]  Agathe Girard,et al.  A case based comparison of identification with neural network and Gaussian process models. , 2003 .

[96]  Robert Babuška,et al.  ON MULTI-OBJECTIVE IDENTIFICATION OF TAKAGI-SUGENO FUZZY MODEL PARAMETERS , 2002 .

[97]  D. Seborg.  A perspective on advanced strategies for process control , 1994 .

[98]  Daniel Sbarbaro,et al.  Neural Networks for Nonlinear Internal Model Control , 1991 .

[99]  David S. Broomhead,et al.  Delay Embeddings for Forced Systems. II. Stochastic Forcing , 2003, J. Nonlinear Sci..

[100]  Robert Babuska,et al.  Neuro-fuzzy methods for nonlinear system identification , 2003, Annu. Rev. Control..

[101]  Tor Arne Johansen,et al.  Operating regime based process modeling and identification , 1997 .

[102]  Lotfi A. Zadeh,et al.  Fuzzy Sets , 1996, Inf. Control..

[103]  Tor Arne Johansen,et al.  Gain-scheduled control of a solar power plant , 2000 .