Non-Parametric Bayesian Methods for Linear System Identification

Recent contributions have tackled the linear system identification problem by means of non-parametric Bayesian methods, which are built on largely adopted machine learning techniques, such as Gaussian Process regression and kernel-based regularized regression. Following the Bayesian paradigm, these procedures treat the impulse response of the system to be estimated as the realization of a Gaussian process. Typically, a Gaussian prior accounting for stability and smoothness of the impulse response is postulated, as a function of some parameters (called hyper-parameters in the Bayesian framework). These are generally estimated by maximizing the so-called marginal likelihood, i.e. the likelihood after the impulse response has been marginalized out. Once the hyper-parameters have been fixed in this way, the final estimator is computed as the conditional expected value of the impulse response w.r.t. the posterior distribution, which coincides with the minimum variance estimator. Assuming that the identification data are corrupted by Gaussian noise, the above-mentioned estimator coincides with the solution of a regularized estimation problem, in which the regularization term is the l2 norm of the impulse response, weighted by the inverse of the prior covariance function (a.k.a. kernel in the machine learning literature). Recent works have shown how such Bayesian approaches are able to jointly perform estimation and model selection, thus overcoming one of the main issues affecting parametric identification procedures, that is complexity selection.
While keeping the classical system identification methods (e.g. Prediction Error Methods and subspace algorithms) as a benchmark for numerical comparison, this thesis extends and analyzes some key aspects of the above-mentioned Bayesian procedure. In particular, four main topics are considered. 1. PRIOR DESIGN. Adopting Maximum Entropy arguments, a new type of l2 regularization is derived: the aim is to penalize the rank of the block Hankel matrix built with Markov coefficients, thus controlling the complexity of the identified model, measured by its McMillan degree. By accounting for the coupling between different input-output channels, this new prior results particularly suited when dealing for the identification of MIMO systems
To speed up the computational requirements of the estimation algorithm, a tailored version of the Scaled Gradient Projection algorithm is designed to optimize the marginal likelihood. 2. CHARACTERIZATION OF UNCERTAINTY. The confidence sets returned by the non-parametric Bayesian identification algorithm are analyzed and compared with those returned by parametric Prediction Error Methods. The comparison is carried out in the impulse response space, by deriving “particle” versions (i.e. Monte-Carlo approximations) of the standard confidence sets. 3. ONLINE ESTIMATION. The application of the non-parametric Bayesian system identification techniques is extended to an online setting, in which new data become available as time goes. Specifically, two key modifications of the original “batch” procedure are proposed in order to meet the real-time requirements. In addition, the identification of time-varying systems is tackled by introducing a forgetting factor in the estimation criterion and by treating it as a hyper-parameter. 4. POST PROCESSING: MODEL REDUCTION. Non-parametric Bayesian identification procedures estimate the unknown system in terms of its impulse response coefficients, thus returning a model with high (possibly infinite) McMillan degree. A tailored procedure is proposed to reduce such model to a lower degree one, which appears more suitable for filtering and control applications. Different criteria for the selection of the order of the reduced model are evaluated and compared.

[1]  Dietmar Bauer,et al.  Asymptotic Distributions of Subspace Estimates under Misspecification of the order , 1998 .

[2]  A. Tsybakov,et al.  Oracle inequalities for inverse problems , 2002 .

[3]  Sun-Yuan Kung,et al.  A new identification and model reduction algorithm via singular value decomposition , 1978 .

[4]  James T. Kwok,et al.  Clustered Nyström Method for Large Scale Manifold Learning and Dimension Reduction , 2010, IEEE Transactions on Neural Networks.

[5]  David P. Wipf,et al.  A New View of Automatic Relevance Determination , 2007, NIPS.

[6]  Benjamin Recht,et al.  Random Features for Large-Scale Kernel Machines , 2007, NIPS.

[7]  Bjorn Ottersten,et al.  A Subspace Based Instrumental Variable Method for State-Space System Identification , 1994 .

[8]  Lennart Ljung,et al.  System Identification Via Sparse Multiple Kernel-Based Regularization Using Sequential Convex Optimization Techniques , 2014, IEEE Transactions on Automatic Control.

[9]  Gene H. Golub,et al.  Generalized cross-validation as a method for choosing a good ridge parameter , 1979, Milestones in Matrix Computation.

[10]  Radford M. Neal Pattern Recognition and Machine Learning , 2007, Technometrics.

[11]  R. Tibshirani Regression Shrinkage and Selection via the Lasso , 1996 .

[12]  V. Verdult Non linear system identification : a state-space approach , 2002 .

[13]  Giulio Bottegal,et al.  Regularized spectrum estimation using stable spline kernels , 2013, Autom..

[14]  Athanasios C. Antoulas,et al.  Approximation of Large-Scale Dynamical Systems , 2005, Advances in Design and Control.

[15]  James Durbin,et al.  The fitting of time series models , 1960 .

[16]  Lennart Ljung,et al.  Some facts about the choice of the weighting matrices in Larimore type of subspace algorithms , 2002, Autom..

[17]  John Lataire,et al.  FRF Smoothing to Improve Initial Estimates for Transfer Function Identification , 2015, IEEE Transactions on Instrumentation and Measurement.

[18]  Paul Tseng,et al.  Hankel Matrix Rank Minimization with Applications to System Identification and Realization , 2013, SIAM J. Matrix Anal. Appl..

[19]  David P. Wipf,et al.  Iterative Reweighted 1 and 2 Methods for Finding Sparse Solutions , 2010, IEEE J. Sel. Top. Signal Process..

[20]  B. Hofmann-Wellenhof,et al.  Introduction to spectral analysis , 1986 .

[21]  B. De Moor,et al.  A unifying theorem for three subspace system identification algorithms , 1994, Proceedings of 1994 American Control Conference - ACC '94.

[22]  Lennart Ljung,et al.  Recursive methods for off-line identification , 1985 .

[23]  Alessandro Chiuso,et al.  A Bayesian approach to sparse plus low rank network identification , 2015, 2015 54th IEEE Conference on Decision and Control (CDC).

[24]  Michel Verhaegen,et al.  A novel algorithm for recursive instrumental variable based subspace identification , 1998, Proceedings of the 37th IEEE Conference on Decision and Control (Cat. No.98CH36171).

[25]  Rich Caruana,et al.  Multitask Learning , 1998, Encyclopedia of Machine Learning and Data Mining.

[26]  Lennart Ljung,et al.  Perspectives on system identification , 2010, Annu. Rev. Control..

[27]  Dietmar Bauer,et al.  Order estimation for subspace methods , 2001, Autom..

[28]  Alessandro Chiuso,et al.  Online identification of time-varying systems: A Bayesian approach , 2016, 2016 IEEE 55th Conference on Decision and Control (CDC).

[29]  Dong-Jo Park,et al.  Fast tracking RLS algorithm using novel variable forgetting factor with unity zone , 1991 .

[30]  Biao Huang,et al.  System Identification , 2000, Control Theory for Physicists.

[31]  V. Verdult,et al.  Filtering and System Identification: A Least Squares Approach , 2007 .

[32]  Bo Wahlberg,et al.  Estimation Of Autoregressive Moving‐Average Models Via High‐Order Autoregressive Approximations , 1989 .

[33]  C. L. Mallows Some comments on C_p , 1973 .

[34]  Thomas M. Cover,et al.  Elements of Information Theory , 2005 .

[35]  Bhaskar D. Rao,et al.  Latent Variable Bayesian Models for Promoting Sparsity , 2011, IEEE Transactions on Information Theory.

[36]  Karen Willcox,et al.  A Survey of Projection-Based Model Reduction Methods for Parametric Dynamical Systems , 2015, SIAM Rev..

[37]  George Kapetanios,et al.  Testing the rank of the Hankel covariance matrix: a statistical approach , 2001, IEEE Trans. Autom. Control..

[38]  W. Larimore System Identification, Reduced-Order Filtering and Modeling via Canonical Variate Analysis , 1983, 1983 American Control Conference.

[39]  Michel Verhaegen,et al.  Identification of the deterministic part of MIMO state space models given in innovations form from input-output data , 1994, Autom..

[40]  Erik Weyer,et al.  Non-asymptotic confidence regions for model parameters in the presence of unmodelled dynamics , 2007, 2007 46th IEEE Conference on Decision and Control.

[41]  R. V. Monopoli,et al.  Stationary Linear and Nonlinear System Identification and Predictor Set Completeness , 1978 .

[42]  S. Hill Reduced gradient computation in prediction error identification , 1985 .

[43]  B. Efron The Estimation of Prediction Error , 2004 .

[44]  Lennart Ljung,et al.  System Identification: Theory for the User , 1987 .

[45]  Rik Pintelon,et al.  System Identification: A Frequency Domain Approach , 2012 .

[46]  Michel Verhaegen,et al.  Subspace identification of multivariable linear parameter-varying systems , 2002, Autom..

[47]  Francesco Dinuzzo,et al.  Kernels for Linear Time Invariant System Identification , 2012, SIAM J. Control. Optim..

[48]  Giuseppe De Nicolao,et al.  A new kernel-based approach for linear system identification , 2010, Autom..

[49]  Bart De Moor,et al.  Continuous-time frequency domain subspace system identification , 1996, Signal Process..

[50]  H. Zeiger,et al.  Approximate linear realizations of given dimension via Ho's algorithm , 1974 .

[51]  R. Genesio,et al.  Identification of reduced models from noisy data , 1975 .

[52]  Bo Wahlberg,et al.  On Consistency of Subspace Methods for System Identification , 1998, Autom..

[53]  Bart De Moor,et al.  Subspace Identification for Linear Systems: Theory ― Implementation ― Applications , 2011 .

[54]  Erik Weyer,et al.  Guaranteed non-asymptotic confidence regions in system identification , 2005, Autom..

[55]  Yiming Ying,et al.  Learning Rates of Least-Square Regularized Regression , 2006, Found. Comput. Math..

[56]  Johan A. K. Suykens,et al.  Identification of stable models in subspace identification by using regularization , 2001, IEEE Trans. Autom. Control..

[57]  J. Lasserre A trace inequality for matrix product , 1995, IEEE Trans. Autom. Control..

[58]  Henrik Ohlsson,et al.  Robust Subspace System Identification via Weighted Nuclear Norm Optimization , 2013, ArXiv.

[59]  A. Tether Construction of minimal linear state-variable models from finite input-output data , 1970 .

[60]  Ashutosh Kumar Singh,et al.  The Elements of Statistical Learning: Data Mining, Inference, and Prediction , 2010 .

[61]  U. Grenander Stochastic processes and statistical inference , 1950 .

[62]  Thilo Penzl Algorithms for model reduction of large dynamical systems , 2006 .

[63]  H. Zou,et al.  Regularization and variable selection via the elastic net , 2005 .

[64]  Michel Verhaegen,et al.  Nuclear Norm Subspace Identification (N2SID) for short data batches , 2014, ArXiv.

[65]  F. Girosi,et al.  Networks for approximation and learning , 1990, Proc. IEEE.

[66]  D.G. Dudley,et al.  Dynamic system identification experiment design and data analysis , 1979, Proceedings of the IEEE.

[67]  P. Whittle,et al.  Estimation and information in stationary time series , 1953 .

[68]  L. A. Zadeh,et al.  From Circuit Theory to System Theory , 1962, Proceedings of the IRE.

[69]  E. Hannan,et al.  The statistical theory of linear systems , 1989 .

[70]  Erik Weyer,et al.  Non-Asymptotic Confidence Sets for the Parameters of Linear Transfer Functions , 2010, IEEE Transactions on Automatic Control.

[71]  R. E. Kalman,et al.  On minimal partial realizations of a linear input/output map , 1971 .

[72]  Franziska Abend,et al.  State Space Modeling Of Time Series , 2016 .

[73]  Norman R. Draper,et al.  Applied regression analysis bibliography update 1994-97 , 1998 .

[74]  Massimo Fornasier,et al.  Low-rank Matrix Recovery via Iteratively Reweighted Least Squares Minimization , 2010, SIAM J. Optim..

[75]  Thomas Kailath,et al.  Fast recursive identification of state space models via exploitation of displacement structure , 1994, Autom..

[76]  Erik Weyer,et al.  Finite sample properties of system identification methods , 2002, IEEE Trans. Autom. Control..

[77]  G. Van Zee,et al.  Gradient computation in prediction error identification of linear discrete-time systems , 1982 .

[78]  Alessandro Chiuso,et al.  Numerical conditioning and asymptotic variance of subspace estimates , 2004, Autom..

[79]  Bjorn Ottersten,et al.  A subspace fitting method for identification of linear state-space models , 1995, IEEE Trans. Autom. Control..

[80]  Bernhard Schölkopf,et al.  Sparse Greedy Matrix Approximation for Machine Learning , 2000, International Conference on Machine Learning.

[81]  Michel Verhaegen,et al.  A Novel Non-Iterative Mimo State Space Model Identification Technique , 1991 .

[82]  M. Fazel,et al.  Reweighted nuclear norm minimization with application to system identification , 2010, Proceedings of the 2010 American Control Conference.

[83]  Manfred Deistler,et al.  Statistical analysis of novel subspace identification methods , 1996, Signal Process..

[84]  P. Kumar,et al.  Theory and practice of recursive identification , 1985, IEEE Transactions on Automatic Control.

[85]  T. Söderström,et al.  Instrumental variable methods for system identification , 1983 .

[86]  Michele Lenza,et al.  Prior Selection for Vector Autoregressions , 2012, Review of Economics and Statistics.

[87]  Alessandro Chiuso,et al.  The role of rank penalties in linear system identification , 2015 .

[88]  Massimo Fornasier,et al.  Compressive Sensing , 2015, Handbook of Mathematical Methods in Imaging.

[89]  Alessandro Chiuso,et al.  Bayesian and regularization approaches to multivariable linear system identification: The role of rank penalties , 2014, 53rd IEEE Conference on Decision and Control.

[90]  Marco Lovera,et al.  CONVERGENCE ANALYSIS OF INSTRUMENTAL VARIABLE RECURSIVE SUBSPACE IDENTIFICATION ALGORITHMS , 2006 .

[91]  A. Chiuso,et al.  The asymptotic variance of subspace estimates , 2004 .

[92]  G. Wahba,et al.  A Correspondence Between Bayesian Estimation on Stochastic Processes and Smoothing by Splines , 1970 .

[93]  Ralph Otto Schmidt,et al.  A signal subspace approach to multiple emitter location and spectral estimation , 1981 .

[94]  David M. Allen,et al.  The Relationship Between Variable Selection and Data Agumentation and a Method for Prediction , 1974 .

[95]  Alessandro Chiuso,et al.  A Scaled Gradient Projection Method for Bayesian Learning in Dynamical Systems , 2014, SIAM J. Sci. Comput..

[96]  Mario Sznaier,et al.  A rank minimization approach to trajectory (in)validation , 2011, Proceedings of the 2011 American Control Conference.

[97]  Athanasios C. Antoulas,et al.  An overview of approximation methods for large-scale dynamical systems , 2005, Annu. Rev. Control..

[98]  J. Borwein,et al.  Two-Point Step Size Gradient Methods , 1988 .

[99]  Bernhard Schölkopf,et al.  A Generalized Representer Theorem , 2001, COLT/EuroCOLT.

[100]  Johan A. K. Suykens,et al.  Least Squares Support Vector Machine Classifiers , 1999, Neural Processing Letters.

[101]  Manfred Morari,et al.  System identification via nuclear norm regularization for simulated moving bed processes from incomplete data sets , 2009, Proceedings of the 48h IEEE Conference on Decision and Control (CDC) held jointly with 2009 28th Chinese Control Conference.

[102]  Giuseppe Carlo Calafiore,et al.  Approximation of n-dimensional data using spherical and ellipsoidal primitives , 2002, IEEE Trans. Syst. Man Cybern. Part A.

[103]  Roman Holenstein,et al.  Particle Markov chain Monte Carlo , 2009 .

[104]  Yucai Zhu,et al.  Identification of Multivariable Industrial Processes: for Simulation, Diagnosis and Control , 1993 .

[105]  C. De Mol,et al.  Forecasting Using a Large Number of Predictors: Is Bayesian Regression a Valid Alternative to Principal Components? , 2006, SSRN Electronic Journal.

[106]  Heinrich Rake,et al.  Identification: Transient- and Frequency-Response Methods , 1992, Concise Encyclopedia of Modelling & Simulation.

[107]  Lennart Ljung,et al.  Stochastic Embedding revisited: A modern interpretation , 2014, 53rd IEEE Conference on Decision and Control.

[108]  N. Davies Multiple Time Series , 2005 .

[109]  Alessandro Chiuso,et al.  Maximum Entropy vector kernels for MIMO system identification , 2015, Autom..

[110]  R. Kálmán Irreducible realizations and the degree of a rational matrix. , 1965 .

[111]  Lennart Ljung,et al.  A statistical perspective on state-space modeling using subspace methods , 1991, [1991] Proceedings of the 30th IEEE Conference on Decision and Control.

[112]  Peter Wellstead,et al.  Instrumental product moment model-order testing: extensions and application , 1982 .

[113]  E. Jaynes,et al.  Confidence Intervals vs Bayesian Intervals , 1976 .

[114]  Stéphane Lecoeuche,et al.  Propagator-based methods for recursive subspace model identification , 2008, Signal Process..

[115]  M.C. Campi,et al.  Non-asymptotic confidence sets for input-output transfer functions , 2006, Proceedings of the 45th IEEE Conference on Decision and Control.

[116]  B. Anderson,et al.  Frequency-weighted optimal Hankel-norm approximation of stable transfer functions , 1985 .

[117]  Dimitri P. Bertsekas,et al.  Constrained Optimization and Lagrange Multiplier Methods , 1982 .

[118]  Lieven Vandenberghe,et al.  Interior-Point Method for Nuclear Norm Approximation with Application to System Identification , 2009, SIAM J. Matrix Anal. Appl..

[119]  I. Daubechies,et al.  Iteratively reweighted least squares minimization for sparse recovery , 2008, 0807.0575.

[120]  Tianshi Chen,et al.  Transfer function and transient estimation by Gaussian process regression in the frequency domain , 2016, Autom..

[121]  Massimiliano Pontil,et al.  Regularized multi--task learning , 2004, KDD.

[122]  F. Agterberg Introduction to Mathematics of Inversion in Remote Sensing and Indirect Measurements , 1979 .

[123]  Christopher K. I. Williams,et al.  Gaussian regression and optimal finite dimensional linear models , 1997 .

[124]  Dietmar Bauer,et al.  Consistency and asymptotic normality of some subspace algorithms for systems without observed inputs , 1999, Autom..

[125]  Giulio Bottegal,et al.  A kernel-based approach to Hammerstein system identification , 2014, ArXiv.

[126]  Howard L. Weinert,et al.  Structure determination and parameter identification for multivariable stochastic linear systems , 1975 .

[127]  Alessandro Chiuso,et al.  Identification of stable models via nonparametric prediction error methods , 2015, 2015 European Control Conference (ECC).

[128]  Anthony Widjaja,et al.  Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond , 2003, IEEE Transactions on Neural Networks.

[129]  S. Bittanti,et al.  Assessing the quality of identified models through the asymptotic theory - when is the result reliable? , 2004, Autom..

[130]  Henrik Ohlsson,et al.  On the estimation of transfer functions, regularizations and Gaussian processes - Revisited , 2012, Autom..

[131]  T. McKelvey Identification of State-Space Models from Time and Frequency Data , 1995 .

[132]  R. Bitmead,et al.  Empirical Estimation of Parameter Distributions in System Identification , 2003 .

[133]  Edward J. Powers,et al.  Time-varying spectral estimation using AR models with variable forgetting factors , 1991, IEEE Trans. Signal Process..

[134]  M. Yuan,et al.  A Reproducing Kernel Hilbert Space Approach to Functional Linear Regression , 2010, 1211.2607.

[135]  Alessandro Chiuso,et al.  Regularization and Bayesian learning in dynamical systems: Past, present and future , 2015, Annu. Rev. Control..

[136]  Lennart Ljung,et al.  Closed-Loop Subspace Identification with Innovation Estimation , 2003 .

[137]  Lennart Ljung,et al.  Maximum entropy properties of discrete-time first-order stable spline kernel , 2014, Autom..

[138]  I. Miller Probability, Random Variables, and Stochastic Processes , 1966 .

[139]  Balázs Csanád Csáji,et al.  Sign-Perturbed Sums: A New System Identification Approach for Constructing Exact Non-Asymptotic Confidence Regions in Linear Regression Models , 2018, IEEE Transactions on Signal Processing.

[140]  Alessandro Chiuso,et al.  A New Kernel-Based Approach for NonlinearSystem Identification , 2011, IEEE Transactions on Automatic Control.

[141]  Bo Wahlberg,et al.  Reweighted nuclear norm regularization: A SPARSEVA approach , 2015, 1507.05718.

[142]  D. Bauer Some asymptotic theory for the estimation of linear systems using maximum likelihood methods or subspace algorithms , 1998 .

[143]  Robert B. Litterman,et al.  Forecasting and Conditional Projection Using Realistic Prior Distributions , 1983 .

[144]  Charles L. Lawson,et al.  Solving least squares problems , 1976, Classics in applied mathematics.

[145]  W. Härdle,et al.  Bootstrapping in Nonparametric Regression: Local Adaptive Smoothing and Confidence Bands , 1988 .

[146]  Tien C. Hsia,et al.  System identification: Least-squares methods , 1977 .

[147]  N. Aronszajn Theory of Reproducing Kernels. , 1950 .

[148]  B. M. Pötscher,et al.  MODEL SELECTION AND INFERENCE: FACTS AND FICTION , 2005, Econometric Theory.

[149]  T. Gustafsson Recursive System Identification Using Instrumental Variable Subspace Tracking , 1997 .

[150]  H. Rake,et al.  Step response and frequency response methods , 1980, Autom..

[151]  Brett Ninness,et al.  Bayesian system identification via Markov chain Monte Carlo techniques , 2010, Autom..

[152]  A. Antoulas,et al.  A Survey of Model Reduction by Balanced Truncation and Some New Results , 2004 .

[153]  J. Mendel,et al.  Discrete Techniques of Parameter Estimation: The Equation Error Formulation , 1973 .

[154]  H. Akaike,et al.  Information Theory and an Extension of the Maximum Likelihood Principle , 1973 .

[155]  David P. Wipf,et al.  Non-Convex Rank Minimization via an Empirical Bayesian Approach , 2012, UAI.

[156]  Mathukumalli Vidyasagar,et al.  A learning theory approach to system identification and stochastic adaptive control , 2008 .

[157]  Wallace E. Larimore,et al.  Canonical variate analysis in identification, filtering, and adaptive control , 1990, 29th IEEE Conference on Decision and Control.

[158]  Giuseppe De Nicolao,et al.  Pitfalls of the parametric approaches exploiting cross-validation for model order selection* , 2012 .

[159]  A. Doucet,et al.  Particle Markov chain Monte Carlo methods , 2010 .

[160]  S. Smale,et al.  Learning Theory Estimates via Integral Operators and Their Approximations , 2007 .

[161]  Karl Johan Åström,et al.  Numerical Identification of Linear Dynamic Systems from Normal Operating Records , 1965 .

[162]  John G. Proakis,et al.  Probability, random variables and stochastic processes , 1985, IEEE Trans. Acoust. Speech Signal Process..

[163]  J. Rissanen,et al.  Modeling By Shortest Data Description* , 1978, Autom..

[164]  Lennart Ljung,et al.  Estimating model variance in the case of undermodeling , 1992 .

[165]  B. Wahlberg System identification using Laguerre models , 1991 .

[166]  T. McKelveyDepartment On the Use of Regularization in System Identification , 1992 .

[167]  A. Caponnetto,et al.  Optimal Rates for the Regularized Least-Squares Algorithm , 2007, Found. Comput. Math..

[168]  Tomaso A. Poggio,et al.  Regularization Theory and Neural Networks Architectures , 1995, Neural Computation.

[169]  K. Glover All optimal Hankel-norm approximations of linear multivariable systems and their L, ∞ -error bounds† , 1984 .

[170]  James Stephen Marron,et al.  BOOTSTRAP SIMULTANEOUS ERROR BARS FOR NONPARAMETRIC REGRESSION , 1991 .

[171]  Steven Kay,et al.  Modern Spectral Estimation: Theory and Application , 1988 .

[172]  Alessandro Chiuso,et al.  Convex vs non-convex estimators for regression and sparse estimation: the mean squared error properties of ARD and GLasso , 2014, J. Mach. Learn. Res..

[173]  Douglas Nychka,et al.  Bayesian Confidence Intervals for Smoothing Splines , 1988 .

[174]  Lennart Ljung,et al.  Kernel methods in system identification, machine learning and function estimation: A survey , 2014, Autom..

[175]  H. Akaike A new look at the statistical model identification , 1974 .

[176]  F. Chatelin Spectral approximation of linear operators , 2011 .

[177]  Zhang Liu,et al.  Nuclear norm system identification with missing inputs and outputs , 2013, Syst. Control. Lett..

[178]  Bart De Moor,et al.  Subspace algorithms for the stochastic identification problem, , 1993, Autom..

[179]  D K Smith,et al.  Numerical Optimization , 2001, J. Oper. Res. Soc..

[180]  Jianming Ye On Measuring and Correcting the Effects of Data Mining and Model Selection , 1998 .

[181]  Peter E. Wellstead Non-parametric methods of system identification , 1981, Autom..

[182]  Andreas Antoniou,et al.  New Improved Recursive Least-Squares Adaptive-Filtering Algorithms , 2013, IEEE Transactions on Circuits and Systems I: Regular Papers.

[183]  Karl Johan Åström,et al.  BOOK REVIEW SYSTEM IDENTIFICATION , 1994, Econometric Theory.

[184]  G. Wahba,et al.  Bootstrap confidence intervals for smoothing splines and their comparison to bayesian confidence intervals , 1995 .

[185]  Lennart Ljung,et al.  Implementation of algorithms for tuning parameters in regularized least squares problems in system identification , 2013, Autom..

[186]  Giuseppe De Nicolao,et al.  Bayesian Function Learning Using MCMC Methods , 1998, IEEE Trans. Pattern Anal. Mach. Intell..

[187]  D. Harville Matrix Algebra From a Statistician's Perspective , 1998 .

[188]  M. Degroot Optimal Statistical Decisions , 1970 .

[189]  H. Hjalmarsson,et al.  Identification of Box-Jenkins Models Using Structured ARX Models and Nuclear Norm Relaxation , 2012 .

[190]  Alessandro Chiuso,et al.  On-line Bayesian system identification , 2016, 2016 European Control Conference (ECC).

[191]  Eric F. Wood,et al.  Review and Unification of Linear Identifiability Concepts , 1982 .

[192]  Petre Stoica,et al.  An indirect prediction error method for system identification , 1991, Autom..

[193]  B. Moore Principal component analysis in linear systems: Controllability, observability, and model reduction , 1981 .

[194]  van den Pmj Paul Hof,et al.  Bayesian system identification based on generalized orthonormal basis functions , 2014 .

[195]  Vladimir Vapnik,et al.  Statistical learning theory , 1998 .

[196]  Don R. Hush,et al.  Optimal Rates for Regularized Least Squares Regression , 2009, COLT.

[197]  Alessandro Chiuso,et al.  Model reduction for linear bayesian system identification , 2015, 2015 54th IEEE Conference on Decision and Control (CDC).

[198]  Johan A. K. Suykens,et al.  Subspace algorithms for system identification and stochastic realization , 1991 .

[199]  Bart De Moor,et al.  N4SID: Subspace algorithms for the identification of combined deterministic-stochastic systems , 1994, Autom..

[200]  Emmanuel J. Candès,et al.  Exact Matrix Completion via Convex Optimization , 2008, Found. Comput. Math..

[201]  Karl Johan Åström,et al.  Lectures on the Identification Problem : The Least Squares Method , 1968 .

[202]  Arild Thowsen,et al.  Structural identifiability , 1977, 1977 IEEE Conference on Decision and Control including the 16th Symposium on Adaptive Processes and A Special Symposium on Fuzzy Set Theory and Applications.

[203]  A. Antoulas,et al.  H 2 Model Reduction for Large-scale Linear Dynamical Systems * , 2022 .

[204]  Charles A. Micchelli,et al.  On Learning Vector-Valued Functions , 2005, Neural Computation.

[205]  Serkan Gugercin,et al.  Model reduction of large-scale systems by least squares , 2006 .

[206]  Maryam Fazel,et al.  Iterative reweighted algorithms for matrix rank minimization , 2012, J. Mach. Learn. Res..

[207]  J.T. Gravdahl,et al.  MPC for Large-Scale Systems via Model Reduction and Multiparametric Quadratic Programming , 2006, Proceedings of the 45th IEEE Conference on Decision and Control.

[208]  B. Schutter,et al.  Minimal state-space realization in linear system theory: an overview , 2000 .

[209]  H. Leeb,et al.  CAN ONE ESTIMATE THE UNCONDITIONAL DISTRIBUTION OF POST-MODEL-SELECTION ESTIMATORS? , 2003, Econometric Theory.

[210]  Michel Verhaegen,et al.  Application of a subspace model identification technique to identify LTI systems operating in closed-loop , 1993, Autom..

[211]  Danny C. Sorensen,et al.  A Modified Low-Rank Smith Method for Large-Scale Lyapunov Equations , 2004, Numerical Algorithms.

[212]  S. Liberty,et al.  Linear Systems , 2010, Scientific Parallel Computing.

[213]  Constantino M. Lagoa,et al.  Parsimonious model identification via atomic norm minimization , 2014, 2014 European Control Conference (ECC).

[214]  Thomas Kailath,et al.  Fast transversal filters with data sequence weighting , 1989, IEEE Trans. Acoust. Speech Signal Process..

[215]  Manfred Deistler,et al.  Consistency and relative efficiency of subspace methods , 1994, Autom..

[216]  B. Moor,et al.  A geometrical strategy for the identification of state space models of linear multivariable systems with singular value decomposition , 1987 .

[217]  Sylvia Richardson,et al.  Markov Chain Monte Carlo in Practice , 1997 .

[218]  E. Parzen An Approach to Time Series Analysis , 1961 .

[219]  Thomas A. Louis,et al.  Empirical Bayes Methods , 2006 .

[220]  M. Lovera,et al.  Recursive subspace identification based on instrumental variable unconstrained quadratic optimization , 2004 .

[221]  Björn E. Ottersten,et al.  Sensor array processing based on subspace fitting , 1991, IEEE Trans. Signal Process..

[222]  Birgit Dietrich,et al.  Model Reduction For Control System Design , 2016 .

[223]  Iven M. Y. Mareels,et al.  Finite sample properties of linear model identification , 1999, IEEE Trans. Autom. Control..

[224]  Arthur E. Hoerl,et al.  Ridge Regression: Biased Estimation for Nonorthogonal Problems , 2000, Technometrics.

[225]  Dietmar Bauer,et al.  Asymptotic properties of subspace estimators , 2005, Autom..

[226]  Lennart Ljung,et al.  On The Consistency of Prediction Error Identification Methods , 1976 .

[227]  M. Verhaegen,et al.  A fast, recursive MIMO state space model identification algorithm , 1991, [1991] Proceedings of the 30th IEEE Conference on Decision and Control.

[228]  Mats Viberg,et al.  Subspace-based methods for the identification of linear time-invariant systems , 1995, Autom..

[229]  Danny C. Sorensen,et al.  The Sylvester equation and approximate balanced reduction , 2002 .

[230]  Wotao Yin,et al.  Iteratively reweighted algorithms for compressive sensing , 2008, 2008 IEEE International Conference on Acoustics, Speech and Signal Processing.

[231]  C. Woodside Estimation of the order of linear systems , 1971 .

[232]  Bo WahlbergS On Weighting in State-space Subspace System Identification , 1995 .

[233]  Thomas Knox,et al.  Empirical Bayes Forecasts of One Time Series Using Many Predictors , 2001 .

[234]  R. Cook,et al.  A Fast Parameter Tracking RLS Algorithm with High Noise Immunity , 1993, 1993 American Control Conference.

[235]  S. Gugercin,et al.  An iterative SVD-Krylov based method for model reduction of large-scale dynamical systems , 2005, Proceedings of the 44th IEEE Conference on Decision and Control.

[236]  D. Sorensen,et al.  A Survey of Model Reduction Methods for Large-Scale Systems , 2000 .

[237]  Bart De Moor,et al.  A unifying theorem for three subspace system identification algorithms , 1995, Autom..

[238]  Alessandro Chiuso,et al.  Classical vs. Bayesian methods for linear system identification: Point estimators and confidence sets , 2015, 2016 European Control Conference (ECC).

[239]  Håkan Hjalmarsson,et al.  A weighted least-squares method for parameter estimation in structured models , 2014, 53rd IEEE Conference on Decision and Control.

[240]  Carl E. Rasmussen,et al.  Gaussian processes for machine learning , 2005, Adaptive computation and machine learning.

[241]  Dietmar Bauer,et al.  Analysis of the asymptotic properties of the MOESP type of subspace algorithms , 2000, Autom..

[242]  J. Rosenthal,et al.  Markov Chain Monte Carlo , 2018 .

[243]  B. Wahlberg Model reductions of high-order estimated models : the asymptotic ML approach , 1989 .

[244]  P. Young,et al.  Time series analysis, forecasting and control , 1972, IEEE Transactions on Automatic Control.

[245]  P. V. D. Hof,et al.  Finite Sample Confidence Regions for Parameters in Prediction Error Identification using Output Error Models , 2008 .

[246]  Lennart Ljung,et al.  Frequency domain versus time domain methods in system identification , 1981, Autom..

[247]  L. Ljung,et al.  Subspace-based multivariable system identification from frequency response data , 1996, IEEE Trans. Autom. Control..

[248]  Felipe Cucker,et al.  On the mathematical foundations of learning , 2001 .

[249]  P. Overschee Subspace Identification: Theory, Implementation, Application , 1995 .

[250]  Stephen P. Boyd,et al.  Segmentation of ARX-models using sum-of-norms regularization , 2010, Autom..

[251]  Joakim Sorelius Subspace-based parameter estimation problems in signal processing , 1999 .

[252]  Erik Weyer,et al.  IDENTIFICATION WITH FINITELY MANY DATA POINTS: THE LSCR APPROACH , 2006 .

[253]  Alessandro Chiuso,et al.  A Bayesian approach to sparse dynamic network identification , 2012, Autom..

[254]  J. Berger Statistical Decision Theory and Bayesian Analysis , 1988 .

[255]  Lennart Ljung,et al.  Regularized linear system identification using atomic, nuclear and kernel-based norms: The role of the stability constraint , 2015, Autom..

[256]  C. Stein Estimation of the Mean of a Multivariate Normal Distribution , 1981 .

[257]  T. Söderström Brief paper: Test of pole-zero cancellation in estimated models , 1975 .

[258]  David Q. Mayne,et al.  Linear identification of ARMA processes , 1982, Autom..

[259]  Francesca P. Carli,et al.  Efficient algorithms for large scale linear system identification using stable spline estimators , 2012 .

[260]  G. Wahba Bayesian "Confidence Intervals" for the Cross-validated Smoothing Spline , 1983 .

[261]  Roy S. Smith,et al.  Frequency Domain Subspace Identification Using Nuclear Norm Minimization and Hankel Matrix Realizations , 2014, IEEE Transactions on Automatic Control.

[262]  Erik Weyer Finite sample properties of system identification of ARX models under mixing conditions , 2000, Autom..

[263]  Alessandro Chiuso,et al.  Tuning complexity in kernel-based linear system identification: The robustness of the marginal likelihood estimator , 2014, 2014 European Control Conference (ECC).

[264]  Zhibiao Zhao Parametric and nonparametric models and methods in financial econometrics , 2008, 0801.1599.

[265]  Michel Gevers,et al.  Uniquely identifiable state-space and ARMA parametrizations for multivariable linear systems , 1984, Autom..

[266]  Alessandro Chiuso,et al.  Consistency analysis of some closed-loop subspace identification methods , 2005, Autom..

[267]  G. Wahba,et al.  Some results on Tchebycheffian spline functions , 1971 .

[268]  Lennart Ljung,et al.  Subspace identification from closed loop data , 1996, Signal Process..

[269]  J. Schoukens,et al.  Parametric identification of transfer functions in the frequency domain-a survey , 1994, IEEE Trans. Autom. Control..

[270]  Lennart Ljung,et al.  Regularization strategies for nonparametric system identification , 2013, 52nd IEEE Conference on Decision and Control.

[271]  Lennart Ljung,et al.  On the design of multiple kernels for nonparametric linear system identification , 2014, 53rd IEEE Conference on Decision and Control.

[272]  J. Burke,et al.  On the MSE Properties of Empirical Bayes Methods for Sparse Estimation , 2012 .

[273]  Zhang Liu,et al.  Subspace system identification via weighted nuclear norm optimization , 2012, 2012 IEEE 51st IEEE Conference on Decision and Control (CDC).

[274]  M. Schervish,et al.  Posterior Consistency in Nonparametric Regression Problems under Gaussian Process Priors , 2004 .

[275]  G. McLachlan,et al.  The EM algorithm and extensions , 1996 .

[276]  L. Ljung,et al.  Constructive state space model induced kernels for regularized system identification , 2014 .

[277]  Pablo A. Parrilo,et al.  Guaranteed Minimum-Rank Solutions of Linear Matrix Equations via Nuclear Norm Minimization , 2007, SIAM Rev..

[278]  Alessandro Chiuso,et al.  On the Asymptotic Properties of Closed-Loop CCA-Type Subspace Algorithms: Equivalence Results and Role of the Future Horizon , 2010, IEEE Transactions on Automatic Control.

[279]  Bart De Moor,et al.  Choice of state-space basis in combined deterministic-stochastic subspace identification , 1995, Autom..

[280]  Bo Wahlberg,et al.  Analysis of state space system identification methods based on instrumental variables and subspace fitting , 1997, Autom..

[281]  Shu Hung Leung,et al.  Gradient-based variable forgetting factor RLS algorithm in time-varying environments , 2005, IEEE Transactions on Signal Processing.

[282]  David R. Anderson,et al.  Model selection and multimodel inference : a practical information-theoretic approach , 2003 .

[283]  Jacob Benesty,et al.  A Robust Variable Forgetting Factor Recursive Least-Squares Algorithm for System Identification , 2008, IEEE Signal Processing Letters.

[284]  Håkan Hjalmarsson,et al.  Sparse estimation based on a validation criterion , 2011, IEEE Conference on Decision and Control and European Control Conference.

[285]  B. Efron Bayesians, Frequentists, and Scientists , 2005 .

[286]  Michel Verhaegen,et al.  Recursive subspace identification of linear and non-linear Wiener state-space models , 2000, Autom..

[287]  Alessandro Chiuso,et al.  Prediction error identification of linear systems: A nonparametric Gaussian regression approach , 2011, Autom..

[288]  K. Sung,et al.  Gauss Newton variable forgetting factor recursive least squares for time varying parameter tracking , 2000 .

[289]  Bin Yang,et al.  Projection approximation subspace tracking , 1995, IEEE Trans. Signal Process..

[290]  Richard A. Davis,et al.  Time Series: Theory and Methods , 2013 .

[291]  T. R. Fortescue,et al.  Implementation of self-tuning regulators with variable forgetting factors , 1981, Autom..

[292]  John E. Dennis,et al.  Numerical methods for unconstrained optimization and nonlinear equations , 1983, Prentice Hall series in computational mathematics.

[293]  Hidenori Kimura,et al.  Recursive 4SID algorithms using gradient type subspace tracking , 2002, Autom..

[294]  E. Parzen STATISTICAL INFERENCE ON TIME SERIES BY RKHS METHODS. , 1970 .

[295]  H. Hotelling Relations Between Two Sets of Variates , 1936 .

[296]  Y. Yao,et al.  On Early Stopping in Gradient Descent Learning , 2007 .

[297]  David Lindley,et al.  CBMS-NSF REGIONAL CONFERENCE SERIES IN APPLIED MATHEMATICS , 2010 .

[298]  G. Marinoschi An identification problem , 2005 .

[299]  Charles A. Micchelli,et al.  Learning Multiple Tasks with Kernel Methods , 2005, J. Mach. Learn. Res..

[300]  D. Rubin,et al.  Maximum likelihood from incomplete data via the EM - algorithm plus discussions on the paper , 1977 .

[301]  Parikshit Shah,et al.  Linear system identification via atomic norm regularization , 2012, 2012 IEEE 51st IEEE Conference on Decision and Control (CDC).

[302]  Henrik Ohlsson,et al.  Four Encounters with System Identification , 2011, Eur. J. Control.

[303]  M. Verhaegen Subspace model identification Part 2. Analysis of the elementary output-error state-space model identification algorithm , 1992 .

[304]  S. Mendelson,et al.  Regularization in kernel learning , 2010, 1001.2094.

[305]  Dennis S. Bernstein,et al.  Subspace identification with guaranteed stability using constrained optimization , 2003, IEEE Trans. Autom. Control..

[306]  Giulio Bottegal,et al.  Robust EM kernel-based methods for linear system identification , 2014, Autom..

[307]  H. Haario,et al.  An adaptive Metropolis algorithm , 2001 .

[308]  Magnus Jansson On subspace methods in system identification and sensor array signal processing , 1997 .

[309]  Maliha S. Nash,et al.  Handbook of Parametric and Nonparametric Statistical Procedures , 2001, Technometrics.

[310]  G. GILBERTt CONTROLLABILITY AND OBSERVABILITY IN MULTIVARIABLE CONTROL SYSTEMS , 2022 .

[311]  Alessandro Chiuso,et al.  Learning sparse dynamic linear systems using stable spline kernels and exponential hyperpriors , 2010, NIPS.

[312]  M. Jansson Asymptotic Variance Analysis of Subspace Identification Methods , 2000 .

[313]  Graham C. Goodwin,et al.  Estimated Transfer Functions with Application to Model Order Selection , 1992 .