Regularized linear system identification using atomic, nuclear and kernel-based norms: The role of the stability constraint

Inspired by ideas taken from the machine learning literature, new regularization techniques have been recently introduced in linear system identification. In particular, all the adopted estimators solve a regularized least squares problem, differing in the nature of the penalty term assigned to the impulse response. Popular choices include atomic and nuclear norms (applied to Hankel matrices) as well as norms induced by the so called stable spline kernels. In this paper, a comparative study of estimators based on these different types of regularizers is reported. Our findings reveal that stable spline kernels outperform approaches based on atomic and nuclear norms since they suitably embed information on impulse response stability and smoothness. This point is illustrated using the Bayesian interpretation of regularization. We also design a new class of regularizers defined by "integral" versions of stable spline/TC kernels. Under quite realistic experimental conditions, the new estimators outperform classical prediction error methods also when the latter are equipped with an oracle for model order selection.

[1]  Parikshit Shah,et al.  Linear system identification via atomic norm regularization , 2012, 2012 IEEE 51st IEEE Conference on Decision and Control (CDC).

[2]  Alessandro Chiuso,et al.  Tuning complexity in regularized kernel-based regression and linear system identification: The robustness of the marginal likelihood estimator , 2015, Autom..

[3]  Ying Li,et al.  Predictive Control Model for Radiant Heating System Based on Neural Network , 2008, 2008 International Conference on Computer Science and Software Engineering.

[4]  Håkan Hjalmarsson,et al.  Sparse Estimation of Polynomial and Rational Dynamical Models , 2014, IEEE Transactions on Automatic Control.

[5]  Philip Haves,et al.  Model predictive control for the operation of building cooling systems , 2010, Proceedings of the 2010 American Control Conference.

[6]  H. Zou.  The Adaptive Lasso and Its Oracle Properties , 2006 .

[7]  Stephen P. Boyd,et al.  A rank minimization heuristic with application to minimum order system approximation , 2001, Proceedings of the 2001 American Control Conference. (Cat. No.01CH37148).

[8]  Xuelong Li,et al.  Tensors in Image Processing and Computer Vision , 2009, Advances in Pattern Recognition.

[9]  Biao Huang,et al.  System Identification , 2000, Control Theory for Physicists.

[10]  B. Efron.  The Estimation of Prediction Error , 2004 .

[11]  Lennart Ljung,et al.  System Identification: Theory for the User , 1987 .

[12]  Peter Green,et al.  Markov chain Monte Carlo in Practice , 1996 .

[13]  Anthony Widjaja,et al.  Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond , 2003, IEEE Transactions on Neural Networks.

[14]  Henrik Ohlsson,et al.  On the estimation of transfer functions, regularizations and Gaussian processes - Revisited , 2012, Autom..

[15]  Felipe Cucker,et al.  On the mathematical foundations of learning , 2001 .

[16]  Henrik Ohlsson,et al.  Robust Subspace System Identification via Weighted Nuclear Norm Optimization , 2013, ArXiv.

[17]  Ashutosh Kumar Singh,et al.  The Elements of Statistical Learning: Data Mining, Inference, and Prediction , 2010 .

[18]  Emmanuel J. Candès,et al.  Exact Matrix Completion via Convex Optimization , 2008, Found. Comput. Math..

[19]  G. Burton.  Sobolev Spaces , 2013 .

[20]  Jianqing Fan,et al.  Variable Selection via Nonconcave Penalized Likelihood and its Oracle Properties , 2001 .

[21]  강승규,et al.  Empirical Bayes Method를 이용한 교통사고 예측모형 , 2009 .

[22]  Emmanuel J. Candès,et al.  Exact Matrix Completion via Convex Optimization , 2009, Found. Comput. Math..

[23]  Lieven Vandenberghe,et al.  Interior-Point Method for Nuclear Norm Approximation with Application to System Identification , 2009, SIAM J. Matrix Anal. Appl..

[24]  M. Fazel,et al.  Reweighted nuclear norm minimization with application to system identification , 2010, Proceedings of the 2010 American Control Conference.

[25]  Lennart Ljung,et al.  Regularized system identification using orthonormal basis functions , 2015, 2015 European Control Conference (ECC).

[26]  S. Frick,et al.  Compressed Sensing , 2014, Computer Vision, A Reference Guide.

[27]  Lennart Ljung,et al.  Regularization strategies for nonparametric system identification , 2013, 52nd IEEE Conference on Decision and Control.

[28]  Lennart Ljung,et al.  On the design of multiple kernels for nonparametric linear system identification , 2014, 53rd IEEE Conference on Decision and Control.

[29]  Alessandro Chiuso,et al.  Regularized estimation of sums of exponentials in spaces generated by stable spline kernels , 2010, Proceedings of the 2010 American Control Conference.

[30]  Stephen P. Boyd,et al.  Graph Implementations for Nonsmooth Convex Programs , 2008, Recent Advances in Learning and Control.

[31]  Giuseppe De Nicolao,et al.  A new kernel-based approach for linear system identification , 2010, Autom..

[32]  Bo Wahlberg,et al.  A sparse estimation technique for general model structures , 2013, 2013 European Control Conference (ECC).

[33]  Trevor Hastie,et al.  Regularization Paths for Generalized Linear Models via Coordinate Descent. , 2010, Journal of statistical software.

[34]  J. F. Kreider.  Prediction Hourly Building Energy Use : The Great Energy Predictor Shootout - Overview and Discussion of Results , 1994 .

[35]  Roy S. Smith,et al.  Frequency Domain Subspace Identification Using Nuclear Norm Minimization and Hankel Matrix Realizations , 2014, IEEE Transactions on Automatic Control.

[36]  Alessandro Chiuso,et al.  Tuning complexity in kernel-based linear system identification: The robustness of the marginal likelihood estimator , 2014, 2014 European Control Conference (ECC).

[37]  Marko Bacic,et al.  Model predictive control , 2003 .

[38]  H. Akaike.  A new look at the statistical model identification , 1974 .

[39]  David J. C. MacKay,et al.  Bayesian Interpolation , 1992, Neural Computation.

[40]  D. Donoho.  For most large underdetermined systems of linear equations the minimal 𝓁1‐norm solution is also the sparsest solution , 2006 .

[41]  Lennart Ljung,et al.  System Identification Via Sparse Multiple Kernel-Based Regularization Using Sequential Convex Optimization Techniques , 2014, IEEE Transactions on Automatic Control.

[42]  R. Tibshirani.  Regression Shrinkage and Selection via the Lasso , 1996 .

[43]  Dennis V. Lindley,et al.  Empirical Bayes Methods , 1974 .

[44]  Philip Haves,et al.  Model predictive control for the operation of building cooling systems , 2010, Proceedings of the 2010 American Control Conference.

[45]  Johan A. K. Suykens,et al.  Convex Estimation of Cointegrated VAR Models by a Nuclear Norm Penalty , 2012 .

[46]  Alessandro Chiuso,et al.  Prediction error identification of linear systems: A nonparametric Gaussian regression approach , 2011, Autom..

[47]  Jason Weston,et al.  Large-scale kernel machines , 2007 .

[48]  N. Aronszajn.  Theory of Reproducing Kernels. , 1950 .

[49]  H. Hjalmarsson,et al.  Identification of Box-Jenkins Models Using Structured ARX Models and Nuclear Norm Relaxation , 2012 .

[50]  C. Stein,et al.  Estimation with Quadratic Loss , 1992 .

[51]  Manfred Morari,et al.  System identification via nuclear norm regularization for simulated moving bed processes from incomplete data sets , 2009, Proceedings of the 48h IEEE Conference on Decision and Control (CDC) held jointly with 2009 28th Chinese Control Conference.

[52]  David Q. Mayne,et al.  Model predictive control: Recent developments and future promise , 2014, Autom..

[53]  Eric R. Ziegel,et al.  The Elements of Statistical Learning , 2003, Technometrics.

[54]  G. Wahba.  Spline models for observational data , 1990 .

[55]  B. Wahlberg.  System identification using Laguerre models , 1991 .

[56]  Alessandro Chiuso,et al.  Convex vs non-convex estimators for regression and sparse estimation: the mean squared error properties of ARD and GLasso , 2014, J. Mach. Learn. Res..

[57]  Lennart Ljung,et al.  Kernel methods in system identification, machine learning and function estimation: A survey , 2014, Autom..

[58]  Alessandro Chiuso,et al.  Bayesian and regularization approaches to multivariable linear system identification: The role of rank penalties , 2014, 53rd IEEE Conference on Decision and Control.

[59]  Pablo A. Parrilo,et al.  The Convex Geometry of Linear Inverse Problems , 2010, Foundations of Computational Mathematics.