Deep Convolutional Networks in System Identification

Recent developments within deep learning are relevant for nonlinear system identification problems. In this paper, we establish connections between the deep learning and the system identification communities. It has recently been shown that convolutional architectures are at least as capable as recurrent architectures when it comes to sequence modeling tasks. Inspired by these results we explore the explicit relationships between the recently proposed temporal convolutional network (TCN) and two classic system identification model structures; Volterra series and block-oriented models. We end the paper with an experimental study where we provide results on two real-world problems, the well-known Silverbox dataset and a newer dataset originating from ground vibration experiments on an F-16 fighter aircraft.

[1]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[2]  Brett Ninness,et al.  Generalised Hammerstein–Wiener system estimation and a benchmark application , 2012 .

[3]  Stephen A. Billings,et al.  Non-linear system identification using neural networks , 1990 .

[4]  Jian Sun,et al.  Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[5]  José de Jesús Rubio,et al.  Stable Kalman filter and neural network for the chaotic systems identification , 2017, J. Frankl. Inst..

[6]  W S McCulloch,et al.  A logical calculus of the ideas immanent in nervous activity , 1990, The Philosophy of Artificial Intelligence.

[7]  Yann LeCun,et al.  Regularization of Neural Networks using DropConnect , 2013, ICML.

[8]  Leon O. Chua,et al.  Fading memory and the problem of approximating nonlinear operators with volterra series , 1985 .

[9]  Johan Schoukens,et al.  On Direct Identification of Physical Parameters in Non-Linear Models , 2004 .

[10]  Yann Dauphin,et al.  A Convolutional Encoder Model for Neural Machine Translation , 2016, ACL.

[11]  G. Palm,et al.  On representation and approximation of nonlinear systems , 1979, Biological Cybernetics.

[12]  Tim Salimans,et al.  Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks , 2016, NIPS.

[13]  Lennart Ljung,et al.  Nonlinear black-box modeling in system identification: a unified overview , 1995, Autom..

[14]  Andrea Garulli,et al.  Identification of Piecewise Affine LFR Models of Interconnected Systems , 2011, IEEE Transactions on Control Systems Technology.

[15]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[16]  Rik Pintelon,et al.  System Identification: A Frequency Domain Approach , 2012 .

[17]  Yann Dauphin,et al.  Language Modeling with Gated Convolutional Networks , 2016, ICML.

[18]  Dumitru Erhan,et al.  Going deeper with convolutions , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[19]  G. Palm.  On representation and approximation of nonlinear systems , 1978, Biological Cybernetics.

[20]  Smriti Srivastava,et al.  Comparative study of neural networks for dynamic nonlinear systems identification , 2019, Soft Comput..

[21]  Razvan Pascanu,et al.  Learned-Norm Pooling for Deep Feedforward and Recurrent Neural Networks , 2013, ECML/PKDD.

[22]  Sergey Ioffe,et al.  Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift , 2015, ICML.

[23]  Andrew L. Maas.  Rectifier Nonlinearities Improve Neural Network Acoustic Models , 2013 .

[24]  Ning Qian,et al.  On the momentum term in gradient descent learning algorithms , 1999, Neural Networks.

[25]  Lennart Ljung,et al.  Estimation of grey box and black box models for non-linear circuit data , 2004 .

[26]  Yoshua Bengio,et al.  Maxout Networks , 2013, ICML.

[27]  Alex Graves,et al.  Neural Machine Translation in Linear Time , 2016, ArXiv.

[28]  Johan Schoukens,et al.  Fast identification of systems with nonlinear feedback , 2004 .

[29]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[30]  Yoshua Bengio,et al.  Gradient-based learning applied to document recognition , 1998, Proc. IEEE.

[31]  Kurt Hornik,et al.  Multilayer feedforward networks are universal approximators , 1989, Neural Networks.

[32]  Koen Tiels,et al.  Identification of block-oriented nonlinear systems starting from linear approximations: A survey , 2016, Autom..

[33]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[34]  Luca Antiga,et al.  Automatic differentiation in PyTorch , 2017 .

[35]  Trevor Darrell,et al.  Fully Convolutional Networks for Semantic Segmentation , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[36]  Johan Schoukens,et al.  Three free data sets for development and benchmarking in nonlinear system identification , 2013, 2013 European Control Conference (ECC).

[37]  E. Bai,et al.  Block Oriented Nonlinear System Identification , 2010 .

[38]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[39]  Nitish Srivastava,et al.  Dropout: a simple way to prevent neural networks from overfitting , 2014, J. Mach. Learn. Res..

[40]  Johan Schoukens,et al.  Identification of systems with localised nonlinearity: From state-space to block-structured models , 2013, Autom..

[41]  Richard D. Braatz,et al.  On the "Identification and control of dynamical systems using neural networks" , 1997, IEEE Trans. Neural Networks.

[42]  Heiga Zen,et al.  WaveNet: A Generative Model for Raw Audio , 2016, SSW.

[43]  Geoffrey E. Hinton,et al.  Phoneme recognition using time-delay neural networks , 1989, IEEE Trans. Acoust. Speech Signal Process..

[44]  Jorge Nocedal,et al.  Optimization Methods for Large-Scale Machine Learning , 2016, SIAM Rev..

[45]  Jeffrey Dean,et al.  Efficient Estimation of Word Representations in Vector Space , 2013, ICLR.

[46]  Johan Schoukens,et al.  Modelling of a slightly nonlinear system: a neural network approach , 2004 .

[47]  Johan Schoukens,et al.  Wiener system identification with generalized orthonormal basis functions , 2014, Autom..

[48]  Yves Rolain,et al.  Parametric identification of parallel Wiener-Hammerstein systems , 2017, Autom..

[49]  Jan Swevers,et al.  Identification of nonlinear systems using Polynomial Nonlinear State Space models , 2010, Autom..

[50]  Tara N. Sainath,et al.  Deep Neural Networks for Acoustic Modeling in Speech Recognition: The Shared Views of Four Research Groups , 2012, IEEE Signal Processing Magazine.

[51]  Ali Farhadi,et al.  You Only Look Once: Unified, Real-Time Object Detection , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[52]  Y. H. Ku,et al.  Book reviewThe Volterra & Wiener theories of nonlinear systems : Martin Schetzen. 531 pages, diagrams, illustr., John Wiley, New York, 1980. Price, $32.50. , 1982 .

[53]  Witold Pedrycz,et al.  Improved learning algorithm for two-layer neural networks for identification of nonlinear systems , 2019, Neurocomputing.

[54]  Tingwen Huang,et al.  Generalized Hybrid Constructive Learning Algorithm for Multioutput RBF Networks , 2017, IEEE Transactions on Cybernetics.

[55]  Johan A. K. Suykens,et al.  Identification of the Silverbox Benchmark Using Nonlinear State-Space Models , 2012 .

[56]  Johan A. K. Suykens,et al.  Kernel based partially linear models and nonlinear identification , 2005, IEEE Transactions on Automatic Control.

[57]  Alberto Bemporad,et al.  Learning Nonlinear State-Space Models Using Deep Autoencoders , 2018, 2018 IEEE Conference on Decision and Control (CDC).

[58]  K. U. Leuven.  A COMPARATIVE STUDY OF LS-SVM’S APPLIED TO THE SILVER BOX IDENTIFICATION PROBLEM , 2004 .

[59]  Geoffrey E. Hinton,et al.  On rectified linear units for speech processing , 2013, 2013 IEEE International Conference on Acoustics, Speech and Signal Processing.

[60]  Jean-Philippe Noël,et al.  F-16 aircraft benchmark based on ground vibration test data , 2017 .

[61]  Gary G. R. Green,et al.  Calculation of the Volterra kernels of non-linear dynamic systems using an artificial neural network , 1994, Biological Cybernetics.

[62]  Vladlen Koltun,et al.  An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling , 2018, ArXiv.

[63]  Mohammad Reza Akbarzadeh-Totonchi,et al.  Extended Fuzzy Logic: Sets and Systems , 2016, IEEE Transactions on Fuzzy Systems.

[64]  Vincent Verdult,et al.  Identification of Local Linear State-Space Models: The Silver-Box Case Study , 2004 .