Deep Boltzmann machine for nonlinear system modelling

Deep Boltzmann machine (DBM) has been successfully applied in classification, regression and time series modeling. For nonlinear system modelling, DBM should also have many advantages over the other neural networks, such as input features extraction and noise tolerance. In this paper, we use DBM to model nonlinear systems by calculating the probability distributions of the input and output. Two novel weight updating algorithms are proposed to obtain these distributions. We use binary encoding and conditional probability transformation methods. The proposed methods are validated with two benchmark nonlinear systems.

[1]  George Cybenko,et al.  Approximation by superpositions of a sigmoidal function , 1992, Math. Control. Signals Syst..

[2]  Xizhao Wang,et al.  Non-iterative approaches in training feed-forward neural networks and their applications , 2018, Soft Computing.

[3]  Yoshua Bengio,et al.  Random Search for Hyper-Parameter Optimization , 2012, J. Mach. Learn. Res..

[4]  Hak-Keung Lam,et al.  Tuning of the structure and parameters of a neural network using an improved genetic algorithm , 2003, IEEE Trans. Neural Networks.

[5]  Gwilym M. Jenkins,et al.  Time series analysis, forecasting and control , 1971 .

[6]  Xiaoou Li,et al.  Nonlinear system identification using deep learning and randomized algorithms , 2015, 2015 IEEE International Conference on Information and Automation.

[7]  Yoshua Bengio,et al.  Justifying and Generalizing Contrastive Divergence , 2009, Neural Computation.

[8]  Geoffrey E. Hinton,et al.  A Learning Algorithm for Boltzmann Machines , 1985, Cogn. Sci..

[9]  Sushmita Mitra,et al.  Neuro-fuzzy rule generation: survey in soft computing framework , 2000, IEEE Trans. Neural Networks Learn. Syst..

[10]  Juan Pardo,et al.  Time-Series Forecasting of Indoor Temperature Using Pre-trained Deep Neural Networks , 2013, ICANN.

[11]  Peter L. Bartlett,et al.  For Valid Generalization the Size of the Weights is More Important than the Size of the Network , 1996, NIPS.

[12]  Wen Yu,et al.  Randomized algorithms for nonlinear system identification with deep learning modification , 2016, Inf. Sci..

[13]  Pascal Vincent,et al.  The Difficulty of Training Deep Architectures and the Effect of Unsupervised Pre-Training , 2009, AISTATS.

[14]  Jyh-Shing Roger Jang,et al.  ANFIS: adaptive-network-based fuzzy inference system , 1993, IEEE Trans. Syst. Man Cybern..

[15]  Xiaoou Li,et al.  Nonlinear system modeling with deep neural networks and autoencoders algorithm , 2016, 2016 IEEE International Conference on Systems, Man, and Cybernetics (SMC).

[16]  Amy Loutfi,et al.  A review of unsupervised feature learning and deep learning for time-series modeling , 2014, Pattern Recognit. Lett..

[17]  Kumpati S. Narendra,et al.  Identification and control of dynamical systems using neural networks , 1990, IEEE Trans. Neural Networks.

[18]  Le Zhang,et al.  Ensemble deep learning for regression and time series forecasting , 2014, 2014 IEEE Symposium on Computational Intelligence in Ensemble Learning (CIEL).

[19]  Yoshua Bengio,et al.  Gradient-based learning applied to document recognition , 1998, Proc. IEEE.

[20]  Frank L. Lewis,et al.  Identification of nonlinear dynamical systems using multilayered neural networks , 1996, Autom..

[21]  Geoffrey E. Hinton,et al.  Deep Learning , 2015, Nature.

[22]  Yee Whye Teh,et al.  A Fast Learning Algorithm for Deep Belief Nets , 2006, Neural Computation.

[23]  George E. P. Box,et al.  Time Series Analysis: Box/Time Series Analysis , 2008 .

[24]  Stephen A. Billings,et al.  International Journal of Control , 2004 .

[25]  Ran Wang,et al.  Noniterative Deep Learning: Incorporating Restricted Boltzmann Machine Into Multilayer Random Weight Neural Networks , 2019, IEEE Transactions on Systems, Man, and Cybernetics: Systems.

[26]  Chee Kheong Siew,et al.  Universal Approximation using Incremental Constructive Feedforward Networks with Random Hidden Nodes , 2006, IEEE Transactions on Neural Networks.

[27]  Wen Yu,et al.  Restricted Boltzmann Machine for Nonlinear System Modeling , 2015, 2015 IEEE 14th International Conference on Machine Learning and Applications (ICMLA).

[28]  Geoffrey E. Hinton,et al.  An Efficient Learning Procedure for Deep Boltzmann Machines , 2012, Neural Computation.

[29]  Yoshua Bengio,et al.  Greedy Layer-Wise Training of Deep Networks , 2006, NIPS.

[30]  Ian Osband,et al.  Deep Learning for Time Series Modeling CS 229 Final Project Report , 2012 .

[31]  Yoshua Bengio,et al.  Why Does Unsupervised Pre-training Help Deep Learning? , 2010, AISTATS.

[32]  Geoffrey E. Hinton,et al.  Reducing the Dimensionality of Data with Neural Networks , 2006, Science.

[33]  Xizhao Wang,et al.  A deep stochastic weight assignment network and its application to chess playing , 2018, J. Parallel Distributed Comput..

[34]  Geoffrey E. Hinton,et al.  Deep Boltzmann Machines , 2009, AISTATS.

[35]  Nicolas Le Roux,et al.  Representational Power of Restricted Boltzmann Machines and Deep Belief Networks , 2008, Neural Computation.

[36]  Léon Personnaz,et al.  Neural-network construction and selection in nonlinear modeling , 2003, IEEE Trans. Neural Networks.

[37]  Yoshua Bengio,et al.  Classification using discriminative restricted Boltzmann machines , 2008, ICML '08.