A review on generative Boltzmann networks applied to dynamic systems

The modelling of dynamic system is a challenging problem in a large number of applications like prediction, bio-data modelling, computer vision or time-series processing. To face the complexity and the non-linearity of data, new models are regularly proposed through the literature. Among proposed models artificial neural network (ANN) have benefit of a large interest in the scientist community. The use of latent variables to extract and diffuse complex features in multilayer feedforward neural networks provide usually excellent results. In 1982, Hopfield proposes a generative and deterministic neural network to model a physical system. His work leads to the emergence of a large number of generative neural networks: Boltzmann Machine and its extensions. Different applications lead researchers to propose new extensions for the Boltzmann machine to handle dynamic systems, continuous variables or systems with complex features. In parallel, a new model named the Diffusion Network has emerged, also inspired from Hopfield network but with continuous stochastic properties and designed to solve stochastic differential equations. This paper has the objective to review the evolution of the Boltzmann Machine's family with a synthetic and historical vision and their development for dynamic problem. To write this review, we selected articles from journals/conferences and review articles (1/3 are <7 years) quoted in meta sources (Scopus and Web-of-Sciences). Once a clearly research question was asked – How generative networks model dynamic systems ? – we defined our search terms for papers. Note that not all extensions to Boltzmann machines are presented in this paper. Only models related with dynamic applications and most salient models were retained.

[1]  Geoffrey E. Hinton,et al.  An Efficient Learning Procedure for Deep Boltzmann Machines , 2012, Neural Computation.

[2]  Paul Mineiro,et al.  A Monte Carlo EM Approach for Partially Observable Diffusion Processes: Theory and Applications to Neural Networks , 2002, Neural Computation.

[3]  Geoffrey E. Hinton,et al.  Massively Parallel Architectures for AI: NETL, Thistle, and Boltzmann Machines , 1983, AAAI.

[4]  Teik C. Lim,et al.  Evaluation of vehicle interior sound quality using a continuous restricted Boltzmann machine-based DBN , 2017 .

[5]  拓海 杉山,et al.  “Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks”の学習報告 , 2017 .

[6]  Emile H. L. Aarts,et al.  Boltzmann machines as a model for parallel annealing , 1991, Algorithmica.

[7]  Siwei Lyu,et al.  Interpretation and Generalization of Score Matching , 2009, UAI.

[8]  Guillaume-Alexandre Bilodeau,et al.  Discriminative conditional restricted Boltzmann machine for discrete choice and latent variable modelling , 2017, ArXiv.

[9]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[10]  J. Sato,et al.  Using deep belief network modelling to characterize differences in brain morphometry in schizophrenia , 2016, Scientific Reports.

[11]  Geoffrey E. Hinton,et al.  Reducing the Dimensionality of Data with Neural Networks , 2006, Science.

[12]  J J Hopfield,et al.  Neurons with graded response have collective computational properties like those of two-state neurons. , 1984, Proceedings of the National Academy of Sciences of the United States of America.

[13]  Geoffrey E. Hinton,et al.  Phone recognition using Restricted Boltzmann Machines , 2010, 2010 IEEE International Conference on Acoustics, Speech and Signal Processing.

[14]  Razvan Pascanu,et al.  Learning Algorithms for the Classification Restricted Boltzmann Machine , 2012, J. Mach. Learn. Res..

[15]  Yee Whye Teh,et al.  A Fast Learning Algorithm for Deep Belief Nets , 2006, Neural Computation.

[16]  Javier R. Movellan,et al.  Learning Continuous Probability Distributions with Symmetric Diffusion Networks , 1993, Cogn. Sci..

[17]  R. A. Leibler,et al.  On Information and Sufficiency , 1951 .

[18]  Haizhou Li,et al.  Conditional restricted Boltzmann machine for voice conversion , 2013, 2013 IEEE China Summit and International Conference on Signal and Information Processing.

[19]  Kwang Y. Lee,et al.  Economic load dispatch for piecewise quadratic cost function using Hopfield neural network , 1993 .

[20]  Tapani Raiko,et al.  Gaussian-Bernoulli deep Boltzmann machine , 2013, The 2013 International Joint Conference on Neural Networks (IJCNN).

[21]  Laurenz Wiskott,et al.  Gaussian-binary restricted Boltzmann machines for modeling natural image statistics , 2014, PloS one.

[22]  Sean R. Eddy,et al.  Biological Sequence Analysis: Probabilistic Models of Proteins and Nucleic Acids , 1998 .

[23]  Duc Truong Pham,et al.  Training of Elman networks and dynamic system modelling , 1996, Int. J. Syst. Sci..

[24]  J. Hopfield,et al.  Computing with neural circuits: a model. , 1986, Science.

[25]  Silvio Savarese,et al.  Structured Recurrent Temporal Restricted Boltzmann Machines , 2014, ICML.

[26]  Christian P. Robert,et al.  Monte Carlo Statistical Methods , 2005, Springer Texts in Statistics.

[27]  Shifei Ding,et al.  Extreme learning machine and its applications , 2013, Neural Computing and Applications.

[28]  Kratarth Goel,et al.  Modeling temporal dependencies in data using a DBN-LSTM , 2015, 2015 IEEE International Conference on Data Science and Advanced Analytics (DSAA).

[29]  Amy Loutfi,et al.  A review of unsupervised feature learning and deep learning for time-series modeling , 2014, Pattern Recognit. Lett..

[30]  Helmut Lütkepohl.  Vector autoregressive models , 2011 .

[31]  Aggelos K. Katsaggelos,et al.  Image restoration using a modified Hopfield network , 1992, IEEE Trans. Image Process..

[32]  Yoshua Bengio,et al.  Justifying and Generalizing Contrastive Divergence , 2009, Neural Computation.

[33]  Honglak Lee,et al.  Learning hierarchical representations for face verification with convolutional deep belief networks , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[34]  Lawrence R. Rabiner,et al.  A tutorial on hidden Markov models and selected applications in speech recognition , 1989, Proc. IEEE.

[35]  Geoffrey E. Hinton.  Deep belief networks , 2009, Scholarpedia.

[36]  Tetsuya Takiguchi,et al.  Voice Conversion Using RNN Pre-Trained by Recurrent Temporal Restricted Boltzmann Machines , 2015, IEEE/ACM Transactions on Audio, Speech, and Language Processing.

[37]  Kunikazu Kobayashi,et al.  Time series forecasting using a deep belief network with restricted Boltzmann machines , 2014, Neurocomputing.

[38]  C. L. Philip Chen,et al.  Fuzzy Restricted Boltzmann Machine for the Enhancement of Deep Learning , 2015, IEEE Transactions on Fuzzy Systems.

[39]  R. E. Kalman,et al.  A New Approach to Linear Filtering and Prediction Problems , 2002 .

[40]  Ruifan Li,et al.  Deep correspondence restricted Boltzmann machine for cross-modal retrieval , 2015, Neurocomputing.

[41]  Chun-Xia Zhang,et al.  Learning ensemble classifiers via restricted Boltzmann machines , 2014, Pattern Recognit. Lett..

[42]  Shifei Ding,et al.  An overview on Restricted Boltzmann Machines , 2018, Neurocomputing.

[43]  C. Giraud.  Introduction to High-Dimensional Statistics , 2014 .

[44]  Robert X. Gao,et al.  Deep learning and its applications to machine health monitoring , 2019, Mechanical Systems and Signal Processing.

[45]  Aapo Hyvärinen,et al.  Some extensions of score matching , 2007, Comput. Stat. Data Anal..

[46]  Stochastic Differential Equations as Dynamical Systems , 1990 .

[47]  Geoffrey E. Hinton.  Training Products of Experts by Minimizing Contrastive Divergence , 2002, Neural Computation.

[48]  Takehisa Yairi,et al.  A review on the application of deep learning in system health management , 2018, Mechanical Systems and Signal Processing.

[49]  B. Øksendal.  Stochastic Differential Equations , 1985 .

[50]  Lukun Wang.  Three-dimensional convolutional restricted Boltzmann machine for human behavior recognition from RGB-D video , 2018, EURASIP J. Image Video Process..

[51]  Alan F. Murray,et al.  Continuous-valued probabilistic behavior in a VLSI generative model , 2006, IEEE Transactions on Neural Networks.

[52]  Jiawei Xiang,et al.  A data indicator-based deep belief networks to detect multiple faults in axial piston pumps , 2018, Mechanical Systems and Signal Processing.

[53]  J J Hopfield,et al.  Neural networks and physical systems with emergent collective computational abilities. , 1982, Proceedings of the National Academy of Sciences of the United States of America.

[54]  Bi Xiaojun,et al.  Contractive Slab and Spike Convolutional Deep Boltzmann Machine , 2018 .

[55]  Alan F. Murray,et al.  Continuous restricted Boltzmann machine with an implementable training algorithm , 2003 .

[56]  Haidong Shao,et al.  Rolling bearing fault feature learning using improved convolutional deep belief network with compressed sensing , 2018 .

[57]  Miguel Á. Carreira-Perpiñán,et al.  On Contrastive Divergence Learning , 2005, AISTATS.

[58]  Mohammad Norouzi,et al.  Stacks of convolutional Restricted Boltzmann Machines for shift-invariant feature learning , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[59]  Yoshua Bengio,et al.  The Spike-and-Slab RBM and Extensions to Discrete and Sparse Data Distributions , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[60]  L. Younes.  Parametric Inference for imperfectly observed Gibbsian fields , 1989 .

[61]  Xiaodong Li,et al.  A spatial–temporal Hopfield neural network approach for super-resolution land cover mapping with multi-temporal different resolution remotely sensed images , 2014 .