Boltzmann machines for time-series

We review Boltzmann machines extended for time-series. These models often have recurrent structure, and back propagration through time (BPTT) is used to learn their parameters. The per-step computational complexity of BPTT in online learning, however, grows linearly with respect to the length of preceding time-series (i.e., learning rule is not local in time), which limits the applicability of BPTT in online learning. We then review dynamic Boltzmann machines (DyBMs), whose learning rule is local in time. DyBM's learning rule relates to spike-timing dependent plasticity (STDP), which has been postulated and experimentally confirmed for biological neural networks.

[1]  Javier R. Movellan,et al.  Diffusion Networks, Products of Experts, and Factor Analysis , 2001 .

[2]  Shun-ichi Amari,et al.  Methods of information geometry , 2000 .

[3]  Silvio Savarese,et al.  Structured Recurrent Temporal Restricted Boltzmann Machines , 2014, ICML.

[4]  Jun Zhu,et al.  Polyphonic Music Modelling with LSTM-RTRBM , 2015, ACM Multimedia.

[5]  Takayuki Osogami,et al.  Seven neurons memorizing sequences of alphabetical images via spike-timing dependent plasticity , 2015, Scientific Reports.

[6]  S. Nelson,et al.  Homeostatic plasticity in the developing nervous system , 2004, Nature Reviews Neuroscience.

[7]  Yoshua Bengio,et al.  Modeling Temporal Dependencies in High-Dimensional Sequences: Application to Polyphonic Music Generation and Transcription , 2012, ICML.

[8]  L. Abbott,et al.  Synaptic plasticity: taming the beast , 2000, Nature Neuroscience.

[9]  Harald Haas,et al.  Harnessing Nonlinearity: Predicting Chaotic Systems and Saving Energy in Wireless Communication , 2004, Science.

[10]  Takayuki Osogami,et al.  Nonlinear Dynamic Boltzmann Machines for Time-Series Prediction , 2017, AAAI.

[11]  Yoshua Bengio,et al.  An objective function for STDP , 2015, ArXiv.

[12]  Gordon Pipa,et al.  SORN: A Self-Organizing Recurrent Neural Network , 2009, Front. Comput. Neurosci..

[13]  Geoffrey E. Hinton,et al.  Learning Multilevel Distributed Representations for High-Dimensional Sequences , 2007, AISTATS.

[14]  Helmut Ltkepohl,et al.  New Introduction to Multiple Time Series Analysis , 2007 .

[15]  Benjamin Schrauwen,et al.  A hierarchy of recurrent networks for speech recognition , 2009, NIPS 2009.

[16]  Geoffrey E. Hinton,et al.  Learning internal representations by error propagation , 1986 .

[17]  Yoshua Bengio,et al.  Equilibrium Propagation: Bridging the Gap between Energy-Based Models and Backpropagation , 2016, Front. Comput. Neurosci..

[18]  Hiroshi Kajino.  A Functional Dynamic Boltzmann Machine , 2017, IJCAI.

[19]  Geoffrey E. Hinton,et al.  Spiking Boltzmann Machines , 1999, NIPS.

[20]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[21]  Geoffrey E. Hinton,et al.  Reducing the Dimensionality of Data with Neural Networks , 2006, Science.

[22]  Yoshua Bengio,et al.  STDP as presynaptic activity times rate of change of postsynaptic activity , 2015, 1509.05936.

[23]  Takayuki Osogami,et al.  Learning binary or real-valued time-series via spike-timing dependent plasticity , 2016, ArXiv.

[24]  Geoffrey E. Hinton,et al.  Modeling Human Motion Using Binary Latent Variables , 2006, NIPS.

[25]  Léon Bottou,et al.  On-line learning and stochastic approximations , 1999 .

[26]  Geoffrey E. Hinton,et al.  OPTIMAL PERCEPTUAL INFERENCE , 1983 .

[27]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[28]  Alex Krizhevsky,et al.  Learning Multiple Layers of Features from Tiny Images , 2009 .

[29]  L. Eon Bottou.  Online Learning and Stochastic Approximations , 1998 .

[30]  G. Bi,et al.  Synaptic Modifications in Cultured Hippocampal Neurons: Dependence on Spike Timing, Synaptic Strength, and Postsynaptic Cell Type , 1998, The Journal of Neuroscience.

[31]  Geoffrey E. Hinton,et al.  Factored conditional restricted Boltzmann Machines for modeling motion style , 2009, ICML '09.

[32]  Geoffrey E. Hinton,et al.  The Recurrent Temporal Restricted Boltzmann Machine , 2008, NIPS.

[33]  Hiroshi Kajino,et al.  Bidirectional Learning for Time-series Models with Hidden Units , 2017, ICML.

[34]  Geoffrey E. Hinton,et al.  A Learning Algorithm for Boltzmann Machines , 1985, Cogn. Sci..

[35]  Geoffrey E. Hinton,et al.  Unsupervised Learning of Image Transformations , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[36]  Ning Qian,et al.  On the momentum term in gradient descent learning algorithms , 1999, Neural Networks.

[37]  Yoram Singer,et al.  Adaptive Subgradient Methods for Online Learning and Stochastic Optimization , 2011, J. Mach. Learn. Res..

[38]  Takayuki Osogami,et al.  Boltzmann machines and energy-based models , 2017, ArXiv.

[39]  Takayuki Osogami,et al.  Regularized dynamic Boltzmann machine with Delay Pruning for unsupervised learning of temporal sequences , 2016, 2016 23rd International Conference on Pattern Recognition (ICPR).

[40]  Takayuki Osogami,et al.  Learning the values of the hyperparameters of a dynamic Boltzmann machine , 2017, IBM J. Res. Dev..

[41]  Takayuki Osogami,et al.  Learning dynamic Boltzmann machines with spike-timing dependent plasticity , 2015, ArXiv.

[42]  Geoffrey E. Hinton,et al.  Exponential Family Harmoniums with an Application to Information Retrieval , 2004, NIPS.

[43]  Wolfgang Maass,et al.  Bayesian Computation Emerges in Generic Cortical Microcircuits through Spike-Timing-Dependent Plasticity , 2013, PLoS Comput. Biol..