Bidirectional Learning in Recurrent Neural Networks Using Equilibrium Propagation

Neurobiologically-plausible learning algorithms for recurrent neural networks that can perform supervised learning are a neglected area of study. Equilibrium propagation is a recent synthesis of several ideas in biological and artificial neural network research that uses a continuous-time, energy-based neural model with a local learning rule. However, despite dealing with recurrent networks, equilibrium propagation has only been applied to discriminative categorization tasks. This thesis generalizes equilibrium propagation to bidirectional learning with asymmetric weights. Simultaneously learning the discriminative as well as generative transformations for a set of data points and their corresponding category labels, bidirectional equilibrium propagation utilizes recurrence and weight asymmetry to share related but non-identical representations within the network. Experiments on an artificial dataset demonstrate the ability to learn both transformations, as well as the ability for asymmetric-weight networks to generalize their discriminative training to the untrained generative task.

[1]  J. Fritz,et al.  Differential Dynamic Plasticity of A1 Receptive Fields during Multiple Spectral Tasks , 2005, The Journal of Neuroscience.

[2]  T. Sejnowski,et al.  Neurocomputational models of working memory , 2000, Nature Neuroscience.

[3]  Joel Z. Leibo,et al.  How Important Is Weight Symmetry in Backpropagation? , 2015, AAAI.

[4]  Michael I. Jordan,et al.  A more biologically plausible learning rule for neural networks. , 1991, Proceedings of the National Academy of Sciences of the United States of America.

[5]  Rob Fergus,et al.  Visualizing and Understanding Convolutional Networks , 2013, ECCV.

[6]  R. O’Reilly.  Six principles for biologically based computational models of cortical cognition , 1998, Trends in Cognitive Sciences.

[7]  Wolfgang Maass,et al.  Emergence of complex computational structures from chaotic neural networks through reward-modulated Hebbian learning. , 2014, Cerebral cortex.

[8]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[9]  D. G. Stork,et al.  Is backpropagation biologically plausible? , 1989, International 1989 Joint Conference on Neural Networks.

[10]  Kurt Hornik,et al.  Approximation capabilities of multilayer feedforward networks , 1991, Neural Networks.

[11]  Arild Nøkland,et al.  Direct Feedback Alignment Provides Learning in Deep Neural Networks , 2016, NIPS.

[12]  A. Gittis,et al.  Intrinsic and synaptic plasticity in the vestibular system , 2006, Current Opinion in Neurobiology.

[13]  L. Abbott,et al.  Competitive Hebbian learning through spike-timing-dependent synaptic plasticity , 2000, Nature Neuroscience.

[14]  Yoshua Bengio,et al.  Towards Biologically Plausible Deep Learning , 2015, ArXiv.

[15]  Xiaohui Xie,et al.  Equivalence of Backpropagation and Contrastive Hebbian Learning in a Layered Network , 2003, Neural Computation.

[16]  Peter Lakatos,et al.  Dynamics of Active Sensing and perceptual selection , 2010, Current Opinion in Neurobiology.

[17]  Ilya Sutskever,et al.  On the Convergence Properties of Contrastive Divergence , 2010, AISTATS.

[18]  David E. Rumelhart,et al.  Product Units: A Computationally Powerful and Biologically Plausible Extension to Backpropagation Networks , 1989, Neural Computation.

[19]  Yoshua Bengio,et al.  Understanding the difficulty of training deep feedforward neural networks , 2010, AISTATS.

[20]  P. Dean,et al.  The importance of Marr’s three levels of analysis for understanding cerebellar function , 2016 .

[21]  Nicholas Furl,et al.  Structural and effective connectivity reveals potential network-based influences on category-sensitive visual areas , 2015, Front. Hum. Neurosci..

[22]  Yoshua Bengio,et al.  Difference Target Propagation , 2014, ECML/PKDD.

[23]  Miguel Á. Carreira-Perpiñán,et al.  On Contrastive Divergence Learning , 2005, AISTATS.

[24]  Pierre Baldi,et al.  Contrastive Learning and Neural Oscillations , 1991, Neural Computation.

[25]  Karl J. Friston,et al.  Dynamic representations and generative models of brain function , 2001, Brain Research Bulletin.

[26]  Yoshua Bengio,et al.  Equilibrium Propagation: Bridging the Gap between Energy-Based Models and Backpropagation , 2016, Front. Comput. Neurosci..

[27]  Geoffrey E. Hinton,et al.  Learning representations by back-propagating errors , 1986, Nature.

[28]  Colin J. Akerman,et al.  Random synaptic feedback weights support error backpropagation for deep learning , 2016, Nature Communications.

[29]  Jochen Triesch,et al.  A Gradient Rule for the Plasticity of a Neuron's Intrinsic Excitability , 2005, ICANN.

[30]  Andrzej Kasiński,et al.  Comparison of supervised learning methods for spike time coding in spiking neural networks , 2006 .

[31]  Karl J. Friston.  The free-energy principle: a unified brain theory? , 2010, Nature Reviews Neuroscience.

[32]  John J. Hopfield,et al.  Neural networks and physical systems with emergent collective computational abilities , 1999 .

[33]  Razvan Pascanu,et al.  Learning Algorithms for the Classification Restricted Boltzmann Machine , 2012, J. Mach. Learn. Res..

[34]  Jochen J. Steil,et al.  Improving reservoirs using intrinsic plasticity , 2008, Neurocomputing.

[35]  Terrence J. Sejnowski,et al.  Unsupervised Learning , 2018, Encyclopedia of GIS.

[36]  Christof Koch,et al.  The role of single neurons in information processing , 2000, Nature Neuroscience.

[37]  F ROSENBLATT,et al.  The perceptron: a probabilistic model for information storage and organization in the brain. , 1958, Psychological review.

[38]  Razvan Pascanu,et al.  On the difficulty of training recurrent neural networks , 2012, ICML.

[39]  Steven A Prescott,et al.  Spike-Rate Coding and Spike-Time Coding Are Affected Oppositely by Different Adaptation Mechanisms , 2008, The Journal of Neuroscience.

[40]  Herbert Jaeger,et al.  Adaptive Nonlinear System Identification with Echo State Networks , 2002, NIPS.

[41]  J. DiCarlo,et al.  Using goal-driven deep learning models to understand sensory cortex , 2016, Nature Neuroscience.

[42]  P J Webros.  BACKPROPAGATION THROUGH TIME: WHAT IT DOES AND HOW TO DO IT , 1990 .

[43]  W. Gerstner,et al.  Spike-Timing-Dependent Plasticity: A Comprehensive Overview , 2012, Front. Syn. Neurosci..

[44]  Michael I. Jordan,et al.  Deterministic Boltzmann Learning Performs Steepest Descent in Weight-Space , 2001 .

[45]  Jürgen Schmidhuber,et al.  Learning to forget: continual prediction with LSTM , 1999 .