SORN: A Self-Organizing Recurrent Neural Network

Understanding the dynamics of recurrent neural networks is crucial for explaining how the brain processes information. In the neocortex, a range of different plasticity mechanisms are shaping recurrent networks into effective information processing circuits that learn appropriate representations for time-varying sensory stimuli. However, it has been difficult to mimic these abilities in artificial neural network models. Here we introduce SORN, a self-organizing recurrent network. It combines three distinct forms of local plasticity to learn spatio-temporal patterns in its input while maintaining its dynamics in a healthy regime suitable for learning. The SORN learns to encode information in the form of trajectories through its high-dimensional state space reminiscent of recent biological findings on cortical coding. All three forms of plasticity are shown to be essential for the network's success.

[1]  Byron M. Yu,et al.  Techniques for extracting single-trial activity patterns from large-scale neural recordings , 2007, Current Opinion in Neurobiology.

[2]  G. Laurent,et al.  Transient Dynamics versus Fixed Points in Odor Representations by Locust Antennal Lobe Projection Neurons , 2005, Neuron.

[3]  Y. Dan,et al.  Stimulus Timing-Dependent Plasticity in Cortical Processing of Orientation , 2001, Neuron.

[4]  H. Markram,et al.  Regulation of Synaptic Efficacy by Coincidence of Postsynaptic APs and EPSPs , 1997, Science.

[5]  Thomas Hofmann,et al.  Temporal dynamics of information content carried by neurons in the primary visual cortex , 2007 .

[6]  Henry Markram,et al.  Real-Time Computing Without Stable States: A New Framework for Neural Computation Based on Perturbations , 2002, Neural Computation.

[7]  J J Hopfield,et al.  Neural networks and physical systems with emergent collective computational abilities. , 1982, Proceedings of the National Academy of Sciences of the United States of America.

[8]  Danko Nikolic,et al.  Temporal dynamics of information content carried by neurons in the primary visual cortex , 2006, NIPS.

[9]  José Carlos Príncipe,et al.  Liquid state machines and cultured cortical networks: The separation property , 2009, Biosyst..

[10]  D Zipser,et al.  Learning the hidden structure of speech. , 1988, The Journal of the Acoustical Society of America.

[11]  Niraj S. Desai,et al.  Activity-dependent scaling of quantal amplitude in neocortical neurons , 1998, Nature.

[12]  Gordon Pipa,et al.  2007 Special Issue: Fading memory and time series prediction in recurrent networks with different forms of plasticity , 2007 .

[13]  Jeffrey L. Elman,et al.  Finding Structure in Time , 1990, Cogn. Sci..

[14]  Gilles Laurent,et al.  Transient Dynamics for Neural Processing , 2008, Science.

[15]  C. Schreiner,et al.  Sequence sensitivity of neurons in cat primary auditory cortex. , 2000, Cerebral cortex.

[16]  Gustavo Deco,et al.  Computational significance of transient dynamics in cortical networks , 2007, The European journal of neuroscience.

[17]  Edward L. Bartlett,et al.  Long-lasting modulation by stimulus context in primate auditory cortex. , 2005, Journal of neurophysiology.

[18]  Jochen J. Steil,et al.  Improving reservoirs using intrinsic plasticity , 2008, Neurocomputing.

[19]  V. Jayaraman,et al.  Encoding and Decoding of Overlapping Odor Sequences , 2006, Neuron.

[20]  Feng Qi Han,et al.  Rapid learning in cortical coding of visual scenes , 2007, Nature Neuroscience.

[21]  Herbert Jaeger,et al.  The''echo state''approach to analysing and training recurrent neural networks , 2001 .

[22]  Niraj S. Desai,et al.  Plasticity in the intrinsic excitability of cortical pyramidal neurons , 1999, Nature Neuroscience.

[23]  G. Bi,et al.  Synaptic Modifications in Cultured Hippocampal Neurons: Dependence on Spike Timing, Synaptic Strength, and Postsynaptic Cell Type , 1998, The Journal of Neuroscience.

[24]  Nils Bertschinger,et al.  Real-Time Computation at the Edge of Chaos in Recurrent Neural Networks , 2004, Neural Computation.

[25]  Vicenç Gómez,et al.  Self-organization using synaptic plasticity , 2008, NIPS.

[26]  Jochen J. Steil,et al.  Online reservoir adaptation by intrinsic plasticity for backpropagation-decorrelation and echo state learning , 2007, Neural Networks.

[27]  Y. Dan,et al.  Temporal Specificity in the Cortical Plasticity of Visual Space Representation , 2002, Science.

[28]  D. Linden,et al.  The other side of the engram: experience-driven changes in neuronal intrinsic excitability , 2003, Nature Reviews Neuroscience.