Attractor dynamics in networks with learning rules inferred from in vivo data

The attractor neural network scenario is a popular scenario for memory storage in association cortex, but there is still a large gap between models based on this scenario and experimental data. We study a recurrent network model in which both learning rules and distribution of stored patterns are inferred from distributions of visual responses for novel and familiar images in inferior temporal cortex (ITC). Unlike classical attractor neural network models, our model exhibits graded activity in retrieval states, with distributions of firing rates that are close to lognormal. Inferred learning rules are close to maximizing the number of stored patterns within a family of unsupervised Hebbian learning rules, suggesting learning rules in ITC are optimized to store a large number of attractor states. Finally, we show that there exists two types of retrieval states: one in which firing rates are constant in time, another in which firing rates fluctuate chaotically.

[1]  G. E. Alexander,et al.  Neuron Activity Related to Short-Term Memory , 1971, Science.

[2]  Marc Mézard,et al.  Solvable models of working memories , 1986 .

[3]  K. Nakamura,et al.  Mnemonic firing of neurons in the monkey temporal pole during a visual recognition memory task. , 1995, Journal of neurophysiology.

[4]  M. Tsodyks,et al.  Working models of working memory , 2014, Current Opinion in Neurobiology.

[5]  J. Knott The organization of behavior: A neuropsychological theory , 1951 .

[6]  Dmitri B. Chklovskii,et al.  Neuronal Circuits Underlying Persistent Representations Despite Time Varying Activity , 2012, Current Biology.

[7]  J. Fuster,et al.  Inferotemporal neurons distinguish and retain behaviorally relevant features of visual stimuli. , 1981, Science.

[8]  Y. Miyashita Neuronal correlate of visual associative long-term memory in the primate temporal cortex , 1988, Nature.

[9]  J J Hopfield,et al.  Neural networks and physical systems with emergent collective computational abilities. , 1982, Proceedings of the National Academy of Sciences of the United States of America.

[10]  Guillaume Hennequin,et al.  Analog Memories in a Balanced Rate-Based Network of E-I Neurons , 2014, NIPS.

[11]  R. Romo,et al.  Neuronal Population Coding of Parametric Working Memory , 2010, The Journal of Neuroscience.

[12]  T. Hromádka,et al.  Sparse Representation of Sounds in the Unanesthetized Auditory Cortex , 2008, PLoS biology.

[13]  Keiji Tanaka,et al.  Statistics of visual responses in primate inferotemporal cortex to object stimuli. , 2011, Journal of neurophysiology.

[14]  Xiao-Jing Wang Synaptic reverberation underlying mnemonic persistent activity , 2001, Trends in Neurosciences.

[15]  Daniel J. Amit,et al.  Modeling brain function: the world of attractor neural networks, 1st Edition , 1989 .

[16]  N. Brunel,et al.  Irregular persistent activity induced by synaptic excitatory feedback , 2007, BMC Neuroscience.

[17]  P. Goldman-Rakic,et al.  Mnemonic coding of visual space in the monkey's dorsolateral prefrontal cortex. , 1989, Journal of neurophysiology.

[18]  Gianluigi Mongillo,et al.  Bistability and spatiotemporal irregularity in neuronal networks with nonlinear synaptic transmission. , 2012, Physical review letters.

[19]  M. Tsodyks Associative Memory in Asymmetric Diluted Network with Low Level of Activity , 1988 .

[20]  David Hansel,et al.  Asynchronous Rate Chaos in Spiking Neuronal Circuits , 2015, bioRxiv.

[21]  R. Romo,et al.  Neuronal correlates of parametric working memory in the prefrontal cortex , 1999, Nature.

[22]  Daniel J. Amit,et al.  Learning in Neural Networks with Material Synapses , 1994, Neural Computation.

[23]  P. Goldman-Rakic,et al.  Temporally irregular mnemonic persistent activity in prefrontal neurons of monkeys during a delayed response task. , 2003, Journal of neurophysiology.

[24]  R. Romo,et al.  Timing and neural encoding of somatosensory parametric working memory in macaque prefrontal cortex. , 2003, Cerebral cortex.

[25]  Srdjan Ostojic,et al.  Two types of asynchronous activity in networks of excitatory and inhibitory spiking neurons , 2014, Nature Neuroscience.

[26]  E. Miller,et al.  Experience-dependent sharpening of visual shape selectivity in inferior temporal cortex. , 2005, Cerebral cortex.

[27]  D. Ferster,et al.  The contribution of noise to contrast invariance of orientation tuning in cat visual cortex. , 2000, Science.

[28]  Alessandro Treves,et al.  Threshold-linear formal neurons in auto-associative nets , 1990 .

[29]  E. Bienenstock,et al.  Theory for the development of neuron selectivity: orientation specificity and binocular interaction in visual cortex , 1982, The Journal of neuroscience : the official journal of the Society for Neuroscience.

[30]  Everton J. Agnes,et al.  Diverse synaptic plasticity mechanisms orchestrated to form and retrieve memories in spiking neural networks , 2015, Nature Communications.

[31]  P. Goldman-Rakic Cellular basis of working memory , 1995, Neuron.

[32]  D. McCormick,et al.  Comparative electrophysiology of pyramidal and sparsely spiny stellate neurons of the neocortex. , 1985, Journal of neurophysiology.

[33]  M. Tsodyks,et al.  Synaptic Theory of Working Memory , 2008, Science.

[34]  J. Cowan,et al.  Excitatory and inhibitory interactions in localized populations of model neurons. , 1972, Biophysical journal.

[35]  Treves,et al.  Graded-response neurons and information encodings in autoassociative memories. , 1990, Physical review. A, Atomic, molecular, and optical physics.

[36]  R. Desimone,et al.  The representation of stimulus familiarity in anterior inferior temporal cortex. , 1993, Journal of neurophysiology.

[37]  G. Parisi A memory which forgets , 1986 .

[38]  W. Senn,et al.  Neocortical pyramidal cells respond as integrate-and-fire neurons to in vivo-like input currents. , 2003, Journal of neurophysiology.

[39]  Nicolas Brunel,et al.  NETWORK MODELS OF MEMORY , 2004 .

[40]  J. Fuster,et al.  Unit activity in monkey parietal cortex related to haptic perception and temporary memory , 2004, Experimental Brain Research.

[41]  P. J. Sjöström,et al.  Rate, Timing, and Cooperativity Jointly Determine Cortical Synaptic Plasticity , 2001, Neuron.

[42]  L. Abbott,et al.  From fixed points to chaos: Three models of delayed discrimination , 2013, Progress in Neurobiology.

[43]  D. Amit The Hebbian paradigm reintegrated: Local reverberations as internal representations , 1995, Behavioral and Brain Sciences.

[44]  A. Destexhe Kinetic Models of Synaptic Transmission , 1997 .

[45]  D. Amit,et al.  Model of global spontaneous activity and local structured activity during delay periods in the cerebral cortex. , 1997, Cerebral cortex.

[46]  Shun-ichi Amari,et al.  Learning Patterns and Pattern Sequences by Self-Organizing Nets of Threshold Elements , 1972, IEEE Transactions on Computers.

[47]  Naoshige Uchida,et al.  Demixed principal component analysis of neural population data , 2014, eLife.

[48]  G. Buzsáki,et al.  The log-dynamic brain: how skewed distributions affect network operations , 2014, Nature Reviews Neuroscience.

[49]  A. Lansner Associative memory models: from the cell-assembly theory to biophysically detailed cortex simulations , 2009, Trends in Neurosciences.

[50]  Sommers,et al.  Chaos in random neural networks. , 1988, Physical review letters.

[51]  Keiji Tanaka,et al.  Effects of shape-discrimination training on the selectivity of inferotemporal cells in adult monkeys. , 1998, Journal of neurophysiology.

[52]  H. Sompolinsky,et al.  Transition to chaos in random neuronal networks , 2015, 1508.06486.

[53]  M. Tsodyks,et al.  The Enhanced Storage Capacity in Neural Networks with Low Activity Level , 1988 .

[54]  T. Sejnowski,et al.  Storing covariance with nonlinearly interacting neurons , 1977, Journal of mathematical biology.

[55]  E. Gardner,et al.  An Exactly Solvable Asymmetric Neural Network Model , 1987 .

[56]  D. Hansel,et al.  On the Distribution of Firing Rates in Networks of Cortical Neurons , 2011, The Journal of Neuroscience.

[57]  J J Hopfield,et al.  Neurons with graded response have collective computational properties like those of two-state neurons. , 1984, Proceedings of the National Academy of Sciences of the United States of America.

[58]  Misha Tsodyks,et al.  Chaos in Highly Diluted Neural Networks , 1991 .

[59]  David L. Sheinberg,et al.  Effects of Long-Term Visual Experience on Responses of Distinct Classes of Single Units in Inferior Temporal Cortex , 2012, Neuron.

[60]  Anders Lansner,et al.  Bistable, Irregular Firing and Population Oscillations in a Modular Attractor Memory Network , 2010, PLoS Comput. Biol..

[61]  Joaquín M. Fuster,et al.  Cortex and Memory: Emergence of a New Paradigm , 2009, Journal of Cognitive Neuroscience.

[62]  E. Gardner,et al.  Maximum Storage Capacity in Neural Networks , 1987 .

[63]  David J. Freedman,et al.  Inferring learning rules from distribution of firing rates in cortical neurons , 2015, Nature Neuroscience.

[64]  A. Artola,et al.  Synaptic Activity Modulates the Induction of Bidirectional Synaptic Changes in Adult Mouse Hippocampus , 2000, The Journal of Neuroscience.

[65]  M. Scanziani,et al.  Distinct recurrent versus afferent dynamics in cortical visual processing , 2015, Nature Neuroscience.

[66]  Nicolas Brunel,et al.  Dynamics and plasticity of stimulus-selective persistent activity in cortical network models. , 2003, Cerebral cortex.

[67]  Carson C. Chow,et al.  Variability in neuronal activity in primate cortex during working memory tasks , 2007, Neuroscience.

[68]  Christos Constantinidis,et al.  Stable population coding for working memory coexists with heterogeneous neural dynamics in prefrontal cortex , 2016, Proceedings of the National Academy of Sciences.

[69]  A. Litwin-Kumar,et al.  Formation and maintenance of neuronal assemblies through synaptic plasticity , 2014, Nature Communications.

[70]  N. Logothetis,et al.  Shape representation in the inferior temporal cortex of monkeys , 1995, Current Biology.

[71]  D. Amit,et al.  Statistical mechanics of neural networks near saturation , 1987 .