Attractor Dynamics in Networks with Learning Rules Inferred from In Vivo Data

The attractor neural network scenario is a popular scenario for memory storage in the association cortex, but there is still a large gap between models based on this scenario and experimental data. We study a recurrent network model in which both learning rules and distribution of stored patterns are inferred from distributions of visual responses for novel and familiar images in the inferior temporal cortex (ITC). Unlike classical attractor neural network models, our model exhibits graded activity in retrieval states, with distributions of firing rates that are close to lognormal. Inferred learning rules are close to maximizing the number of stored patterns within a family of unsupervised Hebbian learning rules, suggesting that learning rules in ITC are optimized to store a large number of attractor states. Finally, we show that there exist two types of retrieval states: one in which firing rates are constant in time and another in which firing rates fluctuate chaotically.

[1]  C. Petersen,et al.  The Excitatory Neuronal Network of the C2 Barrel Column in Mouse Primary Somatosensory Cortex , 2009, Neuron.

[2]  Misha Tsodyks,et al.  Chaos in Highly Diluted Neural Networks , 1991 .

[3]  Nicolas Brunel,et al.  Is cortical connectivity optimized for storing information? , 2016, Nature Neuroscience.

[4]  David L. Sheinberg,et al.  Effects of Long-Term Visual Experience on Responses of Distinct Classes of Single Units in Inferior Temporal Cortex , 2012, Neuron.

[5]  G. E. Alexander,et al.  Neuron Activity Related to Short-Term Memory , 1971, Science.

[6]  S. Grossberg On learning, information, lateral inhibition, and transmitters , 1969 .

[7]  Henning Sprekeler,et al.  Inhibitory Plasticity Balances Excitation and Inhibition in Sensory Pathways and Memory Networks , 2011, Science.

[8]  Marc Mézard,et al.  Solvable models of working memories , 1986 .

[9]  J. Knott The organization of behavior: A neuropsychological theory , 1951 .

[10]  J. Fuster,et al.  Inferotemporal neurons distinguish and retain behaviorally relevant features of visual stimuli. , 1981, Science.

[11]  Y. Miyashita Neuronal correlate of visual associative long-term memory in the primate temporal cortex , 1988, Nature.

[12]  Keiji Tanaka,et al.  Effects of shape-discrimination training on the selectivity of inferotemporal cells in adult monkeys. , 1998, Journal of neurophysiology.

[13]  H. Sompolinsky,et al.  Transition to chaos in random neuronal networks , 2015, 1508.06486.

[14]  Carson C. Chow,et al.  Variability in neuronal activity in primate cortex during working memory tasks , 2007, Neuroscience.

[15]  R. Romo,et al.  Neuronal Population Coding of Parametric Working Memory , 2010, The Journal of Neuroscience.

[16]  R. Romo,et al.  Neuronal correlates of parametric working memory in the prefrontal cortex , 1999, Nature.

[17]  Daniel J. Amit,et al.  Learning in Neural Networks with Material Synapses , 1994, Neural Computation.

[18]  N. Brunel,et al.  Irregular persistent activity induced by synaptic excitatory feedback , 2007, BMC Neuroscience.

[19]  T. Hromádka,et al.  Sparse Representation of Sounds in the Unanesthetized Auditory Cortex , 2008, PLoS biology.

[20]  E. Miller,et al.  Experience-dependent sharpening of visual shape selectivity in inferior temporal cortex. , 2005, Cerebral cortex.

[21]  Treves,et al.  Graded-response neurons and information encodings in autoassociative memories. , 1990, Physical review. A, Atomic, molecular, and optical physics.

[22]  Keiji Tanaka,et al.  Statistics of visual responses in primate inferotemporal cortex to object stimuli. , 2011, Journal of neurophysiology.

[23]  Xiao-Jing Wang Synaptic reverberation underlying mnemonic persistent activity , 2001, Trends in Neurosciences.

[24]  Wulfram Gerstner,et al.  Associative memory in a network of ‘spiking’ neurons , 1992 .

[25]  Christos Constantinidis,et al.  Stable population coding for working memory coexists with heterogeneous neural dynamics in prefrontal cortex , 2016, Proceedings of the National Academy of Sciences.

[26]  W. Senn,et al.  Neocortical pyramidal cells respond as integrate-and-fire neurons to in vivo-like input currents. , 2003, Journal of neurophysiology.

[27]  A. Litwin-Kumar,et al.  Formation and maintenance of neuronal assemblies through synaptic plasticity , 2014, Nature Communications.

[28]  Shun-ichi Amari,et al.  Learning Patterns and Pattern Sequences by Self-Organizing Nets of Threshold Elements , 1972, IEEE Transactions on Computers.

[29]  R. Desimone,et al.  The representation of stimulus familiarity in anterior inferior temporal cortex. , 1993, Journal of neurophysiology.

[30]  David Hansel,et al.  Asynchronous Rate Chaos in Spiking Neuronal Circuits , 2015 .

[31]  J. Fuster,et al.  Unit activity in monkey parietal cortex related to haptic perception and temporary memory , 2004, Experimental Brain Research.

[32]  P. J. Sjöström,et al.  Rate, Timing, and Cooperativity Jointly Determine Cortical Synaptic Plasticity , 2001, Neuron.

[33]  L. Abbott,et al.  From fixed points to chaos: Three models of delayed discrimination , 2013, Progress in Neurobiology.

[34]  N. Logothetis,et al.  Shape representation in the inferior temporal cortex of monkeys , 1995, Current Biology.

[35]  D. Amit,et al.  Statistical mechanics of neural networks near saturation , 1987 .

[36]  K. Miller,et al.  Modeling the Dynamic Interaction of Hebbian and Homeostatic Plasticity , 2014, Neuron.

[37]  Walter J. Freeman,et al.  The Hebbian paradigm reintegrated: Local reverberations as internal representations , 1995, Behavioral and Brain Sciences.

[38]  Alessandro Treves,et al.  Threshold-linear formal neurons in auto-associative nets , 1990 .

[39]  Daniel J. Amit,et al.  Modeling brain function: the world of attractor neural networks, 1st Edition , 1989 .

[40]  Xiao-Jing Wang,et al.  Erratum to: Effects of neuromodulation in a cortical network model of object working memory dominated by recurrent inhibition , 2014, Journal of Computational Neuroscience.

[41]  D. Ferster,et al.  The contribution of noise to contrast invariance of orientation tuning in cat visual cortex. , 2000, Science.

[42]  Naoshige Uchida,et al.  Demixed principal component analysis of neural population data , 2014, eLife.

[43]  G. Buzsáki,et al.  The log-dynamic brain: how skewed distributions affect network operations , 2014, Nature Reviews Neuroscience.

[44]  A. Lansner Associative memory models: from the cell-assembly theory to biophysically detailed cortex simulations , 2009, Trends in Neurosciences.

[45]  Sommers,et al.  Chaos in random neural networks. , 1988, Physical review letters.

[46]  Kree,et al.  Continuous-time dynamics of asymmetrically diluted neural networks. , 1987, Physical review. A, General physics.

[47]  Wenjun Yan,et al.  Medial prefrontal activity during delay period contributes to learning of a working memory task , 2014, Science.

[48]  Dmitri B. Chklovskii,et al.  Neuronal Circuits Underlying Persistent Representations Despite Time Varying Activity , 2012, Current Biology.

[49]  Sandro Romani,et al.  Discrete attractor dynamics underlying selective persistent activity in frontal cortex , 2017, bioRxiv.

[50]  P. Holmes,et al.  Nonlinear Oscillations, Dynamical Systems, and Bifurcations of Vector Fields , 1983, Applied Mathematical Sciences.

[51]  P. Goldman-Rakic,et al.  Mnemonic coding of visual space in the monkey's dorsolateral prefrontal cortex. , 1989, Journal of neurophysiology.

[52]  Ulises Pereira,et al.  Attractor dynamics in networks with learning rules inferred from in vivo data , 2017 .

[53]  Zengcai V. Guo,et al.  Flow of Cortical Activity Underlying a Tactile Decision in Mice , 2014, Neuron.

[54]  E. Bienenstock,et al.  Theory for the development of neuron selectivity: orientation specificity and binocular interaction in visual cortex , 1982, The Journal of neuroscience : the official journal of the Society for Neuroscience.

[55]  P. Goldman-Rakic,et al.  Temporally irregular mnemonic persistent activity in prefrontal neurons of monkeys during a delayed response task. , 2003, Journal of neurophysiology.

[56]  H. Markram,et al.  Physiology and anatomy of synaptic connections between thick tufted pyramidal neurones in the developing rat neocortex. , 1997, The Journal of physiology.

[57]  R. Romo,et al.  Timing and neural encoding of somatosensory parametric working memory in macaque prefrontal cortex. , 2003, Cerebral cortex.

[58]  Srdjan Ostojic,et al.  Two types of asynchronous activity in networks of excitatory and inhibitory spiking neurons , 2014, Nature Neuroscience.

[59]  A. Treves Mean-field analysis of neuronal spike dynamics , 1993 .

[60]  Henry Markram,et al.  An Algorithm for Modifying Neurotransmitter Release Probability Based on Pre- and Postsynaptic Spike Timing , 2001, Neural Computation.

[61]  Anders Lansner,et al.  Bistable, Irregular Firing and Population Oscillations in a Modular Attractor Memory Network , 2010, PLoS Comput. Biol..

[62]  M. Scanziani,et al.  Distinct recurrent versus afferent dynamics in cortical visual processing , 2015, Nature Neuroscience.

[63]  E. Gardner,et al.  Maximum Storage Capacity in Neural Networks , 1987 .

[64]  David J. Freedman,et al.  Inferring learning rules from distribution of firing rates in cortical neurons , 2015, Nature Neuroscience.

[65]  A. Artola,et al.  Synaptic Activity Modulates the Induction of Bidirectional Synaptic Changes in Adult Mouse Hippocampus , 2000, The Journal of Neuroscience.

[66]  G. Parisi A memory which forgets , 1986 .

[67]  M. Tsodyks,et al.  The Enhanced Storage Capacity in Neural Networks with Low Activity Level , 1988 .

[68]  T. Sejnowski,et al.  Storing covariance with nonlinearly interacting neurons , 1977, Journal of mathematical biology.

[69]  E. Gardner,et al.  An Exactly Solvable Asymmetric Neural Network Model , 1987 .

[70]  D. Hansel,et al.  On the Distribution of Firing Rates in Networks of Cortical Neurons , 2011, The Journal of Neuroscience.

[71]  F. Attneave,et al.  The Organization of Behavior: A Neuropsychological Theory , 1949 .

[72]  J J Hopfield,et al.  Neurons with graded response have collective computational properties like those of two-state neurons. , 1984, Proceedings of the National Academy of Sciences of the United States of America.

[73]  Gianluigi Mongillo,et al.  Bistability and spatiotemporal irregularity in neuronal networks with nonlinear synaptic transmission. , 2012, Physical review letters.

[74]  M. Tsodyks Associative Memory in Asymmetric Diluted Network with Low Level of Activity , 1988 .

[75]  D. McCormick,et al.  Comparative electrophysiology of pyramidal and sparsely spiny stellate neurons of the neocortex. , 1985, Journal of neurophysiology.

[76]  M. Tsodyks,et al.  Synaptic Theory of Working Memory , 2008, Science.

[77]  Everton J. Agnes,et al.  Diverse synaptic plasticity mechanisms orchestrated to form and retrieve memories in spiking neural networks , 2015, Nature Communications.

[78]  P. Goldman-Rakic Cellular basis of working memory , 1995, Neuron.

[79]  A. Destexhe Kinetic Models of Synaptic Transmission , 1997 .

[80]  D. Amit,et al.  Model of global spontaneous activity and local structured activity during delay periods in the cerebral cortex. , 1997, Cerebral cortex.

[81]  H. Markram,et al.  The neocortical microcircuit as a tabula rasa. , 2005, Proceedings of the National Academy of Sciences of the United States of America.

[82]  Nicolas Brunel,et al.  NETWORK MODELS OF MEMORY , 2004 .

[83]  John J. Hopfield,et al.  Neural networks and physical systems with emergent collective computational abilities , 1999 .

[84]  K. Nakamura,et al.  Mnemonic firing of neurons in the monkey temporal pole during a visual recognition memory task. , 1995, Journal of neurophysiology.

[85]  M. Tsodyks,et al.  Working models of working memory , 2014, Current Opinion in Neurobiology.

[86]  W. Gerstner,et al.  Triplets of Spikes in a Model of Spike Timing-Dependent Plasticity , 2006, The Journal of Neuroscience.

[87]  Guillaume Hennequin,et al.  Analog Memories in a Balanced Rate-Based Network of E-I Neurons , 2014, NIPS.