The Neural Marketplace: I. General Formalism and Linear Theory

We propose a mathematical theory for how networks of neurons in the brain self-organize into functional networks, similarly to the self-organization of supply networks in a free-market economy. The theory is inspired by recent experimental results showing how information about changes to output synapses can travel backward along axons to affect a neuron’s inputs. In neuronal development, competition for such “retroaxonal” signals determines which neurons live and which die. We suggest that in adults, an analogous form of competition occurs between neurons, to supply their targets with appropriate information in exchange for a “payment” returned to them backward along the axon. We review experimental evidence suggesting that neurotrophins may constitute such a signaling pathway in the adult brain. We construct a mathematical model, in which a small number of “consumer” neurons receive explicit fast error signals while a larger number of “producer” neurons compete to supply them with information, guided by retroaxonal signals from the consumers and from each other. We define a loss function to measure network performance and define the “worth” of a producer to be the increase in loss that would result if that neuron were to fall silent. We show how slow retroaxonal signals can allow producers to estimate their worth, and how these estimates allow the network to perform a form of parallel search over multiple producer cells. We validate our approximations and demonstrate the proposed learning rule using simulations.

[1]  Nello Cristianini,et al.  Kernel Methods for Pattern Analysis , 2003, ICTAI.

[2]  Tai Sing Lee,et al.  Hierarchical Bayesian inference in the visual cortex. , 2003, Journal of the Optical Society of America. A, Optics, image science, and vision.

[3]  Katherine M. Armstrong,et al.  Selective gating of visual signals by microstimulation of frontal cortex , 2003, Nature.

[4]  D. Paré,et al.  Plastic synaptic networks of the amygdala for the acquisition, expression, and extinction of conditioned fear. , 2010, Physiological reviews.

[5]  E. Izhikevich Solving the distal reward problem through linkage of STDP and dopamine signaling , 2007, BMC Neuroscience.

[6]  E. Kandel,et al.  D1/D5 receptor agonists induce a protein synthesis-dependent late potentiation in the CA1 region of the hippocampus. , 1995, Proceedings of the National Academy of Sciences of the United States of America.

[7]  K. Fox,et al.  The role of nitric oxide in pre-synaptic plasticity and homeostasis , 2013, Front. Cell. Neurosci..

[8]  Francis Crick,et al.  The recent excitement about neural networks , 1989, Nature.

[9]  R. Costa Plastic Corticostriatal Circuits for Action Learning , 2007, Annals of the New York Academy of Sciences.

[10]  HighWire Press,et al.  Philosophical transactions of the Royal Society of London. Series B, Containing papers of a biological character , 1934 .

[11]  C Kentros,et al.  Abolition of long-term stability of new hippocampal place cell maps by NMDA receptor blockade. , 1998, Science.

[12]  Walter Senn,et al.  Spatio-Temporal Credit Assignment in Neuronal Population Learning , 2011, PLoS Comput. Biol..

[13]  D. Mumford On the computational architecture of the neocortex , 2004, Biological Cybernetics.

[14]  Rosalind A. Segal,et al.  Neurotrophins use the Erk5 pathway to mediate a retrograde survival response , 2001, Nature Neuroscience.

[15]  J. O'Keefe,et al.  The hippocampus as a spatial map. Preliminary evidence from unit activity in the freely-moving rat. , 1971, Brain research.

[16]  D Mumford,et al.  On the computational architecture of the neocortex. II. The role of cortico-cortical loops. , 1992, Biological cybernetics.

[17]  Eric R Kandel,et al.  The long-term stability of new hippocampal place fields requires new protein synthesis. , 2004, Proceedings of the National Academy of Sciences of the United States of America.

[18]  J. Lisman,et al.  The Input–Output Transformation of the Hippocampal Granule Cells: From Grid Cells to Place Fields , 2009, The Journal of Neuroscience.

[19]  Anja Vogler,et al.  An Introduction to Multivariate Statistical Analysis , 2004 .

[20]  Kenneth D. Harris,et al.  The Convallis Rule for Unsupervised Learning in Cortical Networks , 2013, PLoS Comput. Biol..

[21]  Geoffrey E. Hinton,et al.  Learning representations by back-propagating errors , 1986, Nature.

[22]  Denise J. Cai,et al.  Synaptic tagging during memory allocation , 2014, Nature Reviews Neuroscience.

[23]  R. Morris,et al.  Making memories last: the synaptic tagging and capture hypothesis , 2010, Nature Reviews Neuroscience.

[24]  Michael Frotscher,et al.  Cholinergic innervation of the rat hippocampus as revealed by choline acetyltransferase immunocytochemistry: A combined light and electron microscopic study , 1985, The Journal of comparative neurology.

[25]  C. D. Gelatt,et al.  Optimization by Simulated Annealing , 1983, Science.

[26]  D. Ginty,et al.  Functions and mechanisms of retrograde neurotrophin signalling , 2005, Nature Reviews Neuroscience.

[27]  A. Borst Seeing smells: imaging olfactory learning in bees , 1999, Nature Neuroscience.

[28]  D. Marr A theory for cerebral neocortex , 1970, Proceedings of the Royal Society of London. Series B. Biological Sciences.

[29]  Henning Sprekeler,et al.  Functional Requirements for Reward-Modulated Spike-Timing-Dependent Plasticity , 2010, The Journal of Neuroscience.

[30]  G. Edelman Neural Darwinism: The Theory Of Neuronal Group Selection , 1989 .

[31]  Li I. Zhang,et al.  Selective Presynaptic Propagation of Long-Term Potentiation in Defined Neural Networks , 2000, The Journal of Neuroscience.

[32]  K. Harris,et al.  Spontaneous Events Outline the Realm of Possible Sensory Responses in Neocortical Populations , 2009, Neuron.

[33]  Roland Vollgraf,et al.  From grids to places , 2007, Journal of Computational Neuroscience.

[34]  David M. Lovinger,et al.  Neurotransmitter roles in synaptic modulation, plasticity and learning in the dorsal striatum , 2010, Neuropharmacology.

[35]  S. Sajikumar,et al.  Late-associativity, synaptic tagging, and the role of dopamine during LTP and LTD , 2004, Neurobiology of Learning and Memory.

[36]  Mattias P. Karlsson,et al.  Network Dynamics Underlying the Formation of Sparse, Informative Representations in the Hippocampus , 2008, The Journal of Neuroscience.

[37]  T. W. Anderson,et al.  An Introduction to Multivariate Statistical Analysis , 1959 .

[38]  M. Bentivoglio,et al.  Chapter I The organization and circuits of mesencephalic dopaminergic neurons and the distribution of dopamine receptors in the brain , 2005 .

[39]  Alcino J. Silva,et al.  CREB and memory. , 1998, Annual review of neuroscience.

[40]  G. Strang The Fundamental Theorem of Linear Algebra , 1993 .

[41]  K. Stevens,et al.  Linguistic experience alters phonetic perception in infants by 6 months of age. , 1992, Science.

[42]  Anthony W. Harrington,et al.  Long-Distance Control of Synapse Assembly by Target-Derived NGF , 2010, Neuron.

[43]  Mu-ming Poo,et al.  Neurotrophins as synaptic modulators , 2001, Nature Reviews Neuroscience.

[44]  Kevin R Jones,et al.  Genetic evidence that brain-derived neurotrophic factor mediates competitive interactions between individual cortical neurons , 2012, Proceedings of the National Academy of Sciences.

[45]  Sreedharan Sajikumar,et al.  Making synapses strong: metaplasticity prolongs associativity of long-term memory by switching synaptic tag mechanisms. , 2014, Cerebral cortex.

[46]  Stanley J. Wiegand,et al.  The neurotrophins BDNF, NT-3, and NGF display distinct patterns of retrograde axonal transport in peripheral and central neurons , 1992, Neuron.

[47]  David Balduzzi,et al.  Cortical prediction markets , 2014, AAMAS.

[48]  David J. C. MacKay,et al.  Information Theory, Inference, and Learning Algorithms , 2004, IEEE Transactions on Information Theory.

[49]  Peter Dayan,et al.  A Neural Substrate of Prediction and Reward , 1997, Science.

[50]  Elke Edelmann,et al.  Pre- and postsynaptic twists in BDNF secretion and action in synaptic plasticity , 2014, Neuropharmacology.

[51]  R. Malenka,et al.  AMPA receptor trafficking and synaptic plasticity. , 2002, Annual review of neuroscience.

[52]  U. Frey,et al.  Synaptic tagging and long-term potentiation , 1997, Nature.

[53]  W. Gerstner,et al.  Connectivity reflects coding: a model of voltage-based STDP with homeostasis , 2010, Nature Neuroscience.

[54]  Rafal Butowt,et al.  Anterograde axonal transport, transcytosis, and recycling of neurotrophic factors , 2001, Molecular Neurobiology.

[55]  B. McNaughton,et al.  Differential modulation of CA1 and dentate gyrus interneurons during exploration of novel environments. , 2004, Journal of neurophysiology.

[56]  N. Sigala,et al.  Visual categorization shapes feature selectivity in the primate temporal cortex , 2002, Nature.

[57]  Nathan Intrator,et al.  Theory of Cortical Plasticity , 2004 .

[58]  Gernot Riedel,et al.  Foreground contextual fear memory consolidation requires two independent phases of hippocampal ERK/CREB activation. , 2006, Learning & memory.

[59]  M. Frank Computational models of motivated action selection in corticostriatal circuits , 2011, Current Opinion in Neurobiology.

[60]  M. Scanziani,et al.  Equalizing Excitation-Inhibition Ratios across Visual Cortical Neurons , 2014, Nature.

[61]  J. Simon Wiegert,et al.  Multiple dynamic representations in the motor cortex during sensorimotor learning , 2012, Nature.

[62]  A. Riccio,et al.  An NGF-TrkA-mediated retrograde signal to transcription factor CREB in sympathetic neurons. , 1997, Science.

[63]  M. Fee The role of efference copy in striatal learning , 2014, Current Opinion in Neurobiology.

[64]  Walter Senn,et al.  A Gradient Learning Rule for the Tempotron , 2009, Neural Computation.

[65]  G. Einevoll,et al.  From grid cells to place cells: A mathematical model , 2006, Hippocampus.

[66]  M. Poo,et al.  Propagation of activity-dependent synaptic depression in simple neural networks , 1997, Nature.

[67]  Wulfram Gerstner,et al.  Spike-Based Reinforcement Learning in Continuous State and Action Space: When Policy Gradient Methods Fail , 2009, PLoS Comput. Biol..

[68]  Eric R. Ziegel,et al.  The Elements of Statistical Learning , 2003, Technometrics.

[69]  J. Csicsvari,et al.  Organization of cell assemblies in the hippocampus , 2003, Nature.

[70]  Woong Sun,et al.  Adaptive roles of programmed cell death during nervous system development. , 2006, Annual review of neuroscience.

[71]  J. Wickens Synaptic plasticity in the basal ganglia , 2009, Behavioural Brain Research.

[72]  M. Hasselmo The role of acetylcholine in learning and memory , 2006, Current Opinion in Neurobiology.

[73]  P. J. Sjöström,et al.  Neocortical LTD via Coincident Activation of Presynaptic NMDA and Cannabinoid Receptors , 2003, Neuron.

[74]  Greg D. Gale,et al.  Hippocampus and contextual fear conditioning: Recent controversies and advances , 2001, Hippocampus.

[75]  L. Frank,et al.  Behavioral/Systems/Cognitive Hippocampal Plasticity across Multiple Days of Exposure to Novel Environments , 2022 .

[76]  D. Hubel,et al.  Receptive fields of single neurones in the cat's striate cortex , 1959, The Journal of physiology.

[77]  G. Turrigiano The Self-Tuning Neuron: Synaptic Scaling of Excitatory Synapses , 2008, Cell.

[78]  F. Eckenstein,et al.  An anatomical study of cholinergic innervation in rat cerebral cortex , 1988, Neuroscience.

[79]  Lin Tian,et al.  Activity in motor-sensory projections reveals distributed coding in somatosensation , 2012, Nature.

[80]  Geoffrey E. Hinton,et al.  The "wake-sleep" algorithm for unsupervised neural networks. , 1995, Science.

[81]  Leon N. Cooper ... et al. Theory of cortical plasticity , 2014 .

[82]  B L McNaughton,et al.  Dynamics of the hippocampal ensemble code for space. , 1993, Science.

[83]  A. J. Hill First occurrence of hippocampal spatial firing in a new environment , 1978, Experimental Neurology.

[84]  D. Purves Body and Brain: A Trophic Theory of Neural Connections , 1988 .

[85]  Seungjin Choi,et al.  Independent Component Analysis , 2009, Handbook of Natural Computing.

[86]  Yee Whye Teh,et al.  A Fast Learning Algorithm for Deep Belief Nets , 2006, Neural Computation.

[87]  Spartaco Santi,et al.  Induction of long-term potentiation and depression is reflected by corresponding changes in secretion of endogenous brain-derived neurotrophic factor. , 2004, Proceedings of the National Academy of Sciences of the United States of America.

[88]  Karl J. Friston Hierarchical Models in the Brain , 2008, PLoS Comput. Biol..

[89]  K. Svoboda,et al.  Neural Activity in Barrel Cortex Underlying Vibrissa-Based Object Localization in Mice , 2010, Neuron.

[90]  E. Thorndike SOME EXPERIMENTS ON ANIMAL INTELLIGENCE. , 1898, Science.

[91]  DeLiang Wang,et al.  Unsupervised Learning: Foundations of Neural Computation , 2001, AI Mag..

[92]  K. Doya,et al.  Multiple representations and algorithms for reinforcement learning in the cortico-basal ganglia circuit , 2011, Current Opinion in Neurobiology.

[93]  Razvan V. Florian,et al.  Reinforcement Learning Through Modulation of Spike-Timing-Dependent Synaptic Plasticity , 2007, Neural Computation.

[94]  Narayan Srinivasa,et al.  A Spiking Neural Model for Stable Reinforcement of Synapses Based on Multiple Distal Rewards , 2013, Neural Computation.

[95]  Stephen Maren,et al.  The Amygdala Is Essential for the Development of Neuronal Plasticity in the Medial Geniculate Nucleus during Auditory Fear Conditioning in Rats , 2001, The Journal of Neuroscience.

[96]  V. Hamburger,et al.  The history of the discovery of the nerve growth factor. , 1993, Journal of neurobiology.

[97]  C. M. Davenport,et al.  Mediation by a CREB family transcription factor of NGF-dependent survival of sympathetic neurons. , 1999, Science.

[98]  V. Hamburger,et al.  History of the discovery of neuronal death in embryos. , 1992, Journal of neurobiology.

[99]  E. Kandel,et al.  Increased Attention to Spatial Context Increases Both Place Field Stability and Spatial Memory , 2004, Neuron.

[100]  E. Thorndike “Animal Intelligence” , 1898, Nature.

[101]  Jürgen Schmidhuber,et al.  Market-Based Reinforcement Learning in Partially Observable Worlds , 2001, ICANN.

[102]  N. Metropolis,et al.  Equation of State Calculations by Fast Computing Machines , 1953, Resonance.

[103]  R. Oppenheim Cell death during development of the nervous system. , 1991, Annual review of neuroscience.

[104]  Mu-ming Poo,et al.  Long-range retrograde spread of LTP and LTD from optic tectum to retina , 2009, Proceedings of the National Academy of Sciences.

[105]  K. Harris Stability of the fittest: organizing learning through retroaxonal signals , 2008, Trends in Neurosciences.

[106]  Kenneth D. Harris,et al.  Top-Down Control of Cortical State , 2013, Neuron.

[107]  Doyun Lee,et al.  Hippocampal Place Fields Emerge upon Single-Cell Manipulation of Excitability During Behavior , 2012, Science.

[108]  Mu-ming Poo,et al.  Rapid BDNF-induced retrograde synaptic modification in a developing retinotectal system , 2004, Nature.