The Neuronal Replicator Hypothesis

We propose that replication (with mutation) of patterns of neuronal activity can occur within the brain using known neurophysiological processes. Thereby evolutionary algorithms implemented by neuro- nal circuits can play a role in cognition. Replication of structured neuronal representations is assumed in several cognitive architectures. Replicators overcome some limitations of selectionist models of neuronal search. Hebbian learning is combined with replication to structure exploration on the basis of associations learned in the past. Neuromodulatory gating of sets of bistable neurons allows patterns of activation to be copied with mutation. If the probability of copying a set is related to the utility of that set, then an evolutionary algorithm can be implemented at rapid timescales in the brain. Populations of neuronal replicators can undertake a more rapid and stable search than can be achieved by serial modification of a single solution. Hebbian learning added to neuronal replication allows a powerful structuring of variability capable of learning the location of a global optimum from multiple previously visited local optima. Replication of solutions can solve the problem of catastrophic forgetting in the stability-plasticity dilemma. In short, neuronal replication is essential to explain several features of flexible cognition. Predictions are made for the experimental validation of the neuronal replicator hypothesis.

[1]  D. Rapaport Toward a theory of thinking. , 1951 .

[2]  Eörs Szathmáry,et al.  Natural Selection in the Brain , 2010 .

[3]  Melanie Mitchell,et al.  The Copycat project: a model of mental fluidity and analogy-making , 1995 .

[4]  A. Cooper,et al.  Predictive Reward Signal of Dopamine Neurons , 2011 .

[5]  Pierre-Yves Oudeyer,et al.  Intrinsic Motivation Systems for Autonomous Mental Development , 2007, IEEE Transactions on Evolutionary Computation.

[6]  S Dehaene,et al.  Neural networks that learn temporal sequences by selection. , 1987, Proceedings of the National Academy of Sciences of the United States of America.

[7]  Chris Watkins,et al.  Learning from delayed rewards , 1989 .

[8]  D. Spalding The Principles of Psychology , 1873, Nature.

[9]  R. Sokal Evolutionary Genetics , 1972, The Quarterly Review of Biology.

[10]  O. Hikosaka Models of information processing in the basal Ganglia edited by James C. Houk, Joel L. Davis and David G. Beiser, The MIT Press, 1995. $60.00 (400 pp) ISBN 0 262 08234 9 , 1995, Trends in Neurosciences.

[11]  Marc Toussaint,et al.  The evolution of genetic representations andmodularadapta tion Dissertation zur Erlangung des Grades eines Doktors der Naturwissenschaften in der Fakultat fur Physik und Astronomie der Ruhr-Universitat Bochum , 2003 .

[12]  William S. Cooper,et al.  Book reviews-the evolution of reason. Logic as a branch of biology , 2001 .

[13]  Misha Tsodyks,et al.  Persistent Activity in Neural Networks with Dynamic Synapses , 2007, PLoS Comput. Biol..

[14]  M. Eigen Selforganization of matter and the evolution of biological macromolecules , 1971, Naturwissenschaften.

[15]  Chrisantha Fernando,et al.  Chemical, Neuronal, and Linguistic Replicators , 2009 .

[16]  W. Köhler The Mentality of Apes. , 2018, Nature.

[17]  R. Aunger The Electric Meme: A New Theory of How We Think , 2002 .

[18]  D. Durstewitz,et al.  Beyond bistability: Biophysics and temporal dynamics of working memory , 2006, Neuroscience.

[19]  C. Pennartz The ascending neuromodulatory systems in learning by reinforcement: comparing computational conjectures with experimental findings , 1995, Brain Research Reviews.

[20]  Róbert Ványi,et al.  Compositional evolution: the impact of sex, symbiosis and modularity on the gradualist framework of evolution , 2008, Genetic Programming and Evolvable Machines.

[21]  M. Pigliucci Is evolvability evolvable? , 2008, Nature Reviews Genetics.

[22]  M. Raichle The Brain's Dark Energy , 2006, Science.

[23]  Jürgen Schmidhuber,et al.  Optimal Artificial Curiosity, Creativity, Music, and the Fine Arts , 2005 .

[24]  Richard A. Watson,et al.  The Effect of Hebbian Learning on Optimisation in Hopfield Networks , 2009 .

[25]  G. Reeke Marvin Minsky, The Society of Mind , 1991, Artif. Intell..

[26]  Jürgen Schmidhuber,et al.  Evolino: Hybrid Neuroevolution/Optimal Linear Search for Sequence Learning , 2005, IJCAI.

[27]  J. Born,et al.  Sleep inspires insight , 2004, Nature.

[28]  Dr. Zbigniew Michalewicz,et al.  How to Solve It: Modern Heuristics , 2004 .

[29]  J. Changeux,et al.  A theory of the epigenesis of neuronal networks by selective stabilization of synapses. , 1973, Proceedings of the National Academy of Sciences of the United States of America.

[30]  Samir W. Mahfoud Niching methods for genetic algorithms , 1996 .

[31]  W. Schultz,et al.  A neural network model with dopamine-like reinforcement signal that learns a spatial delayed response task , 1999, Neuroscience.

[32]  Hans-Georg Beyer,et al.  The Theory of Evolution Strategies , 2001, Natural Computing Series.

[33]  Y. Niv Reinforcement learning in the brain , 2009 .

[34]  Xiao-Jing Wang Synaptic reverberation underlying mnemonic persistent activity , 2001, Trends in Neurosciences.

[35]  John R. Anderson How Can the Human Mind Occur in the Physical Universe , 2007 .

[36]  Alex Fraser,et al.  Simulation of Genetic Systems by Automatic Digital Computers I. Introduction , 1957 .

[37]  Chrisantha Fernando,et al.  Copying and Evolution of Neuronal Topology , 2008, PloS one.

[38]  Olivier Michel,et al.  Cyberbotics Ltd. Webots™: Professional Mobile Robot Simulation , 2004 .

[39]  Günter P. Wagner,et al.  Complex Adaptations and the Evolution of Evolvability , 2005 .

[40]  D. Zipser,et al.  A spiking network model of short-term active memory , 1993, The Journal of neuroscience : the official journal of the Society for Neuroscience.

[41]  Joel L. Davis,et al.  A Model of How the Basal Ganglia Generate and Use Neural Signals That Predict Reinforcement , 1994 .

[42]  J. Mark Baldwin The influence of Darwin on theory of knowledge and philosophy. , 1909 .

[43]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[44]  Toshiyuki Kondo,et al.  A reinforcement learning with evolutionary state recruitment strategy for autonomous mobile robots control , 2003, Robotics Auton. Syst..

[45]  S Dehaene,et al.  A neuronal model of a global workspace in effortful cognitive tasks. , 1998, Proceedings of the National Academy of Sciences of the United States of America.

[46]  E. Oja Simplified neuron model as a principal component analyzer , 1982, Journal of mathematical biology.

[47]  Charles Kemp,et al.  The discovery of structural form , 2008, Proceedings of the National Academy of Sciences.

[48]  Jun Wang,et al.  A recurrent neural network for real-time matrix inversion , 1993 .

[49]  B. Elsevier,et al.  The Extended Phenotype: The Gene as the Unit of Selection. By Richard Dawkins, Oxford and San Francisco: W.H. Freeman, 1982, $18.95 hardcover. , 1984 .

[50]  Geoffrey E. Hinton,et al.  Adaptive Mixtures of Local Experts , 1991, Neural Computation.

[51]  Florentin Wörgötter,et al.  Temporal Sequence Learning, Prediction, and Control: A Review of Different Models and Their Relation to Biological Mechanisms , 2005, Neural Computation.

[52]  M. Strauss,et al.  Book ReviewChance and Necessity: An essay on the natural philosophy of modern biology. , 1972 .

[53]  Eugene M. Izhikevich,et al.  Polychronization: Computation with Spikes , 2006, Neural Computation.

[54]  Mitsuo Kawato,et al.  Multiple Model-Based Reinforcement Learning , 2002, Neural Computation.

[55]  W. Brown Animal Intelligence: Experimental Studies , 1912, Nature.

[56]  András Lörincz,et al.  Learning to Play Using Low-Complexity Rule-Based Policies: Illustrations through Ms. Pac-Man , 2007, J. Artif. Intell. Res..

[57]  Jean-Jacques E. Slotine,et al.  Synchronization and Redundancy: Implications for Robustness of Neural Learning and Decision Making , 2010, Neural Computation.

[58]  Jürgen Schmidhuber,et al.  Ultimate Cognition à la Gödel , 2009, Cognitive Computation.

[59]  Misha Tsodyks,et al.  The Emergence of Up and Down States in Cortical Networks , 2006, PLoS Comput. Biol..

[60]  G M Edelman,et al.  The evolution of somatic selection: the antibody tale. , 1994, Genetics.

[61]  M. Hasselmo,et al.  Mechanism of Graded Persistent Cellular Activity of Entorhinal Cortex Layer V Neurons , 2006, Neuron.

[62]  P. Adams Hebb and Darwin. , 1998, Journal of theoretical biology.

[63]  S Dehaene,et al.  A hierarchical neuronal network for planning behavior. , 1997, Proceedings of the National Academy of Sciences of the United States of America.

[64]  G. Edelman,et al.  Spike-timing dynamics of neuronal groups. , 2004, Cerebral cortex.

[65]  Karl J. Friston,et al.  Free-energy and the brain , 2007, Synthese.

[66]  Jürgen Schmidhuber,et al.  Curious model-building control systems , 1991, [Proceedings] 1991 IEEE International Joint Conference on Neural Networks.

[67]  Laurenz Wiskott,et al.  A functional hypothesis for adult hippocampal neurogenesis: Avoidance of catastrophic interference in the dentate gyrus , 2006, Hippocampus.

[68]  G. Edelman,et al.  Reentry and the problem of integrating multiple cortical areas: simulation of dynamic integration in the visual system. , 1992, Cerebral cortex.

[69]  X. Wang,et al.  Synaptic Basis of Cortical Persistent Activity: the Importance of NMDA Receptors to Working Memory , 1999, The Journal of Neuroscience.

[70]  J. Hadamard,et al.  The Psychology of Invention in the Mathematical Field. , 1945 .

[71]  L. Altenberg,et al.  PERSPECTIVE: COMPLEX ADAPTATIONS AND THE EVOLUTION OF EVOLVABILITY , 1996, Evolution; international journal of organic evolution.

[72]  Juergen Schmidhuber,et al.  A General Method For Incremental Self-Improvement And Multi-Agent Learning In Unrestricted Environme , 1999 .

[73]  Thomas Bäck,et al.  Evolutionary computation: Toward a new philosophy of machine intelligence , 1997, Complex..

[74]  Marcus Hutter,et al.  Universal Artificial Intellegence - Sequential Decisions Based on Algorithmic Probability , 2005, Texts in Theoretical Computer Science. An EATCS Series.

[75]  A. Grinvald,et al.  Spontaneously emerging cortical representations of visual attributes , 2003, Nature.

[76]  D. Amit The Hebbian paradigm reintegrated: Local reverberations as internal representations , 1995, Behavioral and Brain Sciences.

[77]  Hod Lipson,et al.  Resilient Machines Through Continuous Self-Modeling , 2006, Science.

[78]  David B. Fogel,et al.  Evolutionary Computation: Towards a New Philosophy of Machine Intelligence , 1995 .

[79]  R. Dawkins,et al.  The extended phenotype: The gene as the unit of selection: By Richard Dawkins, Oxford and San Francisco: W.H. Freeman, 1982, $18.95 hardcover , 1984 .

[80]  G. Edelman Neural Darwinism: The Theory Of Neuronal Group Selection , 1989 .

[81]  Peter Dayan,et al.  Dopamine: generalization and bonuses , 2002, Neural Networks.

[82]  Claes Andersson,et al.  Splitting the replicator: Generalized Darwinism and the place of culture in nature , 2011 .

[83]  Luc Steels,et al.  A (very) Brief Introduction to Fluid Construction Grammar , 2006 .

[84]  H. Heller,et al.  Principles of Life , 2010 .

[85]  M. Tsodyks,et al.  Synaptic Theory of Working Memory , 2008, Science.

[86]  Risto Miikkulainen,et al.  Efficient Reinforcement Learning Through Evolving Neural Network Topologies , 2002, GECCO.

[87]  Peter Ford Dominey Complex sensory-motor sequence learning based on recurrent state representation and reinforcement learning , 1995, Biological Cybernetics.

[88]  P. Redgrave,et al.  Is the short-latency dopamine response too short to signal reward error? , 1999, Trends in Neurosciences.

[89]  S. Becker A computational principle for hippocampal learning and neurogenesis , 2005, Hippocampus.

[90]  Stephen Grossberg,et al.  The ART of adaptive pattern recognition by a self-organizing neural network , 1988, Computer.

[91]  Jeffrey L. Elman,et al.  Finding Structure in Time , 1990, Cogn. Sci..

[92]  Chrisantha Fernando,et al.  Symbol manipulation and rule learning in spiking neuronal networks. , 2011, Journal of theoretical biology.

[93]  H. Barlow Vision: A computational investigation into the human representation and processing of visual information: David Marr. San Francisco: W. H. Freeman, 1982. pp. xvi + 397 , 1983 .

[94]  Merav Parter,et al.  Facilitated Variation: How Evolution Learns from Past Environments To Generalize to New Environments , 2008, PLoS Comput. Biol..

[95]  K. Svoboda,et al.  Experience-dependent structural synaptic plasticity in the mammalian brain , 2009, Nature Reviews Neuroscience.

[96]  J. Hadamard,et al.  The Psychology of Invention in the Mathematical Field. , 1945 .

[97]  Wirt Atmar,et al.  Notes on the simulation of evolution , 1994, IEEE Trans. Neural Networks.

[98]  Takashi Omori,et al.  Adaptive internal state space construction method for reinforcement learning of a real-world agent , 1999, Neural Networks.

[99]  Bartlett W. Mel,et al.  Cortical rewiring and information storage , 2004, Nature.

[100]  L. Altenberg The evolution of evolvability in genetic programming , 1994 .

[101]  D J Amit,et al.  Multiple-object working memory--a model for behavioral performance. , 2003, Cerebral cortex.

[102]  Stuart G. Shanker,et al.  The nature of insight , 1995, Minds and Machines.

[103]  W. Abraham,et al.  Memory retention – the synaptic stability versus plasticity dilemma , 2005, Trends in Neurosciences.

[104]  Eugene M. Izhikevich,et al.  Dynamical Systems in Neuroscience: The Geometry of Excitability and Bursting , 2006 .

[105]  Luc Steels,et al.  Unify and Merge in Fluid Construction Grammar , 2006, EELC.

[106]  Olivier Michel,et al.  Cyberbotics Ltd. Webots™: Professional Mobile Robot Simulation , 2004, ArXiv.

[107]  J. S. F. Barker,et al.  Simulation of Genetic Systems by Automatic Digital Computers , 1958 .

[108]  F. Wörgötter,et al.  Activity-dependent structural plasticity , 2009, Brain Research Reviews.

[109]  John Maynard Smith,et al.  From replicators to reproducers: the first major transitions leading to life. , 1997, Journal of theoretical biology.

[110]  A. Grinvald,et al.  Dynamics of Ongoing Activity: Explanation of the Large Variability in Evoked Cortical Responses , 1996, Science.

[111]  M. Mintun,et al.  Brain work and brain imaging. , 2006, Annual review of neuroscience.

[112]  George Kampis,et al.  Evolvability of Natural and Artificial Systems , 2011, FET.

[113]  Reinhard Bürger,et al.  THE MUTATION MATRIX AND THE EVOLUTION OF EVOLVABILITY , 2007, Evolution; international journal of organic evolution.

[114]  U. Alon,et al.  Spontaneous evolution of modularity and network motifs. , 2005, Proceedings of the National Academy of Sciences of the United States of America.

[115]  C. Ofria,et al.  Genome complexity, robustness and genetic interactions in digital organisms , 1999, Nature.

[116]  G. Marcus The Algebraic Mind: Integrating Connectionism and Cognitive Science , 2001 .

[117]  Christopher C. Byrne,et al.  Self-deception in an evolutionary game. , 2001, Journal of theoretical biology.

[118]  J C Houk,et al.  Action selection and refinement in subcortical loops through basal ganglia and cerebellum , 2007, Philosophical Transactions of the Royal Society B: Biological Sciences.

[119]  E. Szathmáry,et al.  A New Replicator: A theoretical framework for analysing replication , 2010, BMC Biology.

[120]  Isaac Meilijson,et al.  Evolution of Reinforcement Learning in Uncertain Environments: A Simple Explanation for Complex Foraging Behaviors , 2002, Adapt. Behav..

[121]  Nicolas Brunel,et al.  Dynamics and plasticity of stimulus-selective persistent activity in cortical network models. , 2003, Cerebral cortex.

[122]  Dario Floreano,et al.  Evolving neuromodulatory topologies for reinforcement learning-like problems , 2007, 2007 IEEE Congress on Evolutionary Computation.

[123]  G. Bower,et al.  THE PSYCHOLOGY OF LEARNING AND M·OTIVATION , 2001 .

[124]  Sebastian Thrun,et al.  Learning to Learn , 1998, Springer US.

[125]  R. O’Reilly,et al.  Computational Explorations in Cognitive Neuroscience: Understanding the Mind by Simulating the Brain , 2000 .

[126]  T. Bonhoeffer,et al.  Experience leaves a lasting structural trace in cortical circuits , 2008, Nature.

[127]  Hermann J. Muller,et al.  The Gene Material as the Initiator and the Organizing Basis of Life , 1966, The American Naturalist.

[128]  H. Sompolinsky,et al.  Temporal integration by calcium dynamics in a model neuron , 2003, Nature Neuroscience.

[129]  A. Grinvald,et al.  Linking spontaneous activity of single cortical neurons and the underlying functional architecture. , 1999, Science.

[130]  George R. Price,et al.  Selection and Covariance , 1970, Nature.

[131]  Shimon Whiteson,et al.  Empirical Studies in Action Selection with Reinforcement Learning , 2007, Adapt. Behav..

[132]  Hans-Paul Schwefel,et al.  Advantages (and disadvantages) of evolutionary computation over other approaches , 2018, Evolutionary Computation 1.

[133]  G. Edelman,et al.  Solving Bernstein's problem: a proposal for the development of coordinated movement by selection. , 1993, Child development.

[134]  Richard S. Sutton,et al.  Neuronlike adaptive elements that can solve difficult learning control problems , 1983, IEEE Transactions on Systems, Man, and Cybernetics.

[135]  M. Bear,et al.  Metaplasticity: the plasticity of synaptic plasticity , 1996, Trends in Neurosciences.

[136]  J. Schmidhuber,et al.  Evolutionary computation versus reinforcement learning , 2000, 2000 26th Annual Conference of the IEEE Industrial Electronics Society. IECON 2000. 2000 IEEE International Conference on Industrial Electronics, Control and Instrumentation. 21st Century Technologies.

[137]  E. Conklin PROBLEMS OF BIOLOGY. , 1898, Science.

[138]  David B. Fogel,et al.  Evolutionary Computation: Toward a New Philosophy of Machine Intelligence (IEEE Press Series on Computational Intelligence) , 2006 .

[139]  Jordan B. Pollack,et al.  Modeling Building-Block Interdependency , 1998, PPSN.

[140]  L. Gabora Creative Thought as a non Darwinian Evolutionary Process , 2005 .

[141]  Corso Elvezia A General Method for Incremental Self-improvement and Multi-agent Learning in Unrestricted Environments , 1996 .

[142]  J. Mark Baldwin On selective thinking. , 1898 .

[143]  L. Nadel,et al.  Multiple trace theory of human memory: Computational, neuroimaging, and neuropsychological results , 2000, Hippocampus.

[144]  P. Robert,et al.  Reply: HPV in focal cortical dysplasia , 2015, Annals of neurology.

[145]  H. Seung,et al.  Robust persistent neural activity in a model integrator with multiple hysteretic dendrites per neuron. , 2003, Cerebral cortex.

[146]  Jürgen Schmidhuber,et al.  Solving Deep Memory POMDPs with Recurrent Policy Gradients , 2007, ICANN.

[147]  Risto Miikkulainen,et al.  Accelerated Neural Evolution through Cooperatively Coevolved Synapses , 2008, J. Mach. Learn. Res..

[148]  Jürgen Schmidhuber,et al.  Reinforcement Learning with Self-Modifying Policies , 1998, Learning to Learn.

[149]  J. K. Kinnear,et al.  Advances in Genetic Programming , 1994 .

[150]  J. Launer Darwin's dangerous idea. , 2002, QJM : monthly journal of the Association of Physicians.

[151]  Ranulfo Romo,et al.  Flexible Control of Mutual Inhibition: A Neural Model of Two-Interval Discrimination , 2005, Science.

[152]  P. Wagner,et al.  CONTRASTING THE UNDERLYING PATTERNS OF ACTIVE TRENDS IN MORPHOLOGIC EVOLUTION , 1996, Evolution; international journal of organic evolution.

[153]  Peter Dayan,et al.  A Neural Substrate of Prediction and Reward , 1997, Science.

[154]  K. Doya,et al.  Representation of Action-Specific Reward Values in the Striatum , 2005, Science.

[155]  Justin L. Vincent,et al.  Spontaneous neuronal activity distinguishes human dorsal and ventral attention systems. , 2006, Proceedings of the National Academy of Sciences of the United States of America.

[156]  Thomas Voegtlin,et al.  Recursive self-organizing maps , 2002, Neural Networks.

[157]  A. Barto,et al.  Adaptive Critics and the Basal Ganglia , 1994 .

[158]  Andrew W. Moore,et al.  Reinforcement Learning: A Survey , 1996, J. Artif. Intell. Res..

[159]  Hod Lipson,et al.  Automated reverse engineering of nonlinear dynamical systems , 2007, Proceedings of the National Academy of Sciences.

[160]  Jürgen Schmidhuber,et al.  HQ-Learning , 1997, Adapt. Behav..

[161]  Julian Togelius,et al.  Ontogenetic and Phylogenetic Reinforcement Learning , 2009, Künstliche Intell..

[162]  H. Seung,et al.  Learning in Spiking Neural Networks by Reinforcement of Stochastic Synaptic Transmission , 2003, Neuron.

[163]  E. Izhikevich Solving the distal reward problem through linkage of STDP and dopamine signaling , 2007, BMC Neuroscience.

[164]  Jürgen Schmidhuber,et al.  Shifting Inductive Bias with Success-Story Algorithm, Adaptive Levin Search, and Incremental Self-Improvement , 1997, Machine Learning.

[165]  Richard S. Sutton,et al.  Introduction to Reinforcement Learning , 1998 .

[166]  Jun Morimoto,et al.  Acquisition of stand-up behavior by a real robot using hierarchical reinforcement learning , 2000, Robotics Auton. Syst..

[167]  Eugene M. Izhikevich,et al.  Simple model of spiking neurons , 2003, IEEE Trans. Neural Networks.

[168]  Stefan Elfwing,et al.  Embodied Evolution of Learning Ability , 2007 .

[169]  Bartlett W. Mel,et al.  Impact of Active Dendrites and Structural Plasticity on the Memory Capacity of Neural Tissue , 2001, Neuron.

[170]  R. Wilton The Psychology of Learning and Motivation: Advances in Research and Theory. Vol 4. , 1972 .