Critical branching neural networks.

It is now well-established that intrinsic variations in human neural and behavioral activity tend to exhibit scaling laws in their fluctuations and distributions. The meaning of these scaling laws is an ongoing matter of debate between isolable causes versus pervasive causes. A spiking neural network model is presented that self-tunes to critical branching and, in doing so, simulates observed scaling laws as pervasive to neural and behavioral activity. These scaling laws are related to neural and cognitive functions, in that critical branching is shown to yield spiking activity with maximal memory and encoding capacities when analyzed using reservoir computing techniques. The model is also shown to account for findings of pervasive 1/f scaling in speech and cued response behaviors that are difficult to explain by isolable causes. Issues and questions raised by the model and its results are discussed from the perspectives of physics, neuroscience, computer and information sciences, and psychological and cognitive sciences.

[1]  Woodrow L. Shew,et al.  Neuronal Avalanches Imply Maximum Dynamic Range in Cortical Networks at Criticality , 2009, The Journal of Neuroscience.

[2]  Christopher T. Kello,et al.  The Pervasiveness of 1/f Scaling in Speech Reflects the Metastable Basis of Cognition , 2008, Cogn. Sci..

[3]  James A. Dixon,et al.  Strong anticipation: Multifractal cascade dynamics modulate scaling in synchronization behaviors , 2011 .

[4]  D. Turcotte,et al.  Self-organized criticality , 1999 .

[5]  Eduardo D. Sontag,et al.  Computational Aspects of Feedback in Neural Circuits , 2006, PLoS Comput. Biol..

[6]  Kelvin E. Jones,et al.  Neuronal variability: noise or part of the signal? , 2005, Nature Reviews Neuroscience.

[7]  Tang,et al.  Self-Organized Criticality: An Explanation of 1/f Noise , 2011 .

[8]  H. Sompolinsky,et al.  Chaos in Neuronal Networks with Balanced Excitatory and Inhibitory Activity , 1996, Science.

[9]  Christopher T. Kello,et al.  Runword: An IBM-PC software package for the collection and acoustic analysis of speeded naming responses , 1998 .

[10]  J. Lisman,et al.  Synaptic plasticity: A molecular memory switch , 2001, Current Biology.

[11]  Christopher T. Kello,et al.  Visual Motion Perception using Critical Branching Neural Computation , 2011, CogSci.

[12]  E. Marder,et al.  Variability, compensation and homeostasis in neuron and network function , 2006, Nature Reviews Neuroscience.

[13]  Philip L. Smith,et al.  Psychology and neurobiology of simple decisions , 2004, Trends in Neurosciences.

[14]  W. Singer,et al.  Neuronal avalanches in spontaneous activity in vivo. , 2010, Journal of neurophysiology.

[15]  G. V. van Orden,et al.  Self-organization of cognitive performance. , 2003, Journal of experimental psychology. General.

[16]  Christopher T. Kello,et al.  Soft-assembly of sensorimotor function. , 2009, Nonlinear dynamics, psychology, and life sciences.

[17]  C. Anderson.  From molecules to mindfulness: How vertically convergent fractal time fluctuations unify cognition and emotion* , 2000 .

[18]  L. Abbott,et al.  Competitive Hebbian learning through spike-timing-dependent synaptic plasticity , 2000, Nature Neuroscience.

[19]  Usher,et al.  Dynamic pattern formation leads to 1/f noise in neural populations. , 1995, Physical review letters.

[20]  D. McCormick,et al.  Turning on and off recurrent balanced cortical activity , 2003, Nature.

[21]  John M Beggs,et al.  The criticality hypothesis: how local cortical networks might optimize information processing , 2008, Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences.

[22]  J J Hopfield,et al.  Neural networks and physical systems with emergent collective computational abilities. , 1982, Proceedings of the National Academy of Sciences of the United States of America.

[23]  W. Newsome,et al.  The Variable Discharge of Cortical Neurons: Implications for Connectivity, Computation, and Information Coding , 1998, The Journal of Neuroscience.

[24]  David H. Wolpert,et al.  No free lunch theorems for optimization , 1997, IEEE Trans. Evol. Comput..

[25]  V. Torre,et al.  On the Dynamics of the Spontaneous Activity in Neuronal Networks , 2007, PloS one.

[26]  William R. Softky,et al.  The highly irregular firing of cortical cells is inconsistent with temporal integration of random EPSPs , 1993, The Journal of neuroscience : the official journal of the Society for Neuroscience.

[27]  H. Markram,et al.  Regulation of Synaptic Efficacy by Coincidence of Postsynaptic APs and EPSPs , 1997, Science.

[28]  Yee Whye Teh,et al.  A Fast Learning Algorithm for Deep Belief Nets , 2006, Neural Computation.

[29]  Marcus E Raichle,et al.  Intrinsic brain activity sets the stage for expression of motivated behavior , 2005, The Journal of comparative neurology.

[30]  Eric Johnson,et al.  Critical Branching Neural Computation, Neural Avalanches, and 1/f Scaling , 2011, CogSci.

[31]  R. Malenka,et al.  Synaptic scaling mediated by glial TNF-α , 2006, Nature.

[32]  Mark Kröll,et al.  Movement Prediction from Real-World Images Using a Liquid State Machine , 2005, IEA/AIE.

[33]  W. Abraham.  Metaplasticity: tuning synapses and networks for plasticity , 2008, Nature Reviews Neuroscience.

[34]  M. Turvey,et al.  Variability and Determinism in Motor Behavior , 2002, Journal of motor behavior.

[35]  David L. Gilden,et al.  Global Model Analysis of Cognitive Variability , 2009, Cogn. Sci..

[36]  M. Welling,et al.  Statistical inference using weak chaos and infinite memory , 2010 .

[37]  Gal Yadid,et al.  Multifractal statistics and underlying kinetics of neuron spiking time-series , 2001 .

[38]  John M. Beggs,et al.  Neuronal Avalanches in Neocortical Circuits , 2003, The Journal of Neuroscience.

[39]  Quansheng Liu,et al.  On generalized multiplicative cascades , 2000 .

[40]  Nils Bertschinger,et al.  Real-Time Computation at the Edge of Chaos in Recurrent Neural Networks , 2004, Neural Computation.

[41]  Benjamin Schrauwen,et al.  Event detection and localization for small mobile robots using reservoir computing , 2008, Neural Networks.

[42]  David Hsu,et al.  Neuronal avalanches and criticality: A dynamical model for homeostasis , 2006, Neurocomputing.

[43]  J. Touboul,et al.  Can Power-Law Scaling and Neuronal Avalanches Arise from Stochastic Dynamics? , 2009, PloS one.

[44]  E. Wagenmakers,et al.  Theories and models for 1/f(beta) noise in human movement science. , 2009, Human movement science.

[45]  Michael T. Turvey,et al.  Thermodynamic Reasons for Perception--Action Cycles , 1991 .

[46]  B. Schrauwen,et al.  Isolated word recognition with the Liquid State Machine: a case study , 2005, Inf. Process. Lett..

[47]  John F. Kolen,et al.  Resonance and the Perception of Musical Meter , 1994, Connect. Sci..

[48]  C. McClung,et al.  DeltaFosB: a molecular switch for long-term adaptation in the brain. , 2004, Brain research. Molecular brain research.

[49]  C. Granger.  Long memory relationships and the aggregation of dynamic models , 1980 .

[50]  Rangarajan,et al.  Integrated approach to the assessment of long range correlation in time series data , 2000, Physical review. E, Statistical physics, plasmas, fluids, and related interdisciplinary topics.

[51]  Howard Hunt Pattee,et al.  Hierarchy Theory: The Challenge of Complex Systems , 1973 .

[52]  Geoffrey E. Hinton.  Connectionist Learning Procedures , 1989, Artif. Intell..

[53]  J. Kelso,et al.  Cortical coordination dynamics and cognition , 2001, Trends in Cognitive Sciences.

[54]  Marc Benayoun,et al.  Avalanches in a Stochastic Model of Spiking Neurons , 2010, PLoS Comput. Biol..

[55]  Karl J. Friston.  The free-energy principle: a unified brain theory? , 2010, Nature Reviews Neuroscience.

[56]  Brian Mingus,et al.  The Emergent neural modeling system , 2008, Neural Networks.

[57]  Surya Ganguli,et al.  Memory traces in dynamical systems , 2008, Proceedings of the National Academy of Sciences.

[58]  B. Mandelbrot,et al.  RANDOM WALK MODELS FOR THE SPIKE ACTIVITY OF A SINGLE NEURON. , 1964, Biophysical journal.

[59]  Stanley,et al.  Self-organized branching processes: Mean-field theory for avalanches. , 1995, Physical review letters.

[60]  Scott Hotton,et al.  Extending Dynamical Systems Theory to Model Embodied Cognition , 2011, Cogn. Sci..

[61]  S. Nelson,et al.  Homeostatic plasticity in the developing nervous system , 2004, Nature Reviews Neuroscience.

[62]  Paolo Del Giudice,et al.  Efficient Event-Driven Simulation of Large Networks of Spiking Neurons and Dynamical Synapses , 2000, Neural Computation.

[63]  Paolo Santi.  Random Walk Models , 2012 .

[64]  Maya Paczuski,et al.  1/f(alpha) noise from correlations between avalanches in self-organized criticality. , 2002, Physical review. E, Statistical, nonlinear, and soft matter physics.

[65]  C. Bédard,et al.  Macroscopic models of local field potentials and the apparent 1/f noise in brain activity. , 2008, Biophysical journal.

[66]  C. Holmgren,et al.  Coincident Spiking Activity Induces Long-Term Changes in Inhibition of Neocortical Pyramidal Cells , 2001, The Journal of Neuroscience.

[67]  A. Opstal.  Dynamic Patterns: The Self-Organization of Brain and Behavior , 1995 .

[68]  A. Grinvald,et al.  Dynamics of Ongoing Activity: Explanation of the Large Variability in Evoked Cortical Responses , 1996, Science.

[69]  John M Beggs,et al.  Critical branching captures activity in living neural networks and maximizes the number of metastable States. , 2005, Physical review letters.

[70]  M. Teich,et al.  Fractal patterns in auditory nerve-spike trains , 1994, IEEE Engineering in Medicine and Biology Magazine.

[71]  J. Hopfield,et al.  All-or-none potentiation at CA3-CA1 synapses. , 1998, Proceedings of the National Academy of Sciences of the United States of America.

[72]  Geoffrey E. Hinton,et al.  A Learning Algorithm for Boltzmann Machines , 1985, Cogn. Sci..

[73]  B. Vereijken,et al.  Beyond 1 / f α fluctuation-1 Interaction-dominant dynamics in human cognition : Beyond 1 / f α fluctuation , 2010 .

[74]  J. Kehne.  The Neural Basis of Motor Control , 1987, The Yale Journal of Biology and Medicine.

[75]  Henry Markram,et al.  Real-Time Computing Without Stable States: A New Framework for Neural Computation Based on Perturbations , 2002, Neural Computation.

[76]  Seunghwan Kim,et al.  Self-organized criticality and scale-free properties in emergent functional neural networks. , 2004, Physical review. E, Statistical, nonlinear, and soft matter physics.

[77]  John G Holden,et al.  Fractal 1/ƒ dynamics suggest entanglement of measurement and human performance. , 2011, Journal of experimental psychology. Human perception and performance.

[78]  P. Bak,et al.  Complexity, contingency, and criticality. , 1995, Proceedings of the National Academy of Sciences of the United States of America.

[79]  Upinder S. Bhalla,et al.  Molecular Switches at the Synapse Emerge from Receptor and Kinase Traffic , 2005, PLoS Comput. Biol..

[80]  Ramón Huerta,et al.  Transient Cognitive Dynamics, Metastability, and Decision Making , 2008, PLoS Comput. Biol..

[81]  Michael J. Spivey,et al.  The Continuity Of Mind , 2008 .

[82]  Vladimir M. Zatsiorsky,et al.  Long-range correlations in human standing , 2001 .

[83]  K. Linkenkaer-Hansen,et al.  Long-Range Temporal Correlations and Scaling Behavior in Human Brain Oscillations , 2001, The Journal of Neuroscience.

[84]  M. Rabin.  Real time computation , 1963 .

[85]  Christopher G. Langton,et al.  Computation at the edge of chaos: Phase transitions and emergent computation , 1990 .

[86]  Fermín Moscoso del Prado Martín,et al.  Causality, Criticality, and Reading Words: Distinct Sources of Fractal Scaling in Behavioral Sequences , 2011, Cogn. Sci..

[87]  D. Contreras,et al.  Balanced Excitation and Inhibition Determine Spike Timing during Frequency Adaptation , 2006, The Journal of Neuroscience.

[88]  Jeffrey L. Elman,et al.  Finding Structure in Time , 1990, Cogn. Sci..

[89]  Wolfgang Maass,et al.  Neural Dynamics as Sampling: A Model for Stochastic Computation in Recurrent Networks of Spiking Neurons , 2011, PLoS Comput. Biol..

[90]  T. E. Harris,et al.  The Theory of Branching Processes. , 1963 .

[91]  G. V. van Orden,et al.  Dispersion of response times reveals cognitive dynamics. , 2009, Psychological review.

[92]  H. Barlow.  Vision: A computational investigation into the human representation and processing of visual information: David Marr. San Francisco: W. H. Freeman, 1982. pp. xvi + 397 , 1983 .

[93]  Geoffrey E. Hinton,et al.  Learning representations by back-propagating errors , 1986, Nature.

[94]  G L Shulman,et al.  INAUGURAL ARTICLE by a Recently Elected Academy Member:A default mode of brain function , 2001 .

[95]  D. Sornette.  Critical Phenomena in Natural Sciences: Chaos, Fractals, Selforganization and Disorder: Concepts and Tools , 2000 .

[96]  D. Plenz,et al.  Spontaneous cortical activity in awake monkeys composed of neuronal avalanches , 2009, Proceedings of the National Academy of Sciences.

[97]  B. Kaulakys.  Simple model of 1/f noise , 1998 .

[98]  H. Stanley,et al.  Introduction to Phase Transitions and Critical Phenomena , 1972 .

[99]  Christopher T. Kello,et al.  Intrinsic Fluctuations Yield Pervasive 1/f Scaling: Comment on , 2011, Cogn. Sci..

[100]  Surya Ganguli,et al.  Statistical mechanics of compressed sensing. , 2010, Physical review letters.

[101]  Mingzhou Ding,et al.  Statistical Analysis of Timing Errors , 2002, Brain and Cognition.

[102]  R. Ratcliff,et al.  1/f noise in human cognition: Is it ubiquitous, and what does it mean? , 2006, Psychonomic bulletin & review.

[103]  Celso Grebogi,et al.  Dynamically multilayered visual system of the multifractal fly. , 2006, Physical review letters.

[104]  Wofgang Maas,et al.  Networks of spiking neurons: the third generation of neural network models , 1997 .

[105]  L. de Arcangelis,et al.  Self-organized criticality model for brain plasticity. , 2006, Physical review letters.

[106]  Marius Usher,et al.  Network Amplification of Local Fluctuations Causes High Spike Rate Variability, Fractal Firing Patterns and Oscillatory Local Field Potentials , 1994, Neural Computation.

[107]  Satish Iyengar,et al.  Modeling neural activity using the generalized inverse Gaussian distribution , 1997, Biological Cybernetics.

[108]  K. Linkenkaer-Hansen,et al.  Avalanche dynamics of human brain oscillations: Relation to critical branching processes and temporal correlations , 2008, Human brain mapping.

[109]  Michael F. Shlesinger,et al.  Dynamic patterns in complex systems , 1988 .

[110]  M. Teich,et al.  Fractal character of the neural spike train in the visual system of the cat. , 1997, Journal of the Optical Society of America. A, Optics, image science, and vision.

[111]  John M. Beggs,et al.  Aberrant Neuronal Avalanches in Cortical Tissue Removed From Juvenile Epilepsy Patients , 2010, Journal of clinical neurophysiology : official publication of the American Electroencephalographic Society.

[112]  A. N. Mamelak,et al.  Long-range temporal correlations in the spontaneous spiking of neurons in the hippocampal-amygdala complex of humans , 2005, Neuroscience.

[113]  Arjen van Ooyen,et al.  Altered temporal correlations in parietal alpha and prefrontal theta oscillations in early-stage Alzheimer disease , 2009, Proceedings of the National Academy of Sciences.

[114]  L. de Arcangelis,et al.  Learning as a phenomenon occurring in a critical state , 2010, Proceedings of the National Academy of Sciences.

[115]  Klaus Linkenkaer-Hansen,et al.  Breakdown of Long-Range Temporal Correlations in Theta Oscillations in Patients with Major Depressive Disorder , 2005, The Journal of Neuroscience.

[116]  Jianbo Gao,et al.  Multiplicative multifractal modeling and discrimination of human neuronal activity [rapid communication] , 2005 .

[117]  Stephen Grossberg,et al.  A massively parallel architecture for a self-organizing neural pattern recognition machine , 1988, Comput. Vis. Graph. Image Process..

[118]  Mark S. Seidenberg,et al.  Phonology, reading acquisition, and dyslexia: insights from connectionist models. , 1999, Psychological review.

[119]  H. Simon,et al.  The Organization of Complex Systems , 1977 .

[120]  Jeff H. Duyn,et al.  Large-scale spontaneous fluctuations and correlations in brain electrical activity observed with magnetoencephalography , 2010, NeuroImage.

[121]  Garrison W. Cottrell,et al.  2007 Special Issue: Learning grammatical structure with Echo State Networks , 2007 .

[122]  Christopher T. Kello,et al.  Situated Behavior and the Place of Measurement in Psychological Theory , 2010 .

[123]  K. Shockley,et al.  Mutual interpersonal postural constraints are involved in cooperative conversation. , 2003, Journal of experimental psychology. Human perception and performance.

[124]  K. Linkenkaer-Hansen,et al.  Stimulus‐induced change in long‐range temporal correlations and scaling behaviour of sensorimotor oscillations , 2004, The European journal of neuroscience.

[125]  A. Kleinschmidt,et al.  Electroencephalographic signatures of attentional and cognitive default modes in spontaneous brain activity fluctuations at rest , 2003, Proceedings of the National Academy of Sciences of the United States of America.

[126]  Y. Dan,et al.  Spike Timing-Dependent Plasticity of Neural Circuits , 2004, Neuron.

[127]  Christopher T. Kello,et al.  Scaling laws in cognitive sciences , 2010, Trends in Cognitive Sciences.

[128]  Ramesh Balasubramaniam,et al.  Motor Learning Characterized by Changing Lévy Distributions , 2009, PloS one.

[129]  William Bialek,et al.  Spikes: Exploring the Neural Code , 1996 .

[130]  G. Turrigiano,et al.  Rapid Synaptic Scaling Induced by Changes in Postsynaptic Firing , 2008, Neuron.

[131]  D L Gilden,et al.  1/f noise in human cognition. , 1995, Science.

[132]  Barak A. Pearlmutter.  Gradient calculations for dynamic recurrent neural networks: a survey , 1995, IEEE Trans. Neural Networks.

[133]  Simon Farrell,et al.  Estimation and interpretation of 1/falpha noise in human cognition. , 2004, Psychonomic bulletin & review.

[134]  José Carlos Príncipe,et al.  Special issue on echo state networks and liquid state machines , 2007, Neural Networks.

[135]  D. Gilden.  Cognitive emissions of 1/f noise. , 2001, Psychological review.

[136]  Beatrix Vereijken,et al.  Interaction-dominant dynamics in human cognition: beyond 1/f(alpha) fluctuation. , 2010, Journal of experimental psychology. General.

[137]  Gavan Lintern,et al.  Dynamic patterns: The self-organization of brain and behavior , 1997, Complex.

[138]  John M. Beggs,et al.  Universal critical dynamics in high resolution neuronal avalanche data. , 2012, Physical review letters.

[139]  Fausto Guzzetti,et al.  Self-organization, the cascade model, and natural hazards , 2002, Proceedings of the National Academy of Sciences of the United States of America.

[140]  Christopher T. Kello,et al.  Power laws, memory capacity, and self-tuned critical branching in an LIF model with binary synapses , 2011 .

[141]  Wolfgang Maass,et al.  Networks of Spiking Neurons: The Third Generation of Neural Network Models , 1996, Electron. Colloquium Comput. Complex..

[142]  Christopher T. Kello,et al.  The emergent coordination of cognitive function. , 2007, Journal of experimental psychology. General.

[143]  Jean-Michel Deniau,et al.  Assessment of bursting activity and interspike intervals variability: A case study for methodological comparison , 2009, Journal of Neuroscience Methods.

[144]  Kristina D. Micheva,et al.  Single-Synapse Analysis of a Diverse Synapse Population: Proteomic Imaging Methods and Markers , 2010, Neuron.

[145]  Marshall R. Mayberry,et al.  Critical branching neural computation , 2010, The 2010 International Joint Conference on Neural Networks (IJCNN).

[146]  D. Munday.  Edge of chaos. , 2002, Journal of the Royal Society of Medicine.

[147]  Kenneth D. Miller,et al.  Physiological Gain Leads to High ISI Variability in a Simple Model of a Cortical Regular Spiking Cell , 1997, Neural Computation.

[148]  Dmitri B. Chklovskii,et al.  Wiring Optimization in Cortical Circuits , 2002, Neuron.

[149]  A. Katchalsky,et al.  Thermodynamics of flow processes in biological systems. , 1962, Biophysical journal.

[150]  C. Granger,et al.  AN INTRODUCTION TO LONG‐MEMORY TIME SERIES MODELS AND FRACTIONAL DIFFERENCING , 1980 .

[151]  D. Gilden,et al.  Response Variability in Attention-Deficit Disorders , 2007, Psychological science.

[152]  Biyu J. He,et al.  The Temporal Structures and Functional Significance of Scale-free Brain Activity , 2010, Neuron.

[153]  Kate Smith-Miles,et al.  Optimization via Intermittency with a Self-Organizing Neural Network , 2005, Neural Computation.

[154]  B. Scott Jackson,et al.  Including Long-Range Dependence in Integrate-and-Fire Models of the High Interspike-Interval Variability of Cortical Neurons , 2004, Neural Computation.

[155]  S. Wang,et al.  Graded bidirectional synaptic plasticity is composed of switch-like unitary events. , 2005, Proceedings of the National Academy of Sciences of the United States of America.

[156]  H. C. Kwan,et al.  Fractal characteristics of human parkinsonian neuronal spike trains , 2006, Neuroscience.

[157]  Peter F. Rowat,et al.  Interspike Interval Statistics in the Stochastic Hodgkin-Huxley Model: Coexistence of Gamma Frequency Bursts and Highly Irregular Firing , 2007, Neural Computation.

[158]  Heinz Georg Schuster,et al.  Simple model for 1/f(alpha) noise. , 2002, Physical review. E, Statistical, nonlinear, and soft matter physics.

[159]  Wolfgang Maass,et al.  Movement Generation with Circuits of Spiking Neurons , 2005, Neural Computation.

[160]  A. Selverston,et al.  Oscillatory neural networks. , 1985, Annual review of physiology.

[161]  Nicolas Brunel,et al.  Dynamics of Sparsely Connected Networks of Excitatory and Inhibitory Spiking Neurons , 2000, Journal of Computational Neuroscience.

[162]  James P. Crutchfield,et al.  Dynamics, computation, and the “edge of chaos”: a re-examination , 1993, adap-org/9306003.

[163]  Richard J. Rogalski,et al.  A Re-Examination , 1978 .

[164]  H Eugene Stanley,et al.  Non-random fluctuations and multi-scale dynamics regulation of human activity. , 2004, Physica A.

[165]  L. Abbott,et al.  Responses of neurons in primary and inferior temporal visual cortices to natural scenes , 1997, Proceedings of the Royal Society of London. Series B: Biological Sciences.

[166]  Thomas L. Thornton,et al.  Provenance of correlations in psychological data , 2005, Psychonomic bulletin & review.

[167]  O. Kinouchi,et al.  Optimal dynamical range of excitable networks at criticality , 2006, q-bio/0601037.

[168]  J. Cao,et al.  Periodic oscillatory solution of bidirectional associative memory networks with delays. , 2000, Physical review. E, Statistical physics, plasmas, fluids, and related interdisciplinary topics.