Integration of Continuous-Time Dynamics in a Spiking Neural Network Simulator

Contemporary modeling approaches to the dynamics of neural networks include two important classes of models: biologically grounded spiking neuron models and functionally inspired rate-based units. We present a unified simulation framework that supports the combination of the two for multi-scale modeling, enables the quantitative validation of mean-field approaches by spiking network simulations, and provides an increase in reliability by usage of the same simulation code and the same network model specifications for both model classes. While most spiking simulations rely on the communication of discrete events, rate models require time-continuous interactions between neurons. Exploiting the conceptual similarity to the inclusion of gap junctions in spiking network simulations, we arrive at a reference implementation of instantaneous and delayed interactions between rate-based models in a spiking network simulator. The separation of rate dynamics from the general connection and communication infrastructure ensures flexibility of the framework. In addition to the standard implementation we present an iterative approach based on waveform-relaxation techniques to reduce communication and increase performance for large-scale simulations of rate-based models with instantaneous interactions. Finally we demonstrate the broad applicability of the framework by considering various examples from the literature, ranging from random networks to neural-field models. The study provides the prerequisite for interactions between rate-based and spiking models in a joint simulation.

[1]  Nicholas J. Higham,et al.  Blocked Schur Algorithms for Computing the Matrix Square Root , 2012, PARA.

[2]  C. Kelley Iterative Methods for Linear and Nonlinear Equations , 1987 .

[3]  Markus Diesmann,et al.  Limits to the Development of Feed-Forward Structures in Large Recurrent Neuronal Networks , 2011, Front. Comput. Neurosci..

[4]  Michael L. Hines,et al.  Comparison of neuronal spike exchange methods on a Blue Gene/P supercomputer , 2011, Front. Comput. Neurosci..

[5]  Srdjan Ostojic,et al.  Two types of asynchronous activity in networks of excitatory and inhibitory spiking neurons , 2014, Nature Neuroscience.

[6]  Nicolas Brunel,et al.  Dynamics of the Firing Probability of Noisy Integrate-and-Fire Neurons , 2002, Neural Computation.

[7]  Moritz Helias,et al.  Fundamental Activity Constraints Lead to Specific Interpretations of the Connectome , 2015, PLoS Comput. Biol..

[8]  Ernest Montbri'o,et al.  Macroscopic description for networks of spiking neurons , 2015, 1506.06581.

[9]  J. Cowan,et al.  A spherical model for orientation and spatial-frequency tuning in a cortical hypercolumn. , 2003, Philosophical transactions of the Royal Society of London. Series B, Biological sciences.

[10]  S. Grossberg Contour Enhancement , Short Term Memory , and Constancies in Reverberating Neural Networks , 1973 .

[11]  G. Deco,et al.  Emerging concepts for the dynamical organization of resting-state activity in the brain , 2010, Nature Reviews Neuroscience.

[12]  James L. McClelland,et al.  Parallel distributed processing: explorations in the microstructure of cognition, vol. 1: foundations , 1986 .

[13]  O. Nevanlinna,et al.  Convergence of dynamic iteration methods for initial value problems , 1987 .

[14]  Pierre Yger,et al.  PyNN: A Common Interface for Neuronal Network Simulators , 2008, Front. Neuroinform..

[15]  Markus Diesmann,et al.  Advancing the Boundaries of High-Connectivity Network Simulation with Distributed Computing , 2005, Neural Computation.

[16]  Moritz Helias,et al.  Optimal Sequence Memory in Driven Random Networks , 2016, Physical Review X.

[17]  Stefan Rotter,et al.  How Structure Determines Correlations in Neuronal Networks , 2011, PLoS Comput. Biol..

[18]  Laurent U. Perrinet,et al.  Complex dynamics in recurrent cortical networks based on spatially realistic connectivities , 2012, Front. Comput. Neurosci..

[19]  Michael L. Hines,et al.  Neuron splitting in compute-bound parallel network simulations enables runtime scaling with twice as many processors , 2008, Journal of Computational Neuroscience.

[20]  Michael A. Buice,et al.  Dynamic Finite Size Effects in Spiking Neural Networks , 2013, PLoS Comput. Biol..

[21]  Carl van Vreeswijk,et al.  Temporal Correlations in Stochastic Networks of Spiking Neurons , 2002, Neural Computation.

[22]  Moritz Helias,et al.  Identifying Anatomical Origins of Coexisting Oscillations in the Cortical Microcircuit , 2015, PLoS Comput. Biol..

[23]  W. Newsome,et al.  The Variable Discharge of Cortical Neurons: Implications for Connectivity, Computation, and Information Coding , 1998, The Journal of Neuroscience.

[24]  H. Sompolinsky,et al.  Chaos in Neuronal Networks with Balanced Excitatory and Inhibitory Activity , 1996, Science.

[25]  Michael Okun,et al.  Instantaneous correlation of excitation and inhibition during ongoing and sensory-evoked activities , 2008, Nature Neuroscience.

[26]  R. O’Reilly,et al.  Computational Explorations in Cognitive Neuroscience: Understanding the Mind by Simulating the Brain , 2000 .

[27]  Karline Soetaert,et al.  Solving Ordinary Differential Equations in R , 2012 .

[28]  Markus Diesmann,et al.  A multi-scale layer-resolved spiking network model of resting-state dynamics in macaque visual cortical areas , 2018, PLoS Comput. Biol..

[29]  D. Hansel,et al.  On the Distribution of Firing Rates in Networks of Cortical Neurons , 2011, The Journal of Neuroscience.

[30]  Moritz Helias,et al.  A unified view on weakly correlated recurrent networks , 2013, Front. Comput. Neurosci..

[31]  C. Petersen,et al.  Membrane Potential Dynamics of GABAergic Neurons in the Barrel Cortex of Behaving Mice , 2010, Neuron.

[32]  Stephen Grossberg,et al.  A neural model of the saccade generator in the reticular formation , 1998, Neural Networks.

[33]  Markus Diesmann,et al.  Maintaining Causality in Discrete Time Neuronal Network Simulations , 2007 .

[34]  Bernhard Rumpe,et al.  NESTML: a modeling language for spiking neurons , 2016, Modellierung.

[35]  Carson C. Chow,et al.  A Spiking Neuron Model for Binocular Rivalry , 2004, Journal of Computational Neuroscience.

[36]  A. Siegert On the First Passage Time Probability Problem , 1951 .

[37]  James A. Bednar,et al.  Topographica: Building and Analyzing Map-Level Simulations from Python, C/C++, MATLAB, NEST, or NEURON Components , 2008, Front. Neuroinform..

[38]  H. Sompolinsky,et al.  Theory of orientation tuning in visual cortex. , 1995, Proceedings of the National Academy of Sciences of the United States of America.

[39]  Wulfram Gerstner,et al.  Neuronal Dynamics: From Single Neurons To Networks And Models Of Cognition , 2014 .

[40]  William R. Softky,et al.  The highly irregular firing of cortical cells is inconsistent with temporal integration of random EPSPs , 1993, The Journal of neuroscience : the official journal of the Society for Neuroscience.

[41]  M. Golubitsky,et al.  Geometric visual hallucinations, Euclidean symmetry and the functional architecture of striate cortex. , 2001, Philosophical transactions of the Royal Society of London. Series B, Biological sciences.

[42]  Wolfgang Maass,et al.  Spiking Neurons , 1998, NC.

[43]  Carson C. Chow,et al.  Correlations, fluctuations, and stability of a finite-size network of coupled oscillators. , 2007, Physical review. E, Statistical, nonlinear, and soft matter physics.

[44]  Hugh R Wilson,et al.  Computational evidence for a rivalry hierarchy in vision , 2003, Proceedings of the National Academy of Sciences of the United States of America.

[45]  James M. Bower,et al.  GENESIS (simulation environment) , 2007 .

[46]  Wulfram Gerstner,et al.  Population Dynamics of Spiking Neurons: Fast Transients, Asynchronous States, and Locking , 2000, Neural Computation.

[47]  James L. McClelland,et al.  Explorations in parallel distributed processing: a handbook of models, programs, and exercises , 1988 .

[48]  Romain Brette,et al.  Philosophy of the Spike: Rate-Based vs. Spike-Based Theories of the Brain , 2015, Front. Syst. Neurosci..

[49]  Andrew P. Davison,et al.  Efficient generation of connectivity in neuronal networks from simulator-independent descriptions , 2014, Front. Neuroinform..

[50]  M. E. Galassi,et al.  GNU SCIENTI C LIBRARY REFERENCE MANUAL , 2005 .

[51]  Marc-Oliver Gewaltig,et al.  NEST (NEural Simulation Tool) , 2007, Scholarpedia.

[52]  Viktor K. Jirsa,et al.  The Virtual Brain: a simulator of primate brain network dynamics , 2013, Front. Neuroinform..

[53]  Stefan Rotter,et al.  Orientation Selectivity in Inhibition-Dominated Networks of Spiking Neurons: Effect of Single Neuron Properties and Network Dynamics , 2015, PLoS Comput. Biol..

[54]  Karl J. Friston,et al.  The Dynamic Brain: From Spiking Neurons to Neural Masses and Cortical Fields , 2008, PLoS Comput. Biol..

[55]  James G. King,et al.  Reconstruction and Simulation of Neocortical Microcircuitry , 2015, Cell.

[56]  Moritz Helias,et al.  Functional methods for disordered neural networks , 2016 .

[57]  Zachary P. Kilpatrick,et al.  Wilson-Cowan Model , 2014, Encyclopedia of Computational Neuroscience.

[58]  Zhencheng Fan,et al.  SOR waveform relaxation methods for stochastic differential equations , 2013, Appl. Math. Comput..

[59]  Kenichi Ohki,et al.  Conversion of Working Memory to Motor Sequence in the Monkey Premotor Cortex , 2003, Science.

[60]  Moritz Helias,et al.  Neuroinformatics Original Research Article Pynest: a Convenient Interface to the Nest Simulator , 2022 .

[61]  Milton Abramowitz,et al.  Handbook of Mathematical Functions with Formulas, Graphs, and Mathematical Tables , 1964 .

[62]  Xiao-Jing Wang,et al.  A Recurrent Network Mechanism of Time Integration in Perceptual Decisions , 2006, The Journal of Neuroscience.

[63]  P. Bressloff Spatiotemporal dynamics of continuum neural fields , 2012 .

[64]  Nicholas T. Carnevale,et al.  Simulation of networks of spiking neurons: A review of tools and strategies , 2006, Journal of Computational Neuroscience.

[65]  Tomoki Fukai,et al.  Supercomputers Ready for Use as Discovery Machines for Neuroscience , 2012, Front. Neuroinform..

[66]  Sompolinsky,et al.  Theory of correlations in stochastic neural networks. , 1994, Physical review. E, Statistical physics, plasmas, fluids, and related interdisciplinary topics.

[67]  Michael A. Arbib,et al.  The Neural Simulation Language: A System for Brain Modeling , 2002 .

[68]  Pierre Yger,et al.  Topologically invariant macroscopic statistics in balanced networks of conductance-based integrate-and-fire neurons , 2011, Journal of Computational Neuroscience.

[69]  G. Schöner,et al.  Dynamic Field Theory of Movement Preparation , 2022 .

[70]  S. Swain Handbook of Stochastic Methods for Physics, Chemistry and the Natural Sciences , 1984 .

[71]  G Horn,et al.  An analysis of spontaneous impulse activity of units in the striate cortex of unrestrained cats , 1966, The Journal of physiology.

[72]  Jutta Docter,et al.  JUQUEEN: IBM Blue Gene/Q® Supercomputer System at the Jülich Supercomputing Centre , 2015 .

[73]  Romain Brette,et al.  The Brian Simulator , 2009, Front. Neurosci..

[74]  J. Fuster,et al.  Unit activity in monkey parietal cortex related to haptic perception and temporary memory , 2004, Experimental Brain Research.

[75]  K. Burrage,et al.  A stochastic exponential Euler scheme for simulation of stiff biochemical reaction systems , 2014, BIT Numerical Mathematics.

[76]  Alexander S. Ecker,et al.  Decorrelated Neuronal Firing in Cortical Microcircuits , 2010, Science.

[77]  L. Abbott,et al.  Eigenvalue spectra of random matrices for neural networks. , 2006, Physical review letters.

[78]  D. Amit,et al.  Model of global spontaneous activity and local structured activity during delay periods in the cerebral cortex. , 1997, Cerebral cortex.

[79]  Ad Aertsen,et al.  A modeler's view on the spatial structure of intrinsic horizontal connectivity in the neocortex , 2010, Progress in Neurobiology.

[80]  M. Mattia,et al.  Population dynamics of interacting spiking neurons. , 2002, Physical review. E, Statistical, nonlinear, and soft matter physics.

[81]  Awad H. Al-Mohy,et al.  A New Scaling and Squaring Algorithm for the Matrix Exponential , 2009, SIAM J. Matrix Anal. Appl..

[82]  Marc-Oliver Gewaltig,et al.  Dynamics of self-sustained asynchronous-irregular activity in random networks of spiking neurons with strong synapses , 2014, Front. Comput. Neurosci..

[83]  Marc de Kamps,et al.  A Generic Approach to Solving Jump Diffusion Equations with Applications to Neural Populations , 2013, 1309.1654.

[84]  Moritz Helias,et al.  Structural Plasticity Controlled by Calcium Based Correlation Detection , 2008, Frontiers Comput. Neurosci..

[85]  Moritz Helias,et al.  Decorrelation of Neural-Network Activity by Inhibitory Feedback , 2012, PLoS Comput. Biol..

[86]  A. Grinvald,et al.  Relationship between intrinsic connections and functional architecture revealed by optical imaging and in vivo targeted biocytin injections in primate striate cortex. , 1993, Proceedings of the National Academy of Sciences of the United States of America.

[87]  Moritz Helias,et al.  The correlation structure of local cortical networks intrinsically results from recurrent dynamics , 2013 .

[88]  H. Kennedy,et al.  A Large-Scale Circuit Mechanism for Hierarchical Dynamical Processing in the Primate Cortex , 2015, Neuron.

[89]  Nicolas Brunel,et al.  Dynamics of Sparsely Connected Networks of Excitatory and Inhibitory Spiking Neurons , 2000, Journal of Computational Neuroscience.

[90]  Michael A. Buice,et al.  Systematic Fluctuation Expansion for Neural Network Activity Equations , 2009, Neural Computation.

[91]  Tobias C. Potjans,et al.  The Cell-Type Specific Cortical Microcircuit: Relating Structure and Activity in a Full-Scale Spiking Network Model , 2012, Cerebral cortex.

[92]  Sommers,et al.  Chaos in random neural networks. , 1988, Physical review letters.

[93]  Nicolas Brunel,et al.  From Spiking Neuron Models to Linear-Nonlinear Models , 2011, PLoS Comput. Biol..

[94]  Iyabo Ann Adamu Numerical approximation of SDEs and stochastic Swift-Hohenberg equation , 2011 .

[95]  A. Destexhe,et al.  Impact of network activity on the integrative properties of neocortical pyramidal neurons in vivo. , 1999, Journal of neurophysiology.

[96]  Nicolas Brunel,et al.  Rate Models with Delays and the Dynamics of Large Networks of Spiking Neurons(Oscillation, Chaos and Network Dynamics in Nonlinear Science) , 2006 .

[97]  G. Deco,et al.  Ongoing Cortical Activity at Rest: Criticality, Multistability, and Ghost Attractors , 2012, The Journal of Neuroscience.

[98]  Jonathan D. Cohen,et al.  The physics of optimal decision making: a formal analysis of models of performance in two-alternative forced-choice tasks. , 2006, Psychological review.

[99]  J. Cowan,et al.  A mathematical theory of visual hallucination patterns , 1979, Biological Cybernetics.

[100]  J. Cowan,et al.  Field-theoretic approach to fluctuation effects in neural networks. , 2007, Physical review. E, Statistical, nonlinear, and soft matter physics.

[101]  Maurizio Mattia,et al.  Finite-size dynamics of inhibitory and excitatory interacting spiking neurons. , 2004, Physical review. E, Statistical, nonlinear, and soft matter physics.

[102]  L. F. Abbott,et al.  Building functional networks of spiking model neurons , 2016, Nature Neuroscience.

[103]  Bruce W. Knight,et al.  Dynamics of Encoding in a Population of Neurons , 1972, The Journal of general physiology.

[104]  Gregor Schöner,et al.  Dynamic Thinking : A Primer on Dynamic Field Theory , 2015 .

[105]  Marius Usher,et al.  Extending a biologically inspired model of choice: multi-alternatives, nonlinearity and value-based multidimensional choice , 2007, Philosophical Transactions of the Royal Society B: Biological Sciences.

[106]  S. Sharma,et al.  The Fokker-Planck Equation , 2010 .

[107]  S. Amari Dynamics of pattern formation in lateral-inhibition type neural fields , 1977, Biological Cybernetics.

[108]  Fumiyoshi Shoji,et al.  Overview of the K computer System , 2012 .

[109]  Isao Shoji A note on convergence rate of a linearization method for the discretization of stochastic differential equations , 2011 .

[110]  James L. McClelland,et al.  An interactive activation model of context effects in letter perception: I. An account of basic findings. , 1981 .

[111]  Jean-Luc R Stevens,et al.  Mechanisms for Stable, Robust, and Adaptive Development of Orientation Maps in the Primary Visual Cortex , 2013, The Journal of Neuroscience.

[112]  Moritz Helias,et al.  Noise dynamically suppresses chaos in neural networks , 2016 .

[113]  Lorenz Mösenlechner,et al.  The state of MIIND , 2008 .

[114]  Moritz Helias,et al.  The Correlation Structure of Local Neuronal Networks Intrinsically Results from Recurrent Dynamics , 2013, PLoS Comput. Biol..

[115]  Tomoki Fukai,et al.  Spiking network simulation code for petascale computers , 2014, Front. Neuroinform..

[116]  M. Amiri,et al.  Functional contributions of astrocytes in synchronization of a neuronal network model. , 2012, Journal of theoretical biology.

[117]  Moritz Helias,et al.  Correlated fluctuations in strongly-coupled binary networks beyond equilibrium , 2015, 1512.01073.

[118]  Anders Krogh,et al.  Introduction to the theory of neural computation , 1994, The advanced book program.

[119]  Moritz Helias,et al.  Scalability of Asynchronous Networks Is Limited by One-to-One Mapping between Effective Connectivity and Correlations , 2014, PLoS Comput. Biol..

[120]  Andrzej Cichocki,et al.  Nonnegative Matrix and Tensor Factorization T , 2007 .

[121]  Boris S. Gutkin,et al.  Multiple Bumps in a Neuronal Model of Working Memory , 2002, SIAM J. Appl. Math..

[122]  Wulfram Gerstner,et al.  Towards a theory of cortical columns: From spiking neurons to interacting neural populations of finite size , 2016, PLoS Comput. Biol..

[123]  Brent Doiron,et al.  Theory of oscillatory firing induced by spatially correlated noise and delayed inhibitory feedback. , 2005, Physical review. E, Statistical, nonlinear, and soft matter physics.

[124]  W. Gerstner,et al.  Time structure of the activity in neural network models. , 1995, Physical review. E, Statistical physics, plasmas, fluids, and related interdisciplinary topics.

[125]  H. Sompolinsky,et al.  13 Modeling Feature Selectivity in Local Cortical Circuits , 2022 .

[126]  Stephen Coombes,et al.  Waves, bumps, and patterns in neural field theories , 2005, Biological Cybernetics.

[127]  D. Hansel,et al.  Role of delays in shaping spatiotemporal dynamics of neuronal activity in large networks. , 2005, Physical review letters.

[128]  T. Faniran Numerical Solution of Stochastic Differential Equations , 2015 .

[129]  B L McNaughton,et al.  Path Integration and Cognitive Mapping in a Continuous Attractor Neural Network Model , 1997, The Journal of Neuroscience.

[130]  J. Cowan,et al.  Excitatory and inhibitory interactions in localized populations of model neurons. , 1972, Biophysical journal.

[131]  Eric Shea-Brown,et al.  Impact of Network Structure and Cellular Response on Spike Time Correlations , 2011, PLoS Comput. Biol..

[132]  Moritz Helias,et al.  A General and Efficient Method for Incorporating Precise Spike Times in Globally Time-Driven Simulations , 2010, Front. Neuroinform..

[133]  Alexander Peyser,et al.  NEST 2.10.0 , 2015 .

[134]  Nicholas Cain,et al.  The computational properties of a simplified cortical column model , 2014, BMC Neuroscience.

[135]  Simon Haykin,et al.  Neural Networks and Learning Machines , 2010 .

[136]  Moritz Helias,et al.  Modulated escape from a metastable state driven by colored noise. , 2014, Physical review. E, Statistical, nonlinear, and soft matter physics.

[137]  Alberto L. Sangiovanni-Vincentelli,et al.  The Waveform Relaxation Method for Time-Domain Analysis of Large Scale Integrated Circuits , 1982, IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems.

[138]  Philip Heidelberger,et al.  Optimization of applications with non-blocking neighborhood collectives via multisends on the Blue Gene/P supercomputer , 2010, 2010 IEEE International Symposium on Parallel & Distributed Processing (IPDPS).

[139]  Michael L. Hines,et al.  The NEURON Book , 2006 .

[140]  Moritz Helias,et al.  A unified framework for spiking and gap-junction interactions in distributed neuronal network simulations , 2015, Front. Neuroinform..

[141]  Örjan Ekeberg,et al.  Brain-scale simulation of the neocortex on the IBM Blue Gene/L supercomputer , 2008, IBM J. Res. Dev..

[142]  H Sompolinsky,et al.  Dynamics of random neural networks with bistable units. , 2014, Physical review. E, Statistical, nonlinear, and soft matter physics.

[143]  James L. McClelland,et al.  The time course of perceptual choice: the leaky, competing accumulator model. , 2001, Psychological review.

[144]  Axel Hutt,et al.  Neural field simulator: two-dimensional spatio-temporal dynamics involving finite transmission speed , 2015, Front. Neuroinform..

[145]  F. Ashby,et al.  Computational Cognitive Neuroscience , 2017 .

[146]  Jérémy Fix,et al.  DANA: Distributed numerical and adaptive modelling framework , 2012, Network.

[147]  P. Dayan,et al.  Supporting Online Material Materials and Methods Som Text Figs. S1 to S9 References the Asynchronous State in Cortical Circuits , 2022 .

[148]  Paul C. Bressloff,et al.  Path-Integral Methods for Analyzing the Effects of Fluctuations in Stochastic Hybrid Neural Networks , 2015, The Journal of Mathematical Neuroscience (JMN).

[149]  Nicolas Brunel,et al.  Fast Global Oscillations in Networks of Integrate-and-Fire Neurons with Low Firing Rates , 1999, Neural Computation.

[150]  Örjan Ekeberg,et al.  Run-Time Interoperability Between Neuronal Network Simulators Based on the MUSIC Framework , 2010, Neuroinformatics.

[151]  K. R. Schneider,et al.  WAVEFORM RELAXATION METHODS FOR STOCHASTIC DIFFERENTIAL EQUATIONS , 2005 .

[152]  H. Risken Fokker-Planck Equation , 1996 .

[153]  Moritz Helias,et al.  Echoes in correlated neural systems , 2012, 1207.0298.

[154]  Xiao-Jing Wang,et al.  A Model of Visuospatial Working Memory in Prefrontal Cortex: Recurrent Network and Cellular Bistability , 1998, Journal of Computational Neuroscience.

[155]  J. Cowan,et al.  A mathematical theory of the functional dynamics of cortical and thalamic nervous tissue , 1973, Kybernetik.

[156]  Jerome A. Feldman,et al.  Connectionist Models and Their Properties , 1982, Cogn. Sci..