Training deep neural density estimators to identify mechanistic models of neural dynamics

Mechanistic modeling in neuroscience aims to explain observed phenomena in terms of underlying causes. However, determining which model parameters agree with complex and stochastic neural data presents a significant challenge. We address this challenge with a machine learning tool which uses deep neural density estimators—trained using model simulations—to carry out Bayesian inference and retrieve the full space of parameters compatible with raw data or selected data features. Our method is scalable in parameters and data features and can rapidly analyze new data after initial training. We demonstrate the power and flexibility of our approach on receptive fields, ion channels, and Hodgkin–Huxley models. We also characterize the space of circuit configurations giving rise to rhythmic activity in the crustacean stomatogastric ganglion, and use these results to derive hypotheses for underlying compensation mechanisms. Our approach will help close the gap between data-driven and theory-driven models of neural dynamics.

[1]  A. Hodgkin,et al.  A quantitative description of membrane current and its application to conduction and excitation in nerve , 1952, The Journal of physiology.

[2]  J. B. Rosen The Gradient Projection Method for Nonlinear Programming. Part I. Linear Constraints , 1960 .

[3]  D. Rubin Bayesianly Justifiable and Relevant Frequency Calculations for the Applied Statistician , 1984 .

[4]  J. P. Jones,et al.  An evaluation of the two-dimensional Gabor filter model of simple receptive fields in cat striate cortex. , 1987, Journal of neurophysiology.

[5]  W. R. Foster,et al.  Significance of conductances in Hodgkin-Huxley models. , 1993, Journal of neurophysiology.

[6]  H. Sompolinsky,et al.  Chaos in Neuronal Networks with Balanced Excitatory and Inhibitory Activity , 1996, Science.

[7]  Giuseppe De Nicolao,et al.  Nonparametric input estimation in physiological systems: Problems, methods, and case studies , 1997, Autom..

[8]  E N Brown,et al.  A Statistical Paradigm for Neural Spike Train Decoding Applied to Position Prediction from Ensemble Firing Patterns of Rat Hippocampal Place Cells , 1998, The Journal of Neuroscience.

[9]  M. Feldman,et al.  Population growth of human Y chromosomes: a study of Y chromosome microsatellites. , 1999, Molecular biology and evolution.

[10]  E J Chichilnisky,et al.  A simple white noise analysis of neuronal light responses , 2001, Network.

[11]  E. Marder,et al.  Global Structure, Robustness, and Modulation of Neuronal Models , 2001, The Journal of Neuroscience.

[12]  D. Balding,et al.  Approximate Bayesian computation in population genetics. , 2002, Genetics.

[13]  E. Marder,et al.  Failure of averaging in the construction of a conductance-based neuron model. , 2002, Journal of neurophysiology.

[14]  Paul Marjoram,et al.  Markov chain Monte Carlo without likelihoods , 2003, Proceedings of the National Academy of Sciences of the United States of America.

[15]  Eve Marder,et al.  Alternative to hand-tuning conductance-based models: construction and analysis of databases of model neurons. , 2003, Journal of neurophysiology.

[16]  Marco Laumanns,et al.  PISA: A Platform and Programming Language Independent Interface for Search Algorithms , 2003, EMO.

[17]  Bruce R. Johnson,et al.  Activity-Independent Homeostasis in Rhythmically Active Neurons , 2003, Neuron.

[18]  L. Paninski Maximum likelihood estimation of cascade point-process neural encoding models , 2004, Network.

[19]  E. Marder,et al.  Similar network activity from disparate circuit parameters , 2004, Nature Neuroscience.

[20]  Hiroaki Kitano,et al.  Biological robustness , 2008, Nature Reviews Genetics.

[21]  Eckart Zitzler,et al.  Indicator-Based Selection in Multiobjective Search , 2004, PPSN.

[22]  Alain Destexhe,et al.  Nonlinear Thermodynamic Models of Voltage-Dependent Currents , 2000, Journal of Computational Neuroscience.

[23]  E J Chichilnisky,et al.  Prediction and Decoding of Retinal Ganglion Cell Responses with a Probabilistic Spiking Model , 2005, The Journal of Neuroscience.

[24]  Uri T Eden,et al.  A point process framework for relating neural spiking activity to spiking history, neural ensemble, and extrinsic covariate effects. , 2005, Journal of neurophysiology.

[25]  John Guckenheimer,et al.  Activity-independent coregulation of IA and Ih in rhythmically active neurons. , 2005, Journal of neurophysiology.

[26]  L. Abbott,et al.  Neural network dynamics. , 2005, Annual review of neuroscience.

[27]  Liam Paninski,et al.  Efficient estimation of detailed single-neuron models. , 2006, Journal of neurophysiology.

[28]  Erik De Schutter,et al.  Complex Parameter Landscape for a Complex Neuron Model , 2006, PLoS Comput. Biol..

[29]  Donald B. Rubin,et al.  Validation of Software for Bayesian Models Using Posterior Quantiles , 2006 .

[30]  Michael J. Berry,et al.  Weak pairwise correlations imply strongly correlated network states in a neural population , 2005, Nature.

[31]  E. Marder,et al.  Variability, compensation and homeostasis in neuron and network function , 2006, Nature Reviews Neuroscience.

[32]  Tim Gollisch,et al.  Modeling Single-Neuron Dynamics and Computations: A Balance of Detail and Abstraction , 2006, Science.

[33]  Eve Marder,et al.  Structure and visualization of high-dimensional conductance spaces. , 2006, Journal of neurophysiology.

[34]  Jonathan W. Pillow,et al.  Likelihood-based approaches to modeling the neural code , 2007 .

[35]  Henry Markram,et al.  A Novel Multiple Objective Optimization Framework for Constraining Conductance-Based Neuron Models by Experimental Data , 2007, Front. Neurosci..

[36]  Christopher R. Myers,et al.  Universally Sloppy Parameter Sensitivities in Systems Biology Models , 2007, PLoS Comput. Biol..

[37]  J. Gold,et al.  The neural basis of decision making. , 2007, Annual review of neuroscience.

[38]  Mark M. Tanaka,et al.  Sequential Monte Carlo without likelihoods , 2007, Proceedings of the National Academy of Sciences.

[39]  R. Peri,et al.  High-throughput electrophysiology: an emerging paradigm for ion-channel screening and physiology , 2008, Nature Reviews Drug Discovery.

[40]  Roger Ratcliff,et al.  The Diffusion Decision Model: Theory and Data for Two-Choice Decision Tasks , 2008, Neural Computation.

[41]  W. M. Keck,et al.  Highly Selective Receptive Fields in Mouse Visual Cortex , 2008, The Journal of Neuroscience.

[42]  Eero P. Simoncelli,et al.  Spatio-temporal correlations and visual signalling in a complete neuronal population , 2008, Nature.

[43]  Henry Markram,et al.  Minimal Hodgkin–Huxley type models for different classes of cortical and thalamic neurons , 2008, Biological Cybernetics.

[44]  John P. Cunningham,et al.  Gaussian-process factor analysis for low-dimensional single-trial analysis of neural population activity , 2008, NIPS.

[45]  Xiao-Jing Wang Decision Making in Recurrent Neuronal Circuits , 2008, Neuron.

[46]  Liam Paninski,et al.  Smoothing of, and Parameter Estimation from, Noisy Biophysical Recordings , 2009, PLoS Comput. Biol..

[47]  C. Robert,et al.  Adaptive approximate Bayesian computation , 2008, 0805.2256.

[48]  L. F. Abbott,et al.  Generating Coherent Patterns of Activity from Chaotic Neural Networks , 2009, Neuron.

[49]  E. Marder,et al.  How Multiple Conductances Determine Electrophysiological Properties in a Multicompartment Model , 2009, The Journal of Neuroscience.

[50]  S. Wood Statistical inference for noisy nonlinear ecological dynamic systems , 2010, Nature.

[51]  Matthias Bethge,et al.  Bayesian Inference for Generalized Linear Models for Spiking Neurons , 2010, Front. Comput. Neurosci..

[52]  Olivier François,et al.  Non-linear regression models for Approximate Bayesian Computation , 2008, Stat. Comput..

[53]  E. Marder,et al.  Compensation for Variable Intrinsic Neuronal Excitability by Circuit-Synaptic Interactions , 2010, The Journal of Neuroscience.

[54]  Wulfram Gerstner,et al.  The influence of structure on the response properties of biologically plausible neural network models , 2011, BMC Neuroscience.

[55]  John P. Cunningham,et al.  Empirical models of spiking in neural populations , 2011, NIPS.

[56]  Bertrand Fontaine,et al.  Fitting Neuron Models to Spike Trains , 2011, Front. Neurosci..

[57]  E. Marder,et al.  Multiple models to capture the variability in biological neurons and networks , 2011, Nature Neuroscience.

[58]  E. Marder Variability, compensation, and modulation in neurons and circuits , 2011, Proceedings of the National Academy of Sciences.

[59]  Henry Markram,et al.  Models of Neocortical Layer 5b Pyramidal Cells Capturing a Wide Range of Dendritic and Perisomatic Active Properties , 2011, PLoS Comput. Biol..

[60]  Nicolas Chopin,et al.  Expectation Propagation for Likelihood-Free Inference , 2011, 1107.5959.

[61]  A. Litwin-Kumar,et al.  Slow dynamics and high variability in balanced cortical networks with clustered connections , 2012, Nature Neuroscience.

[62]  Wulfram Gerstner,et al.  Theory and Simulation in Neuroscience , 2012, Science.

[63]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[64]  James G. Scott,et al.  Fully Bayesian inference for neural models with negative-binomial spiking , 2012, NIPS.

[65]  Joachim Haß,et al.  An Approximation to the Adaptive Exponential Integrate-and-Fire Neuron Model Allows Fast and Predictive Fitting to Physiological Data , 2012, Front. Comput. Neurosci..

[66]  James G. Scott,et al.  Bayesian Inference for Logistic Models Using Pólya–Gamma Latent Variables , 2012, 1205.0310.

[67]  Michael L. Hines,et al.  Mitral cell spike synchrony modulated by dendrodendritic synapse location , 2012, Front. Comput. Neurosci..

[68]  Mark C. W. van Rossum,et al.  Probabilistic inference of short-term synaptic plasticity in neocortical microcircuits , 2013, Front. Comput. Neurosci..

[69]  E. Marder,et al.  Multiple Mechanisms Switch an Electrically Coupled, Synaptically Inhibited Neuron between Competing Rhythmic Oscillators , 2013, Neuron.

[70]  B. Rodríguez,et al.  Experimentally calibrated population of models predicts and explains intersubject variability in cardiac cellular electrophysiology , 2013, Proceedings of the National Academy of Sciences.

[71]  David Sussillo,et al.  Opening the Black Box: Low-Dimensional Dynamics in High-Dimensional Recurrent Neural Networks , 2013, Neural Computation.

[72]  Mark S. Goldman,et al.  A Modeling Framework for Deriving the Structural and Functional Architecture of a Short-Term Memory Microcircuit , 2013, Neuron.

[73]  W. Newsome,et al.  Context-dependent computation by recurrent dynamics in prefrontal cortex , 2013, Nature.

[74]  S. Sisson,et al.  A comparative review of dimension reduction methods in approximate Bayesian computation , 2012, 1202.3819.

[75]  J. Sethna,et al.  Parameter Space Compression Underlies Emergent Theories and Predictive Models , 2013, Science.

[76]  O. Sporns Contributions and challenges for network models in cognitive neuroscience , 2014, Nature Neuroscience.

[77]  Hao Huang,et al.  Estimating parameters and predicting membrane voltages with conductance-based neuron models , 2014, Biological Cybernetics.

[78]  Eve Marder,et al.  Many Parameter Sets in a Multicompartment Model Oscillator Are Robust to Temperature Perturbations , 2014, The Journal of Neuroscience.

[79]  Max Welling,et al.  GPS-ABC: Gaussian Process Surrogate Approximate Bayesian Computation , 2014, UAI.

[80]  Eve Marder,et al.  Cell types, network homeostasis, and pathological compensation from a biologically plausible ion channel expression model. , 2014, Neuron.

[81]  Tobias C. Potjans,et al.  The Cell-Type Specific Cortical Microcircuit: Relating Structure and Activity in a Full-Scale Spiking Network Model , 2012, Cerebral cortex.

[82]  Byron M. Yu,et al.  Dimensionality reduction for large-scale neural recordings , 2014, Nature Neuroscience.

[83]  Richard Wilkinson,et al.  Accelerating ABC methods using Gaussian processes , 2014, AISTATS.

[84]  Sarah Filippi,et al.  A framework for parameter estimation and model selection from experimental data in systems biology using approximate Bayesian computation , 2014, Nature Protocols.

[85]  Aidan C. Daly,et al.  Hodgkin–Huxley revisited: reparametrization and identifiability analysis of the classic action potential model with approximate Bayesian methods , 2015, Royal Society Open Science.

[86]  Shakir Mohamed,et al.  Variational Inference with Normalizing Flows , 2015, ICML.

[87]  Romain Brette,et al.  What Is the Most Realistic Single-Compartment Model of Spike Initiation? , 2015, PLoS Comput. Biol..

[88]  Eve Marder,et al.  Computational models in the age of large datasets , 2015, Current Opinion in Neurobiology.

[89]  E. Marder,et al.  Robust circuit rhythms in small circuits arise from variable circuit components and mechanisms , 2015, Current Opinion in Neurobiology.

[90]  Wulfram Gerstner,et al.  Automated High-Throughput Characterization of Single Neurons by Means of Simplified Spiking Models , 2015, PLoS Comput. Biol..

[91]  Daniel B. Rubin,et al.  The Stabilized Supralinear Network: A Unifying Circuit Motif Underlying Multi-Input Integration in Sensory Cortex , 2015, Neuron.

[92]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[93]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[94]  Allen Cell Types Database , 2016 .

[95]  Henry Markram,et al.  BluePyOpt: Leveraging Open Source Software and Cloud Infrastructure to Optimise Model Parameters in Neuroscience , 2016, Front. Neuroinform..

[96]  Eve Marder,et al.  Temperature-Robust Neural Function from Activity-Dependent Ion Channel Regulation , 2016, Current Biology.

[97]  Kenneth D Harris,et al.  Inhibitory control of correlated intrinsic variability in cortical networks , 2016, bioRxiv.

[98]  Iain Murray,et al.  Fast $\epsilon$-free Inference of Simulation Models with Bayesian Conditional Density Estimation , 2016, 1605.06376.

[99]  Michael U. Gutmann,et al.  Bayesian Optimization for Likelihood-Free Inference of Simulator-Based Statistical Models , 2015, J. Mach. Learn. Res..

[100]  Christof Koch,et al.  Generalized leaky integrate-and-fire models classify multiple neuron types , 2017, Nature Communications.

[101]  Christof Koch,et al.  Systematic generation of biophysically detailed models for diverse cortical neuron types , 2018, Nature Communications.

[102]  Prabhat,et al.  Improvements to Inference Compilation for Probabilistic Programming in Large-Scale Scientific Simulators , 2017, ArXiv.

[103]  Iain Murray,et al.  Masked Autoregressive Flow for Density Estimation , 2017, NIPS.

[104]  Frank D. Wood,et al.  Using synthetic data to train neural networks is model-based reasoning , 2017, 2017 International Joint Conference on Neural Networks (IJCNN).

[105]  Rajnish Ranjan,et al.  Mapping the function of neuronal ion channels in model and experiment , 2017, eLife.

[106]  John P. Cunningham,et al.  Maximum Entropy Flow Networks , 2017, ICLR.

[107]  Frank D. Wood,et al.  Inference Compilation and Universal Probabilistic Programming , 2016, AISTATS.

[108]  Jakob H. Macke,et al.  Fast amortized inference of neural activity from calcium imaging data with variational autoencoders , 2017, NIPS.

[109]  Jakob H. Macke,et al.  Flexible statistical inference for mechanistic models of neural dynamics , 2017, NIPS.

[110]  J. Gold,et al.  On the nature and use of models in network neuroscience , 2018, Nature Reviews Neuroscience.

[111]  Yun S. Song,et al.  A Likelihood-Free Inference Framework for Population Genetic Data using Exchangeable Neural Networks , 2018, bioRxiv.

[112]  Bai Jiang,et al.  Learning Summary Statistic for Approximate Bayesian Computation via Deep Neural Network , 2015, 1510.02175.

[113]  Jakob H. Macke,et al.  Likelihood-free inference with emulator networks , 2018, AABI.

[114]  Iain Murray,et al.  Sequential Neural Methods for Likelihood-free Inference , 2018, ArXiv.

[115]  Ann B. Lee,et al.  ABC–CDE: Toward Approximate Bayesian Computation With Complex High-Dimensional Data and Limited Simulations , 2018, Journal of Computational and Graphical Statistics.

[116]  Josef Ladenbauer,et al.  Inferring and validating mechanistic models of neural microcircuits based on spike-train data , 2018, Nature Communications.

[117]  Chethan Pandarinath,et al.  Inferring single-trial neural population dynamics using sequential auto-encoders , 2017, Nature Methods.

[118]  Yee Whye Teh,et al.  Faithful Inversion of Generative Models for Effective Amortized Inference , 2017, NeurIPS.

[119]  R. Baker,et al.  Mechanistic models versus machine learning, a fight worth fighting for the biological community? , 2018, Biology Letters.

[120]  Kevin Burrage,et al.  Unlocking data sets by calibrating populations of models to data density: A study in atrial electrophysiology , 2017, Science Advances.

[121]  Maria C. Dadarlat,et al.  Flow stimuli reveal ecologically appropriate responses in mouse visual cortex , 2018, Proceedings of the National Academy of Sciences.

[122]  Aki Vehtari,et al.  Validating Bayesian Inference Algorithms with Simulation-Based Calibration , 2018, 1804.06788.

[123]  Eve Marder,et al.  Circuit Robustness to Temperature Perturbation Is Altered by Neuromodulators , 2017, Neuron.

[124]  Timothy O'Leary,et al.  Homeostasis, failure of homeostasis and degenerate ion channel regulation , 2018 .

[125]  Eve Marder,et al.  Visualization of currents in neural models with similar behavior and different conductance densities , 2019, eLife.

[126]  Surya Ganguli,et al.  Reverse engineering recurrent networks for sentiment classification reveals line attractor dynamics , 2019, NeurIPS.

[127]  Kristofer E. Bouchard,et al.  Inferring neuronal ionic conductances from membrane potentials using CNNs , 2019, bioRxiv.

[128]  Philipp Berens,et al.  Approximate Bayesian Inference for a Mechanistic Model of Vesicle Release at a Ribbon Synapse , 2019, bioRxiv.

[129]  Henry Markram,et al.  A Kinetic Map of the Homomeric Voltage-Gated Potassium Channel (Kv) Family , 2019, Front. Cell. Neurosci..

[130]  Sean R. Bittner,et al.  Interrogating theoretical models of neural computation with deep inference , 2019, bioRxiv.

[131]  David S. Greenberg,et al.  Automatic Posterior Transformation for Likelihood-Free Inference , 2019, ICML.

[132]  Gilles Louppe,et al.  Likelihood-free MCMC with Approximate Likelihood Ratios , 2019, ArXiv.

[133]  Michael U. Gutmann,et al.  Efficient Bayesian Experimental Design for Implicit Models , 2018, AISTATS.

[134]  Iain Murray,et al.  Sequential Neural Likelihood: Fast Likelihood-free Inference with Autoregressive Flows , 2018, AISTATS.

[135]  Edward S. Boyden,et al.  Advances in the automation of whole-cell patch clamp technology , 2019, Journal of Neuroscience Methods.

[136]  Jonathan Oesterle,et al.  Bayesian inference for biophysical neuron models enables stimulus optimization for retinal neuroprosthetics , 2020, bioRxiv.

[137]  Inference of a Mesoscopic Population Model from Population Spike Trains , 2019, Neural Computation.

[138]  Iain Murray,et al.  On Contrastive Learning for Likelihood-free Inference , 2020, ICML.

[139]  David S. Greenberg,et al.  SBI - A toolkit for simulation-based inference , 2020, J. Open Source Softw..

[140]  Gilles Louppe,et al.  The frontier of simulation-based inference , 2019, Proceedings of the National Academy of Sciences.

[141]  Eric Nalisnick,et al.  Normalizing Flows for Probabilistic Modeling and Inference , 2019, J. Mach. Learn. Res..