Data-Driven Approaches to Understanding Visual Neuron Activity.

With modern neurophysiological methods able to record neural activity throughout the visual pathway in the context of arbitrarily complex visual stimulation, our understanding of visual system function is becoming limited by the available models of visual neurons that can be directly related to such data. Different forms of statistical models are now being used to probe the cellular and circuit mechanisms shaping neural activity, understand how neural selectivity to complex visual features is computed, and derive the ways in which neurons contribute to systems-level visual processing. However, models that are able to more accurately reproduce observed neural activity often defy simple interpretations. As a result, rather than being used solely to connect with existing theories of visual processing, statistical modeling will increasingly drive the evolution of more sophisticated theories. Expected final online publication date for the Annual Review of Vision Science, Volume 5 is September 16, 2019. Please see http://www.annualreviews.org/page/journal/pubdates for revised estimates.

[1]  Bartlett W. Mel,et al.  Pyramidal Neuron as Two-Layer Neural Network , 2003, Neuron.

[2]  Nicholas A. Steinmetz,et al.  Diverse coupling of neurons to populations in sensory cortex , 2015, Nature.

[3]  Liam Paninski,et al.  Automating the design of informative sequences of sensory stimuli , 2011, Journal of Computational Neuroscience.

[4]  E J Chichilnisky,et al.  Prediction and Decoding of Retinal Ganglion Cell Responses with a Probabilistic Spiking Model , 2005, The Journal of Neuroscience.

[5]  Robert Shapley,et al.  Linear and nonlinear systems analysis of the visual system: Why does it seem so linear? A review dedicated to the memory of Henk Spekreijse , 2009, Vision Research.

[6]  Christopher C. Pack,et al.  Hierarchical processing of complex motion along the primate dorsal visual pathway , 2012, Proceedings of the National Academy of Sciences.

[7]  Tian-De Shou 寿天德,et al.  The functional roles of feedback projections in the visual system , 2010, Neuroscience Bulletin.

[8]  Michael J. Berry,et al.  Refractoriness and Neural Precision , 1997, The Journal of Neuroscience.

[9]  E. Adelson,et al.  Directionally selective complex cells and the computation of motion energy in cat visual cortex , 1992, Vision Research.

[10]  Nicholas J Priebe,et al.  The accuracy of membrane potential reconstruction based on spiking receptive fields. , 2012, Journal of neurophysiology.

[11]  M. Carandini,et al.  Normalization as a canonical neural computation , 2011, Nature Reviews Neuroscience.

[12]  Fred Rieke,et al.  Synaptic Rectification Controls Nonlinear Spatial Integration of Natural Visual Inputs , 2016, Neuron.

[13]  Yoshua. Bengio,et al.  Learning Deep Architectures for AI , 2007, Found. Trends Mach. Learn..

[14]  Maneesh Sahani,et al.  Input-Specific Gain Modulation by Local Sensory Context Shapes Cortical and Thalamic Responses to Complex Sounds , 2016, Neuron.

[15]  R. Shapley,et al.  The nonlinear pathway of Y ganglion cells in the cat retina , 1979, The Journal of general physiology.

[16]  Fred Rieke,et al.  The spatial structure of a nonlinear receptive field , 2012, Nature Neuroscience.

[17]  James G. King,et al.  Reconstruction and Simulation of Neocortical Microcircuitry , 2015, Cell.

[18]  Ryan J. Prenger,et al.  Bayesian Reconstruction of Natural Images from Human Brain Activity , 2009, Neuron.

[19]  Matthew R Whiteway,et al.  The quest for interpretable models of neural population activity , 2019, Current Opinion in Neurobiology.

[20]  Sarah M. N. Woolley,et al.  A Generalized Linear Model for Estimating Spectrotemporal Receptive Fields from Responses to Natural Sounds , 2011, PloS one.

[21]  James J. DiCarlo,et al.  Evidence that recurrent circuits are critical to the ventral stream’s execution of core object recognition behavior , 2018, Nature Neuroscience.

[22]  Frances S. Chance,et al.  Gain Modulation from Background Synaptic Input , 2002, Neuron.

[23]  Yuan Yu,et al.  TensorFlow: A system for large-scale machine learning , 2016, OSDI.

[24]  Yuwei Cui,et al.  Diverse Suppressive Influences in Area MT and Selectivity to Complex Motion Features , 2013, The Journal of Neuroscience.

[25]  Tim Gollisch,et al.  Rapid Neural Coding in the Retina with Relative Spike Latencies , 2008, Science.

[26]  J. Touryan,et al.  Isolation of Relevant Visual Features from Random Stimuli for Cortical Complex Cells , 2002, The Journal of Neuroscience.

[27]  R. Shapley,et al.  The use of m-sequences in the analysis of visual neurons: Linear receptive field properties , 1997, Visual Neuroscience.

[28]  Stefano Panzeri,et al.  Inference of neuronal functional circuitry with spike-triggered non-negative matrix factorization , 2017, Nature Communications.

[29]  Chethan Pandarinath,et al.  Inferring single-trial neural population dynamics using sequential auto-encoders , 2017, Nature Methods.

[30]  Maneesh Sahani,et al.  Equating information-theoretic and likelihood-based methods for neural dimensionality reduction , 2013 .

[31]  Eero P. Simoncelli,et al.  Maximum Likelihood Estimation of a Stochastic Integrate-and-Fire Neural Encoding Model , 2004, Neural Computation.

[32]  William F. Kindel,et al.  Using deep learning to reveal the neural code for images in primary visual cortex , 2017, ArXiv.

[33]  M. Carandini,et al.  Functional Mechanisms Shaping Lateral Geniculate Responses to Artificial and Natural Stimuli , 2008, Neuron.

[34]  David E. Whitney,et al.  Orientation selectivity and the functional clustering of synaptic inputs in primary visual cortex , 2016, Nature Neuroscience.

[35]  George Cybenko,et al.  Approximation by superpositions of a sigmoidal function , 1989, Math. Control. Signals Syst..

[36]  David A. Leopold,et al.  Functional MRI mapping of dynamic visual features during natural viewing in the macaque , 2015, NeuroImage.

[37]  Yuwei Cui,et al.  Inferring Nonlinear Neuronal Computation Based on Physiologically Plausible Inputs , 2013, PLoS Comput. Biol..

[38]  M. Bethge,et al.  Inhibition decorrelates visual feature representations in the inner retina , 2017, Nature.

[39]  Eero P. Simoncelli,et al.  Mapping nonlinear receptive field structure in primate retina at single cone resolution , 2015, eLife.

[40]  M. Meister,et al.  Fast and Slow Contrast Adaptation in Retinal Circuitry , 2002, Neuron.

[41]  Qing Shi,et al.  Functional characterization of retinal ganglion cells using tailored nonlinear modeling , 2018 .

[42]  Tomaso Poggio,et al.  From Understanding Computation to Understanding Neural Circuitry , 1976 .

[43]  Haluk Öğmen,et al.  Feedforward and feedback processes in vision , 2015, Front. Psychol..

[44]  James A. Bednar,et al.  Model Constrained by Visual Hierarchy Improves Prediction of Neural Responses to Natural Scenes , 2016, PLoS Comput. Biol..

[45]  J. Gallant,et al.  Natural Stimulus Statistics Alter the Receptive Field Structure of V1 Neurons , 2004, The Journal of Neuroscience.

[46]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[47]  R. Reid,et al.  Predicting Every Spike A Model for the Responses of Visual Neurons , 2001, Neuron.

[48]  Bruce G. Cumming,et al.  High-resolution eye tracking using V1 neuron activity , 2014, Nature Communications.

[49]  Nicholas A. Roy,et al.  A comparison of deep learning and linear-nonlinear cascade approaches to neural encoding , 2018 .

[50]  Jonathan W. Pillow,et al.  Spectral methods for neural characterization using generalized quadratic models , 2013, NIPS.

[51]  Nicholas A. Steinmetz,et al.  Spontaneous behaviors drive multidimensional, brain-wide activity , 2018, bioRxiv.

[52]  Rufin van Rullen,et al.  Rate Coding Versus Temporal Order Coding: What the Retinal Ganglion Cells Tell the Visual Cortex , 2001, Neural Computation.

[53]  L. Paninski,et al.  Temporal Precision in the Visual Pathway through the Interplay of Excitation and Stimulus- Driven Suppression , 2022 .

[54]  E H Adelson,et al.  Spatiotemporal energy models for the perception of motion. , 1985, Journal of the Optical Society of America. A, Optics and image science.

[55]  J. Touryan,et al.  Spatial Structure of Complex Cell Receptive Fields Measured with Natural Images , 2005, Neuron.

[56]  Ha Hong,et al.  Explicit information for category-orthogonal object properties increases along the ventral stream , 2016, Nature Neuroscience.

[57]  D. Hubel,et al.  Receptive fields, binocular interaction and functional architecture in the cat's visual cortex , 1962, The Journal of physiology.

[58]  J. Gallant,et al.  A Three-Dimensional Spatiotemporal Receptive Field Model Explains Responses of Area MT Neurons to Naturalistic Movies , 2011, The Journal of Neuroscience.

[59]  Surya Ganguli,et al.  Deep learning models reveal internal structure and diverse computations in the retina under natural scenes , 2018, bioRxiv.

[60]  William Bialek,et al.  Analyzing Neural Responses to Natural Signals: Maximally Informative Dimensions , 2002, Neural Computation.

[61]  Jitendra Malik,et al.  Pixels to Voxels: Modeling Visual Representation in the Human Brain , 2014, ArXiv.

[62]  Sergey L. Gratiy,et al.  Fully integrated silicon probes for high-density recording of neural activity , 2017, Nature.

[63]  Curtis L Baker,et al.  Natural versus Synthetic Stimuli for Estimating Receptive Field Models: A Comparison of Predictive Robustness , 2012, The Journal of Neuroscience.

[64]  Anthony J. Movshon,et al.  Visual Response Properties of Striate Cortical Neurons Projecting to Area MT in Macaque Monkeys , 1996, The Journal of Neuroscience.

[65]  Daniel F. Bossut,et al.  IMPLICATION OF NEURAL NETWORKS FOR HOW WE THINK ABOUT BRAIN FUNCTION , 1992 .

[66]  Olivier Marre,et al.  Features and functions of nonlinear spatial integration by retinal ganglion cells , 2012, Journal of Physiology-Paris.

[67]  Kurt Hornik,et al.  Approximation capabilities of multilayer feedforward networks , 1991, Neural Networks.

[68]  Ha Hong,et al.  Performance-optimized hierarchical models predict neural responses in higher visual cortex , 2014, Proceedings of the National Academy of Sciences.

[69]  William Bialek,et al.  Real-time performance of a movement-sensitive neuron in the blowfly visual system: coding and information transfer in short spike sequences , 1988, Proceedings of the Royal Society of London. Series B. Biological Sciences.

[70]  G D Lewen,et al.  Reproducibility and Variability in Neural Spike Trains , 1997, Science.

[71]  Doris Y. Tsao,et al.  The Code for Facial Identity in the Primate Brain , 2017, Cell.

[72]  Nicole C. Rust,et al.  Do We Know What the Early Visual System Does? , 2005, The Journal of Neuroscience.

[73]  Michael J. Berry,et al.  Selectivity for multiple stimulus features in retinal ganglion cells. , 2006, Journal of neurophysiology.

[74]  L. Optican,et al.  Temporal encoding of two-dimensional patterns by single units in primate inferior temporal cortex. III. Information theoretic analysis. , 1987, Journal of neurophysiology.

[75]  E. Chichilnisky,et al.  Precision of spike trains in primate retinal ganglion cells. , 2004, Journal of neurophysiology.

[76]  Mijung Park,et al.  Bayesian inference for low rank spatiotemporal neural receptive fields , 2013, NIPS.

[77]  David J. Field,et al.  How Close Are We to Understanding V1? , 2005, Neural Computation.

[78]  James J DiCarlo,et al.  Neural population control via deep image synthesis , 2018, Science.

[79]  L. Paninski.  Maximum likelihood estimation of cascade point-process neural encoding models , 2004, Network.

[80]  Tomaso A. Poggio,et al.  A Canonical Neural Circuit for Cortical Nonlinear Operations , 2008, Neural Computation.

[81]  Surya Ganguli,et al.  On simplicity and complexity in the brave new world of large-scale neuroscience , 2015, Current Opinion in Neurobiology.

[82]  J. DiCarlo,et al.  Using goal-driven deep learning models to understand sensory cortex , 2016, Nature Neuroscience.

[83]  Tatyana O Sharpee,et al.  Computational identification of receptive fields. , 2013, Annual review of neuroscience.

[84]  Bruce G Cumming,et al.  Decision-related activity in sensory neurons: correlations among neurons and with behavior. , 2012, Annual review of neuroscience.

[85]  Mark S. Cembrowski,et al.  A Synaptic Mechanism for Retinal Adaptation to Luminance and Contrast , 2011, The Journal of Neuroscience.

[86]  Konrad P Kording,et al.  How advances in neural recording affect data analysis , 2011, Nature Neuroscience.

[87]  Matthew T. Kaufman,et al.  Movement-related activity dominates cortex during sensory-guided decision making , 2018, bioRxiv.

[88]  Tomaso A. Poggio,et al.  Bridging the Gaps Between Residual Learning, Recurrent Neural Networks and Visual Cortex , 2016, ArXiv.

[89]  J. Gallant,et al.  Predicting neuronal responses during natural vision , 2005, Network.

[90]  Eero P. Simoncelli,et al.  Spatiotemporal Elements of Macaque V1 Receptive Fields , 2005, Neuron.

[91]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[92]  L .Paninski,et al.  Neural data science: accelerating the experiment-analysis-theory cycle in large-scale neuroscience , 2017, Current Opinion in Neurobiology.

[93]  J. Gallant,et al.  Complete functional characterization of sensory neurons by system identification. , 2006, Annual review of neuroscience.

[94]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[95]  Nikolaus Kriegeskorte,et al.  Deep neural networks: a new framework for modelling biological vision and brain information processing , 2015, bioRxiv.

[96]  Guy A Orban,et al.  Higher order visual processing in macaque extrastriate cortex. , 2008, Physiological reviews.

[97]  J. B. Demb,et al.  Divisive suppression explains high-precision firing and contrast adaptation in retinal ganglion cells , 2016, bioRxiv.

[98]  Konrad P. Körding,et al.  Modern Machine Learning as a Benchmark for Fitting Neural Responses , 2018, Front. Comput. Neurosci..

[99]  Yves Frégnac,et al.  Adaptation of the simple or complex nature of V1 receptive fields to visual statistics , 2011, Nature Neuroscience.

[100]  Bruce Cumming,et al.  Disparity processing in primary visual cortex , 2016, Philosophical Transactions of the Royal Society B: Biological Sciences.

[101]  Nicole C. Rust,et al.  In praise of artifice , 2005, Nature Neuroscience.

[102]  R. Shapley,et al.  The effect of contrast on the transfer properties of cat retinal ganglion cells. , 1978, The Journal of physiology.

[103]  Brian Lau,et al.  Computational subunits of visual cortical neurons revealed by artificial neural networks , 2002, Proceedings of the National Academy of Sciences of the United States of America.

[104]  J. Movshon,et al.  Receptive field organization of complex cells in the cat's striate cortex. , 1978, The Journal of physiology.

[105]  O. Schwartz,et al.  Flexible Gating of Contextual Influences in Natural Vision , 2015, Nature Neuroscience.

[106]  Carlos R. Ponce,et al.  Evolving Images for Visual Neurons Using a Deep Generative Network Reveals Coding Principles and Neuronal Preferences , 2019, Cell.

[107]  Thomas Serre,et al.  Deep Learning: The Good, the Bad, and the Ugly. , 2019, Annual review of vision science.

[108]  Yves Frégnac,et al.  Hidden Complexity of Synaptic Receptive Fields in Cat V1 , 2014, The Journal of Neuroscience.

[109]  Eero P. Simoncelli,et al.  Attention stabilizes the shared gain of V4 populations , 2015, eLife.

[110]  Lawrence D. Jackel,et al.  Backpropagation Applied to Handwritten Zip Code Recognition , 1989, Neural Computation.

[111]  G C DeAngelis,et al.  The physiology of stereopsis. , 2001, Annual review of neuroscience.

[112]  Yuwei Cui,et al.  Nonlinear computations shaping temporal processing of precortical vision. , 2016, Journal of neurophysiology.

[113]  Eero P. Simoncelli,et al.  Dimensionality reduction in neural models: an information-theoretic generalization of spike-triggered average and covariance analysis. , 2006, Journal of vision.

[114]  Shawn R. Olsen,et al.  Lateral presynaptic inhibition mediates gain control in an olfactory circuit , 2008, Nature.

[115]  Uri T Eden,et al.  A point process framework for relating neural spiking activity to spiking history, neural ensemble, and extrinsic covariate effects. , 2005, Journal of neurophysiology.

[116]  Timothy J. Blanche,et al.  Construction of Direction Selectivity through Local Energy Computations in Primary Visual Cortex , 2013, PloS one.

[117]  Il Memming Park,et al.  Functional dissection of signal and noise in MT and LIP during decision-making , 2017, Nature Neuroscience.

[118]  Liam Paninski,et al.  Multilayer Recurrent Network Models of Primate Retinal Ganglion Cell Responses , 2016, ICLR.

[119]  Ming Li,et al.  Convolutional neural network models of V1 responses to complex patterns , 2018, Journal of Computational Neuroscience.

[120]  Eero P. Simoncelli,et al.  To appear in: The New Cognitive Neurosciences, 3rd edition Editor: M. Gazzaniga. MIT Press, 2004. Characterization of Neural Responses with Stochastic Stimuli , 2022 .

[121]  Liam Paninski,et al.  Statistical models for neural encoding, decoding, and optimal stimulus design. , 2007, Progress in brain research.

[122]  Tai Sing Lee,et al.  Accounting for network effects in neuronal responses using L1 regularized point process models , 2010, NIPS.

[123]  Radford M. Neal.  Pattern Recognition and Machine Learning , 2007, Technometrics.

[124]  Martin Vinck,et al.  Arousal and Locomotion Make Distinct Contributions to Cortical Activity Patterns and Visual Encoding , 2014, Neuron.

[125]  Jian K. Liu,et al.  Spike-Triggered Covariance Analysis Reveals Phenomenological Diversity of Contrast Adaptation in the Retina , 2015, PLoS Comput. Biol..

[126]  Arthur Gretton,et al.  Inferring spike trains from local field potentials. , 2008, Journal of neurophysiology.

[127]  Maneesh Sahani,et al.  Evidence Optimization Techniques for Estimating Stimulus-Response Functions , 2002, NIPS.

[128]  Konrad P. Körding,et al.  Toward an Integration of Deep Learning and Neuroscience , 2016, bioRxiv.

[129]  Jürgen Schmidhuber,et al.  Deep learning in neural networks: An overview , 2014, Neural Networks.

[130]  Chun-I Yeh,et al.  Temporal precision in the neural code and the timescales of natural vision , 2007, Nature.

[131]  Y. Frégnac,et al.  The Role of Delayed Suppression in Slow and Fast Contrast Adaptation in V1 Simple Cells , 2013, The Journal of Neuroscience.

[132]  Yuwei Cui,et al.  Inferring Cortical Variability from Local Field Potentials , 2016, The Journal of Neuroscience.

[133]  Amy M. Ni,et al.  Learning and attention reveal a general relationship between population activity and behavior , 2018, Science.

[134]  James J. DiCarlo,et al.  How Does the Brain Solve Visual Object Recognition? , 2012, Neuron.

[135]  Leon A. Gatys,et al.  Deep convolutional models improve predictions of macaque V1 responses to natural images , 2019, PLoS Comput. Biol..

[136]  Surya Ganguli,et al.  Inferring hidden structure in multilayered neural circuits , 2017, bioRxiv.

[137]  J. Movshon,et al.  Spatial summation in the receptive fields of simple cells in the cat's striate cortex. , 1978, The Journal of physiology.

[138]  Daniel L. K. Yamins,et al.  Deep Neural Networks Rival the Representation of Primate IT Cortex for Core Visual Object Recognition , 2014, PLoS Comput. Biol..

[139]  Nikolaus Kriegeskorte,et al.  Deep Supervised, but Not Unsupervised, Models May Explain IT Cortical Representation , 2014, PLoS Comput. Biol..

[140]  Mehrdad Jazayeri,et al.  Navigating the Neural Space in Search of the Neural Code , 2017, Neuron.

[141]  Eero P. Simoncelli,et al.  How MT cells analyze the motion of visual patterns , 2006, Nature Neuroscience.

[142]  T. Sharpee,et al.  Estimating linear–nonlinear models using Rényi divergences , 2009, Network.

[143]  Eero P. Simoncelli,et al.  Spatio-temporal correlations and visual signalling in a complete neuronal population , 2008, Nature.

[144]  Il Memming Park,et al.  Bayesian Spike-Triggered Covariance Analysis , 2011, NIPS.

[145]  Eero P. Simoncelli,et al.  Spike-triggered neural characterization. , 2006, Journal of vision.

[146]  E J Chichilnisky,et al.  A simple white noise analysis of neuronal light responses , 2001, Network.

[147]  Thomas Serre,et al.  Hierarchical Models of the Visual System , 2014, Encyclopedia of Computational Neuroscience.

[148]  Alexander S. Ecker,et al.  Neural system identification for large populations separating "what" and "where" , 2017, NIPS.

[149]  Bruce G Cumming,et al.  Variability and Correlations in Primary Visual Cortical Neurons Driven by Fixational Eye Movements , 2016, The Journal of Neuroscience.

[150]  Kechen Zhang,et al.  Active Data Collection for Efficient Estimation and Comparison of Nonlinear Neural Models , 2011, Neural Computation.

[151]  Maneesh Sahani,et al.  Equating information-theoretic and likelihood-based methods for neural dimensionality reduction , 2013, 1308.3542.

[152]  Tim Gollisch,et al.  Modeling convergent ON and OFF pathways in the early visual system , 2008, Biological Cybernetics.

[153]  Adrian G Bondy,et al.  Feedback Determines the Structure of Correlated Variability in Primary Visual Cortex , 2016, Nature Neuroscience.

[154]  D. Leopold,et al.  Face-selective neurons maintain consistent visual responses across months , 2014, Proceedings of the National Academy of Sciences.

[155]  Jonathan W. Pillow,et al.  Capturing the Dynamical Repertoire of Single Neurons with Generalized Linear Models , 2016, Neural Computation.

[156]  S. Baccus,et al.  Linking the Computational Structure of Variance Adaptation to Biophysical Mechanisms , 2012, Neuron.

[157]  Nicholas A. Steinmetz,et al.  Spontaneous behaviors drive multidimensional, brainwide activity , 2019, Science.