Sensory cortex is optimized for prediction of future input

Neurons in sensory cortex are tuned to diverse features in natural scenes. But what determines which features neurons become selective to? Here we explore the idea that neuronal selectivity is optimized to represent features in the recent sensory past that best predict immediate future inputs. We tested this hypothesis using simple feedforward neural networks, which were trained to predict the next few moments of video or audio in clips of natural scenes. The networks developed receptive fields that closely matched those of real cortical neurons in different mammalian species, including the oriented spatial tuning of primary visual cortex, the frequency selectivity of primary auditory cortex and, most notably, their temporal tuning properties. Furthermore, the better a network predicted future inputs the more closely its receptive fields resembled those in the brain. This suggests that sensory processing is optimized to extract those features with the most capacity to predict future input.

[1]  K. Harris,et al.  Cortical connectivity and sensory coding , 2013, Nature.

[2]  Terrence J. Sejnowski,et al.  Slow Feature Analysis: Unsupervised Learning of Invariances , 2002, Neural Computation.

[3]  Yoshua Bengio,et al.  Towards Biologically Plausible Deep Learning , 2015, ArXiv.

[4]  S. Laughlin,et al.  Predictive coding: a fresh view of inhibition in the retina , 1982, Proceedings of the Royal Society of London. Series B. Biological Sciences.

[5]  Paul W. Frankland,et al.  The acoustic startle reflex: neurons and connections , 1995, Brain Research Reviews.

[6]  Colin J. Akerman,et al.  Random synaptic feedback weights support error backpropagation for deep learning , 2016, Nature Communications.

[7]  Neil C. Rabinowitz,et al.  Contrast Gain Control in Auditory Cortex , 2011, Neuron.

[8]  Kerry M. M. Walker,et al.  Interdependent Encoding of Pitch, Timbre, and Spatial Location in Auditory Cortex , 2009, The Journal of Neuroscience.

[9]  Michael J. Berry,et al.  Predictive information in a sensory population , 2013, Proceedings of the National Academy of Sciences.

[10]  Adrian Rees,et al.  Responses of neurons in the inferior colliculus of the rat to AM and FM tones , 1983, Hearing Research.

[11]  O. Marre,et al.  Toward a unified theory of efficient, predictive, and sparse coding , 2017, Proceedings of the National Academy of Sciences.

[12]  Gabriel Kreiman,et al.  Deep Predictive Coding Networks for Video Prediction and Unsupervised Learning , 2016, ICLR.

[13]  Marc'Aurelio Ranzato,et al.  Video (language) modeling: a baseline for generative models of natural videos , 2014, ArXiv.

[14]  N. R. Bartlett,et al.  Latency of the blink reflex and stimulus intensity J , 1967 .

[15]  Bruno A. Olshausen,et al.  Learning sparse, overcomplete representations of time-varying natural images , 2003, Proceedings 2003 International Conference on Image Processing (Cat. No.03CH37429).

[16]  J. P. Jones,et al.  An evaluation of the two-dimensional Gabor filter model of simple receptive fields in cat striate cortex. , 1987, Journal of neurophysiology.

[17]  Nicole L. Carlson,et al.  Sparse Codes for Speech Predict Spectrotemporal Receptive Fields in the Inferior Colliculus , 2012, PLoS Comput. Biol..

[18]  Mounya Elhilali,et al.  Sustained Firing of Model Central Auditory Neurons Yields a Discriminative Spectro-temporal Representation for Natural Sounds , 2013, PLoS Comput. Biol..

[19]  Konrad P. Körding,et al.  Sparse Spectrotemporal Coding of Sounds , 2003, EURASIP J. Adv. Signal Process..

[20]  Izumi Ohzawa,et al.  Joint-encoding of motion and depth by visual cortical neurons: neural basis of the Pulfrich effect , 2001, Nature Neuroscience.

[21]  E H Adelson,et al.  Spatiotemporal energy models for the perception of motion. , 1985, Journal of the Optical Society of America. A, Optics and image science.

[22]  N. R. Bartlett,et al.  Erratum to: Latency of the blink reflex and stimulus intensity , 1968 .

[23]  Geraint Rees,et al.  Predicting the stream of human consciousness , 2005 .

[24]  Rajesh P. N. Rao,et al.  An optimal estimation approach to visual perception and learning , 1999, Vision Research.

[25]  C. Baker,et al.  Linear filtering and nonlinear interactions in direction-selective visual cortex neurons: A noise correlation analysis , 2001, Visual Neuroscience.

[26]  Wiktor Mlynarski,et al.  Learning Midlevel Auditory Codes from Natural Sound Statistics , 2017, Neural Computation.

[27]  I. Ohzawa,et al.  Spatiotemporal organization of simple-cell receptive fields in the cat's striate cortex. II. Linearity of temporal and spatial summation. , 1993, Journal of neurophysiology.

[28]  Andrew J. King,et al.  Hearing and Auditory Function in Ferrets , 2014 .

[29]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[30]  David J. Field,et al.  Sparse coding with an overcomplete basis set: A strategy employed by V1? , 1997, Vision Research.

[31]  M. Feller,et al.  Mechanisms underlying development of visual maps and receptive fields. , 2008, Annual review of neuroscience.

[32]  Lynne Kiorpes,et al.  Visual development in primates: Neural mechanisms and critical periods , 2015, Developmental neurobiology.

[33]  Johannes C. Dahmen,et al.  Stimulus-Timing-Dependent Plasticity of Cortical Frequency Representation , 2008, The Journal of Neuroscience.

[34]  D J Field,et al.  Relations between the statistics of natural images and the response properties of cortical cells. , 1987, Journal of the Optical Society of America. A, Optics and image science.

[35]  Rajesh P. N. Rao,et al.  Predictive Coding , 2019, A Blueprint for the Hard Problem of Consciousness.

[36]  Romi Nijhawan,et al.  Motion extrapolation in catching , 1994, Nature.

[37]  B. Willmore,et al.  Incorporating Midbrain Adaptation to Mean Sound Level Improves Models of Auditory Cortical Processing , 2016, The Journal of Neuroscience.

[38]  Antonio Torralba,et al.  Statistics of natural image categories , 2003, Network.

[39]  Honglak Lee,et al.  Action-Conditional Video Prediction using Deep Networks in Atari Games , 2015, NIPS.

[40]  Wulfram Gerstner,et al.  Nonlinear Hebbian Learning as a Unifying Principle in Receptive Field Formation , 2016, PLoS Comput. Biol..

[41]  Richard E. Turner,et al.  A Structured Model of Video Reproduces Primary Visual Cortical Organisation , 2009, PLoS Comput. Biol..

[42]  Timothy Q Gentner,et al.  Central auditory neurons have composite receptive fields , 2016, Proceedings of the National Academy of Sciences.

[43]  M. Merzenich,et al.  Optimizing sound features for cortical neurons. , 1998, Science.

[44]  W. M. Keck,et al.  Highly Selective Receptive Fields in Mouse Visual Cortex , 2008, The Journal of Neuroscience.

[45]  D. Hubel,et al.  Receptive fields of single neurones in the cat's striate cortex , 1959, The Journal of physiology.

[46]  Rajesh P. N. Rao,et al.  Dynamic Model of Visual Recognition Predicts Neural Response Properties in the Visual Cortex , 1997, Neural Computation.

[47]  D Marr,et al.  Early processing of visual information. , 1976, Philosophical transactions of the Royal Society of London. Series B, Biological sciences.

[48]  Maneesh Sahani,et al.  How Linear are Auditory Cortical Responses? , 2002, NIPS.

[49]  Naftali Tishby,et al.  Predictability, Complexity, and Learning , 2000, Neural Computation.

[50]  Rhodri Cusack,et al.  Auditory Perceptual Organization Inside and Outside the Laboratory , 2004 .

[51]  Rajesh P. N. Rao,et al.  Predictive coding in the visual cortex: a functional interpretation of some extra-classical receptive-field effects. , 1999 .

[52]  Andrew J. King,et al.  Network Receptive Field Modeling Reveals Extensive Integration and Multi-feature Selectivity in Auditory Cortical Neurons , 2016, PLoS Comput. Biol..

[53]  R. Shapley,et al.  Linear mechanisms of directional selectivity in simple cells of cat striate cortex. , 1987, Proceedings of the National Academy of Sciences of the United States of America.

[54]  Aapo Hyvärinen,et al.  Bubbles: a unifying framework for low-level statistical properties of natural image sequences. , 2003, Journal of the Optical Society of America. A, Optics, image science, and vision.

[55]  Naftali Tishby,et al.  The Representation of Prediction Error in Auditory Cortex , 2016, PLoS Comput. Biol..

[56]  Nitish Srivastava,et al.  Unsupervised Learning of Video Representations using LSTMs , 2015, ICML.

[57]  Bevil R. Conway,et al.  Spatiotemporal Structure of Nonlinear Subunits in Macaque Visual Cortex , 2006, The Journal of Neuroscience.

[58]  Eero P. Simoncelli,et al.  To appear in: The New Cognitive Neurosciences, 3rd edition Editor: M. Gazzaniga. MIT Press, 2004. Characterization of Neural Responses with Stochastic Stimuli , 2022 .

[59]  Richard Hans Robert Hahnloser,et al.  An Efficient Coding Hypothesis Links Sparsity and Selectivity of Neural Responses , 2011, PloS one.

[60]  M. Sachs,et al.  Rate versus level functions for auditory-nerve fibers in cats: tone-burst stimuli. , 1974, The Journal of the Acoustical Society of America.

[61]  Felix Creutzig,et al.  Predictive Coding and the Slowness Principle: An Information-Theoretic Approach , 2008, Neural Computation.

[62]  D. Ruderman,et al.  Independent component analysis of natural image sequences yields spatio-temporal filters similar to simple cells in primary visual cortex , 1998, Proceedings of the Royal Society of London. Series B: Biological Sciences.

[63]  Johannes C. Dahmen,et al.  Learning to hear: plasticity of auditory cortical processing , 2007, Current Opinion in Neurobiology.

[64]  I. Nelken,et al.  Functional organization of ferret auditory cortex. , 2005, Cerebral cortex.

[65]  I. Ohzawa,et al.  Spatiotemporal organization of simple-cell receptive fields in the cat's striate cortex. I. General characteristics and postnatal development. , 1993, Journal of neurophysiology.

[66]  E J Chichilnisky,et al.  A simple white noise analysis of neuronal light responses , 2001, Network.

[67]  C. Atencio,et al.  Cooperative Nonlinearities in Auditory Cortical Neurons , 2008, Neuron.

[68]  Sarah Marzen,et al.  The evolution of lossy compression , 2015, Journal of The Royal Society Interface.

[69]  A. Aertsen,et al.  Spectro-temporal receptive fields of auditory neurons in the grassfrog , 1980, Biological Cybernetics.

[70]  J. H. Hateren,et al.  Independent component filters of natural images compared with simple cells in primary visual cortex , 1998 .

[71]  Bevil R. Conway,et al.  Contrast affects speed tuning, space-time slant, and receptive-field organization of simple cells in macaque V1. , 2007, Journal of neurophysiology.

[72]  Laurenz Wiskott,et al.  Slow feature analysis yields a rich repertoire of complex cell properties. , 2005, Journal of vision.

[73]  Michael S. Lewicki,et al.  Efficient auditory coding , 2006, Nature.

[74]  J. Kelly,et al.  Hearing in the ferret (Mustela putorius): effects of primary auditory cortical lesions on thresholds for pure tone detection. , 1988, Journal of neurophysiology.

[75]  Rasmus Berg Palm,et al.  Prediction as a candidate for learning deep hierarchical models of data , 2012 .

[76]  David J Heeger,et al.  Theory of cortical function , 2017, Proceedings of the National Academy of Sciences.

[77]  Stephanie E. Palmer,et al.  Optimal Prediction in the Retina and Natural Motion Statistics , 2016 .

[78]  Konrad P. Körding,et al.  Extracting Slow Subspaces from Natural Videos Leads to Complex Cells , 2001, ICANN.

[79]  Chris Eliasmith,et al.  Neural Engineering: Computation, Representation, and Dynamics in Neurobiological Systems , 2004, IEEE Transactions on Neural Networks.

[80]  Karl J. Friston Learning and inference in the brain , 2003, Neural Networks.

[81]  Zhaoping Li,et al.  Understanding Auditory Spectro-Temporal Receptive Fields and Their Changes with Input Statistics by Efficient Coding Principles , 2011, PLoS Comput. Biol..

[82]  David J. Field,et al.  Emergence of simple-cell receptive field properties by learning a sparse code for natural images , 1996, Nature.

[83]  D. Ringach Spatial structure and symmetry of simple-cell receptive fields in macaque primary visual cortex. , 2002, Journal of neurophysiology.

[84]  Surya Ganguli,et al.  Fast large-scale optimization by unifying stochastic gradient and quasi-Newton methods , 2013, ICML.

[85]  F. Attneave Some informational aspects of visual perception. , 1954, Psychological review.

[86]  A. Aertsen,et al.  A comparison of the Spectro-Temporal sensitivity of auditory neurons to tonal and natural stimuli , 1981, Biological Cybernetics.

[87]  Tobias Bonhoeffer,et al.  Altered Visual Experience Induces Instructive Changes of Orientation Preference in Mouse Visual Cortex , 2011, The Journal of Neuroscience.

[88]  Philip Rose,et al.  Protocol for the collection of databases of recordings for forensic-voice-comparison research and practice , 2012 .

[89]  I. Ohzawa,et al.  Encoding of binocular disparity by simple cells in the cat's visual cortex. , 1996, Journal of neurophysiology.