What is changing when: Decoding visual information in movies from human intracranial recordings

ABSTRACT The majority of visual recognition studies have focused on the neural responses to repeated presentations of static stimuli with abrupt and well‐defined onset and offset times. In contrast, natural vision involves unique renderings of visual inputs that are continuously changing without explicitly defined temporal transitions. Here we considered commercial movies as a coarse proxy to natural vision. We recorded intracranial field potential signals from 1,284 electrodes implanted in 15 patients with epilepsy while the subjects passively viewed commercial movies. We could rapidly detect large changes in the visual inputs within approximately 100 ms of their occurrence, using exclusively field potential signals from ventral visual cortical areas including the inferior temporal gyrus and inferior occipital gyrus. Furthermore, we could decode the content of those visual changes even in a single movie presentation, generalizing across the wide range of transformations present in a movie. These results present a methodological framework for studying cognition during dynamic and natural vision. HIGHLIGHTSThis study presents a methodology to examine intracranial field potential responses to continuous movie stimuli.Intracranial field potentials from human ventral visual cortex show strong, selective and consistent responses to changes during a movie.We can decode when visual changes happen and what content changes in the visual input in single events directly from physiological signals.

[1]  Thomas Serre,et al.  A feedforward architecture accounts for rapid categorization , 2007, Proceedings of the National Academy of Sciences.

[2]  G A Ojemann,et al.  Treatment of temporal lobe epilepsy. , 1997, Annual review of medicine.

[3]  Peter J. Ramadge,et al.  Inter-subject alignment of human cortical anatomy using functional connectivity , 2013, NeuroImage.

[4]  J. Gallant,et al.  Reconstructing Visual Experiences from Brain Activity Evoked by Natural Movies , 2011, Current Biology.

[5]  Brian E. Russ,et al.  Single-Unit Activity during Natural Vision: Diversity, Consistency, and Spatial Sensitivity among AF Face Patch Neurons , 2015, The Journal of Neuroscience.

[6]  Barry D. Anderson,et al.  A window on reality? , 2004, Journal of clinical oncology : official journal of the American Society of Clinical Oncology.

[7]  Daniel T. Levin,et al.  A Window on Reality , 2012 .

[8]  P. Fldik,et al.  The Speed of Sight , 2001, Journal of Cognitive Neuroscience.

[9]  Joel Z. Leibo,et al.  The dynamics of invariant object recognition in the human visual system. , 2014, Journal of neurophysiology.

[10]  Gabriel Kreiman,et al.  Visual population codes : toward a common multivariate framework for cell recording and functional imaging , 2012 .

[11]  M. Tarr,et al.  Visual Object Recognition , 1996, ISTCS.

[12]  Andreas Bartels,et al.  Brain dynamics during natural viewing conditions—A new guide for mapping connectivity in vivo , 2005, NeuroImage.

[13]  M. Weliky,et al.  Small modulation of ongoing cortical dynamics by sensory input during natural vision , 2004, Nature.

[14]  Nicole C. Rust,et al.  In praise of artifice , 2005, Nature Neuroscience.

[15]  Nicole C. Rust,et al.  Do We Know What the Early Visual System Does? , 2005, The Journal of Neuroscience.

[16]  Jack L. Gallant,et al.  A Continuous Semantic Space Describes the Representation of Thousands of Object and Action Categories across the Human Brain , 2012, Neuron.

[17]  J L Gallant,et al.  Sparse coding and decorrelation in primary visual cortex during natural vision. , 2000, Science.

[18]  T. Poggio,et al.  Hierarchical models of object recognition in cortex , 1999, Nature Neuroscience.

[19]  Gidon Felsen,et al.  A natural approach to studying vision , 2005, Nature Neuroscience.

[20]  N. Logothetis,et al.  Phase-of-Firing Coding of Natural Visual Stimuli in Primary Visual Cortex , 2008, Current Biology.

[21]  E. Rolls,et al.  View-invariant representations of familiar objects by neurons in the inferior temporal visual cortex. , 1998, Cerebral cortex.

[22]  Yuanye Ma,et al.  Telemetric recordings of single neuron activity and visual scenes in monkeys walking in an open field , 2004, Journal of Neuroscience Methods.

[23]  G. Kreiman,et al.  Timing, Timing, Timing: Fast Decoding of Object Information from Intracranial Field Potentials in Human Visual Cortex , 2009, Neuron.

[24]  Rajesh P. N. Rao,et al.  Broadband changes in the cortical surface potential track activation of functionally diverse neuronal populations , 2014, NeuroImage.

[25]  Tomaso Poggio,et al.  Fast Readout of Object Identity from Macaque Inferior Temporal Cortex , 2005, Science.

[26]  Gabriel Kreiman,et al.  Temporal stability of visually selective responses in intracranial field potentials recorded from human occipital and temporal lobes. , 2012, Journal of neurophysiology.

[27]  Keiji Tanaka,et al.  Inferotemporal cortex and object vision. , 1996, Annual review of neuroscience.

[28]  Yehezkel Yeshurun,et al.  Antagonistic relationship between gamma power and visual evoked potentials revealed in human visual cortex. , 2011, Cerebral cortex.

[29]  Andreas Bartels,et al.  Integration of EEG source imaging and fMRI during continuous viewing of natural movies. , 2010, Magnetic resonance imaging.

[30]  C. Koch,et al.  The origin of extracellular fields and currents — EEG, ECoG, LFP and spikes , 2012, Nature Reviews Neuroscience.

[31]  Yadin Dudai,et al.  The cinema-cognition dialogue: a match made in brain , 2012, Front. Hum. Neurosci..

[32]  Juan R. Vidal,et al.  Category-Specific Visual Responses: An Intracranial Study Comparing Gamma, Beta, Alpha, and ERP Response Selectivity , 2010, Front. Hum. Neurosci..

[33]  Gal Chechik,et al.  Invariant Temporal Dynamics Underlie Perceptual Stability in Human Visual Cortex , 2017, Current Biology.

[34]  L. Optican,et al.  Temporal encoding of two-dimensional patterns by single units in primate inferior temporal cortex. III. Information theoretic analysis. , 1987, Journal of neurophysiology.

[35]  E T Rolls,et al.  Sparseness of the neuronal representation of stimuli in the primate temporal visual cortex. , 1995, Journal of neurophysiology.

[36]  Partha P. Mitra,et al.  Chronux: A platform for analyzing neural signals , 2010, Journal of Neuroscience Methods.

[37]  David A. Leopold,et al.  Functional MRI mapping of dynamic visual features during natural viewing in the macaque , 2015, NeuroImage.

[38]  Anders M. Dale,et al.  Automatic parcellation of human cortical gyri and sulci using standard anatomical nomenclature , 2010, NeuroImage.

[39]  Yehezkel Yeshurun,et al.  Enhanced Category Tuning Revealed by Intracranial Electroencephalograms in High-Order Human Visual Areas , 2007, The Journal of Neuroscience.

[40]  D. Heeger,et al.  Slow Cortical Dynamics and the Accumulation of Information over Long Timescales , 2012, Neuron.

[41]  R. Malach,et al.  Intersubject Synchronization of Cortical Activity During Natural Vision , 2004, Science.

[42]  H. Spitzer,et al.  Temporal encoding of two-dimensional patterns by single units in primate primary visual cortex. I. Stimulus-response relations. , 1990, Journal of neurophysiology.

[43]  Joseph R. Madsen,et al.  Spatiotemporal Dynamics Underlying Object Completion in Human Ventral Visual Cortex , 2014, Neuron.

[44]  Anitha Pasupathy,et al.  Transformation of shape information in the ventral pathway , 2007, Current Opinion in Neurobiology.

[45]  K. Grill-Spector,et al.  The human visual cortex. , 2004, Annual review of neuroscience.

[46]  G D Lewen,et al.  Neural coding of naturalistic motion stimuli , 2001, Network.

[47]  David J. Freedman,et al.  Dynamic population coding of category information in inferior temporal and prefrontal cortex. , 2008, Journal of neurophysiology.

[48]  James J. DiCarlo,et al.  How Does the Brain Solve Visual Object Recognition? , 2012, Neuron.