Activations of deep convolutional neural networks are aligned with gamma band activity of human visual cortex

Recent advances in the field of artificial intelligence have revealed principles about neural processing, in particular about vision. Previous work demonstrated a direct correspondence between the hierarchy of the human visual areas and layers of deep convolutional neural networks (DCNN) trained on visual object recognition. We use DCNN to investigate which frequency bands correlate with feature transformations of increasing complexity along the ventral visual pathway. By capitalizing on intracranial depth recordings from 100 patients we assess the alignment between the DCNN and signals at different frequency bands. We find that gamma activity (30–70 Hz) matches the increasing complexity of visual feature representations in DCNN. These findings show that the activity of the DCNN captures the essential characteristics of biological object recognition not only in space and time, but also in the frequency domain. These results demonstrate the potential that artificial intelligence algorithms have in advancing our understanding of the brain.Ilya Kuzovkin et al. compare intracranial depth recordings from human subjects taken during a visual recognition task to activations of deep convolutional neural networks (DCNNs). They find that signals in gamma frequency bands in the recordings are aligned with the hierarchical layer structure of the DCNN, showing that DCNNs capture important characteristics of biological object recognition.

[1]  Catherine Tallon-Baudry,et al.  Visual Grouping and the Focusing of Attention Induce Gamma-band Oscillations at Different Frequencies in Human Magnetoencephalogram Signals , 2006, Journal of Cognitive Neuroscience.

[2]  Arnaud Delorme,et al.  EEGLAB: an open source toolbox for analysis of single-trial EEG dynamics including independent component analysis , 2004, Journal of Neuroscience Methods.

[3]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[4]  Nikolaus Kriegeskorte,et al.  Deep neural networks: a new framework for modelling biological vision and brain information processing , 2015, bioRxiv.

[5]  Trevor Darrell,et al.  Caffe: Convolutional Architecture for Fast Feature Embedding , 2014, ACM Multimedia.

[6]  Juan R. Vidal,et al.  Category-Specific Visual Responses: An Intracranial Study Comparing Gamma, Beta, Alpha, and ERP Response Selectivity , 2010, Front. Hum. Neurosci..

[7]  P. Roelfsema,et al.  Alpha and gamma oscillations characterize feedback and feedforward processing in monkey visual cortex , 2014, Proceedings of the National Academy of Sciences.

[8]  Wolf Singer,et al.  Neuronal Synchrony: A Versatile Code for the Definition of Relations? , 1999, Neuron.

[9]  O. J. Dunn.  Multiple Comparisons among Means , 1961 .

[10]  F. Varela,et al.  Measuring phase synchrony in brain signals , 1999, Human brain mapping.

[11]  O. Bertrand,et al.  Oscillatory gamma activity in humans and its role in object representation , 1999, Trends in Cognitive Sciences.

[12]  H. Kennedy,et al.  Alpha-Beta and Gamma Rhythms Subserve Feedback and Feedforward Influences among Human Visual Cortical Areas , 2016, Neuron.

[13]  Thomas E. Nichols,et al.  Thresholding of Statistical Maps in Functional Neuroimaging Using the False Discovery Rate , 2002, NeuroImage.

[14]  Michal Irani,et al.  Convergent evolution of face spaces across human face-selective neuronal groups and deep convolutional networks , 2019, Nature Communications.

[15]  Odelia Schwartz,et al.  Faculty of 1000 evaluation for Deep neural networks: A new framework for modeling biological vision and brain information processing. , 2017 .

[16]  Yizhen Zhang,et al.  Deep Recurrent Neural Network Reveals a Hierarchy of Process Memory during Dynamic Natural Vision , 2017, bioRxiv.

[17]  Rob Fergus,et al.  Visualizing and Understanding Convolutional Networks , 2013, ECCV.

[18]  Ingrid Daubechies,et al.  The wavelet transform, time-frequency localization and signal analysis , 1990, IEEE Trans. Inf. Theory.

[19]  P. Fries.  A mechanism for cognitive dynamics: neuronal communication through neuronal coherence , 2005, Trends in Cognitive Sciences.

[20]  Nikolaus Kriegeskorte,et al.  Deep Supervised, but Not Unsupervised, Models May Explain IT Cortical Representation , 2014, PLoS Comput. Biol..

[21]  Antonio Torralba,et al.  Deep Neural Networks predict Hierarchical Spatio-temporal Cortical Dynamics of Human Visual Object Recognition , 2016, ArXiv.

[22]  Yizhen Zhang,et al.  Deep Recurrent Neural Network Reveals a Hierarchy of Process Memory during Dynamic Natural Vision , 2017 .

[23]  Michael S. Bernstein,et al.  ImageNet Large Scale Visual Recognition Challenge , 2014, International Journal of Computer Vision.

[24]  J. Talairach,et al.  Referentially oriented cerebral MRI anatomy : an atlas of stereotaxic anatomical correlations for gray and white matter , 1993 .

[25]  J. Maunsell,et al.  Different Origins of Gamma Rhythm and High-Gamma Activity in Macaque Visual Cortex , 2011, PLoS biology.

[26]  G. Buzsáki,et al.  Mechanisms of gamma oscillations. , 2012, Annual review of neuroscience.

[27]  S. Zeki.  The visual association cortex , 1993, Current Opinion in Neurobiology.

[28]  W. Singer,et al.  Stimulus-specific neuronal oscillations in orientation columns of cat visual cortex. , 1989, Proceedings of the National Academy of Sciences of the United States of America.

[29]  I. Fried,et al.  Neural “Ignition”: Enhanced Activation Linked to Perceptual Awareness in Human Ventral Stream Visual Cortex , 2009, Neuron.

[30]  D. P. Russell,et al.  Increased Synchronization of Neuromagnetic Responses during Conscious Perception , 1999, The Journal of Neuroscience.

[31]  James J. DiCarlo,et al.  How Does the Brain Solve Visual Object Recognition? , 2012, Neuron.

[32]  Alexander Borst,et al.  How does Nature Program Neuron Types? , 2008, Front. Neurosci..

[33]  H. B. Mann,et al.  On a Test of Whether one of Two Random Variables is Stochastically Larger than the Other , 1947 .

[34]  Ha Hong,et al.  Performance-optimized hierarchical models predict neural responses in higher visual cortex , 2014, Proceedings of the National Academy of Sciences.

[35]  James J DiCarlo,et al.  Large-Scale, High-Resolution Comparison of the Core Visual Object Recognition Behavior of Humans, Monkeys, and State-of-the-Art Deep Artificial Neural Networks , 2018, The Journal of Neuroscience.

[36]  Juan R. Vidal,et al.  Selective Neural Synchrony Suppression as a Forward Gatekeeper to Piecemeal Conscious Perception. , 2016, Cerebral cortex.

[37]  Antonio Torralba,et al.  Comparison of deep neural networks to spatio-temporal cortical dynamics of human visual object recognition reveals hierarchical correspondence , 2016, Scientific Reports.

[38]  C. Tallon-Baudry,et al.  Neural Dissociation between Visual Awareness and Spatial Attention , 2008, The Journal of Neuroscience.

[39]  K. Grill-Spector,et al.  The human visual cortex. , 2004, Annual review of neuroscience.

[40]  J. Pernier,et al.  Oscillatory γ-Band (30–70 Hz) Activity Induced by a Visual Search Task in Humans , 1997, The Journal of Neuroscience.

[41]  Marcel A. J. van Gerven,et al.  Deep Neural Networks Reveal a Gradient in the Complexity of Neural Representations across the Ventral Stream , 2014, The Journal of Neuroscience.

[42]  W Singer,et al.  Visual feature integration and the temporal correlation hypothesis. , 1995, Annual review of neuroscience.

[43]  Nikolaus Kriegeskorte,et al.  Frontiers in Systems Neuroscience Systems Neuroscience , 2022 .

[44]  P. Fries.  Rhythms for Cognition: Communication through Coherence , 2015, Neuron.

[45]  Michael Eickenberg,et al.  Convolutional Network Layers Map the Function of the Human Visual Cortex , 2017, ERCIM News.

[46]  Michael Eickenberg,et al.  Seeing it all: Convolutional network layers map the function of the human visual system , 2017, NeuroImage.

[47]  Leon A. Gatys,et al.  Deep convolutional models improve predictions of macaque V1 responses to natural images , 2019, PLoS Comput. Biol..

[48]  J. DiCarlo,et al.  Using goal-driven deep learning models to understand sensory cortex , 2016, Nature Neuroscience.

[49]  K. Seeliger,et al.  CNN-based Encoding and Decoding of Visual Object Recognition in Space and Time , 2017, bioRxiv.

[50]  Jeremy R. Manning,et al.  Broadband Shifts in Local Field Potential Power Spectra Are Correlated with Single-Neuron Spiking in Humans , 2009, The Journal of Neuroscience.

[51]  Ha Hong,et al.  Hierarchical Modular Optimization of Convolutional Networks Achieves Representations Similar to Macaque IT and Human Ventral Stream , 2013, NIPS.

[52]  Hod Lipson,et al.  Understanding Neural Networks Through Deep Visualization , 2015, ArXiv.

[53]  A. Engel,et al.  Cognitive functions of gamma-band activity: memory match and utilization , 2004, Trends in Cognitive Sciences.

[54]  Ha Hong,et al.  A performance-optimized model of neural responses across the ventral visual stream , 2016, bioRxiv.

[55]  H. Kennedy,et al.  Visual Areas Exert Feedforward and Feedback Influences through Distinct Frequency Channels , 2014, Neuron.

[56]  T. Hendler,et al.  A hierarchical axis of object processing stages in the human visual cortex. , 2001, Cerebral cortex.

[57]  Christopher D. Chambers,et al.  Redefine statistical significance , 2017, Nature Human Behaviour.

[58]  Catherine Tallon-Baudry,et al.  The many faces of the gamma band response to complex visual stimuli , 2005, NeuroImage.