Understanding Biological Visual Attention Using Convolutional Neural Networks

Covert visual attention has been shown repeatedly to enhance performance on tasks involving the features and spatial locations to which it is deployed. Many neural correlates of covert attention have been found, but given the complexity of the visual system, connecting these neural effects to performance changes is challenging. Here, we use a deep convolutional neural network as a large-scale model of the visual system to test the effects of applying attention-like neural changes. Particularly, we explore variants of the feature similarity gain model (FSGM) of attention---which relates a cell9s tuning to its attentional modulation. We show that neural modulation of the type and magnitude observed experimentally can lead to performance changes of the type and magnitude observed experimentally. Furthermore, performance enhancements from attention occur for a diversity of tasks: high level object category detection and classification, low level orientation detection, and cross-modal color classification of an attended orientation. Utilizing the full observability of the model we also determine how activity should change to best enhance performance and how activity changes propagate through the network. Through this we find that, for attention applied at certain layers, modulating activity according to tuning performs as well as attentional modulations determined by backpropagation. At other layers, attention applied according to tuning does not successfully propagate through the network, and has a weaker impact on performance than attention determined by backpropagation. This thus highlights a potential discrepancy between neural tuning and function.

[1]  Doris Y. Tsao,et al.  The effect of face patch microstimulation on perception of faces and objects , 2017, Nature Neuroscience.

[2]  J. Maunsell,et al.  Attention improves performance primarily by reducing interneuronal correlations , 2009, Nature Neuroscience.

[3]  G. DeAngelis,et al.  Decoupled choice-driven and stimulus-related activity in parietal neurons may be misrepresented by choice probabilities , 2017, Nature Communications.

[4]  T. Poggio,et al.  What and where: A Bayesian inference theory of attention , 2010, Vision Research.

[5]  Carrie J. McAdams,et al.  Effects of Attention on Orientation-Tuning Functions of Single Neurons in Macaque Cortical Area V4 , 1999, The Journal of Neuroscience.

[6]  R. Desimone,et al.  A backward progression of attentional effects in the ventral stream , 2009, Proceedings of the National Academy of Sciences.

[7]  J. Theeuwes,et al.  Attentional effects on preattentive vision: spatial precues affect the detection of simple features. , 1999, Journal of experimental psychology. Human perception and performance.

[8]  James J. DiCarlo,et al.  Evidence that recurrent circuits are critical to the ventral stream’s execution of core object recognition behavior , 2018, Nature Neuroscience.

[9]  H. Hawkins,et al.  Visual attention modulates signal detectability. , 1990 .

[10]  Ali Borji,et al.  Optimal attentional modulation of a neural population , 2014, Front. Comput. Neurosci..

[11]  Marius V Peelen,et al.  The Neural Dynamics of Attentional Selection in Natural Scenes , 2016, The Journal of Neuroscience.

[12]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[13]  Matthias Bethge,et al.  Comparing deep neural networks against humans: object recognition when the signal gets weaker , 2017, ArXiv.

[14]  Michael Eickenberg,et al.  Seeing it all: Convolutional network layers map the function of the human visual system , 2017, NeuroImage.

[15]  Richard T Born,et al.  Feature attention for binocular disparity in primate area MT depends on tuning strength. , 2015, Journal of neurophysiology.

[16]  H. P. Op de Beeck,et al.  Deep Neural Networks as a Computational Model for Human Shape Sensitivity , 2016 .

[17]  Doris Y. Tsao,et al.  How thoughts arise from sights: inferotemporal and prefrontal contributions to vision , 2017, Current Opinion in Neurobiology.

[18]  N. P. Bichot,et al.  A Source for Feature-Based Attention in the Prefrontal Cortex , 2015, Neuron.

[19]  Yoshua Bengio,et al.  Show, Attend and Tell: Neural Image Caption Generation with Visual Attention , 2015, ICML.

[20]  C Daniel Salzman,et al.  Reward expectation differentially modulates attentional behavior and activity in visual area V4 , 2015, Nature Neuroscience.

[21]  Sabine Kastner,et al.  Visual attention as a multilevel selection process , 2004, Cognitive, affective & behavioral neuroscience.

[22]  K. Cave.  The FeatureGate model of visual selection , 1999, Psychological research.

[23]  Winrich A. Freiwald,et al.  Attention to Surfaces Modulates Motion Processing in Extrastriate Area MT , 2007, Neuron.

[24]  M. Carrasco,et al.  Attention trades off spatial acuity , 2009, Vision Research.

[25]  C. Bundesen.  A theory of visual attention. , 1990, Psychological review.

[26]  Bradley C. Love,et al.  Deep Networks as Models of Human and Animal Categorization , 2017, CogSci.

[27]  Gustavo Deco,et al.  Attention in natural scenes: Neurophysiological and computational bases , 2006, Neural Networks.

[28]  A. Nobre,et al.  Orienting Attention to Locations in Internal Representations , 2003, Journal of Cognitive Neuroscience.

[29]  M. Carrasco.  Visual attention: The past 25 years , 2011, Vision Research.

[30]  Wojciech Zaremba,et al.  Deep Neural Networks Predict Category Typicality Ratings for Images , 2015, CogSci.

[31]  Thomas Serre,et al.  Object decoding with attention in inferior temporal cortex , 2011, Proceedings of the National Academy of Sciences.

[32]  C. J. Downing.  Expectancy and visual-spatial attention: effects on perceptual quality. , 1988, Journal of experimental psychology. Human perception and performance.

[33]  Li Fei-Fei,et al.  Neural mechanisms of rapid natural scene categorization in human visual cortex , 2009, Nature.

[34]  Gary Lupyan,et al.  Correction: Making the Invisible Visible: Verbal but Not Visual Cues Enhance Visual Detection , 2010, PLoS ONE.

[35]  C. Duffy,et al.  Cortical neuronal responses to optic flow are shaped by visual strategies for steering. , 2008, Cerebral cortex.

[36]  Kunihiko Fukushima,et al.  Neocognitron: A hierarchical neural network capable of visual pattern recognition , 1988, Neural Networks.

[37]  Robert Desimone,et al.  Feature-Based Attention in the Frontal Eye Field and Area V4 during Visual Search , 2011, Neuron.

[38]  Marius V. Peelen,et al.  Object detection in natural scenes: Independent effects of spatial and category-based attention , 2017, Attention, perception & psychophysics.

[39]  Jude F. Mitchell,et al.  Differential Attention-Dependent Response Modulation across Cell Classes in Macaque Visual Area V4 , 2007, Neuron.

[40]  J. Gallant,et al.  Combined effects of spatial and feature-based attention on responses of V4 neurons , 2009, Vision Research.

[41]  Samuel Ritter,et al.  Cognitive Psychology for Deep Neural Networks: A Shape Bias Case Study , 2017, ICML.

[42]  H. Spitzer,et al.  Increased attention enhances both behavioral and neuronal performance. , 1988, Science.

[43]  Maneesh Sahani,et al.  Attention in a Bayesian Framework , 2012, Front. Hum. Neurosci..

[44]  J. Maunsell,et al.  The role of attention in visual processing. , 2002, Philosophical transactions of the Royal Society of London. Series B, Biological sciences.

[45]  S. Kastner,et al.  A neural basis for real-world visual search in human occipitotemporal cortex , 2011, Proceedings of the National Academy of Sciences.

[46]  Thomas Serre,et al.  Robust Object Recognition with Cortex-Like Mechanisms , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[47]  Mohammad Reza Daliri,et al.  Attention enhances stimulus representations in macaque visual cortex without affecting their signal-to-noise level , 2016, Scientific Reports.

[48]  Pieter R Roelfsema,et al.  Neuronal Activity in the Visual Cortex Reveals the Temporal Order of Cognitive Operations , 2010, The Journal of Neuroscience.

[49]  Dobromir Rahnev,et al.  Stimulus expectation alters decision criterion but not sensory signal in perceptual decision making , 2017, Scientific Reports.

[50]  Nicole C. Rust,et al.  Signals in inferotemporal and perirhinal cortex suggest an “untangling” of visual target information , 2013, Nature Neuroscience.

[51]  Matthew F. Peterson,et al.  Statistical decision theory to relate neurons to behavior in the study of covert visual attention , 2009, Vision Research.

[52]  Hakwan Lau,et al.  Prior Expectation Modulates the Interaction between Sensory and Prefrontal Regions in the Human Brain , 2011, The Journal of Neuroscience.

[53]  Zenghui Wang,et al.  Deep Convolutional Neural Networks for Image Classification: A Comprehensive Review , 2017, Neural Computation.

[54]  G. DeAngelis,et al.  Cortical area MT and the perception of stereoscopic depth , 1998, Nature.

[55]  R. Desimone,et al.  Neural mechanisms of spatial selective attention in areas V1, V2, and V4 of macaque visual cortex. , 1997, Journal of neurophysiology.

[56]  Timothée Masquelier,et al.  Deep Networks Can Resemble Human Feed-forward Vision in Invariant Object Recognition , 2015, Scientific Reports.

[57]  Leon A. Gatys,et al.  Deep convolutional models improve predictions of macaque V1 responses to natural images , 2017, bioRxiv.

[58]  Katherine M. Armstrong,et al.  Selective gating of visual signals by microstimulation of frontal cortex , 2003, Nature.

[59]  John H. R. Maunsell,et al.  A Refined Neuronal Population Measure of Visual Attention , 2015, PloS one.

[60]  Bryan P. Tripp,et al.  Similarities and differences between stimulus tuning in the inferotemporal visual cortex and convolutional networks , 2016, 2017 International Joint Conference on Neural Networks (IJCNN).

[61]  Marcel van Gerven,et al.  Convolutional neural network-based encoding and decoding of visual object recognition in space and time , 2017, NeuroImage.

[62]  S. Luck,et al.  Feature-based attention modulates feedforward visual processing , 2009, Nature Neuroscience.

[63]  G. Boynton.  A framework for describing the effects of attention on visual responses , 2009, Vision Research.

[64]  D. Bradley,et al.  Neural population code for fine perceptual decisions in area MT , 2005, Nature Neuroscience.

[65]  Colin J. Akerman,et al.  Random synaptic feedback weights support error backpropagation for deep learning , 2016, Nature Communications.

[66]  Mika Koivisto,et al.  Top-down preparation modulates visual categorization but not subjective awareness of objects presented in natural backgrounds , 2017, Vision Research.

[67]  P. Verghese.  Visual Search and Attention A Signal Detection Theory Approach , 2001, Neuron.

[68]  Jens Schwarzbach,et al.  Control of object-based attention in human cortex. , 2004, Cerebral cortex.

[69]  A. Treisman,et al.  A feature-integration theory of attention , 1980, Cognitive Psychology.

[70]  Ramesh Srinivasan,et al.  Distinct Attention Networks for Feature Enhancement and Suppression in Vision , 2012, Psychological science.

[71]  Philipp Schwedhelm.  Feature-based attention in primate visual cortex , 2016 .

[72]  L. Itti,et al.  Search Goal Tunes Visual Features Optimally , 2007, Neuron.

[73]  J. Maunsell,et al.  Using Neuronal Populations to Study the Mechanisms Underlying Spatial and Feature Attention , 2011, Neuron.

[74]  A. Pouget,et al.  Neural correlations, population coding and computation , 2006, Nature Reviews Neuroscience.

[75]  James Worcester,et al.  Object Detection in Natural Scenes by Feedback , 2002, Biologically Motivated Computer Vision.

[76]  John K. Tsotsos,et al.  Modeling Visual Attention via Selective Tuning , 1995, Artif. Intell..

[77]  R. Krauzlis,et al.  Superior colliculus and visual spatial attention. , 2013, Annual review of neuroscience.

[78]  David L. Sheinberg,et al.  The Effects of Prefrontal Cortex Inactivation on Object Responses of Single Neurons in the Inferotemporal Cortex during Visual Search , 2011, The Journal of Neuroscience.

[79]  R. T. Smith,et al.  Covert spatial attention is functionally intact in amblyopic human adults , 2016, Journal of vision.

[80]  B. C. Motter,et al.  Neural correlates of feature selective memory and pop-out in extrastriate area V4 , 1994, The Journal of neuroscience : the official journal of the Society for Neuroscience.

[81]  R. Desimone,et al.  Modulation of Oscillatory Neuronal Synchronization by Selective Visual Attention , 2001, Science.

[82]  Fred H. Hamker,et al.  The role of feedback connections in task-driven visual search , 1999 .

[83]  G. Boynton,et al.  Global effects of feature-based attention in human visual cortex , 2002, Nature Neuroscience.

[84]  Marcel A. J. van Gerven,et al.  Deep Neural Networks Reveal a Gradient in the Complexity of Neural Representations across the Ventral Stream , 2014, The Journal of Neuroscience.

[85]  T. Poggio,et al.  Hierarchical models of object recognition in cortex September 23 , 1999 , 1999 .

[86]  T. Poggio,et al.  Hierarchical models of object recognition in cortex , 1999, Nature Neuroscience.

[87]  Gary Lupyan,et al.  Language can boost otherwise unseen objects into visual awareness , 2013, Proceedings of the National Academy of Sciences.

[88]  William T. Newsome,et al.  Cortical microstimulation influences perceptual judgements of motion direction , 1990, Nature.

[89]  Wei Ji Ma,et al.  Humans flexibly incorporate attention-dependent uncertainty into perceptual decisions and confidence , 2017, bioRxiv.

[90]  S. Treue.  Neural correlates of attention in primate visual cortex , 2001, Trends in Neurosciences.

[91]  Nikolaus Kriegeskorte,et al.  Fixed versus mixed RSA: Explaining visual representations by fixed and mixed feature sets from shallow and deep computational models , 2016 .

[92]  Ha Hong,et al.  Performance-optimized hierarchical models predict neural responses in higher visual cortex , 2014, Proceedings of the National Academy of Sciences.

[93]  Stefan Treue,et al.  Feature-based attention influences motion processing gain in macaque visual cortex , 1999, Nature.

[94]  J. Maunsell,et al.  Attention-related changes in correlated neuronal activity arise from normalization mechanisms , 2017, Nature Neuroscience.

[95]  Robert Desimone,et al.  Cortical Connections of Area V4 in the Macaque , 2008 .

[96]  Alexander G. Huth,et al.  Attention During Natural Vision Warps Semantic Representation Across the Human Brain , 2013, Nature Neuroscience.

[97]  Leslie G. Ungerleider,et al.  A general mechanism for perceptual decision-making in the human brain , 2004, Nature.

[98]  Thomas Serre,et al.  What are the Visual Features Underlying Human Versus Machine Vision? , 2017, 2017 IEEE International Conference on Computer Vision Workshops (ICCVW).

[99]  R. Desimone,et al.  Responses of Neurons in Inferior Temporal Cortex during Memory- Guided Visual Search , 1998 .

[100]  Grace W. Lindsay.  Feature-based Attention in Convolutional Neural Networks , 2015, ArXiv.

[101]  J. Wolfe,et al.  Guided Search 2.0 A revised model of visual search , 1994, Psychonomic bulletin & review.

[102]  Thomas Zhihao Luo,et al.  Neuronal Modulations in Visual Cortex Are Associated with Only One of Multiple Components of Attention , 2015, Neuron.

[103]  Thomas L. Griffiths,et al.  Adapting Deep Network Features to Capture Psychological Representations: An Abridged Report , 2017, IJCAI.

[104]  Nikolaus Kriegeskorte,et al.  Deep Supervised, but Not Unsupervised, Models May Explain IT Cortical Representation , 2014, PLoS Comput. Biol..

[105]  Joonyeol Lee,et al.  Attentional Modulation of MT Neurons with Single or Multiple Stimuli in Their Receptive Fields , 2010, The Journal of Neuroscience.

[106]  J. Maunsell,et al.  Graded Neuronal Modulations Related to Visual Spatial Attention , 2016, The Journal of Neuroscience.