Modeling Bottom-Up and Top-Down Attention with a Neurodynamic Model of V1

Previous studies in that line suggested that lateral interactions of V1 cells are responsible, among other visual effects, of bottom-up visual attention (alternatively named visual salience or saliency). Our objective is to mimic these connections in the visual system with a neurodynamic network of firing-rate neurons. Early subcortical processes (i.e. retinal and thalamic) are functionally simulated. An implementation of the cortical magnification function is included to define the retinotopical projections towards V1, processing neuronal activity for each distinct view during scene observation. Novel computational definitions of top-down inhibition (in terms of inhibition of return and selection mechanisms), are also proposed to predict attention in Free-Viewing and Visual Search conditions. Results show that our model outpeforms other biologically-inpired models of saliency prediction as well as to predict visual saccade sequences during free viewing. We also show how temporal and spatial characteristics of inhibition of return can improve prediction of saccades, as well as how distinct search strategies (in terms of feature-selective or category-specific inhibition) predict attention at distinct image contexts. Author summary Saliency maps are the representations of how certain visual regions attract attention in a visual scene, and these can be measured with eye movements. A myriad of computational models with artificial and biological inspiration have been able to acquire outstanding predictions of human fixations. However, most of these models have been built specifically for visual saliency, a characteristic that denies their biological plausibility for modeling distinct visual processing mechanisms or other visual processes simultaneously. In addition to saliency, our approach is also able to efficiently work for other tasks (without applying any type of training or optimization and keeping the same parametrization) such as Visual Search, Visual Discomfort [1], Brightness [2] and Color Induction [3]. By performing simulations of human physiology and its mechanisms, we propose to build a unified model that could be extended to predict and understand distinct perceptual processes in which V1 is responsible.

[1]  John K. Tsotsos,et al.  Saliency, attention, and visual search: an information theoretic approach. , 2009, Journal of vision.

[2]  Z Li,et al.  Pre-attentive segmentation in the primary visual cortex. , 1998, Spatial vision.

[3]  W. Wescoe The Reticular Formation , 1962, JAMA.

[4]  Iain D. Gilchrist,et al.  Visual correlates of fixation selection: effects of scale and time , 2005, Vision Research.

[5]  John K. Tsotsos,et al.  On computational modeling of visual saliency: Examining what’s right, and what’s left , 2015, Vision Research.

[6]  Ziad M Hafed,et al.  How is visual salience computed in the brain? Insights from behaviour, neurobiology and modelling , 2017, Philosophical Transactions of the Royal Society B: Biological Sciences.

[7]  Lawrence C. Sincich,et al.  The circuitry of V1 and V2: integration of color, form, and motion. , 2005, Annual review of neuroscience.

[8]  John K. Tsotsos,et al.  Saliency Based on Information Maximization , 2005, NIPS.

[9]  B. Wandell,et al.  Visual Field Maps in Human Cortex , 2007, Neuron.

[10]  Dražen Domijan,et al.  A Neurodynamic Model of Feature-Based Spatial Selection , 2018, Front. Psychol..

[11]  Sang Joon Kim,et al.  A Mathematical Theory of Communication , 2006 .

[12]  James W Bisley,et al.  Neural correlates of attention and distractibility in the lateral intraparietal area. , 2006, Journal of neurophysiology.

[13]  I. Rentschler,et al.  Peripheral vision and pattern recognition: a review. , 2011, Journal of vision.

[14]  Alfred Kobsa,et al.  Computer Vision and Graphics , 2012, Lecture Notes in Computer Science.

[15]  Alex D. Hwang,et al.  Semantic guidance of eye movements in real-world scenes , 2011, Vision Research.

[16]  John K. Tsotsos,et al.  Towards the Quantitative Evaluation of Visual Attention Models Bottom−up Top-down Dynamic Static 0 0 0 , 2022 .

[17]  Christof Koch,et al.  A Computational Analysis of the Function of Three Inhibitory Cell Types in Contextual Visual Processing , 2016, Front. Comput. Neurosci..

[18]  Zhaoping Li A saliency map in primary visual cortex , 2002, Trends in Cognitive Sciences.

[19]  Anja K. E. Horn,et al.  Reticular Formation: Eye Movements, Gaze and Blinks , 2004 .

[20]  Zhaoping Li,et al.  A Neural Model of Contour Integration in the Primary Visual Cortex , 1998, Neural Computation.

[21]  Li Zhaoping,et al.  Understanding Vision: Theory, Models, and Data , 2014 .

[22]  Marc-Oliver Gewaltig,et al.  Towards Reproducible Descriptions of Neuronal Network Models , 2009, PLoS Comput. Biol..

[23]  T. Geisel,et al.  Are human scanpaths Levy flights , 1999 .

[24]  Xose Manuel Pardo,et al.  Psychophysical evaluation of individual low-level feature influences on visual attention , 2018, Vision Research.

[25]  Gert Kootstra,et al.  Predicting Eye Fixations on Complex Visual Stimuli Using Local Symmetry , 2011, Cognitive Computation.

[26]  Gabriel Cristóbal,et al.  Self-Invertible 2D Log-Gabor Wavelets , 2007, International Journal of Computer Vision.

[27]  Jeffrey M. Zacks,et al.  Ambient and focal visual processing of naturalistic activity. , 2016, Journal of vision.

[28]  Joel Pokorny,et al.  Spectral sensitivities of the human cones , 2013 .

[29]  Frédo Durand,et al.  What Do Different Evaluation Metrics Tell Us About Saliency Models? , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[30]  John K. Tsotsos,et al.  A Focus on Selection for Fixation , 2016 .

[31]  Christof Koch,et al.  A Model of Saliency-Based Visual Attention for Rapid Scene Analysis , 2009 .

[32]  S. Grossberg,et al.  Resonant Cholinergic Dynamics in Cognitive and Motor Decision-Making: Attention, Category Learning, and Choice in Neocortex, Superior Colliculus, and Optic Tectum , 2016, Front. Neurosci..

[33]  Emily A. Cooper,et al.  Sensitivity and bias in the discrimination of two-dimensional and three-dimensional motion direction. , 2016, Journal of vision.

[34]  David J. Field,et al.  Emergence of simple-cell receptive field properties by learning a sparse code for natural images , 1996, Nature.

[35]  Stefan Winkler,et al.  Overview of Eye tracking Datasets , 2013, 2013 Fifth International Workshop on Quality of Multimedia Experience (QoMEX).

[36]  Harold Pashler,et al.  A Boolean map theory of visual attention. , 2007, Psychological review.

[37]  Weisi Lin,et al.  Selective Visual Attention: Computational Models and Applications , 2013 .

[38]  James M. Rehg,et al.  The Secrets of Salient Object Segmentation , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[39]  Wulfram Gerstner,et al.  Spike-timing dependent plasticity , 2010, Scholarpedia.

[40]  Timothée Masquelier,et al.  Unsupervised Learning of Visual Features through Spike Timing Dependent Plasticity , 2007, PLoS Comput. Biol..

[41]  Zhaoping Li,et al.  Psychophysical Tests of the Hypothesis of a Bottom-Up Saliency Map in Primary Visual Cortex , 2007, PLoS Comput. Biol..

[42]  S. Grossberg,et al.  Where’s Waldo? How perceptual, cognitive, and emotional brain processes cooperate during learning to categorize and find desired objects in a cluttered scene , 2014, Front. Integr. Neurosci..

[43]  John K. Tsotsos,et al.  Modeling task influences for saccade sequence and visual relevance prediction , 2019 .

[44]  KochChristof,et al.  A Model of Saliency-Based Visual Attention for Rapid Scene Analysis , 1998 .

[45]  Naila Murray,et al.  Saliency estimation using a non-parametric low-level vision model , 2011, CVPR 2011.

[46]  Edward M Callaway,et al.  Morphology of superior colliculus‐ and middle temporal area‐projecting neurons in primate primary visual cortex , 2012, The Journal of comparative neurology.

[47]  L. Chalupa,et al.  The new visual neurosciences , 2014 .

[48]  W. Cowan,et al.  Annual Review of Neuroscience , 1995 .

[49]  E. L. Schwartz,et al.  Spatial mapping in the primate sensory projection: Analytic structure and relevance to perception , 1977, Biological Cybernetics.

[50]  R. Shapley,et al.  The Orientation Selectivity of Color-Responsive Neurons in Macaque V1 , 2008, The Journal of Neuroscience.

[51]  P H Schiller,et al.  Look and see: how the brain moves your eyes about. , 2001, Progress in brain research.

[52]  Zhi Liu,et al.  Saccadic model of eye movements for free-viewing condition , 2015, Vision Research.

[53]  E. Rolls Memory, Attention, and Decision-Making: A unifying computational neuroscience approach , 2007 .

[54]  Xavier Otazu,et al.  A Neurodynamical Model of Brightness Induction in V1 , 2013, PloS one.

[55]  Xavier Otazu,et al.  SID4VAM: A Benchmark Dataset With Synthetic Images for Visual Attention Modeling , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[56]  A. Samuel,et al.  Multiple cueing dissociates location- and feature-based repetition effects , 2014, Vision Research.

[57]  J. Antes The time course of picture viewing. , 1974, Journal of experimental psychology.

[58]  Matthew Cook,et al.  Unsupervised learning of digit recognition using spike-timing-dependent plasticity , 2015, Front. Comput. Neurosci..

[59]  R. Desimone,et al.  Neural mechanisms of selective visual attention. , 1995, Annual review of neuroscience.

[60]  Jeffrey D. Schall Frontal Eye Fields , 2009 .

[61]  M. Carrasco Covert attention increases contrast sensitivity: Psychophysical, neurophysiological and neuroimaging studies. , 2006, Progress in brain research.

[62]  Peter Dayan,et al.  Cortical Surround Interactions and Perceptual Salience via Natural Scene Statistics , 2012, PLoS Comput. Biol..

[63]  Ali Borji,et al.  State-of-the-Art in Visual Attention Modeling , 2013, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[64]  Zhaoping Li,et al.  Bottom-up saliency and top-down learning in the primary visual cortex of monkeys , 2018, Proceedings of the National Academy of Sciences.

[65]  P. Lennie,et al.  The machinery of colour vision , 2007, Nature Reviews Neuroscience.

[66]  Thierry Baccino,et al.  Methods for comparing scanpaths and saliency maps: strengths and weaknesses , 2012, Behavior Research Methods.

[67]  Ralf Engbert,et al.  Temporal evolution of the central fixation bias in scene viewing. , 2016, Journal of vision.

[68]  C. Gilbert Horizontal integration and cortical dynamics , 1992, Neuron.

[69]  Alexander Pastukhov,et al.  Believable change: bistable reversals are governed by physical plausibility. , 2012, Journal of vision.

[70]  D. Fitzpatrick,et al.  Patterns of excitation and inhibition evoked by horizontal connections in visual cortex share a common relationship to orientation columns , 1995, Neuron.

[71]  Susan L. Franzel,et al.  Guided search: an alternative to the feature integration model for visual search. , 1989, Journal of experimental psychology. Human perception and performance.

[72]  Olivier Penacchio,et al.  Visual discomfort and the spatial distribution of Fourier energy , 2015, Vision Research.

[73]  A. Samuel,et al.  Inhibition of return: A graphical meta-analysis of its time course and an empirical test of its temporal and spatial properties , 2003, Psychonomic bulletin & review.

[74]  John K. Tsotsos,et al.  Modeling Visual Attention via Selective Tuning , 1995, Artif. Intell..

[75]  A. Treisman,et al.  Search asymmetry: a diagnostic for preattentive processing of separable features. , 1985, Journal of experimental psychology. General.

[76]  M. Ahmadlou,et al.  Functional modulation of primary visual cortex by the superior colliculus in the mouse , 2018, Nature Communications.

[77]  Li Zhaoping,et al.  Gaze capture by eye-of-origin singletons: interdependence with awareness. , 2012, Journal of vision.

[78]  Zhaoping Li V1 mechanisms and some figure-ground and border effects. , 2003, Journal of physiology, Paris.

[79]  S Ullman,et al.  Shifts in selective visual attention: towards the underlying neural circuitry. , 1985, Human neurobiology.

[80]  E. Rolls,et al.  A Neurodynamical cortical model of visual attention and invariant object recognition , 2004, Vision Research.

[81]  Pietro Perona,et al.  Graph-Based Visual Saliency , 2006, NIPS.

[82]  Drew H. Abney,et al.  Journal of Experimental Psychology : Human Perception and Performance Influence of Musical Groove on Postural Sway , 2015 .

[83]  Timothée Masquelier,et al.  Deep Learning in Spiking Neural Networks , 2018, Neural Networks.

[84]  Xavier Otazu,et al.  Comparison between Mallat's and the ‘à trous’ discrete wavelet transform based algorithms for the fusion of multispectral and panchromatic images , 2005 .

[85]  Michael W. Spratling Predictive coding as a model of the V1 saliency map hypothesis , 2012, Neural Networks.

[86]  Giuseppe Boccignone,et al.  Modelling gaze shift as a constrained random walk , 2004 .

[87]  Thomas Serre,et al.  Towards a Theory of Computation in the Visual Cortex , 2017 .

[88]  Eugene M. Izhikevich,et al.  Which model to use for cortical spiking neurons? , 2004, IEEE Transactions on Neural Networks.

[89]  Ali Borji,et al.  CAT2000: A Large Scale Fixation Dataset for Boosting Saliency Research , 2015, ArXiv.

[90]  Xavier Otazu,et al.  A Neurodynamic Model of Saliency Prediction in V1 , 2018, Neural Computation.

[91]  D. C. Essen,et al.  Neurons in monkey visual area V2 encode combinations of orientations , 2007, Nature Neuroscience.

[92]  P. Lennie,et al.  Chromatic mechanisms in striate cortex of macaque , 1990, The Journal of neuroscience : the official journal of the Society for Neuroscience.

[93]  L. SchwartzE. Spatial mapping in the primate sensory projection , 1977 .

[94]  H Barlow,et al.  Redundancy reduction revisited , 2001, Network.

[95]  Michael Breakspear,et al.  Modeling Magnification and Anisotropy in the Primate Foveal Confluence , 2010, PLoS Comput. Biol..

[96]  Zhaoping Li,et al.  From the optic tectum to the primary visual cortex: migration through evolution of the saliency map for exogenous attentional guidance , 2016, Current Opinion in Neurobiology.

[97]  Philippe Gaussier,et al.  Efficient Neural Models for Visual Attention , 2010, ICCVG.

[98]  Z Li,et al.  Contextual influences in V1 as a basis for pop out and asymmetry in visual search. , 1999, Proceedings of the National Academy of Sciences of the United States of America.

[99]  Bruno A. Olshausen,et al.  Selective visual attention , 1998 .

[100]  C. Pierrot-Deseilligny,et al.  Cortical control of ocular saccades in humans: a model for motricity. , 2003, Progress in brain research.

[101]  C. Pierrot-Deseilligny,et al.  Eye movement control by the cerebral cortex , 2004, Current opinion in neurology.

[102]  J. Theeuwes,et al.  Oculomotor capture and Inhibition of Return: Evidence for an oculomotor suppression account of IOR , 2002, Psychological research.

[103]  Yuqiao Gu,et al.  A neural network model of attention-modulated neurodynamics , 2007, Cognitive Neurodynamics.

[104]  S. Nelson,et al.  An emergent model of orientation selectivity in cat visual cortical simple cells , 1995, The Journal of neuroscience : the official journal of the Society for Neuroscience.

[105]  S. Liversedge,et al.  Oxford handbook of eye movements , 2011 .

[106]  Yoshinori Shichida,et al.  Cone visual pigments. , 2014, Biochimica et biophysica acta.

[107]  Patrick Cavanagh,et al.  Remapped visual masking. , 2011, Journal of vision.

[108]  Laurent Itti,et al.  Superior colliculus encodes visual saliency before the primary visual cortex , 2017, Proceedings of the National Academy of Sciences.

[109]  Olivier Penacchio,et al.  Discomfort from urban scenes : metabolic consequences , 2017 .

[110]  A. Treisman,et al.  A feature-integration theory of attention , 1980, Cognitive Psychology.

[111]  S. Yantis,et al.  Visual attention: control, representation, and time course. , 1997, Annual review of psychology.

[112]  Jose-Manuel Alonso,et al.  Fundamentals of vision : low and mid-level processes in perception , 2006 .