Foreground-background segmentation revealed during natural image viewing

One of the major challenges in visual neuroscience is represented by foreground-background segmentation. Data from nonhuman primates show that segmentation leads to two distinct, but associated processes: the enhancement of neural activity during figure processing (i.e., foreground enhancement) and the suppression of background-related activity (i.e., background suppression). To study foreground-background segmentation in ecological conditions, we introduce a novel method based on parametric modulation of low-level image properties followed by application of simple computational image-processing models. By correlating the outcome of this procedure with human fMRI activity measured during passive viewing of 334 natural images, we reconstruct easily interpretable 'neural images' from seven visual areas: V1, V2, V3, V3A, V3B, V4 and LOC. Results show evidence of foreground enhancement for all tested regions, while background suppression specifically occurs in V4 and LOC. 'Neural images' reconstructed from V4 and LOC revealed a preserved spatial resolution of foreground textures, indicating a richer representation of the salient part of natural images, rather than a simplistic model of object shape. Our results indicate that scene segmentation is an automatic process that occurs during natural viewing, even when individuals are not required to perform any particular task.

[1]  Keiji Tanaka,et al.  Matching Categorical Object Representations in Inferior Temporal Cortex of Man and Monkey , 2008, Neuron.

[2]  N. Rubin,et al.  fMRI Activation in Response to Illusory Contours and Salient Regions in the Human Lateral Occipital Complex , 2003, Neuron.

[3]  Irving Biederman,et al.  Cortical representation of medial axis structure. , 2013, Cerebral cortex.

[4]  Andrew Zisserman,et al.  Representing shape with a spatial pyramid kernel , 2007, CIVR '07.

[5]  Kechen Zhang,et al.  A Sparse Object Coding Scheme in Area V4 , 2011, Current Biology.

[6]  Thomas Deselaers,et al.  ClassCut for Unsupervised Class Segmentation , 2010, ECCV.

[7]  David Marr,et al.  VISION A Computational Investigation into the Human Representation and Processing of Visual Information , 2009 .

[8]  G B Stanley,et al.  Reconstruction of Natural Scenes from Ensemble Responses in the Lateral Geniculate Nucleus , 1999, The Journal of Neuroscience.

[9]  Alex R. Wade,et al.  Multivariate Patterns in the Human Object-Processing Pathway Reveal a Shift from Retinotopic to Shape Curvature Representations in Lateral Occipital Areas, LO-1 and LO-2 , 2016, The Journal of Neuroscience.

[10]  Charless C. Fowlkes,et al.  Contour Detection and Hierarchical Image Segmentation , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[11]  Leslie G. Ungerleider,et al.  Texture segregation in the human visual cortex: A functional MRI study. , 2000, Journal of neurophysiology.

[12]  Nikolaus Kriegeskorte,et al.  Deep Supervised, but Not Unsupervised, Models May Explain IT Cortical Representation , 2014, PLoS Comput. Biol..

[13]  Kendrick N. Kay,et al.  Attention Reduces Spatial Uncertainty in Human Ventral Temporal Cortex , 2015, Current Biology.

[14]  Matti Pietikäinen,et al.  A Generalized Local Binary Pattern Operator for Multiresolution Gray Scale and Rotation Invariant Texture Classification , 2001, ICAPR.

[15]  Jack L. Gallant,et al.  Both convolutional neural networks and voxel-wise encoding models of brain activity derived from ConvNets represent boundary-and surface-related features , 2016 .

[16]  Doris Y. Tsao,et al.  Consistency of Border-Ownership Cells across Artificial Stimuli, Natural Stimuli, and Stimuli with Ambiguous Contours , 2016, The Journal of Neuroscience.

[17]  Jack L. Gallant,et al.  Encoding and decoding in fMRI , 2011, NeuroImage.

[18]  J. Gallant,et al.  Identifying natural images from human brain activity , 2008, Nature.

[19]  H. Neumann,et al.  The Role of Attention in Figure-Ground Segregation in Areas V1 and V4 of the Visual Cortex , 2012, Neuron.

[20]  Jitendra Malik,et al.  Local figure-ground cues are valid for natural images. , 2007, Journal of vision.

[21]  Thomas L. Griffiths,et al.  Supplementary Information for Natural Speech Reveals the Semantic Maps That Tile Human Cerebral Cortex , 2022 .

[22]  Emiliano Ricciardi,et al.  How concepts are encoded in the human brain: A modality independent, category-based cortical organization of semantic knowledge , 2016, NeuroImage.

[23]  D. Mumford,et al.  The role of the primary visual cortex in higher level vision , 1998, Vision Research.

[24]  Cordelia Schmid,et al.  Beyond Bags of Features: Spatial Pyramid Matching for Recognizing Natural Scene Categories , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[25]  S. Edelman,et al.  Cue-Invariant Activation in Object-Related Areas of the Human Occipital Lobe , 1998, Neuron.

[26]  F. Qiu,et al.  Figure-ground mechanisms provide structure for selective attention , 2007, Nature Neuroscience.

[27]  Hans Spinnler,et al.  Perceptual and associative disorders of visual recognition , 1969, Neurology.

[28]  Bevil R. Conway,et al.  Toward a Unified Theory of Visual Area V4 , 2012, Neuron.

[29]  Stephen A. Engel,et al.  Neural Response to Perception of Volume in the Lateral Occipital Complex , 2001, Neuron.

[30]  R. Tibshirani,et al.  An introduction to the bootstrap , 1993 .

[31]  R. Malach,et al.  Object-related activity revealed by functional magnetic resonance imaging in human occipital cortex. , 1995, Proceedings of the National Academy of Sciences of the United States of America.

[32]  Edoardo Bisiach,et al.  Recognition of overlapping patterns and focal hemisphere damage , 1976, Neuropsychologia.

[33]  Pieter R. Roelfsema,et al.  Texture Segregation Causes Early Figure Enhancement and Later Ground Suppression in Areas V1 and V4 of Visual Cortex , 2016, Cerebral cortex.

[34]  Victor A. F. Lamme.  The neurophysiology of figure-ground segregation in primary visual cortex , 1995, The Journal of neuroscience : the official journal of the Society for Neuroscience.

[35]  Pierre Kornprobst,et al.  Bio-inspired computer vision: Towards a synergistic approach of artificial and biological vision , 2016, Comput. Vis. Image Underst..

[36]  J. Gallant,et al.  Complete functional characterization of sensory neurons by system identification. , 2006, Annual review of neuroscience.

[37]  Eero P. Simoncelli,et al.  Metamers of the ventral stream , 2011, Nature Neuroscience.

[38]  G. Rees,et al.  The Neural Bases of Multistable Perception , 2022 .

[39]  Masa-aki Sato,et al.  Visual Image Reconstruction from Human Brain Activity using a Combination of Multiscale Local Image Decoders , 2008, Neuron.

[40]  Brian A. Wandell,et al.  Population receptive field estimates in human visual cortex , 2008, NeuroImage.

[41]  I. Biederman.  Recognition-by-components: a theory of human image understanding. , 1987, Psychological review.

[42]  Antonio Torralba,et al.  Modeling the Shape of the Scene: A Holistic Representation of the Spatial Envelope , 2001, International Journal of Computer Vision.

[43]  Pieter R. Roelfsema,et al.  Distinct Roles of the Cortical Layers of Area V1 in Figure-Ground Segregation , 2013, Current Biology.

[44]  Eric T. Carlson,et al.  Medial Axis Shape Coding in Macaque Inferotemporal Cortex , 2012, Neuron.

[45]  H. Komatsu,et al.  Image statistics underlying natural texture selectivity of neurons in macaque V4 , 2014, Proceedings of the National Academy of Sciences.

[46]  B. C. Motter.  Central V4 Receptive Fields Are Scaled by the V1 Cortical Magnification and Correspond to a Constant-Sized Sampling of the V1 Surface , 2009, The Journal of Neuroscience.

[47]  Bart Krekelberg,et al.  The Statistical Analysis of Multi-Voxel Patterns in Functional Imaging , 2013, PloS one.

[48]  F. D. Lange,et al.  Shape Perception Simultaneously Up- and Downregulates Neural Activity in the Primary Visual Cortex , 2014, Current Biology.

[49]  Jonathan R. Williford,et al.  Figure-Ground Organization in Visual Cortex for Natural Scenes , 2016, eNeuro.

[50]  Nicole C. Rust,et al.  Do We Know What the Early Visual System Does? , 2005, The Journal of Neuroscience.

[51]  Jack L. Gallant,et al.  A Continuous Semantic Space Describes the Representation of Thousands of Object and Action Categories across the Human Brain , 2012, Neuron.

[52]  J. Gallant,et al.  Reconstructing Visual Experiences from Brain Activity Evoked by Natural Movies , 2011, Current Biology.

[53]  Giovanni d'Avossa,et al.  Impaired integration of object knowledge and visual input in a case of ventral simultanagnosia with bilateral damage to area V4 , 2012, Cognitive neuropsychology.

[54]  Jonathan Winawer,et al.  Imaging retinotopic maps in the human brain , 2011, Vision Research.

[55]  Y. Benjamini,et al.  Controlling the false discovery rate: a practical and powerful approach to multiple testing , 1995 .

[56]  Johannes J. Fahrenfort,et al.  Feedforward and Recurrent Processing in Scene Segmentation: Electroencephalography and Functional Magnetic Resonance Imaging , 2008, Journal of Cognitive Neuroscience.

[57]  Jean-Baptiste Poline,et al.  Inverse retinotopy: Inferring the visual content of images from brain activation patterns , 2006, NeuroImage.