Visual Noise from Natural Scene Statistics Reveals Human Scene Category Representations

Our perceptions are guided both by the bottom-up information entering our eyes, as well as our top-down expectations of what we will see. Although bottom-up visual processing has been extensively studied, comparatively little is known about top-down signals. Here, we describe REVEAL (Representations Envisioned Via Evolutionary ALgorithm), a method for visualizing an observer's internal representation of a complex, real-world scene, allowing us to, for the first time, visualize the top-down information in an observer's mind. REVEAL rests on two innovations for solving this high dimensional problem: visual noise that samples from natural image statistics, and a computer algorithm that collaborates with human observers to efficiently obtain a solution. In this work, we visualize observers' internal representations of a visual scene category (street) using an experiment in which the observer views the naturalistic visual noise and collaborates with the algorithm to externalize his internal representation. As no scene information was presented, observers had to use their internal knowledge of the target, matching it with the visual features in the noise. We matched reconstructed images with images of real-world street scenes to enhance visualization. Critically, we show that the visualized mental images can be used to predict rapid scene detection performance, as each observer had faster and more accurate responses to detecting real-world images that were the most similar to his reconstructed street templates. These results show that it is possible to visualize previously unobservable mental representations of real world stimuli. More broadly, REVEAL provides a general method for objectively examining the content of previously private, subjective mental experiences.

[1]  Eero P. Simoncelli,et al.  Natural image statistics and neural representation. , 2001, Annual review of neuroscience.

[2]  A. Ahumada,et al.  Stimulus Features in Signal Detection , 1971 .

[3]  V. Lamme,et al.  The distinct modes of vision offered by feedforward and recurrent processing , 2000, Trends in Neurosciences.

[4]  Li Fei-Fei,et al.  Finding “good” features for natural scene classification , 2010 .

[5]  J. Gallant,et al.  Identifying natural images from human brain activity , 2008, Nature.

[6]  Terrence J. Sejnowski,et al.  Unsupervised Learning , 2018, Encyclopedia of GIS.

[7]  J. Davidoff,et al.  Exposure to an urban environment alters the local bias of a remote culture , 2012, Cognition.

[8]  Eric T. Carlson,et al.  A neural code for three-dimensional object shape in macaque inferotemporal cortex , 2008, Nature Neuroscience.

[9]  A. O'Toole,et al.  Prototype-referenced shape encoding revealed by high-level aftereffects , 2001, Nature Neuroscience.

[10]  David J. Field,et al.  What Is the Goal of Sensory Coding? , 1994, Neural Computation.

[11]  D. F. Marks,et al.  Visual imagery differences in the recall of pictures. , 1973, British journal of psychology.

[12]  Krista A. Ehinger,et al.  SUN Database: Exploring a Large Collection of Scene Categories , 2014, International Journal of Computer Vision.

[13]  S. McKelvie Effects of format on the distribution of scores on the Vividness of Visual Imagery Questionnaire: A replication and extension , 1992 .

[14]  Cheryl Olman,et al.  Classification objects, ideal observers & generative models , 2004, Cogn. Sci..

[15]  G. Rhodes,et al.  The timecourse of higher-level face aftereffects , 2007, Vision Research.

[16]  Jakob H Macke,et al.  Estimating predictive stimulus features from psychophysical data: The decision image technique applied to human faces. , 2010, Journal of vision.

[17]  Dorothea Heiss-Czedik,et al.  An Introduction to Genetic Algorithms. , 1997, Artificial Life.

[18]  P. Schyns,et al.  Superstitious Perceptions Reveal Properties of Internal Representations , 2003, Psychological science.

[19]  H Barlow,et al.  Redundancy reduction revisited , 2001, Network.

[20]  P. Montague,et al.  Vividness of mental imagery: Individual variability can be measured objectively , 2007, Vision Research.

[21]  P. Schyns,et al.  Measuring Internal Representations from Behavioral and Brain Data , 2012, Current Biology.

[22]  Paul T. Sowden,et al.  The use of visual information in natural scenes , 2005 .

[23]  Michelle R. Greene,et al.  High-level aftereffects to global scene properties. , 2010, Journal of experimental psychology. Human perception and performance.

[24]  Dirk B. Walther,et al.  Good Exemplars of Natural Scene Categories Elicit Clearer Patterns than Bad Exemplars but Not Greater BOLD Activity , 2013, PloS one.

[25]  R. Nisbett,et al.  Culture and the Physical Environment , 2006, Psychological science.

[26]  J. Victor Analyzing receptive fields, classification images and functional images: challenges with opportunities for synergy , 2005, Nature Neuroscience.

[27]  Adam N. Sanborn,et al.  Uncovering mental representations with Markov chain Monte Carlo , 2010, Cognitive Psychology.

[28]  Jonathan D. Cohen,et al.  Proceedings of the 33rd Annual Conference of the Cognitive Science Society , 2011 .

[29]  Tomaso Poggio,et al.  Fast Readout of Object Identity from Macaque Inferior Temporal Cortex , 2005, Science.

[30]  Antonio Torralba,et al.  Statistics of natural image categories , 2003, Network.

[31]  Frédéric Gosselin,et al.  Bubbles: a technique to reveal the use of information in recognition tasks , 2001, Vision Research.

[32]  S. Yantis,et al.  Visual attention: control, representation, and time course. , 1997, Annual review of psychology.

[33]  Stefan Treue,et al.  Adaptation to statistical properties of visual scenes biases rapid categorization , 2007 .

[34]  D. Ruderman,et al.  Independent component analysis of natural image sequences yields spatio-temporal filters similar to simple cells in primary visual cortex , 1998, Proceedings of the Royal Society of London. Series B: Biological Sciences.

[35]  W. Geisler Visual perception and the statistical properties of natural scenes. , 2008, Annual review of psychology.

[36]  E. Halgren,et al.  Top-down facilitation of visual recognition. , 2006, Proceedings of the National Academy of Sciences of the United States of America.

[37]  R. Shepard Ecological constraints on internal representation: resonant kinematics of perceiving, imagining, thinking, and dreaming. , 1984, Psychological review.

[38]  D J Field,et al.  Relations between the statistics of natural images and the response properties of cortical cells. , 1987, Journal of the Optical Society of America. A, Optics and image science.

[39]  Michelle R. Greene,et al.  PSYCHOLOGICAL SCIENCE Research Article The Briefest of Glances The Time Course of Natural Scene Understanding , 2022 .

[40]  F. Attneave Some informational aspects of visual perception. , 1954, Psychological review.

[41]  K Nakayama,et al.  Visual attention to surfaces in three-dimensional space. , 1995, Proceedings of the National Academy of Sciences of the United States of America.

[42]  Doris Y. Tsao,et al.  What's so special about the average face? , 2006, Trends in Cognitive Sciences.