The lawful imprecision of human surface tilt estimation in natural scenes

Estimating local surface orientation (slant and tilt) is fundamental to recovering the three-dimensional structure of the environment. It is unknown how well humans perform this task in natural scenes. Here, with a database of natural stereo-images having groundtruth surface orientation at each pixel, we find dramatic differences in human tilt estimation with natural and artificial stimuli. Estimates are precise and unbiased with artificial stimuli and imprecise and strongly biased with natural stimuli. An image-computable Bayes optimal model grounded in natural scene statistics predicts human bias, precision, and trial-by-trial errors without fitting parameters to the human data. The similarities between human and model performance suggest that the complex human performance patterns with natural stimuli are lawful, and that human visual systems have internalized local image and scene statistics to optimally infer the three-dimensional structure of the environment. These results generalize our understanding of vision from the lab to the real world.

[1]  S. Appelle.  Perception and discrimination as a function of stimulus orientation: the "oblique effect" in man and animals. , 1972, Psychological bulletin.

[2]  Jeanny Hérault,et al.  Model of Frequency Analysis in the Visual Cortex and the Shape from Texture Problem , 2008, International Journal of Computer Vision.

[3]  M. Hayhoe,et al.  In what ways do eye movements contribute to everyday activities? , 2001, Vision Research.

[4]  Stéphane Mallat,et al.  The Texture Gradient Equation for Recovering Shape from Texture , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[5]  Johannes Burge,et al.  Accuracy Maximization Analysis for Sensory-Perceptual Tasks: Computational Improvements, Filter Robustness, and Coding Advantages for Scaled Additive Noise , 2017, PLoS Comput. Biol..

[6]  Hideko F. Norman,et al.  Visual discrimination of local surface structure: Slant, tilt, and curvedness , 2006, Vision Research.

[7]  Dale Purves,et al.  Image/source statistics of surfaces in natural scenes , 2003, Network.

[8]  M. Landy,et al.  Why Is Spatial Stereoresolution So Low? , 2004, The Journal of Neuroscience.

[9]  R. F. Wagner,et al.  Efficiency of human visual signal discrimination. , 1981, Science.

[10]  James H. Elder,et al.  Texture properties affecting the accuracy of surface attitude judgements , 2006, Vision Research.

[11]  Thomas Martinetz,et al.  Variability of eye movements when viewing dynamic natural scenes. , 2010, Journal of vision.

[12]  Dale Purves,et al.  A statistical explanation of visual space , 2003, Nature Neuroscience.

[13]  Johannes Burge,et al.  Linking normative models of natural tasks to descriptive models of neural response , 2017, bioRxiv.

[14]  Wilson S. Geisler,et al.  Optimal speed estimation in natural image movies predicts human performance , 2015, Nature Communications.

[15]  Wilson S. Geisler,et al.  Optimal defocus estimates from individual images for autofocusing a digital camera , 2012, Electronic Imaging.

[16]  Jitendra Malik,et al.  Surface orientation from texture: Isotropy or homogeneity (or both)? , 1997, Vision Research.

[17]  H. Bülthoff,et al.  3D shape perception from combined depth cues in human visual cortex , 2005, Nature Neuroscience.

[18]  M. Ernst,et al.  Humans integrate visual and haptic information in a statistically optimal fashion , 2002, Nature.

[19]  Christopher W. Tyler,et al.  Binocular cross-correlation in time and space , 1978, Vision Research.

[20]  Qasim Zaidi,et al.  Three-dimensional shape from non-homogeneous textures: carved and stretched surfaces. , 2004, Journal of vision.

[21]  W. Geisler,et al.  Optimal disparity estimation in natural stereo images. , 2014, Journal of vision.

[22]  James M. Hillis,et al.  Slant from texture and disparity cues: optimal cue combination. , 2004, Journal of vision.

[23]  S. McKee,et al.  Disparity increment thresholds for gratings. , 2004, Journal of vision.

[24]  Kent A. Stevens,et al.  Slant-tilt: The visual encoding of surface orientation , 1983, Biological Cybernetics.

[25]  Johannes Burge,et al.  Optimal defocus estimation in individual natural images , 2011, Proceedings of the National Academy of Sciences.

[26]  Jiri Najemnik,et al.  Optimal stimulus encoders for natural tasks. , 2009, Journal of vision.

[27]  M. Landy,et al.  Weighted linear cue combination with possibly correlated error , 2003, Vision Research.

[28]  Takahisa M. Sanada,et al.  Representation of 3-D surface orientation by velocity and disparity gradient cues in area MT. , 2012, Journal of Neurophysiology.

[29]  Adam Binch,et al.  Perception as Bayesian Inference , 2014 .

[30]  H. Sakata,et al.  Integration of perspective and disparity cues in surface-orientation-selective neurons of area CIP. , 2001, Journal of neurophysiology.

[31]  M. Banks,et al.  Visual–Haptic Adaptation Is Determined by Relative Reliability , 2010, The Journal of Neuroscience.

[32]  H. Bülthoff,et al.  Estimation of 3D shape from image orientations , 2011, Proceedings of the National Academy of Sciences.

[33]  M. Ernst,et al.  Focus cues affect perceived depth. , 2005, Journal of vision.

[34]  Edward H. Adelson,et al.  Motion illusions as optimal percepts , 2002, Nature Neuroscience.

[35]  J. Pelz,et al.  Oculomotor behavior in natural and man-made environments , 2007 .

[36]  Charless C. Fowlkes,et al.  Natural-Scene Statistics Predict How the Figure–Ground Cue of Convexity Affects Human Depth Perception , 2010, The Journal of Neuroscience.

[37]  John P. Wann,et al.  Where you look when you learn to steer , 2004 .

[38]  Pascal Mamassian,et al.  Temporal dynamics in bistable perception. , 2005, Journal of vision.

[39]  Eero P. Simoncelli,et al.  Noise characteristics and prior expectations in human visual speed perception , 2006, Nature Neuroscience.

[40]  J. Todd,et al.  Effects of changing viewing conditions on the perceived structure of smoothly curved surfaces. , 1996, Journal of experimental psychology. Human perception and performance.

[41]  Eero P. Simoncelli,et al.  Cardinal rules: Visual orientation perception reflects knowledge of environmental statistics , 2011, Nature Neuroscience.

[42]  Julian Leyland,et al.  The Southampton-York Natural Scenes (SYNS) dataset: Statistics of surface attitude , 2016, Scientific Reports.

[43]  C. Furmanski,et al.  An oblique effect in human primary visual cortex , 2000, Nature Neuroscience.

[44]  Jitendra Malik,et al.  Computing Local Surface Orientation and Shape from Texture for Curved Surfaces , 1997, International Journal of Computer Vision.

[45]  J. Todd.  Review TRENDS in Cognitive Sciences Vol.8 No.3 March 2004 The visual perception of 3D shape q , 2022 .

[46]  Barton L. Anderson,et al.  Coupled computations of three-dimensional shape and material , 2015, Current Biology.

[47]  Ari Rosenberg,et al.  The Visual Representation of 3D Object Orientation in Parietal Cortex , 2013, The Journal of Neuroscience.

[48]  Robin L. Hill,et al.  Eye movements : a window on mind and brain , 2007 .

[49]  Andrea Li,et al.  Perception of three-dimensional shape from texture is based on patterns of oriented energy , 2000, Vision Research.

[50]  Josh H. McDermott,et al.  Psychophysics with junctions in real images. , 2010, Perception.

[51]  David C. Knill,et al.  Surface orientation from texture: ideal observers, generic observers and the information content of texture cues , 1998, Vision Research.

[52]  H. Barlow.  Vision: A computational investigation into the human representation and processing of visual information: David Marr. San Francisco: W. H. Freeman, 1982. pp. xvi + 397 , 1983 .

[53]  Joan Lasenby,et al.  Shape from Texture: Fast Estimation of Planar Surface Orientation via Fourier Analysis , 2007, BMVC.

[54]  J. Saunders,et al.  Perception of 3D surface orientation from skew symmetry , 2001, Vision Research.

[55]  James T Todd,et al.  The visual perception of 3-D shape from multiple cues: Are observers capable of perceiving metric structure? , 2003, Perception & psychophysics.

[56]  Brian C. McCann,et al.  Estimating 3D tilt from local image cues in natural scenes , 2016, Journal of vision.

[57]  D. Knill.  Ideal observer perturbation analysis reveals human strategies for inferring surface orientation from texture , 1998, Vision Research.

[58]  Hiroshi Ban,et al.  Integration of texture and disparity cues to surface slant in dorsal visual cortex. , 2013, Journal of neurophysiology.