Material and shape perception based on two types of intensity gradient information

Visual estimation of the material and shape of an object from a single image includes a hard ill-posed computational problem. However, in our daily life we feel we can estimate both reasonably well. The neural computation underlying this ability remains poorly understood. Here we propose that the human visual system uses different aspects of object images to separately estimate the contributions of the material and shape. Specifically, material perception relies mainly on the intensity gradient magnitude information, while shape perception relies mainly on the intensity gradient order information. A clue to this hypothesis was provided by the observation that luminance-histogram manipulation, which changes luminance gradient magnitudes but not the luminance-order map, effectively alters the material appearance but not the shape of an object. In agreement with this observation, we found that the simulated physical material changes do not significantly affect the intensity order information. A series of psychophysical experiments further indicate that human surface shape perception is robust against intensity manipulations provided they do not disturb the intensity order information. In addition, we show that the two types of gradient information can be utilized for the discrimination of albedo changes from highlights. These findings suggest that the visual system relies on these diagnostic image features to estimate physical properties in a distal world.

[1]  Paul E. Debevec,et al.  Image-based lighting , 2002, IEEE Computer Graphics and Applications.

[2]  Edward H. Adelson,et al.  Luminance re-mapping for the control of apparent material , 2005, APGV '05.

[3]  E. Land,et al.  Lightness and retinex theory. , 1971, Journal of the Optical Society of America.

[4]  Erik Reinhard,et al.  Image-based material editing , 2005, SIGGRAPH '05.

[5]  Shin'ya Nishida,et al.  Visual wetness perception based on image color statistics. , 2017, Journal of vision.

[6]  Bill Triggs,et al.  Histograms of oriented gradients for human detection , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[7]  Erik Reinhard,et al.  Color Transfer between Images , 2001, IEEE Computer Graphics and Applications.

[8]  J. Koenderink,et al.  Photometric Invariants Related to Solid Shape , 1980 .

[9]  William B. Cowan,et al.  An inexpensive scheme for calibration of a colour monitor in terms of CIE standard coordinates , 1983, SIGGRAPH.

[10]  Shuicheng Yan,et al.  An HOG-LBP human detector with partial occlusion handling , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[11]  Naokazu Goda,et al.  Neural Selectivity and Representation of Gloss in the Monkey Inferior Temporal Cortex , 2012, The Journal of Neuroscience.

[12]  Stephen Lin,et al.  A Closed-Form Solution to Retinex with Nonlocal Texture Constraints , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[13]  J T Todd,et al.  Ambiguity and the ‘Mental Eye’ in Pictorial Relief , 2001, Perception.

[14]  Phillip J. Marlow,et al.  The Perception and Misperception of Specular Surface Reflectance , 2012, Current Biology.

[15]  Pascal Barla,et al.  Flow-guided warping for image-based shape manipulation , 2016, ACM Trans. Graph..

[16]  H BülthoffHeinrich,et al.  Low-Level Image Cues in the Perception of Translucent Materials , 2005 .

[17]  Vincent Lepetit,et al.  DAISY: An Efficient Dense Descriptor Applied to Wide-Baseline Stereo , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[18]  Noah Snavely,et al.  Intrinsic images in the wild , 2014, ACM Trans. Graph..

[19]  Romain Vergne,et al.  Predicting the effects of illumination in shape from shading , 2013 .

[20]  E. Adelson,et al.  Image statistics for surface reflectance perception. , 2008, Journal of the Optical Society of America. A, Optics, image science, and vision.

[21]  Barton L Anderson,et al.  Image statistics do not explain the perception of gloss and lightness. , 2009, Journal of vision.

[22]  Atsushi Kimura,et al.  Luminance distribution modifies the perceived freshness of strawberries , 2012, i-Perception.

[23]  Astrid M L Kappers,et al.  Shape-from-shading for matte and glossy objects. , 2006, Acta psychologica.

[24]  Naokazu Goda,et al.  Perceptual Gloss Parameters Are Encoded by Population Responses in the Monkey Inferior Temporal Cortex , 2014, The Journal of Neuroscience.

[25]  Bin Fan,et al.  Local Intensity Order Pattern for feature description , 2011, 2011 International Conference on Computer Vision.

[26]  Edward H. Adelson,et al.  Estimating Intrinsic Component Images using Non-Linear Regression , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[27]  Edward H. Adelson,et al.  Image statistics for 3D shape estimation , 2010 .

[28]  J. Beck,et al.  Highlights and the perception of glossiness , 1981, Perception & psychophysics.

[29]  Edward H. Adelson,et al.  On seeing stuff: the perception of materials by humans and machines , 2001, IS&T/SPIE Electronic Imaging.

[30]  Steven W. Zucker,et al.  How Shading Constrains Surface Patches without Knowledge of Light Sources , 2014, SIAM J. Imaging Sci..

[31]  Erik Reinhard,et al.  Photographic tone reproduction for digital images , 2002, ACM Trans. Graph..

[32]  Shin'ya Nishida,et al.  Human perception of subresolution fineness of dense textures based on image intensity statistics. , 2017, Journal of vision.

[33]  Philipp Berens,et al.  CircStat: AMATLABToolbox for Circular Statistics , 2009, Journal of Statistical Software.

[34]  J. Todd,et al.  Is the perception of 3D shape from shading based on assumed reflectance and illumination? , 2014, i-Perception.

[35]  Edward H. Adelson,et al.  Shape from Sheen , 2009 .

[36]  Katja Doerschner,et al.  Relative flattening between velvet and matte 3D shapes: evidence for similar shape-from-shading computations. , 2012, Journal of vision.

[37]  Barton L. Anderson,et al.  Coupled computations of three-dimensional shape and material , 2015, Current Biology.

[38]  Barton L Anderson,et al.  The role of brightness and orientation congruence in the perception of surface gloss. , 2011, Journal of vision.

[39]  Marc Levoy,et al.  A volumetric method for building complex models from range images , 1996, SIGGRAPH.

[40]  Pascal Barla,et al.  Surface flows for image-based shading design , 2012, ACM Trans. Graph..

[41]  Gregory J. Ward,et al.  Measuring and modeling anisotropic reflection , 1992, SIGGRAPH.

[42]  D H Brainard,et al.  The Psychophysics Toolbox. , 1997, Spatial vision.

[43]  E. Mingolla,et al.  Perception of surface curvature and direction of illumination from patterns of shading. , 1983, Journal of experimental psychology. Human perception and performance.

[44]  Greg Humphreys,et al.  Physically Based Rendering: From Theory to Implementation , 2004 .

[45]  Steven W. Zucker,et al.  Shadows and shading flow fields , 1996, Proceedings CVPR IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[46]  D G Pelli,et al.  The VideoToolbox software for visual psychophysics: transforming numbers into movies. , 1997, Spatial vision.

[47]  Jitendra Malik,et al.  Shape, Illumination, and Reflectance from Shading , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[48]  A. Torralba,et al.  Specular reflections and the perception of shape. , 2004, Journal of vision.

[49]  Qasim Zaidi,et al.  Frequency-based heuristics for material perception. , 2013, Journal of vision.

[50]  Wojciech Matusik,et al.  A data-driven reflectance model , 2003, ACM Trans. Graph..

[51]  Ohad Ben-Shahar,et al.  On the perceptual organization of texture and shading flows: from a geometrical model to coherence computation , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[52]  Eero P. Simoncelli,et al.  Differentiation of discrete multidimensional signals , 2004, IEEE Transactions on Image Processing.

[53]  David J. Kriegman,et al.  The Bas-Relief Ambiguity , 2004, International Journal of Computer Vision.

[54]  Paul A. Viola,et al.  Bayesian Model of Surface Perception , 1997, NIPS.

[55]  Edward H. Adelson,et al.  Band-Sifting Decomposition for Image-Based Material Editing , 2015, ACM Trans. Graph..

[56]  S. Pont,et al.  Material — Illumination Ambiguities and the Perception of Solid Objects , 2006, Perception.

[57]  E. Adelson,et al.  Image statistics and the perception of surface qualities , 2007, Nature.

[58]  Garrett M. Johnson,et al.  Color Imaging: Fundamentals and Applications , 2008 .

[59]  William T. Freeman,et al.  Learning Local Evidence for Shading and Reflectance , 2001, ICCV.

[60]  J. Koenderink,et al.  Surface perception in pictures , 1992, Perception & psychophysics.

[61]  B. Anderson,et al.  Specular Image Structure Modulates the Perception of Three-Dimensional Shape , 2014, Current Biology.

[62]  Astrid M L Kappers,et al.  Shape from Shading from Images Rendered with Various Surface Types and Light Fields , 2007, Perception.

[63]  H. Bülthoff,et al.  Estimation of 3D shape from image orientations , 2011, Proceedings of the National Academy of Sciences.

[64]  Naohisa Miyakawa,et al.  Representation of Glossy Material Surface in Ventral Superior Temporal Sulcal Area of Common Marmosets , 2017, Front. Neural Circuits.

[65]  Phillip J. Marlow,et al.  The perception of gloss depends on highlight congruence with surface shading. , 2011, Journal of vision.

[66]  J. Koenderink,et al.  Local shape of pictorial relief , 2014, i-Perception.

[67]  S. Nishida,et al.  Use of image-based information in judgments of surface-reflectance properties. , 1998, Journal of the Optical Society of America. A, Optics, image science, and vision.