On Visual Realism of Synthesized Imagery

Traditionally, computer graphics has been concerned with producing imagery that is as physically accurate as possible. But accurate physical simulation of geometry, lighting, and material properties of a visual scene can be cumbersome and time consuming. At the same time, human vision is far from accurate, which offers an enormous opportunity to create imagery at a reduced computational cost as well as with less reliance on human modelers. As a result, a recent trend is toward accepting perceptual plausibility instead of physical accuracy as a guiding principle in the design of modeling and rendering systems. This requires us to understand visual realism, which involves both learning statistical regularities of the world, for instance, by employing huge amounts of data, as well as human's visual perception of it. This paper addresses issues related to understanding realism, presents several applications, and discusses what this interesting approach may lead to in the future.

[1]  Alexei A. Efros,et al.  Scene completion using millions of photographs , 2008, Commun. ACM.

[2]  Tom Geller,et al.  Overcoming the Uncanny Valley , 2008, IEEE Computer Graphics and Applications.

[3]  Karol Myszkowski,et al.  Using the visual differences predictor to improve performance of progressive global illumination computation , 2000, TOGS.

[4]  Erik Reinhard,et al.  A Survey of Image Statistics Relevant to Computer Graphics , 2011, Comput. Graph. Forum.

[5]  S. Pont,et al.  Material — Illumination Ambiguities and the Perception of Solid Objects , 2006, Perception.

[6]  S. Zucker,et al.  Shape-from-shading on a cloudy day , 1994 .

[7]  Matthijs C. Dorst Distinctive Image Features from Scale-Invariant Keypoints , 2011 .

[8]  Robert Bridson,et al.  Fluid Simulation for Computer Graphics , 2008 .

[9]  David Salesin,et al.  Image Analogies , 2001, SIGGRAPH.

[10]  Michael Isard,et al.  Total Recall: Automatic Query Expansion with a Generative Feature Model for Object Retrieval , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[11]  Marc Levoy,et al.  Light field rendering , 1996, SIGGRAPH.

[12]  H. Jensen Realistic Image Synthesis Using Photon Mapping , 2001 .

[13]  Shi-Min Hu,et al.  Sketch2Photo: internet image montage , 2009, ACM Trans. Graph..

[14]  Peter Norvig,et al.  The Unreasonable Effectiveness of Data , 2009, IEEE Intelligent Systems.

[15]  Edward Cutrell,et al.  Measuring the Perception of Visual Realism in Images , 2001, Rendering Techniques.

[16]  Jean-Michel Morel,et al.  A non-local algorithm for image denoising , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[17]  Sung Yong Shin,et al.  On pixel-based texture synthesis by non-parametric sampling , 2006, Comput. Graph..

[18]  Steven M. Seitz,et al.  Photo tourism: exploring photo collections in 3D , 2006, ACM Trans. Graph..

[19]  Aapo Hyvärinen,et al.  Natural Image Statistics - A Probabilistic Approach to Early Computational Vision , 2009, Computational Imaging and Vision.

[20]  R. L. Cook,et al.  Stochastic simplification of aggregate detail , 2007, SIGGRAPH 2007.

[21]  A. Torralba,et al.  Specular reflections and the perception of shape. , 2004, Journal of vision.

[22]  Suman K. Mitra,et al.  Color Transfer Using Motion Estimations and Its Application to Video Compression , 2005, CAIP.

[23]  Philippe Bekaert,et al.  Advanced Global Illumination, Second Edition , 2006 .

[24]  Patrick Pérez,et al.  Poisson image editing , 2003, ACM Trans. Graph..

[25]  Jan Kautz,et al.  Is accurate occlusion of glossy reflections necessary? , 2007, APGV.

[26]  Erik Reinhard,et al.  Colour Spaces for Colour Transfer , 2011, CCIW.

[27]  Min H. Kim,et al.  Perceptual influence of approximate visibility in indirect illumination , 2009, TAP.

[28]  Erik Reinhard,et al.  Progressive color transfer for images of arbitrary dynamic range , 2011, Comput. Graph..

[29]  Jan Kautz,et al.  Video-based characters: creating new human performances from a multi-view video database , 2011, SIGGRAPH 2011.

[30]  William Bialek,et al.  Statistics of Natural Images: Scaling in the Woods , 1993, NIPS.

[31]  Roberto Cipolla,et al.  Semantic Photo Synthesis , 2006, Electronic Imaging.

[32]  Luís Paulo Santos,et al.  Selective component-based rendering , 2005, GRAPHITE '05.

[33]  J. Koenderink,et al.  The internal representation of solid shape with respect to vision , 1979, Biological Cybernetics.

[34]  Jitendra Malik,et al.  Recognizing action at a distance , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[35]  Richard Szeliski,et al.  Video textures , 2000, SIGGRAPH.

[36]  I. Motoyoshi Highlight-shading relationship as a cue for the perception of translucent and transparent materials. , 2010, Journal of vision.

[37]  David J. Kriegman,et al.  The Bas-Relief Ambiguity , 2004, International Journal of Computer Vision.

[38]  Alexei A. Efros,et al.  Data-driven visual similarity for cross-domain image matching , 2011, ACM Trans. Graph..

[39]  Ping-Sing Tsai,et al.  Shape from Shading: A Survey , 1999, IEEE Trans. Pattern Anal. Mach. Intell..

[40]  Kavita Bala,et al.  Visual equivalence in dynamic scenes , 2009 .

[41]  M. Landy,et al.  The Plenoptic Function and the Elements of Early Vision , 1991 .

[42]  James Arvo,et al.  A framework for realistic image synthesis , 1997, SIGGRAPH.

[43]  Philip Dutré,et al.  The influence of shape on the perception of material reflectance , 2007, ACM Trans. Graph..

[44]  Alexei A. Efros,et al.  Webcam clip art: appearance and illuminant transfer from time-lapse sequences , 2009, ACM Trans. Graph..

[45]  Jessica K. Hodgins,et al.  Interactive control of avatars animated with human motion data , 2002, SIGGRAPH.

[46]  Shree K. Nayar,et al.  A practical analytic single scattering model for real time rendering , 2005, SIGGRAPH '05.

[47]  J T Todd,et al.  Ambiguity and the ‘Mental Eye’ in Pictorial Relief , 2001, Perception.

[48]  D J Field,et al.  Relations between the statistics of natural images and the response properties of cortical cells. , 1987, Journal of the Optical Society of America. A, Optics and image science.

[49]  Erik Reinhard,et al.  Image Statistics in Visual Computing , 2013 .

[50]  Wojciech Matusik,et al.  Image restoration using online photo collections , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[51]  Andrew Zisserman,et al.  Get Out of my Picture! Internet-based Inpainting , 2009, BMVC.

[52]  Sylvain Lefebvre,et al.  An Interactive Perceptual Rendering Pipeline using Contrast and Spatial Masking , 2007, Rendering Techniques.

[53]  H. Bülthoff,et al.  Depth Discrimination from Shading under Diffuse Lighting , 2000, Perception.

[54]  Lucas Kovar,et al.  Motion graphs , 2002, SIGGRAPH Classes.

[55]  William T. Freeman,et al.  Example-Based Super-Resolution , 2002, IEEE Computer Graphics and Applications.

[56]  Douglas Lanman,et al.  BiDi screen: a thin, depth-sensing LCD for 3D interaction using light fields , 2009, SIGGRAPH 2009.

[57]  Richard Szeliski,et al.  Multi-image matching using multi-scale oriented patches , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[58]  Anselmo Lastra,et al.  Real‐Time Cloud Rendering , 2001, Comput. Graph. Forum.

[59]  Richard Szeliski,et al.  The lumigraph , 1996, SIGGRAPH.

[60]  Erik Reinhard,et al.  Image-based material editing , 2005, SIGGRAPH '05.

[61]  Jitendra Malik,et al.  Modeling and Rendering Architecture from Photographs: A hybrid geometry- and image-based approach , 1996, SIGGRAPH.

[62]  Erik Reinhard Example-Based Image Manipulation , 2012, CGIV.

[63]  Antonio Torralba,et al.  Ieee Transactions on Pattern Analysis and Machine Intelligence 1 80 Million Tiny Images: a Large Dataset for Non-parametric Object and Scene Recognition , 2022 .

[64]  Wojciech Matusik,et al.  CG2Real: Improving the Realism of Computer Generated Images Using a Large Collection of Photographs , 2011, IEEE Transactions on Visualization and Computer Graphics.

[65]  D. Ruderman,et al.  Statistics of cone responses to natural images: implications for visual coding , 1998 .

[66]  Olga Veksler,et al.  Fast Approximate Energy Minimization via Graph Cuts , 2001, IEEE Trans. Pattern Anal. Mach. Intell..

[67]  Olivier Dumont,et al.  A perceptual heuristic for shadow computation in photo-realistic images , 2006, SIGGRAPH '06.

[68]  Alexei A. Efros,et al.  Photo clip art , 2007, ACM Trans. Graph..

[69]  Zhenhua Li,et al.  Color transfer based remote sensing image fusion using non-separable wavelet frame transform , 2005, Pattern Recognit. Lett..

[70]  Hans-Peter Seidel,et al.  Imperfect shadow maps for efficient computation of indirect illumination , 2008, SIGGRAPH Asia '08.

[71]  L. Maloney,et al.  Visual Perception of Thick Transparent Materials , 2011, Psychological science.

[72]  James A. Ferwerda Through a Glass Brightly: Material Appearance and Image Quality , 2012 .

[73]  Ken-ichi Anjyo,et al.  Tour into the picture: using a spidery mesh interface to make animation from a single image , 1997, SIGGRAPH.

[74]  William T. Freeman,et al.  The generic viewpoint assumption in a framework for visual perception , 1994, Nature.

[75]  Antonio Torralba,et al.  Infinite Images: Creating and Exploring a Large Photorealistic Virtual Space , 2008, Proceedings of the IEEE.

[76]  Erik Reinhard,et al.  Statistical regularities in low and high dynamic range images , 2010, APGV '10.

[77]  Heinrich H. Bülthoff,et al.  Low-Level Image Cues in the Perception of Translucent Materials , 2005, TAP.

[78]  Frédo Durand,et al.  Efficient Reflectance and Visibility Approximations for Environment Map Rendering , 2007, Comput. Graph. Forum.

[79]  Sylvia C. Pont,et al.  A comparison of material and illumination discrimination performance for real rough, real smooth and computer generated smooth spheres , 2005, APGV '05.

[80]  William T. Freeman,et al.  Exploiting the generic viewpoint assumption , 1996, International Journal of Computer Vision.

[81]  Antonio Torralba,et al.  Building the gist of a scene: the role of global image features in recognition. , 2006, Progress in brain research.

[82]  Alexei A. Efros,et al.  Automatic photo pop-up , 2005, SIGGRAPH 2005.

[83]  Edward H. Adelson,et al.  On seeing stuff: the perception of materials by humans and machines , 2001, IS&T/SPIE Electronic Imaging.

[84]  Shenchang Eric Chen,et al.  QuickTime VR: an image-based approach to virtual environment navigation , 1995, SIGGRAPH.

[85]  Alexei A. Efros,et al.  Image quilting for texture synthesis and transfer , 2001, SIGGRAPH.

[86]  Bruce Walter,et al.  Visual equivalence: towards a new standard for image fidelity , 2007, SIGGRAPH 2007.

[87]  Erik Reinhard,et al.  Color Transfer between Images , 2001, IEEE Computer Graphics and Applications.

[88]  Roberto Cipolla,et al.  PhotoBuilder-3D models of architectural scenes from uncalibrated images , 1999, Proceedings IEEE International Conference on Multimedia Computing and Systems.

[89]  Bernhard P. Wrobel,et al.  Multiple View Geometry in Computer Vision , 2001 .

[90]  David Mumford,et al.  Statistics of natural images and models , 1999, Proceedings. 1999 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No PR00149).

[91]  Donald P. Greenberg,et al.  A perceptually based physical error metric for realistic image synthesis , 1999, SIGGRAPH.

[92]  Veronica Sundstedt,et al.  Measuring the perception of light inconsistencies , 2010, APGV '10.

[93]  Kavita Bala,et al.  Perception of complex aggregates , 2008, SIGGRAPH 2008.

[94]  Hans-Peter Seidel,et al.  Perception-guided global illumination solution for animation rendering , 2001, SIGGRAPH.

[95]  Okan Arikan,et al.  Interactive motion generation from examples , 2002, ACM Trans. Graph..

[96]  Donald P. Greenberg,et al.  A model of visual masking for computer graphics , 1997, SIGGRAPH.

[97]  Patrick Cavanagh,et al.  Perceiving Illumination Inconsistencies in Scenes , 2005, Perception.

[98]  Alexander Toet,et al.  Natural colour mapping for multiband nightvision imagery , 2003, Inf. Fusion.

[99]  Donald P. Greenberg,et al.  Perceptual illumination components: a new approach to efficient, high quality global illumination rendering , 2004, SIGGRAPH 2004.