Intrinsic Decompositions for Image Editing

Intrinsic images are a mid‐level representation of an image that decompose the image into reflectance and illumination layers. The reflectance layer captures the color/texture of surfaces in the scene, while the illumination layer captures shading effects caused by interactions between scene illumination and surface geometry. Intrinsic images have a long history in computer vision and recently in computer graphics, and have been shown to be a useful representation for tasks ranging from scene understanding and reconstruction to image editing. In this report, we review and evaluate past work on this problem. Specifically, we discuss each work in terms of the priors they impose on the intrinsic image problem. We introduce a new synthetic ground‐truth dataset that we use to evaluate the validity of these priors and the performance of the methods. Finally, we evaluate the performance of the different methods in the context of image‐editing applications.

[1]  Erik Reinhard,et al.  Multiple Light Source Estimation in a Single Image , 2013, Comput. Graph. Forum.

[2]  Xuelong Li,et al.  Intrinsic images using optimization , 2011, CVPR 2011.

[3]  Seungyong Lee,et al.  Intrinsic Image Decomposition using Deep Convolutional Network , 2016 .

[4]  Jonathan T. Barron,et al.  Scene Intrinsics and Depth from a Single Image , 2015, 2015 IEEE International Conference on Computer Vision Workshop (ICCVW).

[5]  Joost van de Weijer,et al.  Intrinsic image evaluation on synthetic complex scenes , 2013, 2013 IEEE International Conference on Image Processing.

[6]  Stella X. Yu,et al.  Direct Intrinsics: Learning Albedo-Shading Decomposition by Convolutional Regression , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[7]  Alexei A. Efros,et al.  Learning Data-Driven Reflectance Priors for Intrinsic Image Decomposition , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[8]  Michael J. Black,et al.  A Naturalistic Open Source Movie for Optical Flow Evaluation , 2012, ECCV.

[9]  John W. Fisher,et al.  Bayesian Nonparametric Intrinsic Image Decomposition , 2014, ECCV.

[10]  Stephen Lin,et al.  A Closed-Form Solution to Retinex with Nonlocal Texture Constraints , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[11]  Edward H. Adelson,et al.  Ground truth dataset and baseline evaluations for intrinsic image algorithms , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[12]  Bernard Ghanem,et al.  Intrinsic Scene Decomposition from RGB-D Images , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[13]  Stephen Lin,et al.  Intrinsic image decomposition with non-local texture cues , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[14]  Edward H. Adelson,et al.  Estimating Intrinsic Component Images using Non-Linear Regression , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[15]  Bogdan Raducanu,et al.  Evaluation of Intrinsic Image Algorithms to Detect the Shadows Cast by Static Objects Outdoors , 2012, Sensors.

[16]  S. Duchêne,et al.  Multi view delighting and relighting , 2015 .

[17]  Edward H. Adelson,et al.  Recovering intrinsic images from a single image , 2005, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[18]  William T. Freeman,et al.  Learning Ordinal Relationships for Mid-Level Vision , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[19]  H. Barrow,et al.  RECOVERING INTRINSIC SCENE CHARACTERISTICS FROM IMAGES , 1978 .

[20]  Alexei A. Efros,et al.  Synthesizing Environment Maps from a Single Image , 2010 .

[21]  Rong Bing,et al.  Prevalence of Echocardiography Use in Patients Hospitalized with Confirmed Acute Pulmonary Embolism: A Real-World Observational Multicenter Study , 2016, PloS one.

[22]  Jitendra Malik,et al.  Shape, Illumination, and Reflectance from Shading , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[23]  Jitendra Malik,et al.  Shape, albedo, and illumination from a single image of an unknown object , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[24]  Stella X. Yu,et al.  Learning lightness from human judgement on relative reflectance , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[25]  Sei-Wang Chen,et al.  Intrinsic Image Extraction from a Single Image , 2009, J. Inf. Sci. Eng..

[26]  Kun Zhou,et al.  Intrinsic Light Field Images , 2016, Comput. Graph. Forum.

[27]  Andrew B. Watson,et al.  Digital images and human vision , 1993 .

[28]  Jitendra Malik,et al.  Intrinsic Scene Properties from a Single RGB-D Image , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[29]  Kun Zhou,et al.  Intrinsic Face Image Decomposition with Human Face Priors , 2014, ECCV.

[30]  Stephen Lin,et al.  Estimating Intrinsic Images from Image Sequences with Biased Illumination , 2004, ECCV.

[31]  Stephen Lin,et al.  Estimation of Intrinsic Image Sequences from Image+Depth Video , 2012, ECCV.

[32]  Noah Snavely,et al.  Photometric Ambient Occlusion for Intrinsic Image Decomposition , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[33]  Andreas Kolb,et al.  Multi-view Multi-illuminant Intrinsic Dataset , 2016, BMVC.

[34]  Andreas Kolb,et al.  A Comprehensive Multi-Illuminant Dataset for Benchmarking of the Intrinsic Image Algorithms , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[35]  Yair Weiss,et al.  Deriving intrinsic images from image sequences , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.

[36]  Michael J. Black,et al.  Intrinsic Depth: Improving Depth Transfer with Intrinsic Images , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[37]  Noah Snavely,et al.  OpenSurfaces , 2013, ACM Trans. Graph..

[38]  Bastian Goldlücke,et al.  A Variational Model for Intrinsic Light Field Decomposition , 2016, ACCV.

[39]  Frédo Durand,et al.  Light mixture estimation for spatially varying white balance , 2008, ACM Trans. Graph..

[40]  Nanning Zheng,et al.  Intrinsic Image Decomposition from Pair-Wise Shading Ordering , 2014, ACCV.

[41]  Peter V. Gehler,et al.  Recovering Intrinsic Images with a Global Sparsity Prior on Reflectance , 2011, NIPS.

[42]  Peter V. Gehler,et al.  Intrinsic Video , 2014, ECCV.

[43]  Xiaoyue Jiang,et al.  Correlation-Based Intrinsic Image Extraction from a Single Image , 2010, ECCV.

[44]  Adrien Bousseau,et al.  Rich Intrinsic Image Decomposition of Outdoor Scenes from Multiple Views , 2012, IEEE Transactions on Visualization and Computer Graphics.

[45]  Qionghai Dai,et al.  Intrinsic video and applications , 2014, ACM Trans. Graph..

[46]  Greg Humphreys,et al.  Physically Based Rendering, Second Edition: From Theory To Implementation , 2010 .

[47]  Pierre-Yves Laffont Intrinsic image decomposition from multiple photographs , 2012 .

[48]  Peter I. Corke,et al.  Dealing with shadows: Capturing intrinsic scene appearance for image-based outdoor localisation , 2013, 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[49]  Noah Snavely,et al.  Reasoning about Photo Collections using Models of Outdoor Illumination , 2014, BMVC.

[50]  Sylvain Paris,et al.  User-assisted intrinsic images , 2009, ACM Trans. Graph..

[51]  K. Hohn,et al.  Determining Lightness from an Image , 2004 .

[52]  Edward H. Adelson,et al.  Band-Sifting Decomposition for Image-Based Material Editing , 2015, ACM Trans. Graph..

[53]  M. Werman,et al.  Color lines: image specific color representation , 2004, Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004..

[54]  Mario Fritz,et al.  Deep Reflectance Maps , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[55]  B K Horn,et al.  Calculating the reflectance map. , 1979, Applied optics.

[56]  Marc Serra Vidal Modeling, estimation and evaluation of intrinsic images considering color information , 2015 .

[57]  Hans-Peter Seidel,et al.  Photometric calibration of high dynamic range cameras , 2005 .

[58]  Michael S. Brown,et al.  Single Image Layer Separation Using Relative Smoothness , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[59]  Noah Snavely,et al.  Intrinsic images in the wild , 2014, ACM Trans. Graph..

[60]  Andrea Fusiello,et al.  Recovering Intrinsic Images by Minimizing Image Complexity , 2015, STAG.

[61]  Maneesh Agrawala,et al.  Illumination decomposition for material recoloring with consistent interreflections , 2011, ACM Trans. Graph..

[62]  Luc Van Gool,et al.  DARN: a Deep Adversial Residual Network for Intrinsic Image Decomposition , 2016, ArXiv.

[63]  Erik Reinhard,et al.  Compositing images through light source detection , 2010, Comput. Graph..

[64]  Frédo Durand,et al.  User-guided white balance for mixed lighting conditions , 2012, ACM Trans. Graph..

[65]  Tony F. Chan,et al.  Structure-Texture Image Decomposition—Modeling, Algorithms, and Parameter Selection , 2006, International Journal of Computer Vision.

[66]  Dmitry Chetverikov,et al.  A Survey of Specularity Removal Methods , 2011, Comput. Graph. Forum.

[67]  Geoffrey E. Hinton,et al.  Deep Lambertian Networks , 2012, ICML.

[68]  Chao Xu,et al.  Intrinsic Image Decomposition Using Color Invariant Edge , 2009, 2009 Fifth International Conference on Image and Graphics.

[69]  Christian Theobalt,et al.  Live intrinsic video , 2016, ACM Trans. Graph..

[70]  E. Land,et al.  Lightness and retinex theory. , 1971, Journal of the Optical Society of America.

[71]  Xuelong Li,et al.  Re-texturing by Intrinsic Video , 2010, 2010 International Conference on Digital Image Computing: Techniques and Applications.

[72]  Joost van de Weijer,et al.  Object recoloring based on intrinsic image estimation , 2011, 2011 International Conference on Computer Vision.

[73]  Bing Zeng,et al.  Intrinsic decomposition for stereoscopic images , 2016, 2016 IEEE International Conference on Image Processing (ICIP).

[74]  Jinze Yu,et al.  Rank-constrained PCA for intrinsic images decomposition , 2016, 2016 IEEE International Conference on Image Processing (ICIP).

[75]  Cheng Lu,et al.  Intrinsic Images by Entropy Minimization , 2004, ECCV.

[76]  Scott Daly,et al.  Digital Images and Human Vision , 1993 .

[77]  Yun-Hong Wang,et al.  Extracting intrinsic images from multi-spectral , 2009, 2009 International Conference on Wavelet Analysis and Pattern Recognition.

[78]  Vladlen Koltun,et al.  A Simple Model for Intrinsic Image Decomposition with Depth Cues , 2013, 2013 IEEE International Conference on Computer Vision.

[79]  Sylvain Lefebvre,et al.  Ieee Transactions on Visualization and Computer Graphics Relighting Photographs of Tree Canopies , 2022 .

[80]  J. Rabin,et al.  Blue-Black or White-Gold? Early Stage Processing and the Color of 'The Dress' , 2016, PloS one.

[81]  Adrien Bousseau,et al.  Coherent intrinsic images from photo collections , 2012, ACM Trans. Graph..

[82]  英樹 藤堂,et al.  Interactive intrinsic video editing , 2014, ACM Trans. Graph..

[83]  Seungyong Lee,et al.  Intrinsic Image Decomposition Using Structure-Texture Separation and Surface Normals , 2014, ECCV.

[84]  P. J. Narayanan,et al.  Intrinsic image decomposition using focal stacks , 2016, ICVGIP '16.

[85]  Bevil R. Conway,et al.  Striking individual differences in color perception uncovered by ‘the dress’ photograph , 2015, Current Biology.

[86]  Scott J. Daly,et al.  Visible differences predictor: an algorithm for the assessment of image fidelity , 1992, Electronic Imaging.

[87]  Sylvain Paris,et al.  Blind video temporal consistency , 2015, ACM Trans. Graph..

[88]  Chuohao Yeo,et al.  Intrinsic images decomposition using a local and global sparse representation of reflectance , 2011, CVPR 2011.

[89]  Adolfo Muñoz,et al.  Intrinsic Images by Clustering , 2012, Comput. Graph. Forum.

[90]  Patrick Pérez,et al.  Poisson image editing , 2003, ACM Trans. Graph..

[91]  Yizhou Yu,et al.  An L1 image transform for edge-preserving smoothing and scene-level intrinsic decomposition , 2015, ACM Trans. Graph..

[92]  Michael S. Brown,et al.  Beyond White: Ground Truth Colors for Color Constancy Correction , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[93]  Kun Zhou,et al.  Simulating makeup through physics-based manipulation of intrinsic image layers , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).