Matting and compositing of transparent and refractive objects

This article introduces a new approach for matting and compositing transparent and refractive objects in photographs. The key to our work is an image-based matting model, termed the Attenuation-Refraction Matte (ARM), that encodes plausible refractive properties of a transparent object along with its observed specularities and transmissive properties. We show that an object's ARM can be extracted directly from a photograph using simple user markup. Once extracted, the ARM is used to paste the object onto a new background with a variety of effects, including compound compositing, Fresnel effect, scene depth, and even caustic shadows. User studies find our results favorable to those obtained with Photoshop as well as perceptually valid in most cases. Our approach allows photo editing of transparent and refractive objects in a manner that produces realistic effects previously only possible via 3D models or environment matting.

[1]  Harry Shum,et al.  Natural shadow matting , 2007, TOGS.

[2]  Patrick Pérez,et al.  Poisson image editing , 2003, ACM Trans. Graph..

[3]  Holly E. Rushmeier,et al.  Evaluation of methods for approximating shapes used to synthesize 3D solid textures , 2008, TAP.

[4]  Jian Sun,et al.  Drag-and-drop pasting , 2006, SIGGRAPH 2006.

[5]  David J. Kriegman,et al.  On Refractive Optical Flow , 2004, ECCV.

[6]  Andrew W. Fitzgibbon,et al.  Image-based environment matting , 2002, SIGGRAPH '02.

[7]  M. Carter.  Computer graphics: Principles and practice , 1997 .

[8]  Erik Reinhard,et al.  Image-based material editing , 2005, SIGGRAPH '05.

[9]  Pieter Peers,et al.  Wavelet Environment matting , 2003, Rendering Techniques.

[10]  Shree K. Nayar,et al.  What does motion reveal about transparency? , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[11]  Katsushi Ikeuchi,et al.  Shape Estimation of Transparent Objects by Using Inverse Polarization Ray Tracing , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[12]  Harry Shum,et al.  Lazy snapping , 2004, ACM Trans. Graph..

[13]  David Salesin,et al.  Environment matting and compositing , 1999, SIGGRAPH.

[14]  Andrew Blake,et al.  "GrabCut" , 2004, ACM Trans. Graph..

[15]  Jian Sun,et al.  Poisson matting , 2004, ACM Trans. Graph..

[16]  Jian Sun,et al.  Lazy snapping , 2004, SIGGRAPH 2004.

[17]  Dan B. Goldman,et al.  Shadow Matting and Compositing , .

[18]  Diego Gutierrez,et al.  Screen-space perceptual rendering of human skin , 2009, TAP.

[19]  Fred L. Bookstein,et al.  Principal Warps: Thin-Plate Splines and the Decomposition of Deformations , 1989, IEEE Trans. Pattern Anal. Mach. Intell..

[20]  David Salesin,et al.  Shadow matting and compositing , 2003, ACM Trans. Graph..

[21]  Wojciech Matusik,et al.  Acquisition and Rendering of Transparent and Refractive Objects , 2002, Rendering Techniques.

[22]  Michael F. Cohen,et al.  Image and Video Matting , 2008 .

[23]  David Salesin,et al.  Environment matting extensions , 2000, SIGGRAPH 2000.

[24]  Vladimir Kolmogorov,et al.  "GrabCut": interactive foreground extraction using iterated graph cuts , 2004, ACM Trans. Graph..

[25]  Kiriakos N. Kutulakos,et al.  Reconstructing the Surface of Inhomogeneous Transparent Scenes by Scatter-Trace Photography , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[26]  David Salesin,et al.  Environment matting extensions: towards higher accuracy and real-time capture , 2000, SIGGRAPH.

[27]  John Hart,et al.  ACM Transactions on Graphics , 2004, SIGGRAPH 2004.

[28]  Erik Reinhard,et al.  Depicting procedural caustics in single images , 2008, SIGGRAPH Asia '08.