Natural shadow matting

This article addresses the problem of natural shadow matting, the removal or extraction of natural shadows from a single image. Because textures are maintained in the shadowless image after the extraction process, our approach produces some of the best results to date among shadow removal techniques. Using the image formation equation typical of computer vision, we advocate a new model for shadow formation where shadow effect is understood as light attenuation instead of a mixture of two colors governed by the conventional matting equation. This leads to a new shadow equation with fewer unknowns to solve, where a three-channel shadow matte and a shadowless image are considered in our optimization. Our problem is formulated as one of energy minimization guided by user-supplied hints in the form of a quadmap which can be specified easily by the user. This formulation allows for robust shadow matte extraction while maintaining texture in the shadowed region by considering color transfer, texture gradient, and shadow smoothness. We demonstrate the usefulness of our approach in shadow removal, image matting, and compositing.

[1]  Tai-Pang Wu,et al.  A Bayesian approach for shadow extraction from a single image , 2005, Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1.

[2]  Daniel Cohen-Or,et al.  Fragment-based image completion , 2003, ACM Trans. Graph..

[3]  David Salesin,et al.  A Bayesian approach to digital matting , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[4]  Jian Sun,et al.  Poisson matting , 2004, ACM Trans. Graph..

[5]  H. Barrow,et al.  RECOVERING INTRINSIC SCENE CHARACTERISTICS FROM IMAGES , 1978 .

[6]  Dan B. Goldman,et al.  Shadow Matting and Compositing , .

[7]  Thomas G. Dietterich.  What is machine learning? , 2020, Archives of Disease in Childhood.

[8]  Vladimir Kolmogorov,et al.  "GrabCut": interactive foreground extraction using iterated graph cuts , 2004, ACM Trans. Graph..

[9]  Jian Sun,et al.  Lazy snapping , 2004, SIGGRAPH 2004.

[10]  Yair Weiss,et al.  Deriving intrinsic images from image sequences , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.

[11]  Guillermo Sapiro,et al.  Image inpainting , 2000, SIGGRAPH.

[12]  Andrew Blake,et al.  "GrabCut" , 2004, ACM Trans. Graph..

[13]  David G. Stork,et al.  Pattern Classification , 1973 .

[14]  Edward H. Adelson,et al.  Recovering intrinsic images from a single image , 2005, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[15]  David Salesin,et al.  Shadow matting and compositing , 2003, ACM Trans. Graph..

[16]  Mark S. Drew,et al.  Removing Shadows from Images , 2002, ECCV.

[17]  Cheng Lu,et al.  Intrinsic Images by Entropy Minimization , 2004, ECCV.

[18]  William A. Barrett,et al.  Intelligent scissors for image composition , 1995, SIGGRAPH.

[19]  Klaus Mueller,et al.  Transferring color to greyscale images , 2002, ACM Trans. Graph..

[20]  Harry Shum,et al.  Lazy snapping , 2004, ACM Trans. Graph..

[21]  Steve Wright.  Digital Compositing for Film and Video, Second Edition , 2006 .

[22]  Masashi Baba,et al.  Shadow removal from a real image based on shadow density , 2004, SIGGRAPH '04.

[23]  E. Land,et al.  Lightness and retinex theory. , 1971, Journal of the Optical Society of America.

[24]  Sung Yong Shin,et al.  On pixel-based texture synthesis by non-parametric sampling , 2006, Comput. Graph..

[25]  Chi-Keung Tang,et al.  Inference of segmented color and texture description by tensor voting , 2004, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[26]  Frédo Durand,et al.  A gentle introduction to bilateral filtering and its applications , 2007, SIGGRAPH Courses.

[27]  Erik Reinhard,et al.  Color Transfer between Images , 2001, IEEE Computer Graphics and Applications.

[28]  Steve Wright.  Digital Compositing for Film and Video , 2001 .

[29]  Tom Duff,et al.  Compositing digital images , 1984, SIGGRAPH.

[30]  John Hart,et al.  ACM Transactions on Graphics , 2004, SIGGRAPH 2004.