Depth-of-field-based alpha-matte extraction

In compositing applications, objects depicted in images frequently have to be separated from their background, so that they can be placed in a new environment. Alpha mattes are important tools aiding the selection of objects, but cannot normally be created in a fully automatic way. We present an algorithm that requires as input two images---one where the object is in focus, and one where the background is in focus---and then automatically produces an alpha matte indicating which pixels belong to the object. This algorithm is inspired by human visual processing and involves nonlinear response compression, center-surround mechanisms as well as a filling-in stage. The output can then be refined with standard computer vision techniques.

[1]  J. Dowling.  The Retina: An Approachable Part of the Brain , 1988 .

[2]  Jitendra Malik,et al.  Scale-Space and Edge Detection Using Anisotropic Diffusion , 1990, IEEE Trans. Pattern Anal. Mach. Intell..

[3]  S. Dakin,et al.  Natural image statistics mediate brightness ‘filling in’ , 2003, Proceedings of the Royal Society of London. Series B: Biological Sciences.

[4]  Shree K. Nayar,et al.  Shape from Focus , 1994, IEEE Trans. Pattern Anal. Mach. Intell..

[5]  Jiaya Jia,et al.  Poisson matting , 2004, SIGGRAPH 2004.

[6]  Jessica K. Hodgins,et al.  Two methods for display of high contrast images , 1999, TOGS.

[7]  K. Naka,et al.  S‐potentials from luminosity units in the retina of fish (Cyprinidae) , 1966, The Journal of physiology.

[8]  Jitendra Malik,et al.  Recovering high dynamic range radiance maps from photographs , 1997, SIGGRAPH.

[9]  Joachim Weickert,et al.  A Review of Nonlinear Diffusion Filtering , 1997, Scale-Space.

[10]  Shree K. Nayar,et al.  Shape from focus: an effective approach for rough surfaces , 1990, Proceedings., IEEE International Conference on Robotics and Automation.

[11]  Erik Reinhard,et al.  Ieee Transactions on Visualization and Computer Graphics 1 Dynamic Range Reduction Inspired by Photoreceptor Physiology , 2022 .

[12]  D. Greig,et al.  Exact Maximum A Posteriori Estimation for Binary Images , 1989 .

[13]  Shree K. Nayar,et al.  Rational Filters for Passive Depth from Defocus , 1998, International Journal of Computer Vision.

[14]  Peter Lawrence,et al.  An Investigation of Methods for Determining Depth from Focus , 1993, IEEE Trans. Pattern Anal. Mach. Intell..

[15]  David Salesin,et al.  Video matting of complex scenes , 2002, SIGGRAPH.

[16]  Edward H. Adelson,et al.  Saturation and adaptation in the rod system , 1982, Vision Research.

[17]  William A. Barrett,et al.  Toboggan-based intelligent scissors with a four-parameter edge model , 1999, Proceedings. 1999 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No PR00149).

[18]  D. Ruderman.  The statistics of natural images , 1994 .

[19]  Christophe Schlick,et al.  Quantization Techniques for Visualization of High Dynamic Range Pictures , 1995 .

[20]  R. Cox,et al.  Journal of the Royal Statistical Society B , 1972 .

[21]  Robert M. Gray,et al.  Automatic object segmentation in images with low depth of field , 2002, Proceedings. International Conference on Image Processing.

[22]  Guillermo Sapiro,et al.  Geodesic Active Contours , 1995, International Journal of Computer Vision.

[23]  David Marr,et al.  VISION A Computational Investigation into the Human Representation and Processing of Visual Information , 2009 .

[24]  Daniel L. Ruderman,et al.  Origins of scaling in natural images , 1996, Vision Research.

[25]  G L WALLS,et al.  The filling-in process. , 1954, American journal of optometry and archives of American Academy of Optometry.

[26]  Andrew Blake,et al.  "GrabCut" , 2004, ACM Trans. Graph..

[27]  Vladimir Kolmogorov,et al.  "GrabCut": interactive foreground extraction using iterated graph cuts , 2004, ACM Trans. Graph..

[28]  Tony Lindeberg,et al.  Scale-Space Theory in Computer Vision , 1993, Lecture Notes in Computer Science.

[29]  James H. Elder,et al.  Are Edges Incomplete? , 1999, International Journal of Computer Vision.

[30]  William A. Barrett,et al.  Intelligent scissors for image composition , 1995, SIGGRAPH.

[31]  Harry Shum,et al.  Lazy snapping , 2004, ACM Trans. Graph..

[32]  Andrew P. Witkin,et al.  Scale-Space Filtering , 1983, IJCAI.

[33]  Joachim Weickert.  A Review of Nonlinear Diiusion Filtering , 1997 .

[34]  Murali Subbarao,et al.  Depth from defocus: A spatial domain approach , 1994, International Journal of Computer Vision.

[35]  Erik Reinhard,et al.  High Dynamic Range Imaging: Acquisition, Display, and Image-Based Lighting (The Morgan Kaufmann Series in Computer Graphics) , 2005 .

[36]  S. W. Kuffler.  Discharge patterns and functional organization of mammalian retina. , 1953, Journal of neurophysiology.

[37]  Jian Sun,et al.  Lazy snapping , 2004, SIGGRAPH 2004.

[38]  Frédo Durand,et al.  Defocus video matting , 2005, SIGGRAPH 2005.

[39]  Takashi Totsuka,et al.  AutoKey: human assisted key extraction , 1995, SIGGRAPH.

[40]  Donald P. Greenberg,et al.  Time-dependent visual adaptation for fast realistic image display , 2000, SIGGRAPH.

[41]  Marie-Pierre Jolly,et al.  Interactive Graph Cuts for Optimal Boundary and Region Segmentation of Objects in N-D Images , 2001, ICCV.

[42]  Greg Ward,et al.  Fast, Robust Image Registration for Compositing High Dynamic Range Photographs from Hand-Held Exposures , 2003, J. Graphics, GPU, & Game Tools.

[43]  E. Reinhard.  Photographic Tone Reproduction for Digital Images , 2002 .

[44]  J. Krauskopf.  Effect of retinal image stabilization on the appearance of heterochromatic targets. , 1963, Journal of the Optical Society of America.

[45]  J. Dowling,et al.  Intracellular recordings from gecko photoreceptors during light and dark adaptation , 1975, The Journal of general physiology.

[46]  David Salesin,et al.  A Bayesian approach to digital matting , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[47]  Jian Sun,et al.  Poisson matting , 2004, ACM Trans. Graph..

[48]  James Ze Wang,et al.  Unsupervised Multiresolution Segmentation for Images with Low Depth of Field , 2001, IEEE Trans. Pattern Anal. Mach. Intell..

[49]  S. Palmer.  Vision Science : Photons to Phenomenology , 1999 .

[50]  Marie-Pierre Jolly,et al.  Interactive graph cuts for optimal boundary & region segmentation of objects in N-D images , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.

[51]  C. A. Burbeck,et al.  Occlusion edge blur: a cue to relative visual depth. , 1996, Journal of the Optical Society of America. A, Optics, image science, and vision.

[52]  D. Hood,et al.  Comparison of changes in sensitivity and sensation: implications for the response-intensity function of the human photopic system. , 1979, Journal of experimental psychology. Human perception and performance.

[53]  Murali Subbarao.  Parallel Depth Recovery By Changing Camera Parameters , 1988, [1988 Proceedings] Second International Conference on Computer Vision.

[54]  Ennio Mingolla,et al.  Neural Dynamics of Form Perception: Boundary Completion, Illusory Figures, And Neon Color Spreading , 1987 .

[55]  Heiko Neumann,et al.  Visual filling-in for computing perceptual surface properties , 2001, Biological Cybernetics.

[56]  Alex Pentland,et al.  A New Sense for Depth of Field , 1985, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[57]  K.,et al.  Two Methods for Display of High Contrast , 2005 .

[58]  S. Grossberg,et al.  Neural dynamics of form perception: boundary completion, illusory figures, and neon color spreading. , 1985 .