Context-Aware Image Matting for Simultaneous Foreground and Alpha Estimation

Natural image matting is an important problem in computer vision and graphics. It is an ill-posed problem when only an input image is available without any external information. While the recent deep learning approaches have shown promising results, they only estimate the alpha matte. This paper presents a context-aware natural image matting method for simultaneous foreground and alpha matte estimation. Our method employs two encoder networks to extract essential information for matting. Particularly, we use a matting encoder to learn local features and a context encoder to obtain more global context information. We concatenate the outputs from these two encoders and feed them into decoder networks to simultaneously estimate the foreground and alpha matte. To train this whole deep neural network, we employ both the standard Laplacian loss and the feature loss: the former helps to achieve high numerical performance while the latter leads to more perceptually plausible results. We also report several data augmentation strategies that greatly improve the network's generalization performance. Our qualitative and quantitative experiments show that our method enables high-quality matting for a single natural image.

[1]  Hany Farid,et al.  Exposing digital forgeries by detecting traces of resampling , 2005, IEEE Transactions on Signal Processing.

[2]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[3]  Min Wu,et al.  Digital image forensics via intrinsic fingerprints , 2008, IEEE Transactions on Information Forensics and Security.

[4]  Peyman Milanfar,et al.  NIMA: Neural Image Assessment , 2017, IEEE Transactions on Image Processing.

[5]  Manuel Menezes de Oliveira Neto,et al.  Shared Sampling for Real‐Time Alpha Matting , 2010, Comput. Graph. Forum.

[6]  Feng Liu,et al.  Context-Aware Synthesis for Video Frame Interpolation , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[7]  Alexei A. Efros,et al.  Generative Visual Manipulation on the Natural Image Manifold , 2016, ECCV.

[8]  Jian Sun,et al.  Fast matting using large kernel matting Laplacian matrices , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[9]  Michael F. Cohen,et al.  Image and Video Matting: A Survey , 2007, Found. Trends Comput. Graph. Vis..

[10]  David Salesin,et al.  A Bayesian approach to digital matting , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[11]  François Chollet,et al.  Xception: Deep Learning with Depthwise Separable Convolutions , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[12]  Jian Sun,et al.  Poisson matting , 2004, ACM Trans. Graph..

[13]  Qingzhong Liu,et al.  Detection of misaligned cropping and recompression with the same quantization matrix and relevant forgery , 2011, MiFor '11.

[14]  Kai Han,et al.  TOM-Net: Learning Transparent Object Matting from a Single Image , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[15]  Hany Farid,et al.  Photo forensics from JPEG dimples , 2017, 2017 IEEE Workshop on Information Forensics and Security (WIFS).

[16]  Yuanjie Zheng,et al.  Learning based digital matting , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[17]  Marc Pollefeys,et al.  Designing Effective Inter-Pixel Information Flow for Natural Image Matting , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[18]  Thomas Brox,et al.  Generating Images with Perceptual Similarity Metrics based on Deep Networks , 2016, NIPS.

[19]  Luc Van Gool,et al.  The Pascal Visual Object Classes (VOC) Challenge , 2010, International Journal of Computer Vision.

[20]  James F. Blinn,et al.  Blue screen matting , 1996, SIGGRAPH.

[21]  Scott Cohen,et al.  Simultaneous foreground, background, and alpha estimation for image matting , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[22]  George Papandreou,et al.  Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation , 2018, ECCV.

[23]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[24]  Chi-Keung Tang,et al.  KNN Matting , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[25]  Jian Sun,et al.  A global sampling method for alpha matting , 2011, CVPR 2011.

[26]  Dani Lischinski,et al.  Spectral Matting , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[27]  Feng Liu,et al.  Video Frame Interpolation via Adaptive Separable Convolution , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[28]  Shih-Fu Chang,et al.  A Data Set of Authentic and Spliced Image Blocks , 2004 .

[29]  Dani Lischinski,et al.  A Closed-Form Solution to Natural Image Matting , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[30]  Ying Wu,et al.  Nonlocal matting , 2011, CVPR 2011.

[31]  Shaziya .P.S. Khan,et al.  Exposing Digital Image Forgeries by Illumination Color Classification , 2015 .

[32]  Vlad I. Morariu,et al.  Detection of Metadata Tampering Through Discrepancy Between Image Content and Metadata Using Multi-task Deep Learning , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[33]  Ping Wang,et al.  Three-layer graph framework with the sumD feature for alpha matting , 2017, Comput. Vis. Image Underst..

[34]  Andrew Owens,et al.  Fighting Fake News: Image Splice Detection via Learned Self-Consistency , 2018, ECCV.

[35]  Jue Wang,et al.  A perceptually motivated online benchmark for image matting , 2009, CVPR.

[36]  Michael F. Cohen,et al.  Optimized Color Sampling for Robust Matting , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[37]  Bernhard Schölkopf,et al.  EnhanceNet: Single Image Super-Resolution Through Automated Texture Synthesis , 2016, 2017 IEEE International Conference on Computer Vision (ICCV).

[38]  Qinping Zhao,et al.  Image Matting with Local and Nonlocal Smooth Priors , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[39]  Pietro Perona,et al.  Microsoft COCO: Common Objects in Context , 2014, ECCV.

[40]  Yochai Blau,et al.  The Perception-Distortion Tradeoff , 2017, CVPR.

[41]  Virginia R. de Sa,et al.  Learning Classification with Unlabeled Data , 1993, NIPS.

[42]  Mark Sandler,et al.  MobileNetV2: Inverted Residuals and Linear Bottlenecks , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[43]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[44]  Ning Xu,et al.  Deep Image Matting , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[45]  Michael F. Cohen,et al.  An iterative optimization approach for unified image segmentation and matting , 2005, Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1.

[46]  Jiwu Huang,et al.  JPEG Error Analysis and Its Applications to Digital Image Forensics , 2010, IEEE Transactions on Information Forensics and Security.

[47]  Jiaya Jia,et al.  Deep Automatic Portrait Matting , 2016, ECCV.

[48]  Guijin Wang,et al.  Iterative transductive learning for alpha matting , 2013, 2013 IEEE International Conference on Image Processing.

[49]  In So Kweon,et al.  Deep Convolutional Neural Network for Natural Image Matting Using Initial Alpha Mattes , 2019, IEEE Transactions on Image Processing.

[50]  In-So Kweon,et al.  Natural Image Matting Using Deep Convolutional Neural Networks , 2016, ECCV.

[51]  Alexei A. Efros,et al.  The Unreasonable Effectiveness of Deep Features as a Perceptual Metric , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[52]  Aljoscha Smolic,et al.  AlphaGAN: Generative adversarial networks for natural image matting , 2018, BMVC.

[53]  Deepu Rajan,et al.  Improving Image Matting Using Comprehensive Sampling Sets , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.