Siamese Dense Network for Reflection Removal with Flash and No-Flash Image Pairs

This work addresses the reflection removal with flash and no-flash image pairs to separate reflection from transmission. When objects are covered by glass, the no-flash image usually contains reflection, and thus flash is used to enhance transmission details. However, the flash image suffers from the specular highlight on the glass surface caused by flash. In this paper, we propose a siamese dense network (SDN) for reflection removal with flash and no-flash image pairs. SDN extracts shareable and complementary features via concatenated siamese dense blocks. We utilize an image fusion block for the SDN to fuse the intermediate output of two branches. Since severe information loss occurs in the specular highlight, we detect the specular highlight in the flash image based on gradient of the maximum chromaticity. Through observations, flash causes various artifacts such as tone distortion and inhomogeneous brightness. Thus, with synthetic datasets we collect 758 pairs of real flash and no-flash image pairs (including their ground truth) by different cameras to gain generalization. Various experiments show that the proposed method successfully removes reflections using flash and no-flash image pairs and outperforms state-of-the-art ones in terms of visual quality and quantitative measurements. Besides, we apply the SDN to color/depth image pairs and achieve both color reflection removal and depth filling.

[1]  F. Durand,et al.  Flash photography enhancement via intrinsic relighting , 2004, ACM Trans. Graph..

[2]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[3]  Yu Liu,et al.  A medical image fusion method based on convolutional neural networks , 2017, 2017 20th International Conference on Information Fusion (Fusion).

[4]  In-So Kweon,et al.  Specular Reflection Separation Using Dark Channel Prior , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[5]  Ju Shen,et al.  Layer Depth Denoising and Completion for Structured-Light RGB-D Cameras , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[6]  Masaaki Ikehara,et al.  Noiseless no-flash photo creation by color transform of flash image , 2011, 2011 18th IEEE International Conference on Image Processing.

[7]  Yoav Y. Schechner,et al.  Overcoming visual reverberations , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[8]  Richard Szeliski,et al.  Layer extraction from multiple images containing reflections and transparency , 2000, Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.PR00662).

[9]  Takahiro Okabe,et al.  Image Enhancement of Low-light Scenes with Near-infrared Flash Images , 2009, IPSJ Trans. Comput. Vis. Appl..

[10]  Marc Pollefeys,et al.  A Dataset of Flash and Ambient Illumination Pairs from the Crowd , 2018, ECCV.

[11]  Kilian Q. Weinberger,et al.  Densely Connected Convolutional Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[12]  Ling-Yu Duan,et al.  CRRN: Multi-scale Guided Concurrent Reflection Removal Network , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[13]  Yang Yang,et al.  Fast Single Image Reflection Suppression via Convex Optimization , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[14]  Nikos Komodakis,et al.  Learning to compare image patches via convolutional neural networks , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[15]  Jan Kautz,et al.  Exposure Fusion , 2009, 15th Pacific Conference on Computer Graphics and Applications (PG'07).

[16]  Rynson W. H. Lau,et al.  Saliency Detection with Flash and No-flash Image Pairs , 2014, ECCV.

[17]  Michael F. Cohen,et al.  Digital photography with flash and no-flash image pairs , 2004, ACM Trans. Graph..

[18]  Katsushi Ikeuchi,et al.  Illumination chromaticity estimation using inverse-intensity chromaticity space , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[19]  Anat Levin,et al.  User Assisted Separation of Reflections from a Single Image Using a Sparsity Prior , 2004, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[20]  Jiaolong Yang,et al.  A Generic Deep Architecture for Single Image Reflection Removal and Image Smoothing (Supplementary Material) , 2017 .

[21]  Harry Shum,et al.  Flash Cut: Foreground Extraction with Flash and No-flash Image Pairs , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[22]  Dani Lischinski,et al.  Colorization using optimization , 2004, ACM Trans. Graph..

[23]  Michael S. Brown,et al.  Single Image Layer Separation Using Relative Smoothness , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[24]  Jae-Young Sim,et al.  Glass Reflection Removal Using Co-Saliency-Based Image Alignment and Low-Rank Matrix Completion in Gradient Domain , 2018, IEEE Transactions on Image Processing.

[25]  Ren Ng,et al.  Single Image Reflection Separation with Perceptual Losses , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[26]  Peyman Milanfar,et al.  Robust flash denoising/deblurring by iterative guided filtering , 2012, EURASIP J. Adv. Signal Process..

[27]  Hang Zhang,et al.  Multi-style Generative Network for Real-time Transfer , 2017, ECCV Workshops.

[28]  Jiaolong Yang,et al.  Single Image Reflection Removal Exploiting Misaligned Training Data and Network Enhancements , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[29]  Yann LeCun,et al.  Signature Verification Using A "Siamese" Time Delay Neural Network , 1993, Int. J. Pattern Recognit. Artif. Intell..

[30]  Cheolkon Jung,et al.  Automatic Contrast-Limited Adaptive Histogram Equalization With Dual Gamma Correction , 2018, IEEE Access.

[31]  Jiaolong Yang,et al.  Robust Optical Flow Estimation of Double-Layer Images under Transparency or Reflection , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[32]  David Zhang,et al.  FSIM: A Feature Similarity Index for Image Quality Assessment , 2011, IEEE Transactions on Image Processing.

[33]  Jae-Young Sim,et al.  Reflection Removal Using Low-Rank Matrix Completion , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[34]  Cheolkon Jung,et al.  Multi-Modal Reflection Removal Using Convolutional Neural Networks , 2019, IEEE Signal Processing Letters.

[35]  Shree K. Nayar,et al.  Separation of Reflection Components Using Color and Polarization , 1997, International Journal of Computer Vision.

[36]  Ramesh Raskar,et al.  Removing photography artifacts using gradient projection and flash-exposure sampling , 2005, ACM Trans. Graph..

[37]  Yann LeCun,et al.  Learning a similarity metric discriminatively, with application to face verification , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[38]  Ronen Basri,et al.  Separation of Transparent Layers using Focus , 2004, International Journal of Computer Vision.

[39]  Cheng Lu,et al.  Shadow Removal via Flash/Noflash Illumination , 2006, 2006 IEEE Workshop on Multimedia Signal Processing.

[40]  Luis Salgado,et al.  Efficient spatio-temporal hole filling strategy for Kinect depth maps , 2012, Electronic Imaging.

[41]  Frédo Durand,et al.  Reflection removal using ghosting cues , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[42]  Yu Liu,et al.  Multi-focus image fusion with a deep convolutional neural network , 2017, Inf. Fusion.

[43]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[44]  Michael S. Brown,et al.  Rain Streak Removal Using Layer Priors , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[45]  Jianxiong Xiao,et al.  SUN RGB-D: A RGB-D scene understanding benchmark suite , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[46]  Sung Yong Shin,et al.  A physically-based approach to reflection separation , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[47]  Jie Yang,et al.  Seeing Deeply and Bidirectionally: A Deep Learning Approach for Single Image Reflection Removal , 2018, ECCV.

[48]  Michael S. Brown,et al.  Reflection Removal Using a Dual-Pixel Sensor , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[49]  Xiaogang Wang,et al.  Deep Learning Face Representation from Predicting 10,000 Classes , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[50]  Michael S. Brown,et al.  Exploiting Reflection Change for Automatic Reflection Removal , 2013, 2013 IEEE International Conference on Computer Vision.

[51]  Derek Hoiem,et al.  Indoor Segmentation and Support Inference from RGBD Images , 2012, ECCV.

[52]  Christian Simon,et al.  Reflection removal for in-vehicle black box videos , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[53]  Xiaochun Cao,et al.  Robust Separation of Reflection from Multiple Images , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[54]  Harry Shum,et al.  Flash matting , 2006, ACM Trans. Graph..

[55]  Edward H. Adelson,et al.  Separating reflections and lighting using independent components analysis , 1999, Proceedings. 1999 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No PR00149).

[56]  J Shamir,et al.  Polarization and statistical analysis of scenes containing a semireflector. , 2000, Journal of the Optical Society of America. A, Optics, image science, and vision.

[57]  Alexei A. Efros,et al.  Context Encoders: Feature Learning by Inpainting , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[58]  Cheolkon Jung,et al.  Single Image Reflection Removal Using Convolutional Neural Networks , 2019, IEEE Transactions on Image Processing.