Learned Dual-View Reflection Removal

Traditional reflection removal algorithms either use a single image as input, which suffers from intrinsic ambiguities, or use multiple images from a moving camera, which is inconvenient for users. We instead propose a learning-based dereflection algorithm that uses stereo images as input. This is an effective trade-off between the two extremes: the parallax between two views provides cues to remove reflections, and two views are easy to capture due to the adoption of stereo cameras in smartphones. Our model consists of a learning-based reflection-invariant flow model for dual-view registration, and a learned synthesis model for combining aligned image pairs. Because no dataset for dual-view reflection removal exists, we render a synthetic dataset of dual-views with and without reflections for use in training. Our evaluation on an additional real-world dataset of stereo pairs shows that our algorithm outperforms existing single-image and multi-image dereflection approaches.

[1]  Alain Trémeau,et al.  Residual Conv-Deconv Grid Network for Semantic Segmentation , 2017, BMVC.

[2]  Jie Yang,et al.  Seeing Deeply and Bidirectionally: A Deep Learning Approach for Single Image Reflection Removal , 2018, ECCV.

[3]  Michael S. Brown,et al.  Single Image Layer Separation Using Relative Smoothness , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[4]  Frédo Durand,et al.  Reflection removal using ghosting cues , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[5]  Ren Ng,et al.  Single Image Reflection Separation with Perceptual Losses , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[6]  Rahul Garg,et al.  Wireless Software Synchronization of Multiple Distributed Cameras , 2018, 2019 IEEE International Conference on Computational Photography (ICCP).

[7]  Ling-Yu Duan,et al.  CRRN: Multi-scale Guided Concurrent Reflection Removal Network , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[8]  Michael S. Brown,et al.  Reflection Removal Using a Dual-Pixel Sensor , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[9]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[10]  Ling-Yu Duan,et al.  Benchmarking Single-Image Reflection Removal Algorithms , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[11]  Jitendra Malik,et al.  Shape, Illumination, and Reflectance from Shading , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[12]  Jan Kautz,et al.  PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volume , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[13]  Guoqiang Han,et al.  Single Image Reflection Removal Beyond Linearity , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[14]  Feng Liu,et al.  Context-Aware Synthesis for Video Frame Interpolation , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[15]  Yoav Y. Schechner,et al.  Polarization-based decorrelation of transparent layers: The inclination angle of an invisible surface , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[16]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.

[17]  Yu-Wing Tai,et al.  A Physically-Based Approach to Reflection Separation: From Physical Modeling to Constrained Optimization , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[18]  William T. Freeman,et al.  A computational approach for obstruction-free photography , 2015, ACM Trans. Graph..

[19]  Li Fei-Fei,et al.  Perceptual Losses for Real-Time Style Transfer and Super-Resolution , 2016, ECCV.

[20]  Michael S. Brown,et al.  Exploiting Reflection Change for Automatic Reflection Removal , 2013, 2013 IEEE International Conference on Computer Vision.

[21]  Michael J. Black,et al.  A Naturalistic Open Source Movie for Optical Flow Evaluation , 2012, ECCV.

[22]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[23]  Richard Szeliski,et al.  A Database and Evaluation Methodology for Optical Flow , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[24]  Yang Yang,et al.  Fast Single Image Reflection Suppression via Convex Optimization , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[25]  Houqiang Li,et al.  Multi-Level Video Frame Interpolation: Exploiting the Interaction Among Different Levels , 2013, IEEE Transactions on Circuits and Systems for Video Technology.

[26]  Assaf Zomet,et al.  Separating reflections from a single image using local features , 2004, Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004..

[27]  Vincent Dumoulin,et al.  Deconvolution and Checkerboard Artifacts , 2016 .

[28]  Mohinder Malhotra.  Single Image Haze Removal Using Dark Channel Prior , 2016 .

[29]  Jae-Young Sim,et al.  Reflection Removal Using Low-Rank Matrix Completion , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[30]  Alexei A. Efros,et al.  The Unreasonable Effectiveness of Deep Features as a Perceptual Metric , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[31]  Richard Szeliski,et al.  Layer extraction from multiple images containing reflections and transparency , 2000, Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.PR00662).

[32]  Andrew Zisserman,et al.  Spatial Transformer Networks , 2015, NIPS.

[33]  Daniel Cremers,et al.  What Makes Good Synthetic Training Data for Learning Disparity and Optical Flow Estimation? , 2018, International Journal of Computer Vision.

[34]  Yair Movshovitz-Attias,et al.  Synthetic depth-of-field with a single-camera mobile phone , 2018, ACM Trans. Graph..

[35]  Jonathan T. Barron,et al.  Learning Single Camera Depth Estimation Using Dual-Pixels , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[36]  Thomas Brox,et al.  FlowNet: Learning Optical Flow with Convolutional Networks , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[37]  Georgios D. Evangelidis,et al.  Parametric Image Alignment Using Enhanced Correlation Coefficient Maximization , 2008, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[38]  Sung-eui Yoon,et al.  Single Image Reflection Removal With Physically-Based Training Images , 2019, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[39]  John E. Hopcroft,et al.  Single Image Reflection Removal Through Cascaded Refinement , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[40]  Michael J. Black,et al.  Slow Flow: Exploiting High-Speed Cameras for Accurate and Diverse Optical Flow Reference Data , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[41]  Jiaolong Yang,et al.  Robust Optical Flow Estimation of Double-Layer Images under Transparency or Reflection , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[42]  Jiaolong Yang,et al.  A Generic Deep Architecture for Single Image Reflection Removal and Image Smoothing (Supplementary Material) , 2017 .

[43]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.