Reflection Removal Using a Dual-Pixel Sensor

Reflection removal is the challenging problem of removing unwanted reflections that occur when imaging a scene that is behind a pane of glass. In this paper, we show that most cameras have an overlooked mechanism that can greatly simplify this task. Specifically, modern DLSR and smartphone cameras use dual pixel (DP) sensors that have two photodiodes per pixel to provide two sub-aperture views of the scene from a single captured image. ``Defocus-disparity'' cues, which are natural by-products of the DP sensor encoded within these two sub-aperture views, can be used to distinguish between image gradients belonging to the in-focus background and those caused by reflection interference. This gradient information can then be incorporated into an optimization framework to recover the background layer with higher accuracy than currently possible from the single captured image. As part of this work, we provide the first image dataset for reflection removal consisting of the sub-aperture views from the DP sensor.

[1]  Xiaokang Yang,et al.  Separation of Weak Reflection from a Single Superimposed Image , 2014, IEEE Signal Processing Letters.

[2]  Anat Levin,et al.  User Assisted Separation of Reflections from a Single Image Using a Sparsity Prior , 2004, ECCV.

[3]  Ronen Basri,et al.  Separation of Transparent Layers using Focus , 2004, International Journal of Computer Vision.

[4]  Michael S. Brown,et al.  Exploiting Reflection Change for Automatic Reflection Removal , 2013, 2013 IEEE International Conference on Computer Vision.

[5]  Ah-Hwee Tan,et al.  Depth of field guided reflection removal , 2016, 2016 IEEE International Conference on Image Processing (ICIP).

[6]  Michal Irani,et al.  Separating Transparent Layers through Layer Information Exchange , 2004, ECCV.

[7]  Yoav Y. Schechner,et al.  Polarization-based decorrelation of transparent layers: The inclination angle of an invisible surface , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[8]  Jiaolong Yang,et al.  A Generic Deep Architecture for Single Image Reflection Removal and Image Smoothing (Supplementary Material) , 2017 .

[9]  Ramesh Raskar,et al.  Removing photography artifacts using gradient projection and flash-exposure sampling , 2005, SIGGRAPH 2005.

[10]  Yu-Wing Tai,et al.  A Physically-Based Approach to Reflection Separation: From Physical Modeling to Constrained Optimization , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[11]  Yoav Y. Schechner,et al.  Overcoming visual reverberations , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[12]  Jie Yang,et al.  Seeing Deeply and Bidirectionally: A Deep Learning Approach for Single Image Reflection Removal , 2018, ECCV.

[13]  Ronen Basri,et al.  Separation of Transparent Layers using Focus , 1998, Sixth International Conference on Computer Vision (IEEE Cat. No.98CH36271).

[14]  Paramanand Chandramouli,et al.  ConvNet-Based Depth Estimation, Reflection Separation and Deblurring of Plenoptic Images , 2016, ACCV.

[15]  Assaf Zomet,et al.  Learning to Perceive Transparency from the Statistics of Natural Scenes , 2002, NIPS.

[16]  Przemys aw liwiński,et al.  A Simple Model for On-Sensor Phase-Detection Autofocusing Algorithm , 2013 .

[17]  Sei-Wang Chen,et al.  Interference reflection separation from a single image , 2009, 2009 Workshop on Applications of Computer Vision (WACV).

[18]  Yair Movshovitz-Attias,et al.  Synthetic depth-of-field with a single-camera mobile phone , 2018, ACM Trans. Graph..

[19]  Joonki Paik,et al.  Sensor-Based Auto-Focusing System Using Multi-Scale Feature Extraction and Phase Correlation Matching , 2015, Sensors.

[20]  Jae-Young Sim,et al.  Glass Reflection Removal Using Co-Saliency-Based Image Alignment and Low-Rank Matrix Completion in Gradient Domain , 2018, IEEE Transactions on Image Processing.

[21]  Ren Ng,et al.  Single Image Reflection Separation with Perceptual Losses , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[22]  Rob Fergus,et al.  Fast Image Deconvolution using Hyper-Laplacian Priors , 2009, NIPS.

[23]  William T. Freeman,et al.  A computational approach for obstruction-free photography , 2015, ACM Trans. Graph..

[24]  Y. Weiss,et al.  Separating reflections from a single image using local features , 2004, CVPR 2004.

[25]  Michael S. Brown,et al.  Single Image Layer Separation Using Relative Smoothness , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[26]  Frédo Durand,et al.  Image and depth from a conventional camera with a coded aperture , 2007, ACM Trans. Graph..

[27]  Arie Yeredor,et al.  Blind Separation of Superimposed Shifted Images Using Parameterized Joint Diagonalization , 2008, IEEE Transactions on Image Processing.

[28]  Bastian Goldlücke,et al.  Variational Separation of Light Field Layers , 2015, VMV.

[29]  Ling-Yu Duan,et al.  CRRN: Multi-scale Guided Concurrent Reflection Removal Network , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[30]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.

[31]  Ling-Yu Duan,et al.  Benchmarking Single-Image Reflection Removal Algorithms , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[32]  Qiaosong Wang,et al.  Automatic Layer Separation using Light Field Imaging , 2015, ArXiv.

[33]  Rama Chellappa,et al.  Edge Suppression by Gradient Field Transformation Using Cross-Projection Tensors , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[34]  Frédo Durand,et al.  Reflection removal using ghosting cues , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[35]  T FreemanWilliam,et al.  Image and depth from a conventional camera with a coded aperture , 2007 .

[36]  Guanghui Liu,et al.  Automatic Reflection Removal using Gradient Intensity and Motion Cues , 2016, ACM Multimedia.

[37]  Jie Chen,et al.  Reflection Removal on Single Light Field Capture Using Focus Manipulation , 2018, IEEE Transactions on Computational Imaging.

[38]  Xiaochun Cao,et al.  Robust Separation of Reflection from Multiple Images , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[39]  Yair Weiss,et al.  Deriving intrinsic images from image sequences , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.

[40]  Changshui Zhang,et al.  Blind Separation of Superimposed Moving Images Using Image Statistics. , 2012, IEEE transactions on pattern analysis and machine intelligence.