Video Reflection Removal Through Spatio-Temporal Optimization

Reflections can obstruct content during video capture and hence their removal is desirable. Current removal techniques are designed for still images, extracting only one reflection (foreground) and one background layer from the input. When extended to videos, unpleasant artifacts such as temporal flickering and incomplete separation are generated. We present a technique for video reflection removal by jointly solving for motion and separation. The novelty of our work is in our optimization formulation as well as the motion initialization strategy. We present a novel spatiotemporal optimization that takes n frames as input and directly estimates 2n frames as output, n for each layer. We aim to fully utilize spatio-temporal information in our objective terms. Our motion initialization is based on iterative frame-to-frame alignment instead of the direct alignment used by current approaches. We compare against advanced video extensions of the state of the art, and we significantly reduce temporal flickering and improve separation. In addition, we reduce image blur and recover moving objects more accurately. We validate our approach through subjective and objective evaluations on real and controlled data.

[1]  Michael Goesele,et al.  Image-based rendering in the gradient domain , 2013, ACM Trans. Graph..

[2]  Marc Levoy,et al.  The Frankencamera: an experimental platform for computational photography , 2010, ACM Trans. Graph..

[3]  Richard Szeliski,et al.  Stereo matching with reflections and translucency , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[4]  Jiaolong Yang,et al.  Robust Optical Flow Estimation of Double-Layer Images under Transparency or Reflection , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[5]  Assaf Zomet,et al.  Separating reflections from a single image using local features , 2004, Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004..

[6]  Ce Liu,et al.  Exploring new representations and applications for motion analysis , 2009 .

[7]  Xiaochun Cao,et al.  Robust Separation of Reflection from Multiple Images , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[8]  Anil C. Kokaram,et al.  User-assisted reflection detection and feature point tracking , 2013, CVMP '13.

[9]  Frédo Durand,et al.  Reflection removal using ghosting cues , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[10]  Anil C. Kokaram,et al.  Motion Estimation for Regions of Reflections through Layer Separation , 2011, 2011 Conference for Visual Media Production.

[11]  William T. Freeman,et al.  A computational approach for obstruction-free photography , 2015, ACM Trans. Graph..

[12]  Changshui Zhang,et al.  Blind Separation of Superimposed Moving Images Using Image Statistics , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[13]  Markus Gross,et al.  Practical temporal consistency for image-based graphics applications , 2012, ACM Trans. Graph..

[14]  Yair Weiss,et al.  Deriving intrinsic images from image sequences , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.

[15]  Yehoshua Y. Zeevi,et al.  Sparse ICA for blind separation of transmitted and reflected images , 2005, Int. J. Imaging Syst. Technol..

[16]  Antonio Torralba,et al.  SIFT Flow: Dense Correspondence across Scenes and Its Applications , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[17]  H Farid,et al.  Separating reflections from images by use of independent component analysis. , 1999, Journal of the Optical Society of America. A, Optics, image science, and vision.

[18]  Michal Irani,et al.  Separating transparent layers of repetitive dynamic behaviors , 2005, Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1.

[19]  Michael Goesele,et al.  Image-based rendering for scenes with reflections , 2012, ACM Trans. Graph..

[20]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.

[21]  Michal Irani,et al.  Separating Transparent Layers through Layer Information Exchange , 2004, ECCV.

[22]  Frédo Durand,et al.  Video magnification in presence of large motions , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[23]  Anat Levin,et al.  User Assisted Separation of Reflections from a Single Image Using a Sparsity Prior , 2004, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[24]  Michael S. Brown,et al.  Exploiting Reflection Change for Automatic Reflection Removal , 2013, 2013 IEEE International Conference on Computer Vision.

[25]  Frédo Durand,et al.  Phase-based video motion processing , 2013, ACM Trans. Graph..

[26]  Anil C. Kokaram,et al.  Reflection detection in image sequences , 2011, CVPR 2011.

[27]  Cordelia Schmid,et al.  Action recognition by dense trajectories , 2011, CVPR 2011.

[28]  Richard Szeliski,et al.  Layer extraction from multiple images containing reflections and transparency , 2000, Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.PR00662).