Simultaneous Stereo Video Deblurring and Scene Flow Estimation

Videos for outdoor scene often show unpleasant blur effects due to the large relative motion between the camera and the dynamic objects and large depth variations. Existing works typically focus monocular video deblurring. In this paper, we propose a novel approach to deblurring from stereo videos. In particular, we exploit the piece-wise planar assumption about the scene and leverage the scene flow information to deblur the image. Unlike the existing approach [31] which used a pre-computed scene flow, we propose a single framework to jointly estimate the scene flow and deblur the image, where the motion cues from scene flow estimation and blur information could reinforce each other, and produce superior results than the conventional scene flow estimation or stereo deblurring methods. We evaluate our method extensively on two available datasets and achieve significant improvement in flow estimation and removing the blur effect over the state-of-the-art methods.

[1]  Antonin Chambolle,et al.  A First-Order Primal-Dual Algorithm for Convex Problems with Applications to Imaging , 2011, Journal of Mathematical Imaging and Vision.

[2]  Stephen Lin,et al.  Image/video deblurring using a hybrid camera , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[3]  Li Xu,et al.  Depth-aware motion deblurring , 2012, 2012 IEEE International Conference on Computational Photography (ICCP).

[4]  Christian Laugier,et al.  The International Journal of Robotics Research (IJRR) - Special issue on ``Field and Service Robotics '' , 2009 .

[5]  Rob Fergus,et al.  Fast Image Deconvolution using Hyper-Laplacian Priors , 2009, NIPS.

[6]  Li Xu,et al.  Forward Motion Deblurring , 2013, 2013 IEEE International Conference on Computer Vision.

[7]  Li Xu,et al.  Unnatural L0 Sparse Representation for Natural Image Deblurring , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[8]  X.C. He,et al.  Motion estimation method for blurred videos and application of deblurring with spatially varying blur kernels , 2010, 5th International Conference on Computer Sciences and Convergence Information Technology.

[9]  Seungyong Lee,et al.  Video deblurring for hand-held cameras using patch-based synthesis , 2012, ACM Trans. Graph..

[10]  Harry Shum,et al.  Full-frame video stabilization with motion inpainting , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[11]  Ming-Hsuan Yang,et al.  Joint Depth Estimation and Camera Shake Removal from Single Blurry Image , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[12]  Ankit Gupta,et al.  Single Image Deblurring Using Motion Density Functions , 2010, ECCV.

[13]  Ian D. Reid,et al.  From Motion Blur to Motion Flow: A Deep Learning Solution for Removing Heterogeneous Motion Blur , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[14]  Michael J. Black,et al.  Modeling Blurred Video with Layers , 2014, ECCV.

[15]  Shree K. Nayar,et al.  Motion-based motion deblurring , 2004, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[16]  Andreas Geiger,et al.  Vision meets robotics: The KITTI dataset , 2013, Int. J. Robotics Res..

[17]  Feng Li,et al.  A hybrid camera for motion deblurring and depth map super-resolution , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[18]  Rob Fergus,et al.  Blind deconvolution using a normalized sparsity measure , 2011, CVPR 2011.

[19]  Tae Hyun Kim,et al.  Generalized video deblurring for dynamic scenes , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[20]  Xin Li,et al.  Fine-Granularity and Spatially-Adaptive Regularization for Projection-Based Image Deblurring , 2011, IEEE Transactions on Image Processing.

[21]  Sunghyun Cho,et al.  Fast motion deblurring , 2009, SIGGRAPH 2009.

[22]  Bernhard Schölkopf,et al.  Blind Correction of Optical Aberrations , 2012, ECCV.

[23]  Jean Ponce,et al.  Learning a convolutional neural network for non-uniform motion blur removal , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[24]  Uwe Franke,et al.  Real-time stereo vision for urban traffic scene understanding , 2000, Proceedings of the IEEE Intelligent Vehicles Symposium 2000 (Cat. No.00TH8511).

[25]  D. Scharstein,et al.  A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms , 2001, Proceedings IEEE Workshop on Stereo and Multi-Baseline Vision (SMBV 2001).

[26]  Kyoung Mu Lee,et al.  Dense 3D Reconstruction from Severely Blurred Images Using a Single Moving Camera , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[27]  Anita Sellent,et al.  Stereo Video Deblurring , 2016, ECCV.

[28]  Tae Hyun Kim,et al.  Segmentation-Free Dynamic Scene Deblurring , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[29]  Sing Bing Kang Automatic Removal of Chromatic Aberration from a Single Image , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[30]  Andreas Geiger,et al.  Are we ready for autonomous driving? The KITTI vision benchmark suite , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[31]  Chun Chen,et al.  Video-based non-uniform object motion blur estimation and deblurring , 2012, Neurocomputing.

[32]  Raquel Urtasun,et al.  Robust Monocular Epipolar Flow Estimation , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[33]  Bernhard Schölkopf,et al.  Fast removal of non-uniform camera shake , 2011, 2011 International Conference on Computer Vision.

[34]  Jiaya Jia,et al.  Mathematical models and practical solvers for uniform motion deblurring , 2014, Motion Deblurring.

[35]  Heiko Hirschmüller,et al.  Stereo Processing by Semiglobal Matching and Mutual Information , 2008, IEEE Trans. Pattern Anal. Mach. Intell..

[36]  Jianping Shi,et al.  Just noticeable defocus blur detection and estimation , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[37]  Andreas Geiger,et al.  Object scene flow for autonomous vehicles , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[38]  Julius Ziegler,et al.  StereoScan: Dense 3d reconstruction in real-time , 2011, 2011 IEEE Intelligent Vehicles Symposium (IV).

[39]  Tae Hyun Kim,et al.  Deep Multi-scale Convolutional Neural Network for Dynamic Scene Deblurring , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[40]  Seungyong Lee,et al.  Recent advances in image deblurring , 2013, SA '13.

[41]  Konrad Schindler,et al.  3D Scene Flow Estimation with a Piecewise Rigid Scene Model , 2015, International Journal of Computer Vision.

[42]  Michal Irani,et al.  Blind Deblurring Using Internal Patch Recurrence , 2014, ECCV.

[43]  Jean Ponce,et al.  Non-uniform Deblurring for Shaken Images , 2012, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.