DAVANet: Stereo Deblurring With View Aggregation

Nowadays stereo cameras are more commonly adopted in emerging devices such as dual-lens smartphones and unmanned aerial vehicles. However, they also suffer from blurry images in dynamic scenes which leads to visual discomfort and hampers further image processing. Previous works have succeeded in monocular deblurring, yet there are few studies on deblurring for stereoscopic images. By exploiting the two-view nature of stereo images, we propose a novel stereo image deblurring network with Depth Awareness and View Aggregation, named DAVANet. In our proposed network, 3D scene cues from the depth and varying information from two views are incorporated, which help to remove complex spatially-varying blur in dynamic scenes. Specifically, with our proposed fusion network, we integrate the bidirectional disparities estimation and deblurring into a unified framework. Moreover, we present a large-scale multi-scene dataset for stereo deblurring, containing 20,637 blurry-sharp stereo image pairs from 135 diverse sequences and their corresponding bidirectional disparities. The experimental results on our dataset demonstrate that DAVANet outperforms state-of-the-art methods in terms of accuracy, speed, and model size.

[1]  Ming-Hsuan Yang,et al.  Joint Depth Estimation and Camera Shake Removal from Single Blurry Image , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[2]  Fatih Murat Porikli,et al.  Simultaneous Stereo Video Deblurring and Scene Flow Estimation , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[3]  Paramanand Chandramouli,et al.  Motion Deblurring in the Wild , 2017, GCPR.

[4]  Daniel P. Huttenlocher,et al.  Generating sharp panoramas from motion-blurred videos , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[5]  Li Xu,et al.  Unnatural L0 Sparse Representation for Natural Image Deblurring , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[6]  Ming-Hsuan Yang,et al.  Learning Recursive Filters for Low-Level Vision via a Hybrid Neural Network , 2016, ECCV.

[7]  Tae Hyun Kim,et al.  Deep Multi-scale Convolutional Neural Network for Dynamic Scene Deblurring , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[8]  Yann LeCun,et al.  Computing the stereo matching cost with a convolutional neural network , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[9]  A. N. Rajagopalan,et al.  Non-uniform Motion Deblurring for Bilayer Scenes , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[10]  Feng Liu,et al.  Video Frame Interpolation via Adaptive Separable Convolution , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[11]  Seung-Hwan Baek,et al.  Enhancing the Spatial Resolution of Stereo Images Using a Parallax Prior , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[12]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[13]  Kyoung Mu Lee,et al.  Joint Estimation of Camera Pose, Depth, Deblurring, and Super-Resolution from a Blurred Image Sequence , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[14]  Li Xu,et al.  Depth-aware motion deblurring , 2012, 2012 IEEE International Conference on Computational Photography (ICCP).

[15]  Deqing Sun,et al.  Blind Image Deblurring Using Dark Channel Prior , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[16]  Frédo Durand,et al.  Burst Image Deblurring Using Permutation Invariant Convolutional Neural Networks , 2018, ECCV.

[17]  Guillermo Sapiro,et al.  Deep Video Deblurring for Hand-Held Cameras , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[18]  Jiri Matas,et al.  DeblurGAN: Blind Motion Deblurring Using Conditional Adversarial Networks , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[19]  Thomas Brox,et al.  A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimation , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[20]  Rynson W. H. Lau,et al.  Learning Fully Convolutional Networks for Iterative Non-blind Deconvolution , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[21]  Yong-Sheng Chen,et al.  Pyramid Stereo Matching Network , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[22]  Yair Weiss,et al.  From learning models of natural image patches to whole image restoration , 2011, 2011 International Conference on Computer Vision.

[23]  Andreas Geiger,et al.  Object scene flow for autonomous vehicles , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[24]  Wen Gao,et al.  Depth-Aware Stereo Video Retargeting , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[25]  Yi Wang,et al.  Scale-Recurrent Network for Deep Image Deblurring , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[26]  Liang Lin,et al.  Zoom and Learn: Generalizing Deep Stereo Matching to Novel Domains , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[27]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[28]  Nenghai Yu,et al.  Stereoscopic Neural Style Transfer , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[29]  Kurt Keutzer,et al.  Dense Point Trajectories by GPU-Accelerated Large Displacement Optical Flow , 2010, ECCV.

[30]  Li Fei-Fei,et al.  Perceptual Losses for Real-Time Style Transfer and Super-Resolution , 2016, ECCV.

[31]  Bernhard Schölkopf,et al.  Spatio-Temporal Transformer Network for Video Restoration , 2018, ECCV.

[32]  Bernhard Schölkopf,et al.  Online Video Deblurring via Dynamic Temporal Blending Network , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[33]  Jean Ponce,et al.  Non-uniform Deblurring for Shaken Images , 2012, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[34]  Luca Antiga,et al.  Automatic differentiation in PyTorch , 2017 .

[35]  Ian D. Reid,et al.  From Motion Blur to Motion Flow: A Deep Learning Solution for Removing Heterogeneous Motion Blur , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[36]  Iasonas Kokkinos,et al.  DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[37]  Ming-Hsuan Yang,et al.  Blind Image Deblurring via Deep Discriminative Priors , 2019, International Journal of Computer Vision.

[38]  Anita Sellent,et al.  Stereo Video Deblurring , 2016, ECCV.

[39]  Dongwoo Lee,et al.  Joint Blind Motion Deblurring and Depth Estimation of Light Field , 2017, ECCV.

[40]  Thomas Brox,et al.  Occlusions, Motion and Depth Boundaries with a Generic Network for Disparity, Optical Flow or Scene Flow Estimation , 2018, ECCV.

[41]  Bernhard Schölkopf,et al.  Fast removal of non-uniform camera shake , 2011, 2011 International Conference on Computer Vision.

[42]  Tong Zhang,et al.  Neural Stereoscopic Image Style Transfer , 2018, ECCV.

[43]  Jean Ponce,et al.  Learning a convolutional neural network for non-uniform motion blur removal , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[44]  Rynson W. H. Lau,et al.  Dynamic Scene Deblurring Using Spatially Variant Recurrent Neural Networks , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.