Region-Adaptive Dense Network for Efficient Motion Deblurring

In this paper, we address the problem of dynamic scene deblurring in the presence of motion blur. Restoration of images affected by severe blur necessitates a network design with a large receptive field, which existing networks attempt to achieve through simple increment in the number of generic convolution layers, kernel-size, or the scales at which the image is processed. However, these techniques ignore the non-uniform nature of blur, and they come at the expense of an increase in model size and inference time. We present a new architecture composed of region adaptive dense deformable modules that implicitly discover the spatially varying shifts responsible for non-uniform blur in the input image and learn to modulate the filters. This capability is complemented by a self-attentive module which captures non-local spatial relationships among the intermediate features and enhances the spatially-varying processing capability. We incorporate these modules into a densely connected encoder-decoder design which utilizes pre-trained Densenet filters to further improve the performance. Our network facilitates interpretable modeling of the spatially-varying deblurring process while dispensing with multi-scale processing and large filters entirely. Extensive comparisons with prior art on benchmark dynamic scene deblurring datasets clearly demonstrate the superiority of the proposed networks via significant improvements in accuracy and speed, enabling almost real-time deblurring.

[1]  Narendra Ahuja,et al.  A Comparative Study for Single Image Blind Deblurring , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[2]  Raquel Urtasun,et al.  Understanding the Effective Receptive Field in Deep Convolutional Neural Networks , 2016, NIPS.

[3]  Han Zhang,et al.  Self-Attention Generative Adversarial Networks , 2018, ICML.

[4]  Deqing Sun,et al.  Blind Image Deblurring Using Dark Channel Prior , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[5]  Andrew Zisserman,et al.  Spatial Transformer Networks , 2015, NIPS.

[6]  Seungyong Lee,et al.  Registration Based Non‐uniform Motion Deblurring , 2012, Comput. Graph. Forum.

[7]  Rob Fergus,et al.  Blind deconvolution using a normalized sparsity measure , 2011, CVPR 2011.

[8]  Tae Hyun Kim,et al.  Generalized video deblurring for dynamic scenes , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[9]  Li Xu,et al.  Two-Phase Kernel Estimation for Robust Motion Deblurring , 2010, ECCV.

[10]  Kilian Q. Weinberger,et al.  Densely Connected Convolutional Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[11]  Haichao Zhang,et al.  Multi-shot Imaging: Joint Alignment, Deblurring, and Resolution-Enhancement , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[12]  Bernhard Schölkopf,et al.  Learning to Deblur , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[13]  Jiri Matas,et al.  DeblurGAN: Blind Motion Deblurring Using Conditional Adversarial Networks , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[14]  Christian Ledig,et al.  Real-Time Video Super-Resolution with Spatio-Temporal Networks and Motion Compensation , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[15]  Abhinav Gupta,et al.  Non-local Neural Networks , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[16]  Xiaochun Cao,et al.  Image Deblurring via Extreme Channels Prior , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[17]  Ming Yang,et al.  Restricted Deformable Convolution-Based Road Scene Semantic Segmentation Using Surround View Cameras , 2018, IEEE Transactions on Intelligent Transportation Systems.

[18]  Ming-Hsuan Yang,et al.  Robust Kernel Estimation with Outliers Handling for Image Deblurring , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[19]  Yanning Zhang,et al.  Multi-image Blind Deblurring Using a Coupled Adaptive Sparse Prior , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[20]  Jian-Feng Cai,et al.  Blind motion deblurring using multiple images , 2009, J. Comput. Phys..

[21]  Jia Xu,et al.  Fast Image Processing with Fully-Convolutional Networks , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[22]  Li Xu,et al.  Unnatural L0 Sparse Representation for Natural Image Deblurring , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[23]  Ming-Hsuan Yang,et al.  Deblurring Text Images via L0-Regularized Intensity and Gradient Prior , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[24]  Gregory Shakhnarovich,et al.  Examining the Impact of Blur on Recognition by Convolutional Networks , 2016, ArXiv.

[25]  Hongdong Li,et al.  Deep Stacked Hierarchical Multi-Patch Network for Image Deblurring , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[26]  Ayan Chakrabarti,et al.  A Neural Approach to Blind Motion Deblurring , 2016, ECCV.

[27]  Tao Shen,et al.  DiSAN: Directional Self-Attention Network for RNN/CNN-free Language Understanding , 2017, AAAI.

[28]  Guillermo Sapiro,et al.  Hand-Held Video Deblurring Via Efficient Fourier Aggregation , 2015, IEEE Transactions on Computational Imaging.

[29]  Frédo Durand,et al.  Burst Image Deblurring Using Permutation Invariant Convolutional Neural Networks , 2018, ECCV.

[30]  Guillermo Sapiro,et al.  Deep Video Deblurring for Hand-Held Cameras , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[31]  Bernhard Schölkopf,et al.  Learning Blind Motion Deblurring , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[32]  Tae Hyun Kim,et al.  Dynamic Scene Deblurring , 2013, 2013 IEEE International Conference on Computer Vision.

[33]  Bernhard Schölkopf,et al.  A Machine Learning Approach for Non-blind Image Deconvolution , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[34]  Xiaoyong Shen,et al.  Dynamic Scene Deblurring With Parameter Selective Sharing and Nested Skip Connections , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[35]  Bernhard Schölkopf,et al.  Online Video Deblurring via Dynamic Temporal Blending Network , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[36]  Lukasz Kaiser,et al.  Attention is All you Need , 2017, NIPS.

[37]  A. N. Rajagopalan,et al.  From Local to Global: Edge Profiles to Camera Motion in Blurred Images , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[38]  Sunghyun Cho,et al.  Fast motion deblurring , 2009, SIGGRAPH 2009.

[39]  Jean Ponce,et al.  Learning a convolutional neural network for non-uniform motion blur removal , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[40]  Dit-Yan Yeung,et al.  Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting , 2015, NIPS.

[41]  Rynson W. H. Lau,et al.  Dynamic Scene Deblurring Using Spatially Variant Recurrent Neural Networks , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[42]  Liang Lin,et al.  Multi-level Wavelet-CNN for Image Restoration , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[43]  Tae Hyun Kim,et al.  Deep Multi-scale Convolutional Neural Network for Dynamic Scene Deblurring , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[44]  Richard Szeliski,et al.  PSF estimation using sharp edge prediction , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[45]  Jiawen Chen,et al.  Bilateral guided upsampling , 2016, ACM Trans. Graph..

[46]  Michael J. Black,et al.  Modeling Blurred Video with Layers , 2014, ECCV.

[47]  Luca Antiga,et al.  Automatic differentiation in PyTorch , 2017 .

[48]  William T. Freeman,et al.  Removing camera shake from a single photograph , 2006, SIGGRAPH 2006.

[49]  Ian D. Reid,et al.  From Motion Blur to Motion Flow: A Deep Learning Solution for Removing Heterogeneous Motion Blur , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[50]  Bowen Zhou,et al.  A Structured Self-attentive Sentence Embedding , 2017, ICLR.

[51]  Rob Fergus,et al.  Fast Image Deconvolution using Hyper-Laplacian Priors , 2009, NIPS.

[52]  Kyoung Mu Lee,et al.  Enhanced Deep Residual Networks for Single Image Super-Resolution , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[53]  Haichao Zhang,et al.  Intra-frame deblurring by leveraging inter-frame camera motion , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[54]  Yi Li,et al.  Deformable Convolutional Networks , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[55]  Yi Wang,et al.  Scale-Recurrent Network for Deep Image Deblurring , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.