End-to-End Learning for Image Burst Deblurring

We present a neural network model approach for multi-frame blind deconvolution. The discriminative approach adopts and combines two recent techniques for image deblurring into a single neural network architecture. Our proposed hybrid-architecture combines the explicit prediction of a deconvolution filter and non-trivial averaging of Fourier coefficients in the frequency domain. In order to make full use of the information contained in all images in one burst, the proposed network embeds smaller networks, which explicitly allow the model to transfer information between images in early layers. Our system is trained end-to-end using standard backpropagation on a set of artificially generated training examples, enabling competitive performance in multi-frame blind deconvolution, both with respect to quality and runtime.

[1]  Kiriakos N. Kutulakos,et al.  Time-constrained photography , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[2]  Stefan Harmeling,et al.  Image denoising: Can plain neural networks compete with BM3D? , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[3]  P. Baudoz,et al.  A Highly Efficient Lucky Imaging Algorithm: Image Synthesis Based on Fourier Amplitude Selection , 2012 .

[4]  Haichao Zhang,et al.  Intra-frame deblurring by leveraging inter-frame camera motion , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[5]  Bernhard Schölkopf,et al.  Efficient filter flow for space-variant multiframe blind deconvolution , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[6]  Viorica Patraucean,et al.  Spatio-temporal video autoencoder with differentiable memory , 2015, ArXiv.

[7]  Ayan Chakrabarti,et al.  A Neural Approach to Blind Motion Deblurring , 2016, ECCV.

[8]  Yoram Singer,et al.  Adaptive Subgradient Methods for Online Learning and Stochastic Optimization , 2011, J. Mach. Learn. Res..

[9]  Shmuel Peleg,et al.  Two motion-blurred images are better than one , 2005, Pattern Recognit. Lett..

[10]  Tae Hyun Kim,et al.  Dynamic Scene Deblurring using a Locally Adaptive Linear Blur Model , 2016, ArXiv.

[11]  Andrew Zisserman,et al.  Spatial Transformer Networks , 2015, NIPS.

[12]  Yair Weiss,et al.  The Return of the Gating Network: Combining Generative Models and Discriminative Training in Natural Image Priors , 2015, NIPS.

[13]  Haichao Zhang,et al.  Multi-shot Imaging: Joint Alignment, Deblurring, and Resolution-Enhancement , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[14]  Sunghyun Cho,et al.  Edge-based blur kernel estimation using patch priors , 2013, IEEE International Conference on Computational Photography (ICCP).

[15]  Jan Kotera,et al.  Convolutional Neural Networks for Direct Text Deblurring , 2015, BMVC.

[16]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[17]  Bernhard Schölkopf,et al.  A Machine Learning Approach for Non-blind Image Deconvolution , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[18]  Svoboda Pavel,et al.  CNN for license plate motion deblurring , 2016 .

[19]  Dacheng Tao,et al.  Recent Progress in Image Deblurring , 2014, ArXiv.

[20]  Guillermo Sapiro,et al.  Burst deblurring: Removing camera shake through fourier burst accumulation , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[21]  Bernhard Schölkopf,et al.  Recording and Playback of Camera Shake: Benchmarking Blind Deconvolution with a Real-World Database , 2012, ECCV.

[22]  Jia Chen,et al.  Robust dual motion deblurring , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[23]  Xiang Zhu,et al.  Deconvolving PSFs for a Better Motion Deblurring Using Multiple Images , 2012, ECCV.

[24]  Bernhard Schölkopf,et al.  Learning to Deblur , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[25]  Peyman Milanfar,et al.  Robust Multichannel Blind Deconvolution via Fast Alternating Minimization , 2012, IEEE Transactions on Image Processing.

[26]  Guillermo Sapiro,et al.  Hand-Held Video Deblurring Via Efficient Fourier Aggregation , 2015, IEEE Transactions on Computational Imaging.

[27]  Martín Abadi,et al.  TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems , 2016, ArXiv.

[28]  Pietro Perona,et al.  Microsoft COCO: Common Objects in Context , 2014, ECCV.

[29]  Yanning Zhang,et al.  Multi-Observation Blind Deconvolution with an Adaptive Sparse Prior , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[30]  Yanning Zhang,et al.  Multi-image Blind Deblurring Using a Coupled Adaptive Sparse Prior , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[31]  Jian-Feng Cai,et al.  Blind motion deblurring using multiple images , 2009, J. Comput. Phys..

[32]  Jean Ponce,et al.  Learning a convolutional neural network for non-uniform motion blur removal , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[33]  Michal Irani,et al.  Blind Deblurring Using Internal Patch Recurrence , 2014, ECCV.

[34]  Ce Liu,et al.  Deep Convolutional Neural Network for Image Deconvolution , 2014, NIPS.

[35]  Aswin C. Sankaranarayanan,et al.  BlurBurst : Removing Blur Due to Camera Shake using Multiple Images , 2013 .

[36]  Bernhard Schölkopf,et al.  Retrospective Motion Correction of Magnitude-Input MR Images , 2015, MLMMI@ICML.

[37]  Yair Weiss,et al.  From learning models of natural image patches to whole image restoration , 2011, 2011 International Conference on Computer Vision.