Gyroscope-Aided Motion Deblurring with Deep Networks

We propose a deblurring method that incorporates gyroscope measurements into a convolutional neural network (CNN). With the help of such measurements, it can handle extremely strong and spatially-variant motion blur. At the same time, the image data is used to overcome the limitations of gyro-based blur estimation. To train our network, we also introduce a novel way of generating realistic training data using the gyroscope. The evaluation shows a clear improvement in visual quality over the state-of-the-art while achieving real-time performance. Furthermore, the method is shown to improve the performance of existing feature detectors and descriptors against the motion blur.

[1]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[2]  Jiri Matas,et al.  Fast Motion Deblurring for Feature Detection and Matching Using Inertial Measurements , 2018, 2018 24th International Conference on Pattern Recognition (ICPR).

[3]  Jean Ponce,et al.  Learning a convolutional neural network for non-uniform motion blur removal , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[4]  Jiri Matas,et al.  DeblurGAN: Blind Motion Deblurring Using Conditional Adversarial Networks , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[5]  Jörg Stückler,et al.  The TUM VI Benchmark for Evaluating Visual-Inertial Odometry , 2018, 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[6]  Marc Levoy,et al.  Gyro-Based Multi-image Deconvolution for Removing Handshake Blur , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[7]  Stephen Lin,et al.  Correction of Spatially Varying Image and Video Motion Blur Using a Hybrid Camera , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[8]  Dacheng Tao,et al.  Training Very Deep CNNs for General Non-Blind Deconvolution , 2018, IEEE Transactions on Image Processing.

[9]  Frédo Durand,et al.  Burst Image Deblurring Using Permutation Invariant Convolutional Neural Networks , 2018, ECCV.

[10]  Guillermo Sapiro,et al.  Deep Video Deblurring for Hand-Held Cameras , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[11]  Deqing Sun,et al.  Blind Image Deblurring Using Dark Channel Prior , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[12]  Ian D. Reid,et al.  From Motion Blur to Motion Flow: A Deep Learning Solution for Removing Heterogeneous Motion Blur , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[13]  Hui Ma,et al.  Image Deblurring with Blurred / Noisy Image Pairs , 2013 .

[14]  Keigo Hirakawa,et al.  Combining Inertial Measurements With Blind Image Deblurring Using Distance Transform , 2016, IEEE Transactions on Computational Imaging.

[15]  Arno Solin,et al.  Inertial Odometry on Handheld Smartphones , 2017, 2018 21st International Conference on Information Fusion (FUSION).

[16]  Tae Hyun Kim,et al.  Deep Multi-scale Convolutional Neural Network for Dynamic Scene Deblurring , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[17]  Filip Sroubek,et al.  Image deblurring in smartphone devices using built-in inertial measurement sensors , 2013, J. Electronic Imaging.

[18]  John Weston,et al.  Strapdown Inertial Navigation Technology , 1997 .

[19]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[20]  S. B. Kang,et al.  Image deblurring using inertial measurement sensors , 2010, ACM Trans. Graph..

[21]  A. Vedaldi.  An open implementation of the SIFT detector and descriptor , 2007 .

[22]  Thekke Madam Nimisha,et al.  Blur-Invariant Deep Learning for Blind-Deblurring , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[23]  Jan Kautz,et al.  Reblur2Deblur: Deblurring videos via self-supervised learning , 2018, 2018 IEEE International Conference on Computational Photography (ICCP).

[24]  Stephen Lin,et al.  Image Deblurring Using Smartphone Inertial Sensors , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[25]  Bart Thomee,et al.  New trends and ideas in visual concept detection: the MIR flickr retrieval evaluation initiative , 2010, MIR '10.

[26]  Alexei A. Efros,et al.  Image-to-Image Translation with Conditional Adversarial Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[27]  Bernhard P. Wrobel,et al.  Multiple View Geometry in Computer Vision , 2001 .

[28]  Xiaochun Cao,et al.  Image Deblurring via Extreme Channels Prior , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[29]  Tobias Höllerer,et al.  Evaluation of Interest Point Detectors and Feature Descriptors for Visual Tracking , 2011, International Journal of Computer Vision.

[30]  Seungyong Lee,et al.  Fast non-blind deconvolution via regularized residual networks with long/short skip-connections , 2017, 2017 IEEE International Conference on Computational Photography (ICCP).

[31]  Cordelia Schmid,et al.  A performance evaluation of local descriptors , 2005, IEEE Transactions on Pattern Analysis and Machine Intelligence.