Inertial-aided Motion Deblurring with Deep Networks

We propose an inertial-aided deblurring method that incorporates gyroscope measurements into a convolutional neural network (CNN). With the help of inertial measurements, it can handle extremely strong and spatially-variant motion blur. At the same time, the image data is used to overcome the limitations of gyro-based blur estimation. To train our network, we also introduce a novel way of generating realistic training data using the gyroscope. The evaluation shows a clear improvement in visual quality over the state-of-the-art while achieving real-time performance. Furthermore, the method is shown to improve the performance of existing feature detectors and descriptors against the motion blur.

[1]  Frédo Durand,et al.  Burst Image Deblurring Using Permutation Invariant Convolutional Neural Networks , 2018, ECCV.

[2]  Guillermo Sapiro,et al.  Deep Video Deblurring for Hand-Held Cameras , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[3]  A. Vedaldi.  An open implementation of the SIFT detector and descriptor , 2007 .

[4]  Jiri Matas,et al.  Fast Motion Deblurring for Feature Detection and Matching Using Inertial Measurements , 2018, 2018 24th International Conference on Pattern Recognition (ICPR).

[5]  Jörg Stückler,et al.  The TUM VI Benchmark for Evaluating Visual-Inertial Odometry , 2018, 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[6]  Marc Levoy,et al.  Gyro-Based Multi-image Deconvolution for Removing Handshake Blur , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[7]  Stephen Lin,et al.  Image Deblurring Using Smartphone Inertial Sensors , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[8]  Cordelia Schmid,et al.  A Performance Evaluation of Local Descriptors , 2005, IEEE Trans. Pattern Anal. Mach. Intell..

[9]  Jiri Matas,et al.  DeblurGAN: Blind Motion Deblurring Using Conditional Adversarial Networks , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[10]  Alexei A. Efros,et al.  Image-to-Image Translation with Conditional Adversarial Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[11]  A N Rajagopalan,et al.  Blur-Invariant Deep Learning for Blind-Deblurring ( Supplementary Material ) , 2017 .

[12]  Stephen Lin,et al.  Correction of Spatially Varying Image and Video Motion Blur Using a Hybrid Camera , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[13]  Bart Thomee,et al.  New trends and ideas in visual concept detection: the MIR flickr retrieval evaluation initiative , 2010, MIR '10.

[14]  Dacheng Tao,et al.  Training Very Deep CNNs for General Non-Blind Deconvolution , 2018, IEEE Transactions on Image Processing.

[15]  Xiaochun Cao,et al.  Image Deblurring via Extreme Channels Prior , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[16]  Ian D. Reid,et al.  From Motion Blur to Motion Flow: A Deep Learning Solution for Removing Heterogeneous Motion Blur , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[17]  Arno Solin,et al.  Inertial Odometry on Handheld Smartphones , 2017, 2018 21st International Conference on Information Fusion (FUSION).

[18]  S. B. Kang,et al.  Image deblurring using inertial measurement sensors , 2010, SIGGRAPH 2010.

[19]  Jan Kautz,et al.  Reblur2Deblur: Deblurring videos via self-supervised learning , 2018, 2018 IEEE International Conference on Computational Photography (ICCP).

[20]  Tobias Höllerer,et al.  Evaluation of Interest Point Detectors and Feature Descriptors for Visual Tracking , 2011, International Journal of Computer Vision.

[21]  Seungyong Lee,et al.  Fast non-blind deconvolution via regularized residual networks with long/short skip-connections , 2017, 2017 IEEE International Conference on Computational Photography (ICCP).

[22]  Jean Ponce,et al.  Learning a convolutional neural network for non-uniform motion blur removal , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[23]  Tae Hyun Kim,et al.  Deep Multi-scale Convolutional Neural Network for Dynamic Scene Deblurring , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[24]  Keigo Hirakawa,et al.  Combining Inertial Measurements With Blind Image Deblurring Using Distance Transform , 2016, IEEE Transactions on Computational Imaging.

[25]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[26]  Bernhard P. Wrobel,et al.  Multiple View Geometry in Computer Vision , 2001 .

[27]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[28]  Deqing Sun,et al.  Blind Image Deblurring Using Dark Channel Prior , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[29]  Hui Ma,et al.  Image Deblurring with Blurred / Noisy Image Pairs , 2013 .

[30]  Filip Sroubek,et al.  Image deblurring in smartphone devices using built-in inertial measurement sensors , 2013, J. Electronic Imaging.

[31]  John Weston,et al.  Strapdown Inertial Navigation Technology , 1997 .