Deblur and deep depth from single defocus image

In this paper, we tackle depth estimation and blur removal from a single out-of-focus image. Previously, depth is estimated, and blurred is removed using multiple images; for example, from multiview or stereo scenes, but doing so with a single image is challenging. Earlier works of monocular images for depth estimated and deblurring either exploited geometric characteristics or priors using hand-crafted features. Lately, there is enough evidence that deep convolutional neural networks (CNN) significantly improved numerous vision applications; hence, in this article, we present a depth estimation method that leverages rich representations learned from cascaded convolutional and fully connected neural networks operating on a patch-pooled set of feature maps. Furthermore, from this depth, we computationally reconstruct an all-focus image, i.e. , removing the blur and achieve synthetic re-focusing, all from a single image. Our method is fast, and it substantially improves depth accuracy over the state-of-the-art alternatives. Our proposed depth estimation approach can be utilized for everyday scenes without any geometric priors or extra information. Furthermore, our experiments on two benchmark datasets consist images of indoor and outdoor scenes, i.e. , Make3D and NYU-v2 demonstrate superior performance in comparison with other available depth estimation state-of-the-art methods by reducing the root-mean-squared error by 57% and 46% , and state-of-the-art blur removal methods by 0.36 dB and 0.72 dB in PSNR, respectively. This improvement in-depth estimation and deblurring is further demonstrated by the superior performance using real defocus images against images captured with a prototype lens.

[1]  Li Xu,et al.  Depth-aware motion deblurring , 2012, 2012 IEEE International Conference on Computational Photography (ICCP).

[2]  Tae-Sun Choi,et al.  Nonlinear Approach for Enhancement of Image Focus Volume in Shape From Focus , 2012, IEEE Transactions on Image Processing.

[3]  Subhasis Chaudhuri,et al.  Recovery of relative depth from a single observation using an uncalibrated (real-aperture) camera , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[4]  Guosheng Lin,et al.  Deep convolutional neural fields for depth estimation from a single image , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[5]  Leonidas J. Guibas,et al.  Estimating image depth using shape collections , 2014, ACM Trans. Graph..

[6]  Domenec Puig,et al.  Analysis of focus measure operators for shape-from-focus , 2013, Pattern Recognit..

[7]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[8]  Sylvain Paris,et al.  Blur kernel estimation using the radon transform , 2011, CVPR 2011.

[9]  Trevor Darrell,et al.  Fully Convolutional Networks for Semantic Segmentation , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[10]  Tae-Sun Choi,et al.  A fast and robust depth estimation method for 3D cameras , 2012, 2012 IEEE International Conference on Consumer Electronics (ICCE).

[11]  Sunghyun Cho,et al.  Good Image Priors for Non-blind Deconvolution - Generic vs. Specific , 2014, ECCV.

[12]  Paul A. Viola,et al.  Rapid object detection using a boosted cascade of simple features , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[13]  A. N. Rajagopalan,et al.  Non-uniform Motion Deblurring for Bilayer Scenes , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[14]  Stephen Lin,et al.  Coded Aperture Pairs for Depth from Defocus and Defocus Deblurring , 2011, International Journal of Computer Vision.

[15]  Arif Mahmood,et al.  Multi-focus image fusion using Content Adaptive Blurring , 2019, Inf. Fusion.

[16]  Deva Ramanan,et al.  Face detection, pose estimation, and landmark localization in the wild , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[17]  Assaf Zomet,et al.  Learning how to inpaint from global image statistics , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[18]  Michal Irani,et al.  Blind Deblurring Using Internal Patch Recurrence , 2014, ECCV.

[19]  Yang Cao,et al.  Single Image Multi-focusing Based on Local Blur Estimation , 2011, 2011 Sixth International Conference on Image and Graphics.

[20]  Trevor Darrell,et al.  Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation , 2013, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[21]  Muralidhara Subbarao,et al.  Accurate Recovery of Three-Dimensional Shape from Image Focus , 1995, IEEE Trans. Pattern Anal. Mach. Intell..

[22]  Anat Levin,et al.  Analyzing Depth from Coded Aperture Sets , 2010, ECCV.

[23]  Feng Wu,et al.  DRPL: Deep Regression Pair Learning for Multi-Focus Image Fusion , 2020, IEEE Transactions on Image Processing.

[24]  Li Xu,et al.  Two-Phase Kernel Estimation for Robust Motion Deblurring , 2010, ECCV.

[25]  Fatih Murat Porikli,et al.  Depth Estimation and Blur Removal from a Single Out-of-focus Image , 2017, BMVC.

[26]  Seungyong Lee,et al.  Fast motion deblurring , 2009, ACM Trans. Graph..

[27]  Ce Liu,et al.  Depth Transfer: Depth Extraction from Video Using Non-Parametric Sampling , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[28]  Richard Szeliski,et al.  PSF estimation using sharp edge prediction , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[29]  Jean Ponce,et al.  Non-uniform Deblurring for Shaken Images , 2012, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[30]  Rob Fergus,et al.  Blind deconvolution using a normalized sparsity measure , 2011, CVPR 2011.

[31]  Rob Fergus,et al.  Fast Image Deconvolution using Hyper-Laplacian Priors , 2009, NIPS.

[32]  Stephen Lin,et al.  Image/video deblurring using a hybrid camera , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[33]  Feng Li,et al.  A hybrid camera for motion deblurring and depth map super-resolution , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[34]  Shree K. Nayar,et al.  Depth from Diffusion , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[35]  Andrew Zisserman,et al.  Deblurring shaken and partially saturated images , 2011, ICCV Workshops.

[36]  Terence Sim,et al.  Defocus map estimation from a single image , 2011, Pattern Recognit..

[37]  Jiaya Jia,et al.  High-quality motion deblurring from a single image , 2008, ACM Trans. Graph..

[38]  Ramesh Raskar,et al.  Dappled photography: mask enhanced cameras for heterodyned light fields and coded aperture refocusing , 2007, ACM Trans. Graph..

[39]  Ming-Hsuan Yang,et al.  Deblurring Face Images with Exemplars , 2014, ECCV.

[40]  Frédo Durand,et al.  Image and depth from a conventional camera with a coded aperture , 2007, ACM Trans. Graph..

[41]  Dani Lischinski,et al.  Deblurring by Example Using Dense Correspondence , 2013, 2013 IEEE International Conference on Computer Vision.

[42]  Ayan Chakrabarti,et al.  A Neural Approach to Blind Motion Deblurring , 2016, ECCV.

[43]  K NayarShree,et al.  Motion-Based Motion Deblurring , 2004 .

[44]  Rob Fergus,et al.  Predicting Depth, Surface Normals and Semantic Labels with a Common Multi-scale Convolutional Architecture , 2014, 2015 IEEE International Conference on Computer Vision (ICCV).

[45]  Ming-Hsuan Yang,et al.  Deblurring Text Images via L0-Regularized Intensity and Gradient Prior , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[46]  Stefan Carlsson,et al.  CNN Features Off-the-Shelf: An Astounding Baseline for Recognition , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition Workshops.

[47]  Vicent Caselles,et al.  Recovering Relative Depth from Low-Level Features Without Explicit T-junction Detection and Interpretation , 2013, International Journal of Computer Vision.

[48]  Sundaresh Ram,et al.  Removing Camera Shake from a Single Photograph , 2009 .

[49]  Derek Hoiem,et al.  Indoor Segmentation and Support Inference from RGBD Images , 2012, ECCV.

[50]  Cong Phuoc Huynh,et al.  Class-Specific Image Deblurring , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[51]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[52]  P. Belhumeur,et al.  Active refocusing of images and videos , 2007, ACM Trans. Graph..

[53]  Jitendra Malik,et al.  Category-specific object reconstruction from a single image , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[54]  Ross B. Girshick,et al.  Fast R-CNN , 2015, 1504.08083.

[55]  Jian Sun,et al.  Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[56]  Frédo Durand,et al.  Defocus Magnification , 2007, Comput. Graph. Forum.

[57]  Frédo Durand,et al.  Understanding Blind Deconvolution Algorithms , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[58]  B. Schölkopf,et al.  Blind Motion Deblurring Using Image Statistics , 2007 .

[59]  Bernhard Schölkopf,et al.  Learning to Deblur , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[60]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[61]  Cong Phuoc Huynh,et al.  Image Deblurring with a Class-Specific Prior , 2019, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[62]  Yair Weiss,et al.  From learning models of natural image patches to whole image restoration , 2011, 2011 International Conference on Computer Vision.

[63]  Sunghyun Cho,et al.  Edge-based blur kernel estimation using patch priors , 2013, IEEE International Conference on Computational Photography (ICCP).

[64]  Hui Ma,et al.  Image Deblurring with Blurred / Noisy Image Pairs , 2013 .

[65]  S. Nayar,et al.  What are good apertures for defocus deblurring? , 2009, 2009 IEEE International Conference on Computational Photography (ICCP).

[66]  Kun Zhou,et al.  Bayesian Depth-from-Defocus with Shading Constraints , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[67]  Xuming He,et al.  Discrete-Continuous Depth Estimation from a Single Image , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[68]  Ayan Chakrabarti,et al.  Depth and Deblurring from a Spectrally-Varying Depth-of-Field , 2012, ECCV.

[69]  Edward H. Adelson,et al.  Personal photo enhancement using example images , 2010, TOGS.