Depth from Defocus in the Wild

We consider the problem of two-frame depth from defocus in conditions unsuitable for existing methods yet typical of everyday photography: a handheld cellphone camera, a small aperture, a non-stationary scene and sparse surface texture. Our approach combines a global analysis of image content—3D surfaces, deformations, figure-ground relations, textures—with local estimation of joint depth-flow likelihoods in tiny patches. To enable local estimation we (1) derive novel defocus-equalization filters that induce brightness constancy across frames and (2) impose a tight upper bound on defocus blur—just three pixels in radius—through an appropriate choice of the second frame. For global analysis we use a novel piecewise-spline scene representation that can propagate depth and flow across large irregularly-shaped regions. Our experiments show that this combination preserves sharp boundaries and yields good depth and flow maps in the face of significant noise, uncertainty, non-rigidity, and data sparsity.

[1]  Rama Chellappa,et al.  Variable focus video: Reconstructing depth and video for dynamic scenes , 2012, 2012 IEEE International Conference on Computational Photography (ICCP).

[2]  Sanja Fidler,et al.  Real-time coarse-to-fine topologically preserving segmentation , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[3]  Alex Pentland,et al.  A New Sense for Depth of Field , 1985, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[4]  Yoav Y. Schechner,et al.  The optimal axial interval in estimating depth from defocus , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[5]  Luc Van Gool,et al.  Convolutional Oriented Boundaries , 2016, ECCV.

[6]  Stephen Lin,et al.  Coded Aperture Pairs for Depth from Defocus and Defocus Deblurring , 2011, International Journal of Computer Vision.

[7]  Shree K. Nayar,et al.  Rational Filters for Passive Depth from Defocus , 1998, International Journal of Computer Vision.

[8]  Konrad Schindler,et al.  Piecewise Rigid Scene Flow , 2013, 2013 IEEE International Conference on Computer Vision.

[9]  Steven M. Seitz,et al.  Depth from focus with your mobile phone , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[10]  Bernhard P. Wrobel,et al.  Multiple View Geometry in Computer Vision , 2001 .

[11]  Peyman Milanfar,et al.  Robust Multichannel Blind Deconvolution via Fast Alternating Minimization , 2012, IEEE Transactions on Image Processing.

[12]  Simon Baker,et al.  Lucas-Kanade 20 Years On: A Unifying Framework , 2004, International Journal of Computer Vision.

[13]  Li Zhang,et al.  Optical flow in the presence of spatially-varying motion blur , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[14]  Carlo Tomasi,et al.  Multiway cut for stereo and motion with slanted surfaces , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[15]  Raquel Urtasun,et al.  Efficient Joint Segmentation, Occlusion Labeling, Stereo and Flow Estimation , 2014, ECCV.

[16]  Stefano Soatto,et al.  This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE IEEE TRANSACTION OF PATTERN RECO , 2022 .

[17]  Jonathan T. Barron,et al.  Multiscale Combinatorial Grouping , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[18]  Subhasis Chaudhuri,et al.  A Variational Approach to Recovering Depth From Defocused Images , 1997, IEEE Trans. Pattern Anal. Mach. Intell..

[19]  Qi Guo,et al.  Focal Flow: Measuring Distance and Velocity with Defocus and Differential Motion , 2016, ECCV.

[20]  Edward H. Adelson,et al.  Crisp Boundary Detection Using Pointwise Mutual Information , 2014, ECCV.

[21]  Olga Veksler,et al.  Fast Approximate Energy Minimization via Graph Cuts , 2001, IEEE Trans. Pattern Anal. Mach. Intell..

[22]  Feng Li,et al.  Dual-focus stereo imaging , 2010, J. Electronic Imaging.

[23]  Michael Möller,et al.  Variational Depth From Focus Reconstruction , 2014, IEEE Transactions on Image Processing.

[24]  Murali Subbarao,et al.  Depth-from-defocus: blur equalization technique , 2006, SPIE Optics East.

[25]  Daniel Cremers,et al.  LSD-SLAM: Large-Scale Direct Monocular SLAM , 2014, ECCV.

[26]  Carsten Rother,et al.  FusionFlow: Discrete-continuous optimization for optical flow estimation , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[27]  Didier Stricker,et al.  Flow Fields: Dense Correspondence Fields for Highly Accurate Large Displacement Optical Flow Estimation , 2019, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[28]  Heiko Hirschmüller,et al.  Evaluation of Cost Functions for Stereo Matching , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[29]  Ying Xiong,et al.  Low-level vision by consensus in a spatial hierarchy of regions , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[30]  Fisher Yu,et al.  3D Reconstruction from Accidental Motion , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[31]  Paolo Favaro,et al.  Recovering thin structures via nonlocal-means regularization with application to depth from defocus , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[32]  Horst Bischof,et al.  Image Guided Depth Upsampling Using Anisotropic Total Generalized Variation , 2013, 2013 IEEE International Conference on Computer Vision.

[33]  Subhasis Chaudhuri,et al.  Optimal selection of camera parameters for recovery of depth from defocused images , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.