Learning Wavefront Coding for Extended Depth of Field Imaging

Depth of field is an important factor of imaging systems that highly affects the quality of the acquired spatial information. Extended depth of field (EDoF) imaging is a challenging ill-posed problem and has been extensively addressed in the literature. We propose a computational imaging approach for EDoF, where we employ wavefront coding via a diffractive optical element (DOE) and we achieve deblurring through a convolutional neural network. Thanks to the end-to-end differentiable modeling of optical image formation and computational post-processing, we jointly optimize the optical design, i.e., DOE, and the deblurring through standard gradient descent methods. Based on the properties of the underlying refractive lens and the desired EDoF range, we provide an analytical expression for the search space of the DOE, which is instrumental in the convergence of the end-to-end network. We achieve superior EDoF imaging performance compared to the state of the art, where we demonstrate results with minimal artifacts in various scenarios, including deep 3D scenes and broadband imaging.

[1]  Stephen P. Boyd,et al.  End-to-end optimization of optics and image processing for achromatic extended depth of field and super-resolution imaging , 2018, ACM Trans. Graph..

[2]  Frédo Durand,et al.  Image and depth from a conventional camera with a coded aperture , 2007, ACM Trans. Graph..

[3]  J. Goodman Introduction to Fourier optics , 1969 .

[4]  Nikolas P. Galatsanos,et al.  A variational approach for Bayesian blind image deconvolution , 2004, IEEE Transactions on Signal Processing.

[5]  Prasan A. Shedligeri,et al.  Data driven coded aperture design for depth recovery , 2017, 2017 IEEE International Conference on Image Processing (ICIP).

[6]  Seungyong Lee,et al.  Fast non-blind deconvolution via regularized residual networks with long/short skip-connections , 2017, 2017 IEEE International Conference on Computational Photography (ICCP).

[7]  Ren Ng Fourier slice photography , 2005, ACM Trans. Graph..

[8]  W. Welford Use of Annular Apertures to Increase Focal Depth , 1960 .

[9]  Jan Kautz,et al.  Loss Functions for Image Restoration With Neural Networks , 2017, IEEE Transactions on Computational Imaging.

[10]  W. Cathey,et al.  Extended depth of field through wave-front coding. , 1995, Applied optics.

[11]  Raja Giryes,et al.  Depth Estimation From a Single Image Using Deep Learned Phase Coded Mask , 2018, IEEE Transactions on Computational Imaging.

[12]  Cordelia Schmid,et al.  Hamming Embedding and Weak Geometric Consistency for Large Scale Image Search , 2008, ECCV.

[13]  Robert D. Nowak,et al.  An EM algorithm for wavelet-based image restoration , 2003, IEEE Trans. Image Process..

[14]  Zhigang Fan,et al.  Optimized asymmetrical tangent phase mask to obtain defocus invariant modulation transfer function in incoherent imaging systems. , 2014, Optics letters.

[15]  Wen Gao,et al.  Spatially variant defocus blur map estimation and deblurring from a single image , 2016, J. Vis. Commun. Image Represent..

[16]  Erdem Sahin,et al.  Learning Optimal Phase-Coded Aperture for Depth of Field Extension , 2019, 2019 IEEE International Conference on Image Processing (ICIP).

[17]  Raja Giryes,et al.  Learned phase coded aperture for the benefit of depth of field extension. , 2018, Optics express.

[18]  Charless C. Fowlkes,et al.  Contour Detection and Hierarchical Image Segmentation , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[19]  Masatoshi Okutomi,et al.  A Practical One-Shot Multispectral Imaging System Using a Single Image Sensor , 2015, IEEE Transactions on Image Processing.

[20]  Sheng Liu,et al.  Extended depth-of-field microscopic imaging with a variable focus microscope objective. , 2011, Optics express.

[21]  Tao Zhang,et al.  HyperReconNet: Joint Coded Aperture Optimization and Image Reconstruction for Compressive Hyperspectral Imaging , 2019, IEEE Transactions on Image Processing.

[22]  Hui Zhao,et al.  Optimized sinusoidal phase mask to extend the depth of field of an incoherent imaging system. , 2010, Optics letters.

[23]  Stephen Lin,et al.  Coded Aperture Pairs for Depth from Defocus and Defocus Deblurring , 2011, International Journal of Computer Vision.

[24]  Diego Gutierrez,et al.  Perceptually Optimized Coded Apertures for Defocus Deblurring , 2012, Comput. Graph. Forum.

[25]  Gordon Wetzstein,et al.  Hybrid optical-electronic convolutional neural networks with optimized diffractive optics for image classification , 2018, Scientific Reports.

[26]  Wangmeng Zuo,et al.  Learning Deep CNN Denoiser Prior for Image Restoration , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[27]  P. Hanrahan,et al.  Light Field Photography with a Hand-held Plenoptic Camera , 2005 .

[28]  Rajesh Menon,et al.  Multi-plane, multi-band image projection via broadband diffractive optics. , 2019, Applied optics.

[29]  Oliver Cossairt,et al.  Spectral Focal Sweep: Extended depth of field from chromatic aberrations , 2010, 2010 IEEE International Conference on Computational Photography (ICCP).

[30]  W. Cathey,et al.  Phase plate to extend the depth of field of incoherent hybrid imaging systems. , 2004, Applied optics.

[31]  Luís B. Almeida,et al.  Blind and Semi-Blind Deblurring of Natural Images , 2010, IEEE Transactions on Image Processing.

[32]  Ramesh Raskar,et al.  Dappled photography: mask enhanced cameras for heterodyned light fields and coded aperture refocusing , 2007, ACM Trans. Graph..

[33]  Lennart Wietzke,et al.  Single lens 3D-camera with extended depth-of-field , 2012, Electronic Imaging.

[34]  S. Nayar,et al.  What are good apertures for defocus deblurring? , 2009, 2009 IEEE International Conference on Computational Photography (ICCP).

[35]  Frédo Durand,et al.  4D frequency analysis of computational cameras for depth of field extension , 2009, SIGGRAPH '09.

[36]  Changyin Zhou,et al.  Diffusion coded photography for extended depth of field , 2010, ACM Trans. Graph..

[37]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[38]  N. George,et al.  Hybrid diffractive-refractive lenses and achromats. , 1988, Applied optics.

[39]  Toshiaki Fujii,et al.  Learning to Capture Light Fields Through a Coded Aperture Camera , 2018, ECCV.

[40]  Rynson W. H. Lau,et al.  FormResNet: Formatted Residual Learning for Image Restoration , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[41]  Andrew Lumsdaine,et al.  The focused plenoptic camera , 2009, 2009 IEEE International Conference on Computational Photography (ICCP).

[42]  Horng Chang,et al.  Design of mobile phone lens with extended depth of field based on point-spread function focus invariance , 2008, Optical Engineering + Applications.

[43]  Wen Gao,et al.  Graph-Based Blind Image Deblurring From a Single Photograph , 2018, IEEE Transactions on Image Processing.