Deep Sparse Light Field Refocusing

Light field photography enables to record 4D images, containing angular information alongside spatial information of the scene. One of the important applications of light field imaging is post-capture refocusing. Current methods require for this purpose a dense field of angle views; those can be acquired with a micro-lens system or with a compressive system. Both techniques have major drawbacks to consider, including bulky structures and angular-spatial resolution trade-off. We present a novel implementation of digital refocusing based on sparse angular information using neural networks. This allows recording high spatial resolution in favor of the angular resolution, thus, enabling to design compact and simple devices with improved hardware as well as better performance of compressive systems. We use a novel convolutional neural network whose relatively small structure enables fast reconstruction with low memory consumption. Moreover, it allows handling without re-training various refocusing ranges and noise levels. Results show major improvement compared to existing methods.

[1]  R. Bracewell Strip Integration in Radio Astronomy , 1956 .

[2]  Marc Levoy,et al.  Light field rendering , 1996, SIGGRAPH.

[3]  Marc Levoy,et al.  Synthetic aperture confocal imaging , 2004, ACM Trans. Graph..

[4]  Marc Levoy,et al.  Using plane + parallax for calibrating dense camera arrays , 2004, CVPR 2004.

[5]  Marc Levoy,et al.  Using plane + parallax for calibrating dense camera arrays , 2004, Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004..

[6]  Marc Levoy,et al.  High performance imaging using large camera arrays , 2005, ACM Trans. Graph..

[7]  P. Hanrahan,et al.  Light Field Photography with a Hand-held Plenoptic Camera , 2005 .

[8]  Ren Ng Fourier slice photography , 2005, ACM Trans. Graph..

[9]  Ramesh Raskar,et al.  Dappled photography: mask enhanced cameras for heterodyned light fields and coded aperture refocusing , 2007, ACM Trans. Graph..

[10]  Chia-Kai Liang,et al.  Programmable aperture photography: multiplexed light field acquisition , 2008, SIGGRAPH 2008.

[11]  Yoshua Bengio,et al.  Understanding the difficulty of training deep feedforward neural networks , 2010, AISTATS.

[12]  José Gil Marichal-Hernández,et al.  Fast approximate 4D:3D discrete Radon transform, from light field to focal stack with O(N4) sums , 2011, Electronic Imaging.

[13]  Ashok Veeraraghavan,et al.  Light field denoising, light field superresolution and stereo camera based refocussing using a GMM light field patch prior , 2012, 2012 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops.

[14]  Gordon Wetzstein,et al.  Compressive light field photography using overcomplete dictionaries and optimized projections , 2013, ACM Trans. Graph..

[15]  Zhaolin Xiao,et al.  Aliasing Detection and Reduction in Plenoptic Imaging , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[16]  Nirvair Neeru,et al.  Performance Comparison of Various Image Denoising Filters Under Spatial Domain , 2014 .

[17]  In-So Kweon,et al.  Accurate depth map estimation from a lenslet light field camera , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[18]  Sergey Ioffe,et al.  Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift , 2015, ICML.

[19]  Wenxing Fu,et al.  Implementing light field image refocusing algorithm , 2015, 2015 2nd International Conference on Opto-Electronics and Applied Optics (IEM OPTRONIX).

[20]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[21]  Jonathan T. Barron,et al.  Fast bilateral-space stereo for synthetic defocus , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[22]  Steven M. Seitz,et al.  Depth from focus with your mobile phone , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[23]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[24]  Miseon Han,et al.  Depth-based refocusing for reducing directional aliasing artifacts. , 2016, Optics express.

[25]  Ting-Chun Wang,et al.  Learning-based view synthesis for light field cameras , 2016, ACM Trans. Graph..

[26]  Thomas Pock,et al.  Convolutional Networks for Shape from Light Field , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[27]  In-So Kweon,et al.  A Unified Approach of Multi-scale Deep and Hand-Crafted Features for Defocus Estimation , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[28]  Li Zhang,et al.  Soft 3D reconstruction for view synthesis , 2017, ACM Trans. Graph..

[29]  Kilian Q. Weinberger,et al.  Densely Connected Convolutional Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[30]  Ravi Ramamoorthi,et al.  Learning to Synthesize a 4D RGBD Light Field from a Single Image , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[31]  Alexander M. Bronstein,et al.  Deep class-aware image denoising , 2017, 2017 IEEE International Conference on Image Processing (ICIP).

[32]  Pavan K. Turaga,et al.  Compressive Light Field Reconstructions Using Deep Learning , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[33]  Jie Chen,et al.  Light Field Compressed Sensing Over a Disparity-Aware Dictionary , 2017, IEEE Transactions on Circuits and Systems for Video Technology.

[34]  Qionghai Dai,et al.  Light Field Reconstruction Using Deep Convolutional Network on EPI , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[35]  Tieniu Tan,et al.  End-to-End View Synthesis for Light Field Imaging with Pseudo 4DCNN , 2018, ECCV.

[36]  Raja Giryes,et al.  Fast and accurate reconstruction of compressed color light field , 2018, 2018 IEEE International Conference on Computational Photography (ICCP).

[37]  Raja Giryes,et al.  Class-Aware Fully Convolutional Gaussian and Poisson Denoising , 2018, IEEE Transactions on Image Processing.

[38]  Huchuan Lu,et al.  DeepLens: Shallow Depth Of Field From A Single Image , 2018, ACM Trans. Graph..

[39]  Williem,et al.  Synthesizing a 4D Spatio-Angular Consistent Light Field from a Single Image , 2019, ArXiv.

[40]  Yuxiang Sun,et al.  RTFNet: RGB-Thermal Fusion Network for Semantic Segmentation of Urban Scenes , 2019, IEEE Robotics and Automation Letters.

[41]  You Zhang,et al.  Synthetic defocus and look-ahead autofocus for casual videography , 2019, ACM Trans. Graph..

[42]  In So Kweon,et al.  Depth from a Light Field Image with Learning-Based Matching Costs , 2019, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[43]  Christine Guillemot,et al.  A Fourier Disparity Layer Representation for Light Fields , 2019, IEEE Transactions on Image Processing.

[44]  Aljosa Smolic,et al.  Interactive Light Field Tilt-Shift Refocus with Generalized Shift-and-Sum , 2019, ArXiv.

[45]  Horst Possegger,et al.  FuseSeg: LiDAR Point Cloud Segmentation Fusing Multi-Modal Data , 2019, 2020 IEEE Winter Conference on Applications of Computer Vision (WACV).