Spatial and Angular Reconstruction of Light Field Based on Deep Generative Networks

Light field (LF) cameras often have significant limitations in spatial and angular resolutions due to their design. Many techniques that attempt to reconstruct LF images at a higher resolution only consider either spatial or angular resolution, but not both. We propose a generative network using high-dimensional convolution to improve both aspects. Our experimental results on both synthetic and real-world data demonstrate that the proposed model outperforms existing state-of-the-art methods in terms of both peak signal-to-noise ratio (PSNR) and visual quality. The proposed method can also generate more realistic spatial details with better fidelity.

[1]  Mike Brookes,et al.  Plenoptic Layer-Based Modeling for Image Based Rendering , 2013, IEEE Transactions on Image Processing.

[2]  Ting-Chun Wang,et al.  Learning-based view synthesis for light field cameras , 2016, ACM Trans. Graph..

[3]  E. Lam,et al.  Light Field Superresolution Reconstruction in Computational Photography , 2011 .

[4]  Tieniu Tan,et al.  LFNet: A Novel Bidirectional Recurrent Convolutional Neural Network for Light-Field Image Super-Resolution , 2018, IEEE Transactions on Image Processing.

[5]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[6]  P. Hanrahan,et al.  Light Field Photography with a Hand-held Plenoptic Camera , 2005 .

[7]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[8]  Edmund Y. Lam,et al.  End-to-end deep learning framework for digital holographic reconstruction , 2019, Advanced Photonics.

[9]  Narendra Ahuja,et al.  Fast and Accurate Image Super-Resolution with Deep Laplacian Pyramid Networks , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[10]  Bahadir K. Gunturk,et al.  Spatial and Angular Resolution Enhancement of Light Fields Using Convolutional Neural Networks , 2017, IEEE Transactions on Image Processing.

[11]  Reuben A. Farrugia,et al.  Super Resolution of Light Field Images Using Linear Subspace Projection of Patch-Volumes , 2017, IEEE Journal of Selected Topics in Signal Processing.

[12]  In-So Kweon,et al.  Light-Field Image Super-Resolution Using Convolutional Neural Network , 2017, IEEE Signal Processing Letters.

[13]  Edmund Y. Lam,et al.  Computational Light Field Generation Using Deep Nonparametric Bayesian Learning , 2019, IEEE Access.

[14]  Edmund Y Lam,et al.  Computational photography with plenoptic camera and light field capture: tutorial. , 2015, Journal of the Optical Society of America. A, Optics, image science, and vision.

[15]  Sven Wanner,et al.  Datasets and Benchmarks for Densely Sampled 4D Light Fields , 2013, VMV.

[16]  Qionghai Dai,et al.  Light Field Reconstruction Using Deep Convolutional Network on EPI , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[17]  Yun Fu,et al.  Residual Dense Network for Image Restoration , 2018, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[18]  Joachim Keinert,et al.  Acquisition system for dense lightfield of large scenes , 2017, 2017 3DTV Conference: The True Vision - Capture, Transmission and Display of 3D Video (3DTV-CON).

[19]  E. Boyden,et al.  Simultaneous whole-animal 3D-imaging of neuronal activity using light-field microscopy , 2014, Nature Methods.

[20]  Edmund Y. Lam,et al.  Learning-based nonparametric autofocusing for digital holography , 2018 .

[21]  Edmund Y. Lam,et al.  A Spatial Projection Analysis of Light Field Capture , 2010 .

[22]  Edmund Y. Lam,et al.  Data-driven light field depth estimation using deep Convolutional Neural Networks , 2016, 2016 International Joint Conference on Neural Networks (IJCNN).

[23]  Sven Wanner,et al.  Variational Light Field Analysis for Disparity Estimation and Super-Resolution , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[24]  Edmund Y. Lam,et al.  High-Dimensional Dense Residual Convolutional Neural Network for Light Field Reconstruction , 2019, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[25]  Christian Ledig,et al.  Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[26]  Ashok Veeraraghavan,et al.  Light field denoising, light field superresolution and stereo camera based refocussing using a GMM light field patch prior , 2012, 2012 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops.