Light Field Super-Resolution: A Benchmark

Lenslet-based light field imaging generally suffers from a fundamental trade-off between spatial and angular resolutions, which limits its promotion to practical applications. To this end, a substantial amount of efforts have been dedicated to light field super-resolution (SR) in recent years. Despite the demonstrated success, existing light field SR methods are often evaluated based on different degradation assumptions using different datasets, and even contradictory results are reported in literature. In this paper, we conduct the first systematic benchmark evaluation for representative light field SR methods on both synthetic and real-world datasets with various downsampling kernels and scaling factors. We then analyze and discuss the advantages and limitations of each kind of method from different perspectives. Especially, we find that CNN-based single image SR without using any angular information outperforms most light field SR methods even including learning-based ones. This benchmark evaluation, along with the comprehensive analysis and discussion, sheds light on the future researches in light field SR.

[1]  Alexei A. Efros,et al.  Occlusion-Aware Depth Estimation Using Light-Field Cameras , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[2]  Andrew Lumsdaine,et al.  Superresolution with the focused plenoptic camera , 2011, Electronic Imaging.

[3]  Max Welling,et al.  Semi-Supervised Classification with Graph Convolutional Networks , 2016, ICLR.

[4]  In-So Kweon,et al.  Learning a Deep Convolutional Network for Light-Field Image Super-Resolution , 2015, 2015 IEEE International Conference on Computer Vision Workshop (ICCVW).

[5]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.

[6]  Ravi Ramamoorthi,et al.  A Light Transport Framework for Lenslet Light Field Cameras , 2015, TOGS.

[7]  Tieniu Tan,et al.  LFNet: A Novel Bidirectional Recurrent Convolutional Neural Network for Light-Field Image Super-Resolution , 2018, IEEE Transactions on Image Processing.

[8]  E. Adelson,et al.  The Plenoptic Function and the Elements of Early Vision , 1991 .

[9]  Lennart Wietzke,et al.  Single lens 3D-camera with extended depth-of-field , 2012, Electronic Imaging.

[10]  Ting-Chun Wang,et al.  Learning-based view synthesis for light field cameras , 2016, ACM Trans. Graph..

[11]  Pascal Frossard,et al.  Graph-based light field super-resolution , 2017, 2017 IEEE 19th International Workshop on Multimedia Signal Processing (MMSP).

[12]  Dong Liu,et al.  LF-fusion: Dense and accurate 3D reconstruction from light field images , 2017, 2017 IEEE Visual Communications and Image Processing (VCIP).

[13]  P. Hanrahan,et al.  Light Field Photography with a Hand-held Plenoptic Camera , 2005 .

[14]  Yochai Blau,et al.  The Perception-Distortion Tradeoff , 2017, CVPR.

[15]  Ashok Veeraraghavan,et al.  Light field denoising, light field superresolution and stereo camera based refocussing using a GMM light field patch prior , 2012, 2012 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops.

[16]  Xiaoou Tang,et al.  Learning a Deep Convolutional Network for Image Super-Resolution , 2014, ECCV.

[17]  Yan Yuan,et al.  Light-Field Image Superresolution Using a Combined Deep CNN Based on EPI , 2018, IEEE Signal Processing Letters.

[18]  Qionghai Dai,et al.  Light Field Reconstruction Using Deep Convolutional Network on EPI , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[19]  Ashok Veeraraghavan,et al.  Improving resolution and depth-of-field of light field cameras using a hybrid imaging system , 2014, 2014 IEEE International Conference on Computational Photography (ICCP).

[20]  Dong Liu,et al.  Light field super-resolution using internal and external similarities , 2017, 2017 IEEE International Conference on Image Processing (ICIP).

[21]  Kyoung Mu Lee,et al.  Enhanced Deep Residual Networks for Single Image Super-Resolution , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[22]  Tieniu Tan,et al.  End-to-End View Synthesis for Light Field Imaging with Pseudo 4DCNN , 2018, ECCV.

[23]  Chih-Yuan Yang,et al.  Learning a No-Reference Quality Metric for Single-Image Super-Resolution , 2016, Comput. Vis. Image Underst..

[24]  W. Freeman,et al.  Understanding Camera Trade-Offs through a Bayesian Analysis of Light Field Projections , 2008, ECCV.

[25]  In-So Kweon,et al.  Accurate depth map estimation from a lenslet light field camera , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[26]  Luc Van Gool,et al.  NTIRE 2017 Challenge on Single Image Super-Resolution: Methods and Results , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[27]  Sven Wanner,et al.  Datasets and Benchmarks for Densely Sampled 4D Light Fields , 2013, VMV.

[28]  Stefan B. Williams,et al.  Decoding, Calibration and Rectification for Lenselet-Based Plenoptic Cameras , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[29]  Michal Irani,et al.  Super-resolution from a single image , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[30]  Pascal Frossard,et al.  A Nonsmooth Graph-Based Approach to Light Field Super-Resolution , 2018, 2018 25th IEEE International Conference on Image Processing (ICIP).

[31]  Lei Zhang,et al.  An edge-guided image interpolation algorithm via directional filtering and data fusion , 2006, IEEE Transactions on Image Processing.

[32]  Alexei A. Efros,et al.  The Unreasonable Effectiveness of Deep Features as a Perceptual Metric , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[33]  Touradj Ebrahimi,et al.  New Light Field Image Dataset , 2016, QoMEX 2016.

[34]  Bahadir K. Gunturk,et al.  Spatial and Angular Resolution Enhancement of Light Fields Using Convolutional Neural Networks , 2017, IEEE Transactions on Image Processing.

[35]  J. P. Luke,et al.  Simultaneous estimation of super-resolved depth and all-in-focus images from a plenoptic camera , 2009, 2009 3DTV Conference: The True Vision - Capture, Transmission and Display of 3D Video.

[36]  Aljoscha Smolic,et al.  Light field denoising by sparse 5D transform domain collaborative filtering , 2017, 2017 IEEE 19th International Workshop on Multimedia Signal Processing (MMSP).

[37]  Xiaoming Chen,et al.  Fast Light Field Reconstruction with Deep Coarse-to-Fine Modeling of Spatial-Angular Clues , 2018, ECCV.

[38]  Tom E. Bishop,et al.  The Light Field Camera: Extended Depth of Field, Aliasing, and Superresolution , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[39]  Sven Wanner,et al.  Variational Light Field Analysis for Disparity Estimation and Super-Resolution , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[40]  Yoshua Bengio,et al.  Domain Adaptation for Large-Scale Sentiment Classification: A Deep Learning Approach , 2011, ICML.

[41]  Touradj Ebrahimi,et al.  JPEG Pleno: Toward an Efficient Representation of Visual Reality , 2016, IEEE MultiMedia.

[42]  Saber Farag,et al.  A NOVEL DISPARITY-ASSISTED BLOCK MATCHING-BASED APPROACH FOR SUPER-RESOLUTION OF LIGHT FIELD IMAGES , 2018, 2018 - 3DTV-Conference: The True Vision - Capture, Transmission and Display of 3D Video (3DTV-CON).

[43]  Reuben A. Farrugia,et al.  Super Resolution of Light Field Images Using Linear Subspace Projection of Patch-Volumes , 2017, IEEE Journal of Selected Topics in Signal Processing.

[44]  Lu Fang,et al.  Combining Exemplar-Based Approach and learning-Based Approach for Light Field Super-Resolution Using a Hybrid Imaging System , 2017, 2017 IEEE International Conference on Computer Vision Workshops (ICCVW).

[45]  Ming Yang,et al.  3D Convolutional Neural Networks for Human Action Recognition , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[46]  Seong-Deok Lee,et al.  Improving the spatail resolution based on 4D light field data , 2009, 2009 16th IEEE International Conference on Image Processing (ICIP).

[47]  Alexei A. Efros,et al.  A 4D Light-Field Dataset and CNN Architectures for Material Recognition , 2016, ECCV.

[48]  Aljoscha Smolic,et al.  Light Field Super-Resolution via LFBM5D Sparse Coding , 2018, 2018 25th IEEE International Conference on Image Processing (ICIP).

[49]  Dong Liu,et al.  Two-stage convolutional neural network for light field super-resolution , 2017, 2017 IEEE International Conference on Image Processing (ICIP).

[50]  Rama Chellappa,et al.  Domain adaptation for object recognition: An unsupervised approach , 2011, 2011 International Conference on Computer Vision.

[51]  Dong Liu,et al.  Unsupervised Depth Estimation from Light Field Using a Convolutional Neural Network , 2018, 2018 International Conference on 3D Vision (3DV).

[52]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[53]  Edmond Boyer,et al.  FeaStNet: Feature-Steered Graph Convolutions for 3D Shape Analysis , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[54]  Lu Fang,et al.  Cross-Scale Reference-Based Light Field Super-Resolution , 2018, IEEE Transactions on Computational Imaging.