Spatial-Angular Interaction for Light Field Image Super-Resolution

Light field (LF) cameras record both intensity and directions of light rays, and capture scenes from a number of viewpoints. Both information within each perspective (i.e., spatial information) and among different perspectives (i.e., angular information) is beneficial to image super-resolution (SR). In this paper, we propose a spatial-angular interactive network (namely, LF-InterNet) for LF image SR. Specifically, spatial and angular features are first separately extracted from input LFs, and then repetitively interacted to progressively incorporate spatial and angular information. Finally, the interacted features are fused to superresolve each sub-aperture image. Experimental results demonstrate the superiority of LF-InterNet over the state-of-the-art methods, i.e., our method can achieve high PSNR and SSIM scores with low computational cost, and recover faithful details in the reconstructed images.

[1]  Pascal Frossard,et al.  Geometry-Consistent Light Field Super-Resolution via Graph-Based Regularization , 2017, IEEE Transactions on Image Processing.

[2]  Bastian Goldlücke,et al.  A Dataset and Evaluation Methodology for Depth Estimation on 4D Light Fields , 2016, ACCV.

[3]  Liang Wang,et al.  Bidirectional Recurrent Convolutional Networks for Multi-Frame Super-Resolution , 2015, NIPS.

[4]  Kilian Q. Weinberger,et al.  Densely Connected Convolutional Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[5]  In-So Kweon,et al.  EPINET: A Fully-Convolutional Neural Network Using Epipolar Geometry for Depth from Light Field Images , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[6]  Christine Guillemot,et al.  Light Field Inpainting Propagation via Low Rank Matrix Completion , 2018, IEEE Transactions on Image Processing.

[7]  Panos M. Pardalos,et al.  Machine Learning Methods for Data Association in Multi-Object Tracking , 2020, ACM Comput. Surv..

[8]  Qionghai Dai,et al.  Learning Sheared EPI Structure for Light Field Reconstruction , 2019, IEEE Transactions on Image Processing.

[9]  In-So Kweon,et al.  Learning a Deep Convolutional Network for Light-Field Image Super-Resolution , 2015, 2015 IEEE International Conference on Computer Vision Workshop (ICCVW).

[10]  Alexei A. Efros,et al.  A 4D Light-Field Dataset and CNN Architectures for Material Recognition , 2016, ECCV.

[11]  Nick Barnes,et al.  A Deep Journey into Super-resolution , 2019, ACM Comput. Surv..

[12]  Luc Van Gool,et al.  Anchored Neighborhood Regression for Fast Example-Based Super-Resolution , 2013, 2013 IEEE International Conference on Computer Vision.

[13]  Shree K. Nayar,et al.  PiCam , 2013, ACM Trans. Graph..

[14]  Haibin Ling,et al.  Saliency Detection on Light Field , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[15]  Wei Wang,et al.  Deep Learning for Single Image Super-Resolution: A Brief Review , 2018, IEEE Transactions on Multimedia.

[16]  Yulan Guo,et al.  DeOccNet: Learning to See Through Foreground Occlusions in Light Fields , 2019, 2020 IEEE Winter Conference on Applications of Computer Vision (WACV).

[17]  Xiaoou Tang,et al.  Learning a Deep Convolutional Network for Image Super-Resolution , 2014, ECCV.

[18]  Abhishek Das,et al.  Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization , 2016, 2017 IEEE International Conference on Computer Vision (ICCV).

[19]  Reuben A. Farrugia,et al.  Super Resolution of Light Field Images Using Linear Subspace Projection of Patch-Volumes , 2017, IEEE Journal of Selected Topics in Signal Processing.

[20]  Michael Elad,et al.  On Single Image Scale-Up Using Sparse-Representations , 2010, Curves and Surfaces.

[21]  Wei An,et al.  Selective Light Field Refocusing for Camera Arrays Using Bokeh Rendering and Superresolution , 2019, IEEE Signal Processing Letters.

[22]  In Kyu Park,et al.  Robust Light Field Depth Estimation Using Occlusion-Noise Aware Data Costs , 2018, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[23]  Qionghai Dai,et al.  Light Field Reconstruction Using Deep Convolutional Network on EPI , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[24]  Jingjing Li,et al.  Memory-oriented Decoder for Light Field Salient Object Detection , 2019, NeurIPS.

[25]  Yan Yuan,et al.  Light-Field Image Superresolution Using a Combined Deep CNN Based on EPI , 2018, IEEE Signal Processing Letters.

[26]  Sam Kwong,et al.  Learning Light Field Angular Super-Resolution via a Geometry-Aware Network , 2020, AAAI.

[27]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[28]  Sahin Isik,et al.  SRLibrary: Comparing different loss functions for super-resolution over various convolutional architectures , 2019, J. Vis. Commun. Image Represent..

[29]  Shu-Tao Xia,et al.  Second-Order Attention Network for Single Image Super-Resolution , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[30]  Thomas S. Huang,et al.  Image Super-Resolution Via Sparse Representation , 2010, IEEE Transactions on Image Processing.

[31]  Zhibo Chen,et al.  Light Field Spatial Super-Resolution Using Deep Efficient Spatial-Angular Separable Convolution , 2019, IEEE Transactions on Image Processing.

[32]  Xu-dong Zhang,et al.  Light-field Image Super-resolution Using Convolutional Neural Network , 2017 .

[33]  Yoshua Bengio,et al.  Understanding the difficulty of training deep feedforward neural networks , 2010, AISTATS.

[34]  Karen O. Egiazarian,et al.  Single image super-resolution via BM3D sparse coding , 2015, 2015 23rd European Signal Processing Conference (EUSIPCO).

[35]  Touradj Ebrahimi,et al.  New Light Field Image Dataset , 2016, QoMEX 2016.

[36]  Hongdong Li,et al.  Revisiting Spatio-Angular Trade-off in Light Field Cameras and Extended Applications in Super-Resolution , 2019, IEEE Transactions on Visualization and Computer Graphics.

[37]  Jing Jin,et al.  Light Field Spatial Super-Resolution via Deep Combinatorial Geometry Embedding and Structural Consistency Regularization , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[38]  Marc Levoy,et al.  Light field rendering , 1996, SIGGRAPH.

[39]  Steven C. H. Hoi,et al.  Deep Learning for Image Super-Resolution: A Survey , 2019, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[40]  Marc Levoy,et al.  High performance imaging using large camera arrays , 2005, ACM Trans. Graph..

[41]  Yun Fu,et al.  Image Super-Resolution Using Very Deep Residual Channel Attention Networks , 2018, ECCV.

[42]  Kyoung Mu Lee,et al.  Accurate Image Super-Resolution Using Very Deep Convolutional Networks , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[43]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[44]  Edmund Y. Lam,et al.  High-Order Residual Network for Light Field Super-Resolution , 2020, AAAI.

[45]  Hao Sheng,et al.  Residual Networks for Light Field Image Super-Resolution , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[46]  Tom E. Bishop,et al.  The Light Field Camera: Extended Depth of Field, Aliasing, and Superresolution , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[47]  Sven Wanner,et al.  Variational Light Field Analysis for Disparity Estimation and Super-Resolution , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[48]  Christian Ledig,et al.  Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[49]  Christine Guillemot,et al.  Learning Fused Pixel and Feature-Based View Reconstructions for Light Fields , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[50]  Daniel Rueckert,et al.  Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[51]  Enhua Wu,et al.  Squeeze-and-Excitation Networks , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[52]  Sven Wanner,et al.  Datasets and Benchmarks for Densely Sampled 4D Light Fields , 2013, VMV.

[53]  Aljoscha Smolic,et al.  Light Field Super-Resolution via LFBM5D Sparse Coding , 2018, 2018 25th IEEE International Conference on Image Processing (ICIP).

[54]  KhanSalman,et al.  A Deep Journey into Super-resolution: A Survey , 2020 .

[55]  Tiantian Wang,et al.  Deep Learning for Light Field Saliency Detection , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[56]  Edmund Y. Lam,et al.  High-Dimensional Dense Residual Convolutional Neural Network for Light Field Reconstruction , 2019, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[57]  Tieniu Tan,et al.  LFNet: A Novel Bidirectional Recurrent Convolutional Neural Network for Light-Field Image Super-Resolution , 2018, IEEE Transactions on Image Processing.

[58]  Yun Fu,et al.  Residual Dense Network for Image Super-Resolution , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[59]  Kyoung Mu Lee,et al.  Enhanced Deep Residual Networks for Single Image Super-Resolution , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[60]  Yu Qiao,et al.  ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks , 2018, ECCV Workshops.

[61]  Luc Van Gool,et al.  NTIRE 2017 Challenge on Single Image Super-Resolution: Methods and Results , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).