The Accurate Estimation of Disparity Maps from Cross-Scale Reference-Based Light Field

This paper addresses the problem of disparity map accurate estimation in the cross-scale reference-based light field, which consists several low-quality images arranged around one central high-resolution (HR) image. In the framework, we use a HR image-guidance CNN (HRIG-CNN) for estimating the disparity map in the HR level. Specifically, we first calculate the coarse disparity map using our cross-pattern strategy, which can blend the multiple disparity maps. And then, we refine this coarse disparity map using HRIG-CNN for obtaining high-quality disparity map, which contains detail information and preserve edge information. With the HR image guidance, our HRIG-CNN achieves state-of-the-art for obtaining disparity map in such hybrid light field condition. In the end, we provide both quantitative and qualitative evaluations on different methods, and demonstrate the high performance and robustness of the proposed framework compared with the state-of-the-arts algorithms.

[1]  G. Lippmann.  Epreuves reversibles donnant la sensation du relief , 1908 .

[2]  Jian Sun,et al.  Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[3]  Ce Liu,et al.  Exploring new representations and applications for motion analysis , 2009 .

[4]  Marc Levoy,et al.  Light field rendering , 1996, SIGGRAPH.

[5]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[6]  Sven Wanner,et al.  Datasets and Benchmarks for Densely Sampled 4D Light Fields , 2013, VMV.

[7]  P. Hanrahan,et al.  Digital light field photography , 2006 .

[8]  M. Levoy,et al.  The light field , 1939 .

[9]  Trevor Darrell,et al.  Caffe: Convolutional Architecture for Fast Feature Embedding , 2014, ACM Multimedia.

[10]  Kyoung Mu Lee,et al.  Accurate Image Super-Resolution Using Very Deep Convolutional Networks , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[11]  Tom E. Bishop,et al.  Plenoptic depth estimation from multiple aliased views , 2009, 2009 IEEE 12th International Conference on Computer Vision Workshops, ICCV Workshops.

[12]  Alexei A. Efros,et al.  Light field video capture using a learning-based hybrid imaging system , 2017, ACM Trans. Graph..

[13]  Mandan Zhao,et al.  Light Field Super-Resolution Using Cross-Resolution Input Based on PatchMatch and Learning Method , 2017, CCCV.

[14]  Zhan Yu,et al.  Line Assisted Light Field Triangulation and Stereo Matching , 2013, 2013 IEEE International Conference on Computer Vision.

[15]  Rob Fergus,et al.  Depth Map Prediction from a Single Image using a Multi-Scale Deep Network , 2014, NIPS.

[16]  Robert C. Bolles,et al.  Epipolar-plane image analysis: An approach to determining structure from motion , 1987, International Journal of Computer Vision.

[17]  Richard Szeliski,et al.  Extracting layers and analyzing their specular properties using epipolar-plane-image analysis , 2005, Comput. Vis. Image Underst..

[18]  Jitendra Malik,et al.  Depth from Combining Defocus and Correspondence Using Light-Field Cameras , 2013, 2013 IEEE International Conference on Computer Vision.

[19]  Marc Levoy,et al.  High performance imaging using large camera arrays , 2005, SIGGRAPH 2005.

[20]  Xiaoou Tang,et al.  Learning a Deep Convolutional Network for Image Super-Resolution , 2014, ECCV.

[21]  Tom E. Bishop,et al.  Full-Resolution Depth Map Estimation from an Aliased Plenoptic Light Field , 2010, ACCV.

[22]  In-So Kweon,et al.  Accurate depth map estimation from a lenslet light field camera , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[23]  Tom E. Bishop,et al.  The Light Field Camera: Extended Depth of Field, Aliasing, and Superresolution , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[24]  Zhengyou Zhang,et al.  Virtual View Generation with a Hybrid Camera Array , 2009 .

[25]  Lennart Wietzke,et al.  Single lens 3D-camera with extended depth-of-field , 2012, Electronic Imaging.

[26]  Shree K. Nayar,et al.  Motion deblurring using hybrid imaging , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[27]  Stephen Lin,et al.  High resolution multispectral video capture with a hybrid camera system , 2011, CVPR 2011.

[28]  Scott McCloskey.  Masking Light Fields to Remove Partial Occlusion , 2014, 2014 22nd International Conference on Pattern Recognition.