Perception-Oriented Stereo Image Super-Resolution

Recent studies of deep learning based stereo image super-resolution (StereoSR) have promoted the development of StereoSR. However, existing StereoSR models mainly concentrate on improving quantitative evaluation metrics and neglect the visual quality of super-resolved stereo images. To improve the perceptual performance, this paper proposes the first perception-oriented stereo image super-resolution approach by exploiting the feedback, provided by the evaluation on the perceptual quality of StereoSR results. To provide accurate guidance for the StereoSR model, we develop the first special stereo image super-resolution quality assessment (StereoSRQA) model, and further construct a StereoSRQA database. Extensive experiments demonstrate that our StereoSR approach significantly improves the perceptual quality and enhances the reliability of stereo images for disparity estimation.

[1]  Jeng-Shyang Pan,et al.  No-Reference Image Quality Assessment in Spatial Domain , 2014, ICGEC.

[2]  JiangGangyi,et al.  Perceptual Full-Reference Quality Assessment of Stereoscopic Images by Considering Binocular Visual Characteristics , 2013 .

[3]  Weimin Tan,et al.  Disparity-Aware Domain Adaptation in Stereo Image Restoration , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[4]  Wei Zhou,et al.  Dual-Stream Interactive Networks for No-Reference Stereoscopic Image Quality Assessment , 2019, IEEE Transactions on Image Processing.

[5]  Mingkui Tan,et al.  Closed-Loop Matters: Dual Regression Networks for Single Image Super-Resolution , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[6]  Shahram Izadi,et al.  StereoNet: Guided Hierarchical Refinement for Real-Time Edge-Aware Depth Prediction , 2018, ECCV.

[7]  Joost van de Weijer,et al.  RankIQA: Learning from Rankings for No-Reference Image Quality Assessment , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[8]  Nanfeng Xiao,et al.  Parallax-Based Spatial and Channel Attention for Stereo Image Super-Resolution , 2019, IEEE Access.

[9]  Kyoung Mu Lee,et al.  Accurate Image Super-Resolution Using Very Deep Convolutional Networks , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[10]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[11]  Atsuto Maki,et al.  Towards a simulation driven stereo vision system , 2012, Proceedings of the 21st International Conference on Pattern Recognition (ICPR2012).

[12]  Li Fei-Fei,et al.  Perceptual Losses for Real-Time Style Transfer and Super-Resolution , 2016, ECCV.

[13]  Xinbo Gao,et al.  Fast and Accurate Single Image Super-Resolution via Information Distillation Network , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[14]  Hua Huang,et al.  No-reference image quality assessment based on spatial and spectral entropies , 2014, Signal Process. Image Commun..

[15]  Alan C. Bovik,et al.  Subjective evaluation of stereoscopic image quality , 2013, Signal Process. Image Commun..

[16]  Haoyu Chen,et al.  PIPAL: a Large-Scale Image Quality Assessment Dataset for Perceptual Image Restoration , 2020, ECCV.

[17]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[18]  Jie Zhou,et al.  Structure-Preserving Super Resolution With Gradient Guidance , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[19]  Christian Ledig,et al.  Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[20]  Thomas Brox,et al.  A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimation , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[21]  Yu Qiao,et al.  ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks , 2018, ECCV Workshops.

[22]  Wangmeng Zuo,et al.  Learning a Single Convolutional Super-Resolution Network for Multiple Degradations , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[23]  Do-Kyoung Kwon,et al.  Full-reference quality assessment of stereopairs accounting for rivalry , 2013, Signal Process. Image Commun..

[24]  Alan C. Bovik,et al.  Image information and visual quality , 2004, 2004 IEEE International Conference on Acoustics, Speech, and Signal Processing.

[25]  Alan C. Bovik,et al.  Making a “Completely Blind” Image Quality Analyzer , 2013, IEEE Signal Processing Letters.

[26]  Wei An,et al.  Learning Parallax Attention for Stereo Image Super-Resolution , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[27]  Alan C. Bovik,et al.  No-Reference Image Quality Assessment in the Spatial Domain , 2012, IEEE Transactions on Image Processing.

[28]  Yi Li,et al.  Convolutional Neural Networks for No-Reference Image Quality Assessment , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[29]  Chih-Yuan Yang,et al.  Learning a No-Reference Quality Metric for Single-Image Super-Resolution , 2016, Comput. Vis. Image Underst..

[30]  Andreas Geiger,et al.  Are we ready for autonomous driving? The KITTI vision benchmark suite , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[31]  C.-C. Jay Kuo,et al.  MCL-3D: A Database for Stereoscopic Image Quality Assessment using 2D-Image-Plus-Depth Source , 2014, J. Inf. Sci. Eng..

[32]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.

[33]  Bo Yan,et al.  Deep Objective Quality Assessment Driven Single Image Super-Resolution , 2019, IEEE Transactions on Multimedia.

[34]  Wei Wu,et al.  Feedback Network for Image Super-Resolution , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[35]  Weisi Lin,et al.  Perceptual Full-Reference Quality Assessment of Stereoscopic Images by Considering Binocular Visual Characteristics , 2013, IEEE Transactions on Image Processing.

[36]  Kyoung Mu Lee,et al.  Enhanced Deep Residual Networks for Single Image Super-Resolution , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[37]  Weisi Lin,et al.  Which Has Better Visual Quality: The Clear Blue Sky or a Blurry Animal? , 2019, IEEE Transactions on Multimedia.

[38]  Alan C. Bovik,et al.  Image information and visual quality , 2006, IEEE Trans. Image Process..

[39]  Sumei Li,et al.  Adaptive Cyclopean Image-Based Stereoscopic Image-Quality Assessment Using Ensemble Learning , 2019, IEEE Transactions on Multimedia.

[40]  Seung-Hwan Baek,et al.  Enhancing the Spatial Resolution of Stereo Images Using a Parallax Prior , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[41]  Gustavo de Veciana,et al.  An information fidelity criterion for image quality assessment using natural scene statistics , 2005, IEEE Transactions on Image Processing.

[42]  Guangming Shi,et al.  End-to-End Blind Image Quality Prediction With Cascaded Deep Neural Network , 2020, IEEE Transactions on Image Processing.

[43]  Xiaoou Tang,et al.  Image Super-Resolution Using Deep Convolutional Networks , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.