No-reference stereoscopic image quality assessment using 3D visual saliency maps fused with three-channel convolutional neural network

In this paper, we present a depth-perceived 3D visual saliency map and propose a no-reference stereoscopic image quality assessment (NR SIQA) algorithm using 3D visual saliency maps and convolutional neural network (CNN). Firstly, the 2D salient region of stereoscopic image is generated, and the depth saliency map is calculated, and then, they are combined to compute 3D visual saliency map by linear weighted method, which can better use depth and disparity information of 3D image. Finally, 3D visual saliency map, together with distorted stereoscopic pairs, is fed into a three-channel CNN to learn human subjective perception. We call proposed depth perception and CNN-based SIQA method DPCNN. The performances of DPCNN are evaluated over the popular LIVE 3D Phase I and LIVE 3D Phase II databases, which demonstrates to be competitive with the state-of-the-art NR SIQA algorithms.

[1]  Nitish Srivastava,et al.  Improving neural networks by preventing co-adaptation of feature detectors , 2012, ArXiv.

[2]  Ying Wu,et al.  A unified approach to salient object detection via low rank matrix recovery , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[3]  Alan C. Bovik,et al.  No-Reference Quality Assessment of Natural Stereopairs , 2013, IEEE Transactions on Image Processing.

[4]  Geoffrey E. Hinton,et al.  Rectified Linear Units Improve Restricted Boltzmann Machines , 2010, ICML.

[5]  Weisi Lin,et al.  Saliency detection for stereoscopic images , 2013, 2013 Visual Communications and Image Processing (VCIP).

[6]  Sebastian Bosse,et al.  A deep neural network for image quality assessment , 2016, 2016 IEEE International Conference on Image Processing (ICIP).

[7]  Michael J. Black,et al.  Secrets of optical flow estimation and their principles , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[8]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.

[9]  Yael Pritch,et al.  Saliency filters: Contrast based filtering for salient region detection , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[10]  Bin Jiang,et al.  No-reference stereoscopic image quality assessment based on hue summation-difference mapping image and binocular joint mutual filtering. , 2018, Applied optics.

[11]  Lin Ma,et al.  Multimodal learning for facial expression recognition , 2015, Pattern Recognit..

[12]  Nianyi Li,et al.  A weighted sparse coding framework for saliency detection , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[13]  Soo-Chang Pei,et al.  Blind Stereoscopic Image Quality Assessment Based on Hierarchical Learning , 2019, IEEE Access.

[14]  Ting Luo,et al.  Blind Binocular Visual Quality Predictor Using Deep Fusion Network , 2020, IEEE Transactions on Computational Imaging.

[15]  Jun Du,et al.  An Experimental Study on Speech Enhancement Based on Deep Neural Networks , 2014, IEEE Signal Processing Letters.

[16]  Ting Luo,et al.  Deep Binocular Fixation Prediction using a Hierarchical Multimodal Fusion Network , 2021 .

[17]  Zhihan Lv,et al.  Stereoscopic image quality assessment method based on binocular combination saliency model , 2016, Signal Process..

[18]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[19]  Sanghoon Lee,et al.  Blind Deep S3D Image Quality Evaluation via Local to Global Feature Aggregation , 2017, IEEE Transactions on Image Processing.

[20]  Kwan-Liu Ma,et al.  Stereoscopic Thumbnail Creation via Efficient Stereo Saliency Detection , 2017, IEEE Transactions on Visualization and Computer Graphics.

[21]  Wei Zhang,et al.  The Application of Visual Saliency Models in Objective Image Quality Assessment: A Statistical Evaluation , 2016, IEEE Transactions on Neural Networks and Learning Systems.

[22]  S. Süsstrunk,et al.  Frequency-tuned salient region detection , 2009, CVPR 2009.

[23]  Lin Ma,et al.  Learning structure of stereoscopic image for no-reference quality assessment with convolutional neural network , 2016, Pattern Recognit..

[24]  Yang Zhao,et al.  Blind assessment for stereo images considering binocular characteristics and deep perception map based on deep belief network , 2019, Inf. Sci..

[25]  Qionghai Dai,et al.  Learning Sparse Representation for No-Reference Quality Assessment of Multiply Distorted Stereoscopic Images , 2017, IEEE Transactions on Multimedia.

[26]  Roushain Akhter,et al.  No-reference stereoscopic image quality assessment , 2010, Electronic Imaging.

[27]  Hongyu Li,et al.  SDSP: A novel saliency detection method by combining simple priors , 2013, 2013 IEEE International Conference on Image Processing.

[28]  Yu Zhou,et al.  Quaternion representation based visual saliency for stereoscopic image quality assessment , 2018, Signal Process..

[29]  Kai Li,et al.  No-reference stereo image quality assessment by learning gradient dictionary-based color visual characteristics , 2018, 2018 IEEE International Symposium on Circuits and Systems (ISCAS).

[30]  Alan C. Bovik,et al.  Subjective evaluation of stereoscopic image quality , 2013, Signal Process. Image Commun..

[31]  Peter König,et al.  Influence of disparity on fixation and saccades in free viewing of natural scenes. , 2009, Journal of vision.

[32]  Wujie Zhou,et al.  Salient Object Detection in Stereoscopic 3D Images Using a Deep Convolutional Residual Autoencoder , 2020, IEEE Transactions on Multimedia.

[33]  Heeseok Oh,et al.  Blind Deep S3D Image Quality Evaluation via Local to Global Feature Aggregation. , 2017, IEEE transactions on image processing : a publication of the IEEE Signal Processing Society.

[34]  Min Gao,et al.  No-Reference Stereoscopic Image Quality Assessment Based on Visual Attention and Perception , 2019, IEEE Access.

[35]  A. Mizuno,et al.  A change of the leading player in flow Visualization technique , 2006, J. Vis..

[36]  Yi Li,et al.  Convolutional Neural Networks for No-Reference Image Quality Assessment , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[37]  Ting Luo,et al.  TSNet: Three-Stream Self-Attention Network for RGB-D Indoor Semantic Segmentation , 2020, IEEE Intelligent Systems.

[38]  Frédo Durand,et al.  Learning to predict where humans look , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[39]  Patrick Le Callet,et al.  Quality Assessment of Stereoscopic Images , 2008, EURASIP J. Image Video Process..

[40]  Jari Takatalo,et al.  What do people look at when they watch stereoscopic movies? , 2010, Electronic Imaging.

[41]  Sumohana S. Channappayya,et al.  No-reference Stereoscopic Image Quality Assessment Using Natural Scene Statistics , 2016, Signal Process. Image Commun..