Siamese-Network-Based Learning to Rank for No-Reference 2D and 3D Image Quality Assessment

2D image quality assessment (IQA) and stereoscopic 3D IQA are considered as two different tasks in the literature. In this paper, we present an index for both no-reference 2D and 3D IQA. We propose to transform the IQA task into a task of quality comparison between images. By generating image pairs, the amount of training data reaches the square of the original amount of data, effectively solving the lacking of training samples. We also propose a learning to rank model using Siamese convolutional neural networks (LRSN) for quality comparison. The presented LRSN has two branches that have the same structure, share weights with each other, and take two image patches as inputs. The goal of LRSN is learning to rank the quality scores between the two input image patches. The relative quality score of a test image is obtained by first comparing its image patches with many image patches of other images and counts the number of times that its image patches are ranked superior to other patches. The experimental results on three 2D (LIVE, CSIQ, and TID2013) and three 3D (LIVE 3D Phase-I, LIVE 3D Phase-II, and NBU) IQA databases demonstrate that the proposed LRSN model works well for both 2D and 3D IQA and outperforms the state-of-the-art no-reference 2D and 3D IQA metrics.

[1]  Sebastian Bosse,et al.  Deep Neural Networks for No-Reference and Full-Reference Image Quality Assessment , 2016, IEEE Transactions on Image Processing.

[2]  Tongliang Liu,et al.  dipIQ: Blind Image Quality Assessment by Learning-to-Rank Discriminable Image Pairs. , 2017, IEEE transactions on image processing : a publication of the IEEE Signal Processing Society.

[3]  Weisi Lin,et al.  A Fast Reliable Image Quality Predictor by Fusing Micro- and Macro-Structures , 2017, IEEE Transactions on Industrial Electronics.

[4]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[5]  Ting Luo,et al.  Blind quality estimator for 3D images based on binocular combination and extreme learning machine , 2017, Pattern Recognit..

[6]  Weisi Lin,et al.  Analysis of Distortion Distribution for Pooling in Image Quality Prediction , 2016, IEEE Transactions on Broadcasting.

[7]  Weisi Lin,et al.  Perceptual Full-Reference Quality Assessment of Stereoscopic Images by Considering Binocular Visual Characteristics , 2013, IEEE Transactions on Image Processing.

[8]  Zhuowen Tu,et al.  Deeply Supervised Salient Object Detection with Short Connections , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[9]  David Zhang,et al.  FSIM: A Feature Similarity Index for Image Quality Assessment , 2011, IEEE Transactions on Image Processing.

[10]  Sanghoon Lee,et al.  Fully Deep Blind Image Quality Predictor , 2017, IEEE Journal of Selected Topics in Signal Processing.

[11]  Lei Zhang,et al.  A Probabilistic Quality Representation Approach to Deep Blind Image Quality Prediction , 2017, ArXiv.

[12]  Lin Ma,et al.  Learning structure of stereoscopic image for no-reference quality assessment with convolutional neural network , 2016, Pattern Recognit..

[13]  Christophe Charrier,et al.  Blind Image Quality Assessment: A Natural Scene Statistics Approach in the DCT Domain , 2012, IEEE Transactions on Image Processing.

[14]  Yan Yang,et al.  Region-Aware Image Denoising by Exploring Parameter Preference , 2018, IEEE Transactions on Circuits and Systems for Video Technology.

[15]  Gangyi Jiang,et al.  Toward Domain Transfer for No-Reference Quality Prediction of Asymmetrically Distorted Stereoscopic Images , 2018, IEEE Transactions on Circuits and Systems for Video Technology.

[16]  Alan C. Bovik,et al.  No-Reference Quality Assessment of Natural Stereopairs , 2013, IEEE Transactions on Image Processing.

[17]  Alan C. Bovik,et al.  Making a “Completely Blind” Image Quality Analyzer , 2013, IEEE Signal Processing Letters.

[18]  Do-Kyoung Kwon,et al.  Full-reference quality assessment of stereopairs accounting for rivalry , 2013, Signal Process. Image Commun..

[19]  Qionghai Dai,et al.  Learning Blind Quality Evaluator for Stereoscopic Images Using Joint Sparse Representation , 2016, IEEE Transactions on Multimedia.

[20]  Qionghai Dai,et al.  Full-Reference Quality Assessment of Stereoscopic Images by Learning Binocular Receptive Field Properties , 2015, IEEE Transactions on Image Processing.

[21]  Nikolay N. Ponomarenko,et al.  Color image database TID2013: Peculiarities and preliminary results , 2013, European Workshop on Visual Information Processing (EUVIP).

[22]  Eric C. Larson,et al.  Most apparent distortion: full-reference image quality assessment and the role of strategy , 2010, J. Electronic Imaging.

[23]  Chunping Hou,et al.  No-reference stereoscopic 3D image quality assessment via combined model , 2018, Multimedia Tools and Applications.

[24]  David S. Doermann,et al.  Unsupervised feature learning framework for no-reference image quality assessment , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[25]  Leida Li,et al.  No Reference Quality Assessment for Screen Content Images With Both Local and Global Feature Representation , 2018, IEEE Transactions on Image Processing.

[26]  Patrick Le Callet,et al.  Quality Assessment of Stereoscopic Images , 2008, EURASIP J. Image Video Process..

[27]  Mohamed-Chaker Larabi,et al.  A perceptual metric for stereoscopic image quality assessment based on the binocular energy , 2013, Multidimens. Syst. Signal Process..

[28]  Alan C. Bovik,et al.  Subjective evaluation of stereoscopic image quality , 2013, Signal Process. Image Commun..

[29]  Alan C. Bovik,et al.  No-Reference Image Quality Assessment in the Spatial Domain , 2012, IEEE Transactions on Image Processing.

[30]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.

[31]  Lei Zhang,et al.  A Feature-Enriched Completely Blind Image Quality Evaluator , 2015, IEEE Transactions on Image Processing.

[32]  Wenzhong Guo,et al.  2D and 3D Image Quality Assessment: A Survey of Metrics and Challenges , 2019, IEEE Access.

[33]  Lei Zhang,et al.  Gradient Magnitude Similarity Deviation: A Highly Efficient Perceptual Image Quality Index , 2013, IEEE Transactions on Image Processing.

[34]  Alan C. Bovik,et al.  Blind Image Quality Assessment: From Natural Scene Statistics to Perceptual Quality , 2011, IEEE Transactions on Image Processing.

[35]  Weisi Lin,et al.  Learning a referenceless stereopair quality engine with deep nonnegativity constrained sparse autoencoder , 2018, Pattern Recognit..

[36]  Alan C. Bovik,et al.  A Statistical Evaluation of Recent Full Reference Image Quality Assessment Algorithms , 2006, IEEE Transactions on Image Processing.

[37]  Weisi Lin,et al.  Pairwise comparison and rank learning for image quality assessment , 2016, Displays.

[38]  Rongrong Ji,et al.  Image Quality Assessment for Color Correction Based on Color Contrast Similarity and Color Value Difference , 2018, IEEE Transactions on Circuits and Systems for Video Technology.

[39]  David S. Doermann,et al.  Beyond Human Opinion Scores: Blind Image Quality Assessment Based on Synthetic Scores , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[40]  Kai Zeng,et al.  Quality Prediction of Asymmetrically Distorted Stereoscopic 3D Images , 2015, IEEE Transactions on Image Processing.

[41]  Yi Li,et al.  Convolutional Neural Networks for No-Reference Image Quality Assessment , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[42]  Yair Moshe,et al.  Image quality assessment based on DCT subband similarity , 2015, 2015 IEEE International Conference on Image Processing (ICIP).

[43]  Zhengfang Duanmu,et al.  End-to-End Blind Image Quality Assessment Using Deep Neural Networks , 2018, IEEE Transactions on Image Processing.

[44]  Lei Zhang,et al.  Learning without Human Scores for Blind Image Quality Assessment , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.