Blind quality estimator for 3D images based on binocular combination and extreme learning machine

Abstract Over recent years, blind quality estimators for three-dimensional (3D) images have received increasing attention. In this paper, we describe a blind quality estimator for 3D images based on binocular combination and an extreme learning machine (ELM) for more accurate alignment with subjective human experience. First, two binocular combinations of stimuli are generated using different combination strategies. Various binocular quality-aware features of these combinations are then extracted by local binary pattern operators, which give an effective description of the degradation pattern. Finally, these features are mapped to the subjective quality score of the distorted 3D image using an ELM. Experimental results using three benchmark databases confirm that the proposed metric is effective and achieves competitive prediction performance when compared with most current full reference and blind metrics.

[1]  Weisi Lin,et al.  Saliency detection for stereoscopic images , 2013, 2013 Visual Communications and Image Processing (VCIP).

[2]  Lu Yu,et al.  Perceptual quality measurement of 3D images based on binocular vision. , 2015, Applied optics.

[3]  Alan C. Bovik,et al.  Subjective evaluation of stereoscopic image quality , 2013, Signal Process. Image Commun..

[4]  Alan C. Bovik,et al.  No-Reference Image Quality Assessment in the Spatial Domain , 2012, IEEE Transactions on Image Processing.

[5]  Yong Yang,et al.  Leukocyte image segmentation by visual attention and extreme learning machine , 2011, Neural Computing and Applications.

[6]  Octavia A. Dobre,et al.  Fold-based Kolmogorov–Smirnov Modulation Classifier , 2016, IEEE Signal Processing Letters.

[7]  Ting Luo,et al.  Utilizing binocular vision to facilitate completely blind 3D image quality measurement , 2016, Signal Process..

[8]  Chee Kheong Siew,et al.  Extreme learning machine: Theory and applications , 2006, Neurocomputing.

[9]  Weisi Lin,et al.  Fourier Transform-Based Scalable Image Quality Measure , 2012, IEEE Transactions on Image Processing.

[10]  D. Simmons The Binocular Combination of Chromatic Contrast , 2005, Perception.

[11]  Robert F Hess,et al.  When stereopsis does not improve with increasing contrast 1 Portions of this work were presented at The European Conference on Visual Perception, Helsinki, 1997. 1 , 1998, Vision Research.

[12]  Chang-Su Kim,et al.  FDQM: Fast Quality Metric for Depth Maps Without View Synthesis , 2015, IEEE Transactions on Circuits and Systems for Video Technology.

[13]  A. E. Hoerl,et al.  Ridge regression: biased estimation for nonorthogonal problems , 2000 .

[14]  Baojun Zhao,et al.  NMF-Based Image Quality Assessment Using Extreme Learning Machine , 2017, IEEE Transactions on Cybernetics.

[15]  Hongming Zhou,et al.  Extreme Learning Machine for Regression and Multiclass Classification , 2012, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[16]  Peter L. Bartlett,et al.  The Sample Complexity of Pattern Classification with Neural Networks: The Size of the Weights is More Important than the Size of the Network , 1998, IEEE Trans. Inf. Theory.

[17]  Kwanghyun Lee,et al.  3D Perception Based Quality Pooling: Stereopsis, Binocular Rivalry, and Binocular Suppression , 2015, IEEE Journal of Selected Topics in Signal Processing.

[18]  Lei Zhang,et al.  Blind Image Quality Assessment Using Joint Statistics of Gradient Magnitude and Laplacian Features , 2014, IEEE Transactions on Image Processing.

[19]  Jiawei Zhou,et al.  Linear binocular combination of responses to contrast modulation: contrast-weighted summation in first- and second-order vision. , 2014, Journal of vision.

[20]  Ja-Ling Wu,et al.  Quality Assessment of Stereoscopic 3D Image Compression by Binocular Integration Behaviors , 2014, IEEE Transactions on Image Processing.

[21]  Lijuan Duan,et al.  Extreme Learning Machine with Gaussian Kernel Based Relevance Feedback Scheme for Image Retrieval , 2016 .

[22]  Jason Jianjun Gu,et al.  Learning to Predict High-Quality Edge Maps for Room Layout Estimation , 2017, IEEE Transactions on Multimedia.

[23]  Kwanghoon Sohn,et al.  No-Reference Quality Assessment for Stereoscopic Images Based on Binocular Quality Perception , 2014, IEEE Transactions on Circuits and Systems for Video Technology.

[24]  Behzad Mansouri,et al.  Binocular summation of contrast remains intact in strabismic amblyopia. , 2007, Investigative ophthalmology & visual science.

[25]  Do-Kyoung Kwon,et al.  Full-reference quality assessment of stereopairs accounting for rivalry , 2013, Signal Process. Image Commun..

[26]  Ke Gu,et al.  Learning a No-Reference Quality Assessment Model of Enhanced Images With Big Data , 2018, IEEE Transactions on Neural Networks and Learning Systems.

[27]  Alan C. Bovik,et al.  Making a “Completely Blind” Image Quality Analyzer , 2013, IEEE Signal Processing Letters.

[28]  David J. Fleet,et al.  Neural encoding of binocular disparity: Energy models, position shifts and phase shifts , 1996, Vision Research.

[29]  Matti Pietikäinen,et al.  Multiresolution Gray-Scale and Rotation Invariant Texture Classification with Local Binary Patterns , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[30]  Kwanghoon Sohn,et al.  Visual Comfort Enhancement for Stereoscopic Video Based on Binocular Fusion Characteristics , 2013, IEEE Transactions on Circuits and Systems for Video Technology.

[31]  Mounir Kaaniche,et al.  Efficient Inter-View Bit Allocation Methods for Stereo Image Coding , 2015, IEEE Transactions on Multimedia.

[32]  K. Mullen,et al.  Cross-orientation masking in human color vision: application of a two-stage model to assess dichoptic and monocular sources of suppression. , 2013, Journal of vision.

[33]  Jiawei Zhou,et al.  Stereopsis and mean luminance. , 2013, Journal of vision.

[34]  Chaminda T. E. R. Hewage,et al.  Reduced-reference quality assessment for 3D video compression and transmission , 2011, IEEE Transactions on Consumer Electronics.

[35]  Joydeep Ghosh,et al.  Blind Image Quality Assessment Without Human Training Using Latent Quality Factors , 2012, IEEE Signal Processing Letters.

[36]  Frances S. Chance,et al.  Complex cells as cortically amplified simple cells , 1999, Nature Neuroscience.

[37]  Geoffrey E. Hinton,et al.  Learning representations by back-propagating errors , 1986, Nature.

[38]  Chi-Man Vong,et al.  Local Receptive Fields Based Extreme Learning Machine , 2015, IEEE Computational Intelligence Magazine.

[39]  Min Zhang,et al.  Blind Image Quality Assessment Using the Joint Statistics of Generalized Local Binary Pattern , 2015, IEEE Signal Processing Letters.

[40]  Gangyi Jiang,et al.  Research on subjective stereoscopic image quality assessment , 2009, Electronic Imaging.

[41]  J. P. Jones,et al.  An evaluation of the two-dimensional Gabor filter model of simple receptive fields in cat striate cortex. , 1987, Journal of neurophysiology.

[42]  Wujie Zhou,et al.  Local gradient patterns (LGP): An effective local-statistical-feature extraction scheme for no-reference image quality assessment , 2017, Inf. Sci..

[43]  Lin Ma,et al.  Learning structure of stereoscopic image for no-reference quality assessment with convolutional neural network , 2016, Pattern Recognit..

[44]  C.-C. Jay Kuo,et al.  MCL-3D: A Database for Stereoscopic Image Quality Assessment using 2D-Image-Plus-Depth Source , 2014, J. Inf. Sci. Eng..

[45]  Wujie Zhou,et al.  Utilizing Dictionary Learning and Machine Learning for Blind Quality Assessment of 3-D Images , 2017, IEEE Transactions on Broadcasting.

[46]  David Zhang,et al.  FSIM: A Feature Similarity Index for Image Quality Assessment , 2011, IEEE Transactions on Image Processing.

[47]  Weisi Lin,et al.  Objective Image Quality Assessment Based on Support Vector Regression , 2010, IEEE Transactions on Neural Networks.

[48]  Nick Holliman,et al.  Stereoscopic image quality metrics and compression , 2008, Electronic Imaging.

[49]  Kai Zeng,et al.  Quality Prediction of Asymmetrically Distorted Stereoscopic 3D Images , 2015, IEEE Transactions on Image Processing.

[50]  Mei Yu,et al.  PMFS: A Perceptual Modulated Feature Similarity Metric for Stereoscopic Image Quality Assessment , 2014, IEEE Signal Processing Letters.

[51]  Nuno Roma,et al.  GHEVC: An Efficient HEVC Decoder for Graphics Processing Units , 2017, IEEE Transactions on Multimedia.

[52]  K. S. Banerjee Generalized Inverse of Matrices and Its Applications , 1973 .

[53]  Bok-Suk Shin,et al.  Evaluation of two stereo matchers on long real-world video sequences , 2015, Pattern Recognit..

[54]  Qian Li,et al.  Saliency structure stereoscopic image quality assessment method , 2014 .

[55]  Alan C. Bovik,et al.  Oriented Correlation Models of Distorted Natural Images With Application to Natural Stereopair Quality Evaluation , 2015, IEEE Transactions on Image Processing.

[56]  Ding Yuan,et al.  Cross-trees, edge and superpixel priors-based cost aggregation for stereo matching , 2015, Pattern Recognit..

[57]  Patrick Le Callet,et al.  Quality Assessment of Stereoscopic Images , 2008, EURASIP J. Image Video Process..

[58]  Weisi Lin,et al.  Using Binocular Feature Combination for Blind Quality Assessment of Stereoscopic Images , 2015, IEEE Signal Processing Letters.

[59]  Lei Zhang,et al.  A Feature-Enriched Completely Blind Image Quality Evaluator , 2015, IEEE Transactions on Image Processing.

[60]  Alexandros Iosifidis,et al.  Minimum Class Variance Extreme Learning Machine for Human Action Recognition , 2013, IEEE Transactions on Circuits and Systems for Video Technology.

[61]  Weisi Lin,et al.  Scalable image quality assessment with 2D mel-cepstrum and machine learning approach , 2012, Pattern Recognit..

[62]  Weisi Lin,et al.  On Predicting Visual Comfort of Stereoscopic Images: A Learning to Rank Based Approach , 2016, IEEE Signal Processing Letters.

[63]  Masaaki Kawahashi,et al.  Renovation of Journal of Visualization , 2010, J. Vis..

[64]  Ting Luo,et al.  Blind 3D image quality assessment based on self-similarity of binocular features , 2017, Neurocomputing.

[65]  Changming Sun,et al.  Feature matching in stereo images encouraging uniform spatial distribution , 2015, Pattern Recognit..

[66]  Lu Yu,et al.  Simulating binocular vision for no-reference 3D visual quality measurement. , 2015, Optics express.

[67]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.

[68]  S. Grossberg,et al.  Neural dynamics of binocular brightness perception , 1999, Vision Research.

[69]  Ahmet M. Kondoz,et al.  Quality analysis for 3D video using 2D video quality models , 2008, IEEE Transactions on Consumer Electronics.

[70]  Wijnand A. IJsselsteijn,et al.  Perceived quality of compressed stereoscopic images: Effects of symmetric and asymmetric JPEG coding and camera separation , 2006, TAP.

[71]  Munchurl Kim,et al.  Binocular Suppression-Based Stereoscopic Video Coding by Joint Rate Control With KKT Conditions for a Hybrid Video Codec System , 2015, IEEE Transactions on Circuits and Systems for Video Technology.

[72]  Roushain Akhter,et al.  No-reference stereoscopic image quality assessment , 2010, Electronic Imaging.

[73]  D V Meegan,et al.  Unequal weighting of monocular inputs in binocular combination: implications for the compression of stereoscopic imagery. , 2001, Journal of experimental psychology. Applied.

[74]  Wujie Zhou,et al.  No-reference stereoscopic image quality measurement based on generalized local ternary patterns of binocular energy response , 2015 .

[75]  R. Hess,et al.  Binocular Combination of Second-Order Stimuli , 2014, PloS one.

[76]  Weisi Lin,et al.  SVD-Based Quality Metric for Image and Video Using Machine Learning , 2012, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[77]  A. Murat Tekalp,et al.  Evaluation of Asymmetric Stereo Video Coding and Rate Scaling for Adaptive 3D Video Streaming , 2011, IEEE Transactions on Broadcasting.

[78]  Alan C. Bovik,et al.  No-Reference Quality Assessment of Natural Stereopairs , 2013, IEEE Transactions on Image Processing.

[79]  Wujie Zhou,et al.  Binocular Responses for No-Reference 3D Image Quality Assessment , 2016, IEEE Transactions on Multimedia.

[80]  Damon M. Chandler,et al.  3D-MAD: A Full Reference Stereoscopic Image Quality Estimator Based on Binocular Lightness and Contrast Perception , 2015, IEEE Transactions on Image Processing.

[81]  Sanghoon Lee,et al.  Transition of Visual Attention Assessment in Stereoscopic Images With Evaluation of Subjective Visual Quality and Discomfort , 2015, IEEE Transactions on Multimedia.