Strategy for Boosting Pair Comparison and Improving Quality Assessment Accuracy

The development of rigorous quality assessment model relies on the collection of reliable subjective data, where the perceived quality of visual multimedia is rated by the human observers. Different subjective assessment protocols can be used according to the objectives, which determine the discriminability and accuracy of the subjective data. Single stimulus methodology, e.g., the Absolute Category Rating (ACR) has been widely adopted due to its simplicity and efficiency. However, Pair Comparison (PC) is of significant advantage over ACR in terms of discriminability. In addition, PC avoids the influence of observers' bias regarding their understanding of the quality scale. Nevertheless, full pair comparison is much more time-consuming. In this study, we therefore 1) employ a generic model to bridge the pair comparison data and ACR data, where the variance term could be recovered and the obtained information is more complete; 2) propose a fusion strategy to boost pair comparisons by utilizing the ACR results as initialization information; 3) develop a novel active batch sampling strategy based on Minimum Spanning Tree (MST) for PC. In such a way, the proposed methodology could achieve the same accuracy of pair comparison but with the compelxity as low as ACR. Extensive experimental results demonstrate the efficiency and accuracy of the proposed approach, which outperforms the state of the art approaches.

[1]  Martin J. Wainwright,et al.  Estimation from Pairwise Comparisons: Sharp Minimax Bounds with Topology Dependence , 2015, J. Mach. Learn. Res..

[2]  Deepthi Nandakumar,et al.  On the accuracy of video quality measurement techniques , 2019, 2019 IEEE 21st International Workshop on Multimedia Signal Processing (MMSP).

[3]  Peter Emerson,et al.  The original Borda count and partial voting , 2013, Soc. Choice Welf..

[4]  Lucjan Janowski,et al.  A Simple Model for Subject Behavior in Subjective Experiments , 2020, ArXiv.

[5]  Andrew B. Watson,et al.  Measurement of visual impairment scales for digital video , 2001, IS&T/SPIE Electronic Imaging.

[6]  Jing Li,et al.  GPM: A Generic Probabilistic Model to Recover Annotator's Behavior and Ground Truth Labeling , 2020, ArXiv.

[7]  Patrick Le Callet,et al.  Towards a New Quality Metric for 3-D Synthesized View Assessment , 2011, IEEE Journal of Selected Topics in Signal Processing.

[8]  Thomas Pfeiffer,et al.  Adaptive Polling for Information Aggregation , 2012, AAAI.

[9]  Yong Man Ro,et al.  Predicting Visual Discomfort of Stereoscopic Images Using Human Attention Model , 2013, IEEE Transactions on Circuits and Systems for Video Technology.

[10]  Yee Whye Teh,et al.  Inferring ground truth from multi-annotator ordinal data: a probabilistic approach , 2013, ArXiv.

[11]  David C. Parkes,et al.  Generalized Method-of-Moments for Rank Aggregation , 2013, NIPS.

[12]  Paul N. Bennett,et al.  Pairwise ranking aggregation in a crowdsourced setting , 2013, WSDM.

[13]  David C. Parkes,et al.  Random Utility Theory for Social Choice , 2012, NIPS.

[14]  Christian Schmid,et al.  A Matlab function to estimate choice model parameters from paired-comparison data , 2004, Behavior research methods, instruments, & computers : a journal of the Psychonomic Society, Inc.

[15]  Marius Pedersen,et al.  The influence of short-term memory in subjective image quality assessment , 2016, 2016 IEEE International Conference on Image Processing (ICIP).

[16]  F. Mosteller.  Remarks on the method of paired comparisons: I. The least squares solution assuming equal standard deviations and equal correlations , 1951 .

[17]  Patrick Le Callet,et al.  Pseudo no reference image quality metric using perceptual data hiding , 2006, Electronic Imaging.

[18]  Robert D. Nowak,et al.  Active Ranking using Pairwise Comparisons , 2011, NIPS.

[19]  Karel Fliegel,et al.  Quality Assessment of Sharpened Images: Challenges, Methodology, and Objective Metrics , 2017, IEEE Transactions on Image Processing.

[20]  R. Plackett.  The Analysis of Permutations , 1975 .

[21]  R. A. Bradley,et al.  RANK ANALYSIS OF INCOMPLETE BLOCK DESIGNS , 1952 .

[22]  Qingming Huang,et al.  HodgeRank on Random Graphs for Subjective Video Quality Assessment , 2012, IEEE Transactions on Multimedia.

[23]  Junle Wang,et al.  Exploring the effects of subjective methodology on assessing visual discomfort in immersive multimedia , 2018, HVEI.

[24]  Yoram Singer,et al.  An Efficient Boosting Algorithm for Combining Preferences by , 2013 .

[25]  Xi Chen,et al.  HodgeRank With Information Maximization for Crowdsourced Pairwise Ranking Aggregation , 2017, AAAI.

[26]  R. A. Bradley,et al.  RANK ANALYSIS OF INCOMPLETE BLOCK DESIGNS THE METHOD OF PAIRED COMPARISONS , 1952 .

[27]  Tom Minka,et al.  TrueSkill Through Time: Revisiting the History of Chess , 2007, NIPS.

[28]  María Pérez-Ortiz,et al.  From Pairwise Comparisons and Rating to a Unified Quality Scale , 2020, IEEE Transactions on Image Processing.

[29]  Jing Li,et al.  Boosting paired comparison methodology in measuring visual discomfort of 3DTV: performances of three different designs , 2013, Electronic Imaging.

[30]  Cristina Hava Muntean,et al.  User-centered EEG-based multimedia quality assessment , 2013, 2013 IEEE International Symposium on Broadband Multimedia Systems and Broadcasting (BMSB).

[31]  Sugato Chakravarty,et al.  Methodology for the subjective assessment of the quality of television pictures , 1995 .

[32]  F. Mosteller,et al.  Remarks on the method of paired comparisons: III. A test of significance for paired comparisons when equal standard deviations and equal correlations are assumed , 1951, Psychometrika.

[33]  D. Amnon Silverstein,et al.  Quantifying Perceptual Image Quality , 1998, PICS.

[34]  David S. Doermann,et al.  Active Sampling for Subjective Image Quality Assessment , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[35]  Nebojsa Jojic,et al.  Efficient Ranking from Pairwise Comparisons , 2013, ICML.

[36]  Craig Boutilier,et al.  Learning Mallows Models with Pairwise Preferences , 2011, ICML.

[37]  Junle Wang,et al.  Study on visual discomfort induced by stimulus movement at fixed depth on stereoscopic displays using shutter glasses , 2011, 2011 17th International Conference on Digital Signal Processing (DSP).

[38]  L. Thurstone.  A law of comparative judgment. , 1994 .

[39]  F. Mosteller.  Remarks on the method of paired comparisons: I. The least squares solution assuming equal standard deviations and equal correlations , 1951 .

[40]  O. Dykstra.  Rank Analysis of Incomplete Block Designs: A Method of Paired Comparisons Employing Unequal Repetitions on Pairs , 1960 .

[41]  Junle Wang,et al.  Hybrid-MST: A Hybrid Active Sampling Strategy for Pairwise Preference Aggregation , 2018, NeurIPS.