Analysing the Impact of Cross-Content Pairs on Pairwise Comparison Scaling

Pairwise comparisons (PWC) methodology is one of the most commonly used methods for subjective quality assessment, especially for computer graphics and multimedia applications. Unlike rating methods, a psychometric scaling operation is required to convert PWC results to numerical subjective quality values. Due to the nature of this scaling operation, the obtained quality scores are relative to the set they are computed in. While it is customary to compare different versions of the same content, in this work we study how cross-content comparisons may benefit psychometric scaling. For this purpose, we use two different video quality databases which have both rating and PWC experiment results. The results show that despite same-content comparisons play a major role in the accuracy of psychometric scaling, the use of a small portion of cross-content comparison pairs is indeed beneficial to obtain more accurate quality estimates.

[1]  Sheila S. Hemami,et al.  Effects of spatial correlations and global precedence on the visual fidelity of distorted images , 2006, Electronic Imaging.

[2]  Pan Gao,et al.  Subjective and Objective Quality Assessment for Volumetric Video Compression , 2019, IQSP.

[3]  Patrick Le Callet,et al.  Tradeoffs in subjective testing methods for image and video quality assessment , 2010, Electronic Imaging.

[4]  Emin Zerman,et al.  The Relation Between MOS and Pairwise Comparisons and the Importance of Cross-Content Comparisons , 2018, HVEI.

[5]  L. Thurstone,et al.  A low of comparative judgement , 1927 .

[6]  Nikolay N. Ponomarenko,et al.  Image database TID2013: Peculiarities, results and perspectives , 2015, Signal Process. Image Commun..

[7]  Giuseppe Valenzise,et al.  Effect of color space on high dynamic range video compression performance , 2017, 2017 Ninth International Conference on Quality of Multimedia Experience (QoMEX).

[8]  R. A. Bradley,et al.  Rank Analysis of Incomplete Block Designs: I. The Method of Paired Comparisons , 1952 .

[9]  Sugato Chakravarty,et al.  Methodology for the subjective assessment of the quality of television pictures , 1995 .

[10]  Junle Wang,et al.  Hybrid-MST: A Hybrid Active Sampling Strategy for Pairwise Preference Aggregation , 2018, NeurIPS.

[11]  Amy R. Reibman.  A strategy to jointly test image quality estimators subjectively , 2012, 2012 19th IEEE International Conference on Image Processing.

[12]  Rafal Mantiuk,et al.  Comparison of Four Subjective Methods for Image Quality Assessment , 2012, Comput. Graph. Forum.

[13]  David S. Doermann,et al.  Active Sampling for Subjective Image Quality Assessment , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[14]  ITU-T Rec. P.910 (04/2008) Subjective video quality assessment methods for multimedia applications , 2009 .

[15]  Tim Weyrich,et al.  A study of image colourfulness , 2014, CAe@Expressive.

[16]  María Pérez-Ortiz,et al.  Psychometric scaling of TID2013 dataset , 2018, 2018 Tenth International Conference on Quality of Multimedia Experience (QoMEX).