Subjective Quality Evaluation via Paired Comparison: Application to Scalable Video Coding

Scalable video coding is a powerful solution for content delivery in many interactive multimedia services due to its adaptability to varying terminal and network constraints. In order to successfully exploit such adaptability, it is necessary to understand users' preference among various scalability options and consequently develop an optimal bit rate adaptation strategy. In this paper, we present a study of subjective quality assessment of scalable video coding, which investigates the influence of the combination of scalability options on perceived quality with the goal of providing guidelines for an adaptive strategy that selects the optimal combination for a given bandwidth constraint. In particular, the study is based on paired comparison of stimuli that is suitable for our goal due to its simplicity and easiness. We propose a new method, called Paired Evaluation via Analysis of Reliability (PEAR), which analyzes paired comparison results and produces not only quality scores but also intuitive measures of confidence of the scores for significance analysis. Results and analysis of extensive subjective tests for two different scalable video codecs and high definition contents are described, from which general consistent conclusions are drawn. The video and subjective data used in the paper are publicly available to the research community.

[1]  Tse-Wei Wang,et al.  A rate-distortion optimization model for SVC inter-layer encoding and bitstream extraction , 2008, J. Vis. Commun. Image Represent..

[2]  Yong Wang,et al.  Subjective preference of spatio-temporal rate in video adaptation using multi-dimensional scalable coding , 2004, 2004 IEEE International Conference on Multimedia and Expo (ICME) (IEEE Cat. No.04TH8763).

[3]  R. A. Bradley,et al.  RANK ANALYSIS OF INCOMPLETE BLOCK DESIGNS THE METHOD OF PAIRED COMPARISONS , 1952 .

[4]  R. Davidson.  On Extending the Bradley-Terry Model to Accommodate Ties in Paired Comparison Experiments , 1970 .

[5]  Chin-Laung Lei,et al.  A crowdsourceable QoE evaluation framework for multimedia content , 2009, ACM Multimedia.

[6]  Mihaela van der Schaar,et al.  FGS+: optimizing the joint SNR-temporal video quality in MPEG-4 fine grained scalable coding , 2002, 2002 IEEE International Symposium on Circuits and Systems. Proceedings (Cat. No.02CH37353).

[7]  Sugato Chakravarty,et al.  Methodology for the subjective assessment of the quality of television pictures , 1995 .

[8]  P. V. Rao,et al.  Ties in Paired-Comparison Experiments: A Generalization of the Bradley-Terry Model , 1967 .

[9]  Touradj Ebrahimi,et al.  Subjective evaluation of scalable video coding for content distribution , 2010, ACM Multimedia.

[10]  Jens-Rainer Ohm,et al.  Advances in Scalable Video Coding , 2005, Proceedings of the IEEE.

[11]  Ebroul Izquierdo,et al.  An efficient optimisation scheme for scalable surveillance centric video communications , 2009, Signal Process. Image Commun..

[12]  D. Amnon Silverstein,et al.  Quantifying Perceptual Image Quality , 1998, PICS.

[13]  Andrew Catellier,et al.  Gradient ascent paired-comparison subjective quality testing , 2009, 2009 International Workshop on Quality of Multimedia Experience.

[14]  Liam Murphy,et al.  Optimum adaptation trajectories for streamed multimedia , 2005, Multimedia Systems.

[15]  Alberto Signoroni,et al.  State-of-the-Art and Trends in Scalable Video Compression With Wavelet-Based Approaches , 2007, IEEE Transactions on Circuits and Systems for Video Technology.

[16]  P. Moran.  On the method of paired comparisons. , 1947, Biometrika.

[17]  Heiko Schwarz,et al.  Overview of the Scalable Video Coding Extension of the H.264/AVC Standard , 2007, IEEE Transactions on Circuits and Systems for Video Technology.

[18]  Jianfei Cai,et al.  Cross-Dimensional Perceptual Quality Assessment for Low Bit-Rate Videos , 2008, IEEE Transactions on Multimedia.

[19]  M. Glickman.  Parameter Estimation in Large Dynamic Paired Comparison Experiments , 1999 .

[20]  ITU-T Rec. P.910 (04/2008) Subjective video quality assessment methods for multimedia applications , 2009 .