Active Sampling for Subjective Video Quality Assessment

This paper presents an active sampling framework to achieve economic and robust subjective Video Quality Assessment (VQA). To overcome the main drawback of paired comparison that the number of pairs grows exponentially with the number of videos under test, the proposed methodology does not require the participants to perform the complete comparison of all the paired videos. Instead, we first ask some participants to perform random sampling of all possible paired comparisons. With a sufficiency of coverage satisfied motivated by Erdos-Renyi random graph, HodgeRank may give reliable results that can be used to pick out confusing pairs. Subsequently, participants will only need to commit to these confusing ones thus could save much time and labor. In other words, our interest is minimizing the number of pairs needed to learn the ranking. We demonstrate the effectiveness of the proposed framework on LIVE Database and experimental results show that it is a promising and applicable method for efficient subjective VOA.

[1]  Trevor Darrell,et al.  Active Learning with Gaussian Processes for Object Categorization , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[2]  Herbert A. Simon,et al.  Problem solving and rule induction: A unified view. , 1974 .

[3]  Thomas S. Huang,et al.  Relevance feedback in image retrieval: A comprehensive review , 2003, Multimedia Systems.

[4]  Qingming Huang,et al.  Random partial paired comparison for subjective video quality assessment via hodgerank , 2011, ACM Multimedia.

[5]  Moni Naor,et al.  Rank aggregation methods for the Web , 2001, WWW '01.

[6]  Chin-Laung Lei,et al.  A crowdsourceable QoE evaluation framework for multimedia content , 2009, ACM Multimedia.

[7]  Arun Rajkumar,et al.  A Statistical Convergence Perspective of Algorithms for Rank Aggregation from Pairwise Data , 2014, ICML.

[8]  Fei-Fei Li,et al.  Towards Scalable Dataset Construction: An Active Learning Approach , 2008, ECCV.

[9]  Matthieu Cord,et al.  Active Learning Methods for Interactive Image Retrieval , 2008, IEEE Transactions on Image Processing.

[10]  Yoram Singer,et al.  An Efficient Boosting Algorithm for Combining Preferences by , 2013 .

[11]  Patrick Henry Winston,et al.  Learning structural descriptions from examples , 1970 .

[12]  Rong Yan,et al.  Automatically labeling video data using multi-class active learning , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[13]  Devavrat Shah,et al.  Iterative ranking from pair-wise comparisons , 2012, NIPS.

[14]  Tsuhan Chen,et al.  Active learning for piecewise planar 3D reconstruction , 2011, CVPR 2011.

[15]  Yuan Yao,et al.  Statistical ranking and combinatorial Hodge theory , 2008, Math. Program..

[16]  M. Kendall,et al.  ON THE METHOD OF PAIRED COMPARISONS , 1940 .

[17]  David A. Cohn,et al.  Improving generalization with active learning , 1994, Machine Learning.

[18]  Prateek Jain,et al.  Far-sighted active learning on a budget for image and video recognition , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[19]  METHODS FOR SUBJECTIVE DETERMINATION OF TRANSMISSION QUALITY Summary , 2022 .

[20]  K. Arrow.  A Difficulty in the Concept of Social Welfare , 1950, Journal of Political Economy.

[21]  Jiebo Luo,et al.  iCoseg: Interactive co-segmentation with intelligent scribble guidance , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[22]  Gunnar E. Carlsson,et al.  Topology and data , 2009 .

[23]  D. Amnon Silverstein,et al.  Efficient method for paired comparison , 2001, J. Electronic Imaging.

[24]  S. Osher,et al.  Statistical ranking using the $l^{1}$-norm on graphs , 2013 .

[25]  Herbert Edelsbrunner,et al.  Computational Topology - an Introduction , 2009 .

[26]  Qingming Huang,et al.  From Social to Individuals: A Parsimonious Path of Multi-Level Models for Crowdsourced Preference Aggregation , 2018, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[27]  Ragnhild Eg,et al.  Randomised pair comparison: an economic and robust method for audiovisual quality assessment , 2010, NOSSDAV.

[28]  Thorsten Joachims,et al.  Optimizing search engines using clickthrough data , 2002, KDD.

[29]  Xiaochun Cao,et al.  Parsimonious Mixed-Effects HodgeRank for Crowdsourced Preference Aggregation , 2016, ACM Multimedia.

[30]  T. L. Saaty.  A Scaling Method for Priorities in Hierarchical Structures , 1977 .

[31]  Tao Xiang,et al.  Robust Subjective Visual Property Prediction from Crowdsourced Pairwise Labels , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[32]  Christopher J. C. Burges,et al.  From RankNet to LambdaRank to LambdaMART: An Overview , 2010 .

[33]  Stella X. Yu,et al.  Affinity CNN: Learning Pixel-Centric Pairwise Relations for Figure/Ground Embedding , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[34]  Robert Tibshirani,et al.  Classification by Pairwise Coupling , 1997, NIPS.

[35]  L. Thurstone.  The method of paired comparisons for social values , 1927 .