Subjective video quality evaluation of different content types under different impairments

ABSTRACT Nowadays, access to multimedia content is one of the most demanded services on the Internet. However, the transmission of audio and video over these networks is not free of problems that negatively affect user experience. Factors such as low image quality, cuts during playback or losses of audio or video, among others, can occur and there is no clear idea about the level of distortion introduced in the perceived quality. For that reason, different impairments should be evaluated based on user opinions, with the aim of analyzing the impact in the perceived quality. In this work, we carried out a subjective evaluation of different types of impairments with different types of contents, including news, cartoons, sports and action movies. A total of 100 individuals, between the ages of 20 and 68, participated in the subjective study. Results show that short-term rebuffering events negatively affect the quality of experience and that desynchronization between audio and video is the least annoying impairment. Moreover, we found that the content type determines the subjective results according to the impairment present during the playback.

[1]  Stefan Winkler,et al.  Analysis of Public Image and Video Databases for Quality Assessment , 2012, IEEE Journal of Selected Topics in Signal Processing.

[2]  ITU-T Rec. P.910 (04/2008) Subjective video quality assessment methods for multimedia applications , 2009 .

[3]  Lea Skorin-Kapov,et al.  Survey and Challenges of QoE Management Issues in Wireless Networks , 2013, J. Comput. Networks Commun..

[4]  Christian Timmerer,et al.  Automated QoE evaluation of Dynamic Adaptive Streaming over HTTP , 2013, 2013 Fifth International Workshop on Quality of Multimedia Experience (QoMEX).

[5]  Markus Fiedler,et al.  The memory effect and its implications on Web QoE modeling , 2011, 2011 23rd International Teletraffic Congress (ITC).

[6]  John G. Beerends,et al.  The Influence of Video Quality on Perceived Audio Quality and Vice Versa , 1999 .

[7]  Philip T. Kortum,et al.  The Effect of Content Desirability on Subjective Video Quality Ratings , 2010, Hum. Factors.

[8]  Xabiel G. Pañeda,et al.  Adaptation engine for a streaming service based on MPEG-DASH , 2015, Multimedia Tools and Applications.

[9]  Qingming Huang,et al.  Robust evaluation for quality of experience in crowdsourcing , 2013, ACM Multimedia.

[10]  Rik Van de Walle,et al.  Assessing the importance of audio/video synchronization for simultaneous translation of video sequences , 2012, Multimedia Systems.

[11]  Xabiel G. Pañeda,et al.  Adaptable system based on Scalable Video Coding for high-quality video service , 2013, Comput. Electr. Eng..

[12]  Alan C. Bovik,et al.  Study of the effects of stalling events on the quality of experience of mobile streaming videos , 2014, 2014 IEEE Global Conference on Signal and Information Processing (GlobalSIP).

[13]  Xabiel G. Pañeda,et al.  A Non-intrusive Estimation for High-Quality Internet TV Services , 2009, FMN.

[14]  Hai Jin,et al.  Guaranteeing QoS of media-based applications in virtualized environment , 2013, New Rev. Hypermedia Multim..

[15]  Rocky K. C. Chang,et al.  Measuring the quality of experience of HTTP video streaming , 2011, 12th IFIP/IEEE International Symposium on Integrated Network Management (IM 2011) and Workshops.

[16]  Yining Qi,et al.  The Effect of Frame Freezing and Frame Skipping on Video Quality , 2006, 2006 International Conference on Intelligent Information Hiding and Multimedia.

[17]  Soo Young Shin,et al.  The Impact of Temporal Impairment on Quality of Experience (QoE) in Video Streaming: A No Reference (NR) Subjective and Objective Study , 2015 .

[18]  James Nightingale,et al.  The impact of network impairment on quality of experience (QoE) in H.265/HEVC video streaming , 2014, IEEE Transactions on Consumer Electronics.

[19]  Ahmet M. Kondoz,et al.  Investigation into spatial audio quality of experience in the presence of accompanying video cues with spatial mismatch , 2013, 2013 IEEE Globecom Workshops (GC Wkshps).

[20]  Xabiel G. Pañeda,et al.  Subjective evaluation of critical success factors for a QoE aware adaptive system , 2013, Comput. Commun..

[21]  Yang Geng,et al.  An objective multi-factor QoE evaluation based on content classification for H.264/AVC encoded video , 2013, 2013 IEEE Symposium on Computers and Communications (ISCC).

[22]  Markus Fiedler,et al.  Impact of Disturbance Locations on Video Quality of Experience , 2011 .

[23]  Michael J. Crawley,et al.  The R book , 2022 .

[24]  M. Ries,et al.  Impact of screening technique on crowdsourcing QoE assessments , 2012, Proceedings of 22nd International Conference Radioelektronika 2012.

[25]  Sugato Chakravarty,et al.  Methodology for the subjective assessment of the quality of television pictures , 1995 .

[26]  Gerardo Rubino,et al.  A new approach for the prediction of end-to-end performance of multimedia streams , 2004, First International Conference on the Quantitative Evaluation of Systems, 2004. QEST 2004. Proceedings..

[27]  Toon De Pessemier,et al.  Quantifying the Influence of Rebuffering Interruptions on the User's Quality of Experience During Mobile Video Watching , 2013, IEEE Transactions on Broadcasting.

[28]  Federica Battisti,et al.  A study on the effects of quality of service parameters on perceived video quality , 2014, 2014 5th European Workshop on Visual Information Processing (EUVIP).

[29]  John Woods,et al.  Survey on QoE\QoS Correlation Models For Multimedia Services , 2013, ArXiv.

[30]  Touradj Ebrahimi,et al.  Towards high efficiency video coding: Subjective evaluation of potential coding technologies , 2011, J. Vis. Commun. Image Represent..

[31]  Piet Demeester,et al.  Lessons learned during real-life QoE assessment , 2012, euroITV 2012.

[32]  C. Arun,et al.  An Interlacing Technique-Based Blind Video Watermarking Using Wavelet , 2015 .

[33]  Michael J. Crawley.  The R Book: Crawley/The R Book , 2012 .

[34]  Liam Murphy,et al.  Subjective assessment of the quality-oriented adaptive scheme , 2005, IEEE Transactions on Broadcasting.

[35]  Philip J. Corriveau,et al.  Study of Rating Scales for Subjective Quality Assessment of High-Definition Video , 2011, IEEE Transactions on Broadcasting.

[36]  Markus Fiedler,et al.  Mobile video sensitivity to packet loss and packet delay variation in terms of QoE , 2012, 2012 19th International Packet Video Workshop (PV).

[37]  Markus Fiedler,et al.  Initial delay vs. interruptions: Between the devil and the deep blue sea , 2012, 2012 Fourth International Workshop on Quality of Multimedia Experience.

[38]  Stefano Tubaro,et al.  Subjective Quality Assessment of H.264/AVC Video Streaming with Packet Losses , 2011, EURASIP J. Image Video Process..

[39]  Gabriel Alfredo Baum,et al.  A new approach for the prediction of end-to-end performance of multimedia streams , 2004 .

[40]  Oliver Hohlfeld,et al.  Impact of frame rate and resolution on objective QoE metrics , 2010, 2010 Second International Workshop on Quality of Multimedia Experience (QoMEX).

[41]  Enrique Barra,et al.  Using multimedia and peer assessment to promote collaborative e-learning , 2014, New Rev. Hypermedia Multim..

[42]  Jon Boley,et al.  The Influence of Individual Audio Impairments on Perceived Video Quality , 2010 .

[43]  Ragnhild Eg,et al.  Audiovisual robustness: exploring perceptual tolerance to asynchrony and quality distortion , 2014, Multimedia Tools and Applications.