Measurement of Quality of Experience of Video-on-Demand Services: A Survey

Video-on-demand streaming services have gained popularity over the past few years. An increase in the speed of the access networks has also led to a larger number of users watching videos online. Online video streaming traffic is estimated to further increase from the current value of 57% to 69% by 2017, Cisco, 2014. In order to retain the existing users and attract new users, service providers attempt to satisfy the user's expectations and provide a satisfactory viewing experience. The first step toward providing a satisfactory service is to be able to quantify the users' perception of the current service level. Quality of experience (QoE) is a quality metric that provides a holistic measure of the users' perception of the quality. In this survey, we first present a tutorial overview of the popular video streaming techniques deployed for stored videos, followed by identifying various metrics that could be used to quantify the QoE for video streaming services; finally, we present a comprehensive survey of the literature on various tools and measurement methodologies that have been proposed to measure or predict the QoE of online video streaming services.

[1]  Lilly Irani,et al.  Amazon Mechanical Turk , 2018, Advances in Intelligent Systems and Computing.

[2]  Christian Timmerer,et al.  Automated QoE evaluation of Dynamic Adaptive Streaming over HTTP , 2013, 2013 Fifth International Workshop on Quality of Multimedia Experience (QoMEX).

[3]  Deep Medhi,et al.  Pytomo: A tool for analyzing playback quality of YouTube videos , 2011, 2011 23rd International Teletraffic Congress (ITC).

[4]  B. Staehle,et al.  YoMo: A YouTube Application Comfort Monitoring Tool , 2010 .

[5]  Deep Medhi,et al.  Viewing YouTube from a metropolitan area: What do users accessing from residential ISPs experience? , 2013, 2013 IFIP/IEEE International Symposium on Integrated Network Management (IM 2013).

[6]  Zhou Wang,et al.  Video quality assessment using structural distortion measurement , 2002, Proceedings. International Conference on Image Processing.

[7]  Richard Nelson,et al.  Application flow control in YouTube video streams , 2011, CCRV.

[8]  Mark Claypool,et al.  RealTracer—Tools for Measuring the Performance of RealVideo on the Internet , 2005, Multimedia Tools and Applications.

[9]  ITU-T Rec. P.910 (04/2008) Subjective video quality assessment methods for multimedia applications , 2009 .

[10]  Amy Csizmar Dalal,et al.  Discerning user-perceived media stream quality through application-layer measurements , 2005, 2005 1st International Conference on Multimedia Services Access Networks, 2005. MSAN '05..

[11]  Gerardo Rubino,et al.  QoE Monitoring Platform for Video Delivery Networks , 2007, IPOM.

[12]  David C. Robinson,et al.  Subjective video quality assessment of HTTP adaptive streaming technologies , 2012, Bell Labs Technical Journal.

[13]  Adlen Ksentini,et al.  A_PSQA: Efficient real-time video streaming QoE tool in a future media internet context , 2011, 2011 IEEE International Conference on Multimedia and Expo.

[14]  Marco Mellia,et al.  YouTube everywhere: impact of device and infrastructure synergies on user experience , 2011, IMC '11.

[15]  Gustavo de Veciana,et al.  Video Quality Assessment on Mobile Devices: Subjective, Behavioral and Objective Studies , 2012, IEEE Journal of Selected Topics in Signal Processing.

[16]  Lea Skorin-Kapov,et al.  A multi-dimensional view of QoE: the ARCU model , 2012, 2012 Proceedings of the 35th International Convention MIPRO.

[17]  Stefan Winkler,et al.  Perceptual Video Quality and Blockiness Metrics for Multimedia Streaming Applications , 2001 .

[18]  Peter Schelkens,et al.  Qualinet White Paper on Definitions of Quality of Experience , 2013 .

[19]  Monia Ghobadi,et al.  Trickle: Rate Limiting YouTube Video Streaming , 2012, USENIX Annual Technical Conference.

[20]  Srinivasan Seshan,et al.  Developing a predictive model of quality of experience for internet video , 2013, SIGCOMM.

[21]  I. Stoica,et al.  A case for a coordinated internet video control plane , 2012, CCRV.

[22]  Thouraya Bouabana-Tebibel,et al.  Empirical QoE/QoS correlation model based on multiple parameters for VoD flows , 2012, 2012 IEEE Global Communications Conference (GLOBECOM).

[23]  Dilip Kumar Krishnappa,et al.  DASHing YouTube: An analysis of using DASH in YouTube video service , 2013, 38th Annual IEEE Conference on Local Computer Networks.

[24]  M. Hannuksela,et al.  ACCEPTANCE OF AUDIOVISUAL QUALITY IN ERRONEOUS TELEVISION SEQUENCES OVER A DVB-H CHANNEL , 2006 .

[25]  David R. Musicant,et al.  Predicting User-Perceived Quality Ratings from Streaming Media Data , 2007, 2007 IEEE International Conference on Communications.

[26]  Tobias Hoßfeld,et al.  Internet Video Delivery in YouTube: From Traffic Measurements to Quality of Experience , 2013, Data Traffic Monitoring and Analysis.

[27]  Sonia Fahmy,et al.  Analyzing video services in Web 2.0: a global perspective , 2008, NOSSDAV.

[28]  Markus Fiedler,et al.  A generic quantitative relationship between quality of experience and quality of service , 2010, IEEE Network.

[29]  Koenraad Laevens,et al.  Session reconstruction for HTTP adaptive streaming: Laying the foundation for network-based QoE monitoring , 2012, 2012 IEEE 20th International Workshop on Quality of Service.

[30]  Sebastian Möller,et al.  On the way to high-quality video calls in future mobile networks , 2011, 2011 Third International Workshop on Quality of Multimedia Experience.

[31]  Shuai Wan,et al.  Bitstream-based quality assessment for networked video: a review , 2012, IEEE Communications Magazine.

[32]  Hong Ren Wu,et al.  No-Reference Quality Assessment for Networked Video via Primary Analysis of Bit Stream , 2010, IEEE Transactions on Circuits and Systems for Video Technology.

[33]  Ramesh K. Sitaraman,et al.  Video Stream Quality Impacts Viewer Behavior: Inferring Causality Using Quasi-Experimental Designs , 2012, IEEE/ACM Transactions on Networking.

[34]  Jie Lin,et al.  Real time video QoE analysis of RTMP streams , 2011, 30th IEEE International Performance Computing and Communications Conference.

[35]  Peter Lambert,et al.  Assessing Quality of Experience of IPTV and Video on Demand Services in Real-Life Environments , 2010, IEEE Transactions on Broadcasting.

[36]  Jing Liu,et al.  A study on Quality of Experience for adaptive streaming service , 2013, ICC Workshops.

[37]  Rocky K. C. Chang,et al.  Measuring the quality of experience of HTTP video streaming , 2011, 12th IFIP/IEEE International Symposium on Integrated Network Management (IM 2011) and Workshops.

[38]  Yining Qi,et al.  The Effect of Frame Freezing and Frame Skipping on Video Quality , 2006, 2006 International Conference on Intelligent Information Hiding and Multimedia.

[39]  Anastasios Kourtis,et al.  An adaptive system for real-time scalable video streaming with end-to-end QOS control , 2010, 11th International Workshop on Image Analysis for Multimedia Interactive Services WIAMIS 10.

[40]  Zongpeng Li,et al.  Youtube traffic characterization: a view from the edge , 2007, IMC '07.

[41]  Ernst W. Biersack,et al.  Analyzing the impact of YouTube delivery policies on user experience , 2012, 2012 24th International Teletraffic Congress (ITC 24).

[42]  Vyas Sekar,et al.  Understanding the impact of video quality on user engagement , 2011, SIGCOMM.

[43]  Mark Claypool,et al.  The effects of jitter on the peceptual quality of video , 1999, MULTIMEDIA '99.

[44]  Iraj Sodagar,et al.  The MPEG-DASH Standard for Multimedia Streaming Over the Internet , 2011, IEEE MultiMedia.

[45]  Antonio Liotta,et al.  QoE-aware QoS management , 2008, MoMM.

[46]  Preben E. Mogensen,et al.  Subjective evaluation of packet service performance in UMTS and heterogeneous networks , 2006, Q2SWinet '06.

[47]  Markus Fiedler,et al.  Initial delay vs. interruptions: Between the devil and the deep blue sea , 2012, 2012 Fourth International Workshop on Quality of Multimedia Experience.

[48]  Mark Claypool,et al.  An empirical study of realvideo performance across the internet , 2001, IMW '01.

[49]  Phuoc Tran-Gia,et al.  A Survey on Quality of Experience of HTTP Adaptive Streaming , 2015, IEEE Communications Surveys & Tutorials.

[50]  Tobias Hoßfeld,et al.  QoE of YouTube Video Streaming for Current Internet Transport Protocols , 2014, MMB/DFT.

[51]  Gerardo Rubino,et al.  Quality of experience estimation for adaptive HTTP/TCP video streaming using H.264/AVC , 2012, 2012 IEEE Consumer Communications and Networking Conference (CCNC).

[52]  Christian Timmerer,et al.  A novel monitoring architecture for media services adaptation based on network QoS to perceived QoS mapping , 2008, Signal Image Video Process..

[53]  H Parmar,et al.  Adobe's Real Time Messaging Protocol , 2022 .

[54]  Moncef Gabbouj,et al.  Rate adaptation for adaptive HTTP streaming , 2011, MMSys.

[55]  Xiapu Luo,et al.  Inferring the QoE of HTTP video streaming from user-viewing activities , 2011, W-MUST '11.

[56]  Weisi Lin,et al.  Perceptual visual quality metrics: A survey , 2011, J. Vis. Commun. Image Represent..

[57]  Jianping Pan,et al.  Adaptive scalable video streaming in wireless networks , 2012, MMSys '12.

[58]  Fang Hao,et al.  Unreeling netflix: Understanding and improving multi-CDN movie delivery , 2012, 2012 Proceedings IEEE INFOCOM.

[59]  Phuoc Tran-Gia,et al.  Quantification of YouTube QoE via Crowdsourcing , 2011, 2011 IEEE International Symposium on Multimedia.

[60]  Amy Csizmar Dalal,et al.  Towards Real-Time Stream Quality Prediction: Predicting Video Stream Quality from Partial Stream Information , 2009, QSHINE.

[61]  Mahbub Hassan,et al.  HTTP-Based Adaptive Streaming for Mobile Clients using Markov Decision Process , 2013, 2013 20th International Packet Video Workshop.

[62]  Alessandro Neri,et al.  Objective quality evaluation of video services , 2006 .

[63]  Cisco Visual Networking Index: Forecast and Methodology 2016-2021.(2017) http://www.cisco.com/c/en/us/solutions/collateral/service-provider/visual- networking-index-vni/complete-white-paper-c11-481360.html. High Efficiency Video Coding (HEVC) Algorithms and Architectures https://jvet.hhi.fraunhofer. , 2017 .

[64]  Taner Arsan.  Review of bandwidth estimation tools and application to bandwidth adaptive video streaming , 2012, High Capacity Optical Networks and Emerging/Enabling Technologies.

[65]  Hong Ren Wu,et al.  Content-Adaptive Packet-Layer Model for Quality Assessment of Networked Video Services , 2012, IEEE Journal of Selected Topics in Signal Processing.

[66]  Filip De Turck,et al.  Enabling autonomic access network QoE management through TCP connection monitoring. , 2007 .

[67]  Adam Wolisz,et al.  Adaptation algorithm for adaptive streaming over HTTP , 2012, 2012 19th International Packet Video Workshop (PV).

[68]  Markus Fiedler,et al.  Quality of Experience from user and network perspectives , 2010, Ann. des Télécommunications.

[69]  Henning Schulzrinne,et al.  RTP: A Transport Protocol for Real-Time Applications , 1996, RFC.

[70]  Marília Curado,et al.  An Overview of Quality of Experience Measurement Challenges for Video Applications in IP Networks , 2010, WWIC.

[71]  H.-J. Zepernick,et al.  Perceptual-based Quality Metrics for Image and Video Services: A Survey , 2007, 2007 Next Generation Internet Networks.

[72]  Amy Csizmar Dalal,et al.  A New Architecture for Measuring and Assessing Streaming Media Quality , 2003 .

[73]  Vincent Barriac,et al.  Standardization activities in the ITU for a QoE assessment of IPTV , 2008, IEEE Communications Magazine.

[74]  Tobias Hoßfeld,et al.  Passive YouTube QoE Monitoring for ISPs , 2012, 2012 Sixth International Conference on Innovative Mobile and Internet Services in Ubiquitous Computing.

[75]  Yanjiao Chen,et al.  From QoS to QoE: A Survey and Tutorial on State of Art, Evolution and Future Directions of Video Quality Analysis , 2014 .

[76]  Martin Reisslein,et al.  Objective Video Quality Assessment Methods: A Classification, Review, and Performance Comparison , 2011, IEEE Transactions on Broadcasting.

[77]  O. Oyman,et al.  Quality of experience for HTTP adaptive streaming services , 2012, IEEE Communications Magazine.

[78]  Tobias Hoßfeld,et al.  From Packets to People: Quality of Experience as a New Measurement Challenge , 2013, Data Traffic Monitoring and Analysis.

[79]  Miska M. Hannuksela,et al.  Perceptual-based quality assessment for audio-visual services: A survey , 2010, Signal Process. Image Commun..

[80]  Marco Mellia,et al.  TStat: TCP STatistic and Analysis Tool , 2003, QoS-IP.

[81]  Fang Hao,et al.  A tale of three CDNs: An active measurement study of Hulu and its CDNs , 2012, 2012 Proceedings IEEE INFOCOM Workshops.

[82]  Filip De Turck,et al.  On-line Estimation of the QoE of Progressive Download Based Services in Multimedia Access Networks , 2008, International Conference on Internet Computing.