Perceptual Quality of HTTP Adaptive Streaming Strategies: Cross-Experimental Analysis of Multi-Laboratory and Crowdsourced Subjective Studies

Today's packet-switched networks are subject to bandwidth fluctuations that cause degradation of the user experience of multimedia services. In order to cope with this problem, HTTP adaptive streaming (HAS) has been proposed in recent years as a video delivery solution for the future Internet and being adopted by an increasing number of streaming services, such as Netflix and Youtube. HAS enables service providers to improve users' quality of experience (QoE) and network resource utilization by adapting the quality of the video stream to the current network conditions. However, the resulting time-varying video quality caused by adaptation introduces a new type of impairment and thus novel QoE research challenges. Despite various recent attempts to investigate these challenges, many fundamental questions regarding HAS perceptual performance are still open. In this paper, the QoE impact of different technical adaptation parameters, including chunk length, switching amplitude, switching frequency, and temporal recency, are investigated. In addition, the influence of content on perceptual quality of these parameters is analyzed. To this end, a large number of adaptation scenarios have been subjectively evaluated in four laboratory experiments and one crowdsourcing study. A statistical analysis of the combined data set reveals results that partly contradict widely held assumptions and provide novel insights in perceptual quality of adapted video sequences, e.g., interaction effects between quality switching direction (up/down) and switching strategy (smooth/abrupt). The large variety of experimental configurations across different studies ensures the consistency and external validity of the presented results that can be utilized for enhancing the perceptual performance of adaptive streaming services.

[1]  Ralf Steinmetz,et al.  Layer-encoded video in scalable adaptive streaming , 2005, IEEE Transactions on Multimedia.

[2]  Michael Seufert,et al.  The Impact of Adaptation Strategies on Perceived Quality of HTTP Adaptive Streaming , 2014, VideoNext '14.

[3]  Jun-Qing Yu,et al.  Content-Based Adaptive Transmission for Soccer Video , 2012 .

[4]  Lucjan Janowski,et al.  QoE as a Function of Frame Rate and Resolution Changes , 2010, FMN.

[5]  Narciso García,et al.  Quality of Experience of adaptive video streaming: Investigation in service parameters and subjective quality assessment methodology , 2015, Signal Process. Image Commun..

[6]  ITU-T Rec. P.910 (04/2008) Subjective video quality assessment methods for multimedia applications , 2009 .

[7]  Phuoc Tran-Gia,et al.  Quantification of YouTube QoE via Crowdsourcing , 2011, 2011 IEEE International Symposium on Multimedia.

[8]  Taner Arsan.  Review of bandwidth estimation tools and application to bandwidth adaptive video streaming , 2012, High Capacity Optical Networks and Emerging/Enabling Technologies.

[9]  Fernando Jaureguizar,et al.  Subjective assessment of the impact of transmission errors in 3DTV compared to HDTV , 2011, 2011 3DTV Conference: The True Vision - Capture, Transmission and Display of 3D Video (3DTV-CON).

[10]  Kjell Brunnström,et al.  Subjective quality assessment of an adaptive video streaming model , 2014, Electronic Imaging.

[11]  Touradj Ebrahimi,et al.  Subjective Quality Evaluation via Paired Comparison: Application to Scalable Video Coding , 2011, IEEE Transactions on Multimedia.

[12]  Philip T. Kortum,et al.  The Effect of Content Desirability on Subjective Video Quality Ratings , 2010, Hum. Factors.

[13]  Christian Timmerer,et al.  Representation Switch Smoothing for Adaptive HTTP Streaming , 2013 .

[14]  Christian Keimel,et al.  Crowdsourcing in QoE Evaluation , 2014, Quality of Experience.

[15]  Michael Seufert,et al.  Crowdsourcing 2.0: Enhancing execution speed and reliability of web-based QoE testing , 2014, 2014 IEEE International Conference on Communications (ICC).

[16]  Gabriel-Miro Muntean,et al.  Subjective Assessment of Region of Interest-Aware Adaptive Multimedia Streaming Quality , 2009, IEEE Transactions on Broadcasting.

[17]  Christian Timmerer,et al.  Using Scalable Video Coding for Dynamic Adaptive Streaming over HTTP in mobile environments , 2012, 2012 Proceedings of the 20th European Signal Processing Conference (EUSIPCO).

[18]  Rik Van de Walle,et al.  Subjective Quality Assessment of Longer Duration Video Sequences Delivered Over HTTP Adaptive Streaming to Tablet Devices , 2014, IEEE Transactions on Broadcasting.

[19]  Michael Seufert,et al.  Assessing effect sizes of influence factors towards a QoE model for HTTP adaptive streaming , 2014, 2014 Sixth International Workshop on Quality of Multimedia Experience (QoMEX).

[20]  Alexander Raake,et al.  Quality of experience and HTTP adaptive streaming: A review of subjective studies , 2014, 2014 Sixth International Workshop on Quality of Multimedia Experience (QoMEX).

[21]  Ragnhild Eg,et al.  Flicker effects in adaptive video streaming to handheld devices , 2011, ACM Multimedia.

[22]  Narciso García,et al.  Subjective Quality Study of Adaptive Streaming of Monoscopic and Stereoscopic Video , 2014, IEEE Journal on Selected Areas in Communications.

[23]  Jacob Cohen.  Statistical Power Analysis for the Behavioral Sciences , 1969, The SAGE Encyclopedia of Research Design.

[24]  Chao Gui,et al.  Content-aware adaptation scheme for QoE optimized dash applications , 2014, 2014 IEEE Global Communications Conference.

[25]  Narciso García,et al.  Effect of content characteristics on quality of experience of adaptive streaming , 2014, 2014 Sixth International Workshop on Quality of Multimedia Experience (QoMEX).

[26]  David C. Robinson,et al.  Subjective video quality assessment of HTTP adaptive streaming technologies , 2012, Bell Labs Technical Journal.

[27]  Gustavo de Veciana,et al.  Video Quality Assessment on Mobile Devices: Subjective, Behavioral and Objective Studies , 2012, IEEE Journal of Selected Topics in Signal Processing.

[28]  Filip De Turck,et al.  On the merits of SVC-based HTTP Adaptive Streaming , 2013, 2013 IFIP/IEEE International Symposium on Integrated Network Management (IM 2013).

[29]  Ralf Steinmetz,et al.  Subjective impression of variations in layer encoded videos , 2003, IWQoS'03.

[30]  George Ghinea,et al.  QoS impact on user perception and understanding of multimedia video clips , 1998, MULTIMEDIA '98.

[31]  Sugato Chakravarty,et al.  Methodology for the subjective assessment of the quality of television pictures , 1995 .

[32]  Xiapu Luo,et al.  QDASH: a QoE-aware DASH system , 2012, MMSys '12.

[33]  Christian Timmerer,et al.  Dynamic adaptive streaming over HTTP dataset , 2012, MMSys '12.

[34]  Blazej Lewcio,et al.  Video quality in next generation mobile networks — Perception of time-varying transmission , 2011, 2011 IEEE International Workshop Technical Committee on Communications Quality and Reliability (CQR).

[35]  Gustavo de Veciana,et al.  Modeling the Time—Varying Subjective Quality of HTTP Video Streams With Rate Adaptations , 2013, IEEE Transactions on Image Processing.