Practical Evaluation of VMAF Perceptual Video Quality for WebRTC Applications

WebRTC is the umbrella term for several emergent technologies aimed to exchange real-time media in the Web. Like other media-related services, the perceived quality of WebRTC communication can be measured using Quality of Experience (QoE) indicators. QoE assessment methods can be classified as subjective (users’ evaluation scores) or objective (models computed as a function of different parameters). In this paper, we focus on VMAF (Video Multi-method Assessment Fusion), which is an emergent full-reference objective video quality assessment model developed by Netflix. VMAF is typically used to assess video streaming services. This paper evaluates the use of VMAF in a different type of application: WebRTC. To that aim, we present a practical use case built on the top of well-known open source technologies, such as JUnit, Selenium, Docker, and FFmpeg. In addition to VMAF, we also calculate other objective QoE video metrics such as Visual Information Fidelity in the pixel domain (VIFp), Structural Similarity (SSIM), or Peak Signal-to-Noise Ratio (PSNR) applied to a WebRTC communication in different network conditions in terms of packet loss. Finally, we compare these objective results with a subjective evaluation using a Mean Opinion Score (MOS) scale to the same WebRTC streams. As a result, we found a strong correlation of the subjective video quality perceived in WebRTC video calls with the objective results computed with VMAF and VIFp in comparison with SSIM and PSNR and their variants.

[1]  Touradj Ebrahimi,et al.  Toward a New Assessment of Quality , 2015, Computer.

[2]  Martin Reisslein,et al.  Objective Video Quality Assessment Methods: A Classification, Review, and Performance Comparison , 2011, IEEE Transactions on Broadcasting.

[3]  Yanjiao Chen,et al.  From QoS to QoE: A Tutorial on Video Quality Assessment , 2015, IEEE Communications Surveys & Tutorials.

[4]  N. Lazar,et al.  The ASA Statement on p-Values: Context, Process, and Purpose , 2016 .

[5]  Weisi Lin,et al.  Visual quality assessment: recent developments, coding applications and future trends , 2013, APSIPA Transactions on Signal and Information Processing.

[6]  Mohammed Ghanbari,et al.  Scope of validity of PSNR in image/video quality assessment , 2008 .

[7]  Boni García,et al.  Understanding and estimating quality of experience in WebRTC applications , 2018, Computing.

[8]  Christian Timmerer,et al.  Challenges of QoE management for cloud applications , 2012, IEEE Communications Magazine.

[9]  Nada Philip,et al.  Multilayer perceptron neural network-based QoS-aware, content-aware and device-Aware QoE prediction model: a proposed prediction model for medical ultrasound streaming over small cell networks , 2019 .

[10]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.

[11]  Boni García,et al.  Kurento: The Swiss Army Knife of WebRTC Media Servers , 2017, IEEE Communications Standards Magazine.

[12]  Jan Nedoma,et al.  A Hybrid QoS-QoE Estimation System for IPTV Service , 2019 .

[13]  Swapna Devi,et al.  A New Method for Color Image Quality Assessment , 2011 .

[14]  Fan Zhang,et al.  Image Quality Assessment by Separately Evaluating Detail Losses and Additive Impairments , 2011, IEEE Transactions on Multimedia.