SR360: boosting 360-degree video streaming with super-resolution

360-degree videos have gained increasing popularity due to its capability to provide users with immersive viewing experience. Given the limited network bandwidth, it is a common approach to only stream video tiles in the user's Field-of-View (FoV) with high quality. However, it is difficult to perform accurate FoV prediction due to diverse user behaviors and time-varying network conditions. In this paper, we re-design the 360-degree video streaming systems by leveraging the technique of super-resolution (SR). The basic idea of our proposed SR360 framework is to utilize abundant computation resources on the user devices to trade off a reduction of network bandwidth. In the SR360 framework, a video tile with low resolution can be boosted to a video tile with high resolution using SR techniques at the client side. We adopt the theory of deep reinforcement learning (DRL) to make a set of decisions jointly, including user FoV prediction, bitrate allocation and SR enhancement. By conducting extensive trace-driven evaluations, we compare the performance of our proposed SR360 with other state-of-the-art methods and the results show that SR360 significantly outperforms other methods by at least 30% on average under different QoE metrics.

[1]  Kyoung Mu Lee,et al.  Enhanced Deep Residual Networks for Single Image Super-Resolution , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[2]  Cyril Concolato,et al.  Tiled-based adaptive streaming using MPEG-DASH , 2016, MMSys.

[3]  Mohammad Hosseini,et al.  View-aware tile-based adaptations in 360 virtual reality video streaming , 2017, 2017 IEEE Virtual Reality (VR).

[4]  Kaigui Bian,et al.  DRL360: 360-degree Video Streaming with Deep Reinforcement Learning , 2019, IEEE INFOCOM 2019 - IEEE Conference on Computer Communications.

[5]  Thomas Stockhammer,et al.  Dynamic adaptive streaming over HTTP --: standards and design principles , 2011, MMSys.

[6]  Feng Qian,et al.  Optimizing 360 video delivery over cellular networks , 2016, ATC@MobiCom.

[7]  Vyas Sekar,et al.  Improving fairness, efficiency, and stability in HTTP-based adaptive video streaming with FESTIVE , 2012, CoNEXT '12.

[8]  Alex Graves,et al.  Asynchronous Methods for Deep Reinforcement Learning , 2016, ICML.

[9]  Filip De Turck,et al.  An HTTP/2-Based Adaptive Streaming Framework for 360° Virtual Reality Videos , 2017, ACM Multimedia.

[10]  Anh Nguyen,et al.  Your Attention is Unique: Detecting 360-Degree Video Saliency in Head-Mounted Display for Head Movement Prediction , 2018, ACM Multimedia.

[11]  Gwendal Simon,et al.  360-Degree Video Head Movement Dataset , 2017, MMSys.

[12]  Cyril Concolato,et al.  MPEG DASH SRD: spatial relationship description , 2016, MMSys.

[13]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.

[14]  Filip De Turck,et al.  HTTP/2-Based Adaptive Streaming of HEVC Video Over 4G/LTE Networks , 2016, IEEE Communications Letters.

[15]  Kyoung Mu Lee,et al.  Accurate Image Super-Resolution Using Very Deep Convolutional Networks , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[16]  Marc Peter Deisenroth,et al.  Deep Reinforcement Learning: A Brief Survey , 2017, IEEE Signal Processing Magazine.

[17]  Jinwoo Shin,et al.  Neural Adaptive Content-aware Internet Video Delivery , 2018, OSDI.

[18]  Zongming Guo,et al.  CLS: A Cross-user Learning based System for Improving QoE in 360-degree Video Adaptive Streaming , 2018, ACM Multimedia.

[19]  Feng Li,et al.  Rubiks: Practical 360-Degree Streaming for Smartphones , 2018, MobiSys.

[20]  Junchen Jiang,et al.  Pano: optimizing 360° video streaming with a better understanding of quality perception , 2019, SIGCOMM.

[21]  Yusheng Ji,et al.  Plato: Learning-based Adaptive Streaming of 360-Degree Videos , 2018, 2018 IEEE 43rd Conference on Local Computer Networks (LCN).