CrowdSR: enabling high-quality video ingest in crowdsourced livecast via super-resolution

The prevalence of personal devices motivates the rapid development of crowdsourced livecast in recent years. However, there exists huge diversity of upstream bandwidth among amateur broadcasters. Moreover, the highest video quality that can be streamed is limited by the hardware configuration of broadcaster devices (e.g., 540p for low-end mobile devices). The above factors pose significant challenges to the ingestion of high-resolution live video streams, and result in poor quality-of-experience (QoE) for viewers. In this paper, we propose a novel live video ingest approach called CrowdSR for crowdsourced livecast. CrowdSR can transform a low-resolution video stream uploaded by weak devices into a high-resolution video stream via super-resolution, and then deliver the stream to viewers. CrowdSR can exploit crowdsourced high-resolution video patches from similar broadcasters to speedup model training. Different from previous work, our approach does not require any modification at the client side, and thus is more practical and easy to implement. Finally, we implement and evaluate CrowdSR by conducting a series of real-world experiments. The results show that CrowdSR significantly outperforms the baseline approaches by 0.42-1.09 dB in terms of PSNR and 0.006-0.014 in terms of SSIM.

[1]  Miao Hu,et al.  SR360: boosting 360-degree video streaming with super-resolution , 2020, NOSSDAV.

[2]  Haitian Pang,et al.  Content Harvest Network: Optimizing First Mile for Crowdsourced Live Streaming , 2019, IEEE Transactions on Circuits and Systems for Video Technology.

[3]  Dongsu Han,et al.  Neural-Enhanced Live Streaming: Improving Live Video Ingest via Online Learning , 2020, SIGCOMM.

[4]  Chih-Yu Wang,et al.  Live Video Streaming with Joint User Association and Caching Placement in Mobile Edge Computing , 2020, 2020 International Conference on Computing, Networking and Communications (ICNC).

[5]  Jian-Wen Chen,et al.  Introduction to H.264 advanced video coding , 2006, Asia and South Pacific Conference on Design Automation, 2006..

[6]  Ravi Netravali,et al.  Edge-based Transcoding for Adaptive Live Video Streaming , 2019, HotEdge.

[7]  Qi Qi,et al.  Vabis: Video Adaptation Bitrate System for Time-Critical Live Streaming , 2020, IEEE Transactions on Multimedia.

[8]  Viktor Gruev,et al.  Bilinear and bicubic interpolation methods for division of focal plane polarimeters. , 2011, Optics express.

[9]  Yiannis Kompatsiaris,et al.  ViSiL: Fine-Grained Spatio-Temporal Video Similarity Learning , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[10]  Kyoung Mu Lee,et al.  Accurate Image Super-Resolution Using Very Deep Convolutional Networks , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[11]  Gang Peng,et al.  CDN: Content Distribution Network , 2004, ArXiv.

[12]  Jiashi Feng,et al.  Central Similarity Quantization for Efficient Image and Video Retrieval , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[13]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[14]  Yi-Hsuan Hung,et al.  Optimizing Social Welfare of Live Video Streaming Services in Mobile Edge Computing , 2020, IEEE Transactions on Mobile Computing.

[15]  Nicholas D. Lane,et al.  Neural Enhancement in Content Delivery Systems: The State-of-the-Art and Future Directions , 2020, DistributedML@CoNEXT.

[16]  Lifeng Sun,et al.  Look Ahead at the First-mile in Livecast with Crowdsourced Highlight Prediction , 2020, IEEE INFOCOM 2020 - IEEE Conference on Computer Communications.

[17]  Yifan Zhang,et al.  A Distance-Driven Alliance for a P2P Live Video System , 2020, IEEE Transactions on Multimedia.

[18]  Mohamed Hefeeda,et al.  Mobile Streaming of Live 360-Degree Videos , 2020, IEEE Transactions on Multimedia.

[19]  Srinivasan Seshan,et al.  Vantage: optimizing video upload for time-shifted viewing of social live streams , 2019, SIGCOMM.

[20]  Zhimin Xu,et al.  QoE-Driven Adaptive K-Push for HTTP/2 Live Streaming , 2019, IEEE Transactions on Circuits and Systems for Video Technology.

[21]  Haihong Zhu,et al.  Optimal Strategies for Live Video Streaming in the Low-latency Regime , 2019, 2019 IEEE 27th International Conference on Network Protocols (ICNP).

[22]  Yida Wang,et al.  Optimizing CNN Model Inference on CPUs , 2018, USENIX Annual Technical Conference.

[23]  Pan Hu,et al.  Dejavu: Enhancing Videoconferencing with Prior Knowledge , 2019, HotMobile.

[24]  Meng Wang,et al.  Video Retrieval with Similarity-Preserving Deep Temporal Hashing , 2019, ACM Trans. Multim. Comput. Commun. Appl..

[25]  Michal Irani,et al.  "Zero-Shot" Super-Resolution Using Deep Internal Learning , 2017, CVPR.

[26]  Wei Liu,et al.  SSD: Single Shot MultiBox Detector , 2015, ECCV.

[27]  Jinwoo Shin,et al.  Neural Adaptive Content-aware Internet Video Delivery , 2018, OSDI.

[28]  Samir R. Das,et al.  Streaming 360-Degree Videos Using Super-Resolution , 2020, IEEE INFOCOM 2020 - IEEE Conference on Computer Communications.

[29]  Feng Qian,et al.  Efficient Volumetric Video Streaming Through Super Resolution , 2021, HotMobile.

[30]  Zhenzhong Chen,et al.  Learned Image Downscaling for Upscaling Using Content Adaptive Resampler , 2019, IEEE Transactions on Image Processing.

[31]  Kyoung Mu Lee,et al.  Enhanced Deep Residual Networks for Single Image Super-Resolution , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[32]  Eirikur Agustsson,et al.  NTIRE 2017 Challenge on Single Image Super-Resolution: Dataset and Study , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[33]  Ali Farhadi,et al.  You Only Look Once: Unified, Real-Time Object Detection , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[34]  Lei Zhang,et al.  Continuous Bitrate & Latency Control with Deep Reinforcement Learning for Live Video Streaming , 2019, ACM Multimedia.

[35]  Lili Qiu,et al.  Jigsaw: Robust Live 4K Video Streaming , 2019, MobiCom.

[36]  Zhong Ming,et al.  TCLiVi: Transmission Control in Live Video Streaming Based on Deep Reinforcement Learning , 2021, IEEE Transactions on Multimedia.