Neural Enhancement in Content Delivery Systems: The State-of-the-Art and Future Directions

Internet-enabled smartphones and ultra-wide displays are transforming a variety of visual apps spanning from on-demand movies and 360° videos to video-conferencing and live streaming. However, robustly delivering visual content under fluctuating networking conditions on devices of diverse capabilities remains an open problem. In recent years, advances in the field of deep learning on tasks such as superresolution and image enhancement have led to unprecedented performance in generating high-quality images from low-quality ones, a process we refer to as neural enhancement. In this paper, we survey state-of-the-art content delivery systems that employ neural enhancement as a key component in achieving both fast response time and high visual quality. We first present the deployment challenges of neural enhancement models. We then cover systems targeting diverse use-cases and analyze their design decisions in overcoming technical challenges. Moreover, we present promising directions based on the latest insights from deep learning research to further boost the quality of experience of these systems.

[1]  Cheng-Hsin Hsu,et al.  Fixation Prediction for 360° Video Streaming in Head-Mounted Virtual Reality , 2017, NOSSDAV.

[2]  Jiwon Kim,et al.  Fast Adaptation to Super-Resolution Networks via Meta-Learning , 2020, ECCV.

[3]  Paramvir Bahl,et al.  Focus: Querying Large Video Datasets with Low Latency and Low Cost , 2018, OSDI.

[4]  Michal Irani,et al.  "Zero-Shot" Super-Resolution Using Deep Internal Learning , 2017, CVPR.

[5]  Xin Deng.  Enhancing Image Quality via Style Transfer for Single Image Super-Resolution , 2018, IEEE Signal Processing Letters.

[6]  Kyung-Ah Sohn,et al.  Fast, Accurate, and, Lightweight Super-Resolution with Cascading Residual Network , 2018, ECCV.

[7]  Nicholas D. Lane,et al.  MobiSR: Efficient On-Device Super-Resolution through Heterogeneous Mobile Processors , 2019, MobiCom.

[8]  Nam Ik Cho,et al.  Meta-Transfer Learning for Zero-Shot Super-Resolution , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[9]  Michal Irani,et al.  Across Scales \& Across Dimensions: Temporal Super-Resolution using Deep Internal Learning , 2020, ECCV.

[10]  Jan P. Allebach,et al.  Zooming Slow-Mo: Fast and Accurate One-Stage Space-Time Video Super-Resolution , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[11]  Paramvir Bahl,et al.  Glimpse: Continuous, Real-Time Object Recognition on Mobile Devices , 2015, SenSys.

[12]  Jinwoo Shin,et al.  Neural Adaptive Content-aware Internet Video Delivery , 2018, OSDI.

[13]  Xiaojuan Qi,et al.  ICNet for Real-Time Semantic Segmentation on High-Resolution Images , 2017, ECCV.

[14]  Keith Winstein,et al.  Salsify: Low-Latency Network Video through Tighter Integration between a Video Codec and a Transport Protocol , 2018, NSDI.

[15]  Houqiang Li,et al.  Learning a Convolutional Neural Network for Image Compact-Resolution , 2019, IEEE Transactions on Image Processing.

[16]  Tie-Yan Liu,et al.  Invertible Image Rescaling , 2020, ECCV.

[17]  Sem C. Borst,et al.  Distributed Caching Algorithms for Content Distribution Networks , 2010, 2010 Proceedings IEEE INFOCOM.

[18]  Bruno Sinopoli,et al.  A Control-Theoretic Approach for Dynamic Adaptive Video Streaming over HTTP , 2015, Comput. Commun. Rev..

[19]  Junchen Jiang,et al.  Bridging the Edge-Cloud Barrier for Real-time Advanced Vision Analytics , 2019, HotCloud.

[20]  L. Ginsberg,et al.  Cisco Systems , 2003 .

[21]  Xiaoou Tang,et al.  Image Super-Resolution Using Deep Convolutional Networks , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[22]  Luc Van Gool,et al.  AI Benchmark: All About Deep Learning on Smartphones in 2019 , 2019, 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW).

[23]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.

[24]  Nicholas D. Lane,et al.  Journey Towards Tiny Perceptual Super-Resolution , 2020, ECCV.

[25]  Pan Hu,et al.  Dejavu: Enhancing Videoconferencing with Prior Knowledge , 2019, HotMobile.

[26]  Christian Ledig,et al.  Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[27]  Jie Li,et al.  Binarized Neural Network for Single Image Super Resolution , 2020, ECCV.

[28]  Yochai Blau,et al.  The Perception-Distortion Tradeoff , 2017, CVPR.

[29]  Jie Zhou,et al.  Structure-Preserving Super Resolution With Gradient Guidance , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[30]  Te-Yuan Huang,et al.  A buffer-based approach to rate adaptation: evidence from a large video streaming service , 2015, SIGCOMM 2015.

[31]  Samir R. Das,et al.  Streaming 360-Degree Videos Using Super-Resolution , 2020, IEEE INFOCOM 2020 - IEEE Conference on Computer Communications.

[32]  Kyoung Mu Lee,et al.  Enhanced Deep Residual Networks for Single Image Super-Resolution , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[33]  Xiaoou Tang,et al.  Deep Network Interpolation for Continuous Imagery Effect Transition , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[34]  Xinbo Gao,et al.  Fast and Accurate Single Image Super-Resolution via Information Distillation Network , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[35]  Alan C. Bovik,et al.  Making a “Completely Blind” Image Quality Analyzer , 2013, IEEE Signal Processing Letters.

[36]  Aakanksha Chowdhery,et al.  Server-Driven Video Streaming for Deep Learning Inference , 2020, SIGCOMM.

[37]  Vyas Sekar,et al.  Improving fairness, efficiency, and stability in HTTP-based adaptive video streaming with FESTIVE , 2012, CoNEXT '12.

[38]  Alexei A. Efros,et al.  The Unreasonable Effectiveness of Deep Features as a Perceptual Metric , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[39]  Kyoung Mu Lee,et al.  Task-Aware Image Downscaling , 2018, ECCV.

[40]  Lizhuang Ma,et al.  Efficient Super Resolution Using Binarized Neural Network , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[41]  Yu Qiao,et al.  ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks , 2018, ECCV Workshops.

[42]  L. Rudin,et al.  Nonlinear total variation based noise removal algorithms , 1992 .

[43]  Xiaoou Tang,et al.  Accelerating the Super-Resolution Convolutional Neural Network , 2016, ECCV.

[44]  Sergey Ioffe,et al.  Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift , 2015, ICML.

[45]  Justin Manweiler,et al.  OverLay: Practical Mobile Augmented Reality , 2015, MobiSys.

[46]  Victor Lempitsky,et al.  Fast Bi-layer Neural Synthesis of One-Shot Realistic Head Avatars , 2020, ECCV.

[47]  Nicholas D. Lane,et al.  Poster: MobiSR -- Efficient On-Device Super-Resolution through Heterogeneous Mobile Processors , 2019, MobiCom.

[48]  Ilias Leontiadis,et al.  EmBench: Quantifying Performance Variations of Deep Neural Networks across Modern Commodity Devices , 2019, EMDL '19.

[49]  Hongzi Mao,et al.  Neural Adaptive Video Streaming with Pensieve , 2017, SIGCOMM.

[50]  Marco Gruteser,et al.  Edge Assisted Real-time Object Detection for Mobile Augmented Reality , 2019, MobiCom.

[51]  Quoc V. Le,et al.  Searching for MobileNetV3 , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[52]  Seongwon Kim,et al.  Supremo: Cloud-Assisted Low-Latency Super-Resolution in Mobile Devices , 2020, IEEE Transactions on Mobile Computing.

[53]  Dongsu Han,et al.  How will Deep Learning Change Internet Video Delivery? , 2017, HotNets.

[54]  Paramvir Bahl,et al.  Live Video Analytics at Scale with Approximation and Delay-Tolerance , 2017, NSDI.

[55]  Nam Ik Cho,et al.  Natural and Realistic Single Image Super-Resolution With Explicit Natural Manifold Discrimination , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[56]  Yun Fu,et al.  Image Super-Resolution Using Very Deep Residual Channel Attention Networks , 2018, ECCV.

[57]  Xu Jia,et al.  Efficient Residual Dense Block Search for Image Super-Resolution , 2020, AAAI.

[58]  Kyoung Mu Lee,et al.  Accurate Image Super-Resolution Using Very Deep Convolutional Networks , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[59]  Dongsu Han,et al.  Neural-Enhanced Live Streaming: Improving Live Video Ingest via Online Learning , 2020, SIGCOMM.