Efficient Volumetric Video Streaming Through Super Resolution

Volumetric videos allow viewers to exercise 6-DoF (degrees of freedom) movement when consuming fully 3D content (e.g., point clouds). Due to their truly immersive nature, streaming volumetric videos is highly bandwidth-demanding. In this work, we present to our knowledge a first volumetric video streaming system that leverages 3D super resolution (SR) of point clouds to boost the video quality on commodity devices, and to facilitate the distribution of volumetric content over bandwidth-constrained wireless networks. However, directly applying off-the-shelf 3D SR models leads to unacceptably low performance (~ 0.1 FPS even on a powerful GPU). To overcome this limitation, we propose a series of optimizations to make SR efficient. Our preliminary results indicate that for an edge-assisted (standalone mobile) setup, a small subset of our proposed optimizations can already drastically improve the FPS by a factor of 131× (53×) and reduce GPU memory usage by 83% (76%), while maintaining the same or even better SR inference accuracy, compared to using an off-the-shelf SR model.

[1]  Feng Qian,et al.  Flare: Practical Viewport-Adaptive 360-Degree Video Streaming for Mobile Devices , 2018, MobiCom.

[2]  Dan Chen.  Lossless Compression of Point-Based 3 D Models , 2022 .

[3]  Dongsu Han,et al.  NEMO: enabling neural-enhanced video streaming on commodity mobile devices , 2020, MobiCom.

[4]  Rufael Mekuria,et al.  Design, Implementation, and Evaluation of a Point Cloud Codec for Tele-Immersive Video , 2017, IEEE Transactions on Circuits and Systems for Video Technology.

[5]  Leonidas J. Guibas,et al.  The Earth Mover's Distance as a Metric for Image Retrieval , 2000, International Journal of Computer Vision.

[6]  Gary K. L. Tam,et al.  Registration of 3D Point Clouds and Meshes: A Survey from Rigid to Nonrigid , 2013, IEEE Transactions on Visualization and Computer Graphics.

[7]  Kaiqi Huang,et al.  Point Cloud Super Resolution with Adversarial Residual Graph Networks , 2019, BMVC.

[8]  Mohammad Hosseini,et al.  Dynamic Adaptive Point Cloud Streaming , 2018, PV@MMSys.

[9]  Samir R. Das,et al.  Streaming 360-Degree Videos Using Super-Resolution , 2020, IEEE INFOCOM 2020 - IEEE Conference on Computer Communications.

[10]  P. Bekaert,et al.  The Quantized kd-Tree: Efficient Ray Tracing of Compressed Point Clouds , 2006, 2006 IEEE Symposium on Interactive Ray Tracing.

[11]  Daniel Cohen-Or,et al.  PU-GAN: A Point Cloud Upsampling Adversarial Network , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[12]  Reinhard Klein,et al.  Eurographics Symposium on Point-based Graphics (2006) Octree-based Point-cloud Compression , 2022 .

[13]  Jinwoo Shin,et al.  Neural Adaptive Content-aware Internet Video Delivery , 2018, OSDI.

[14]  Daniel Cohen-Or,et al.  PU-Net: Point Cloud Upsampling Network , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[15]  Feng Qian,et al.  ViVo: visibility-aware mobile volumetric video streaming , 2020, MobiCom.

[16]  Neil A. Dodgson,et al.  Fast Marching farthest point sampling , 2003, Eurographics.

[17]  Bruno Sinopoli,et al.  A Control-Theoretic Approach for Dynamic Adaptive Video Streaming over HTTP , 2015, Comput. Commun. Rev..

[18]  Jenq-Neng Hwang,et al.  Rate-Utility Optimized Streaming of Volumetric Media for Augmented Reality , 2018, IEEE Journal on Emerging and Selected Topics in Circuits and Systems.

[19]  Ajmal Mian,et al.  Spherical Kernel for Efficient Graph Convolution on 3D Point Clouds , 2020, IEEE transactions on pattern analysis and machine intelligence.

[20]  Nico Blodow,et al.  Real-time compression of point cloud streams , 2012, 2012 IEEE International Conference on Robotics and Automation.

[21]  C. Hellge,et al.  Cloud rendering-based volumetric video streaming system for mixed reality services , 2020, MMSys.

[22]  Thomas S. Huang,et al.  Image Super-Resolution Via Sparse Representation , 2010, IEEE Transactions on Image Processing.

[23]  Ersin Yumer,et al.  Learning Blind Video Temporal Consistency , 2018, ECCV.

[24]  Feng Qian,et al.  A close examination of performance and power characteristics of 4G LTE networks , 2012, MobiSys '12.

[25]  Daniel Cohen-Or,et al.  Edge-aware point set resampling , 2013, ACM Trans. Graph..

[26]  Hong Chang,et al.  Super-resolution through neighbor embedding , 2004, Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004..

[27]  Steven Fortune,et al.  Voronoi Diagrams and Delaunay Triangulations , 2004, Handbook of Discrete and Computational Geometry, 2nd Ed..

[28]  Dongsu Han,et al.  Neural-Enhanced Live Streaming: Improving Live Video Ingest via Online Learning , 2020, SIGCOMM.

[29]  Meenakshisundaram Gopi,et al.  A Generic Scheme for Progressive Point Cloud Coding , 2008, IEEE Transactions on Visualization and Computer Graphics.

[30]  Marc Alexa,et al.  Computing and Rendering Point Set Surfaces , 2003, IEEE Trans. Vis. Comput. Graph..

[31]  Ayellet Tal,et al.  On the Visibility of Point Clouds , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[32]  Daniel Cohen-Or,et al.  Patch-Based Progressive 3D Point Set Upsampling , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[33]  Filip De Turck,et al.  Towards 6DoF HTTP Adaptive Streaming Through Point Cloud Compression , 2019, ACM Multimedia.

[34]  Feng Qian,et al.  Toward Practical Volumetric Video Streaming on Commodity Smartphones , 2019, HotMobile.