ViVo: visibility-aware mobile volumetric video streaming

In this paper, we perform a first comprehensive study of mobile volumetric video streaming. Volumetric videos are truly 3D, allowing six degrees of freedom (6DoF) movement for their viewers during playback. Such flexibility enables numerous applications in entertainment, healthcare, education, etc. However, volumetric video streaming is extremely bandwidth-intensive. We conduct a detailed investigation of each of the following aspects for point cloud streaming (a popular volumetric data format): encoding, decoding, segmentation, viewport movement patterns, and viewport prediction. Motivated by the observations from the above study, we propose ViVo, which is to the best of our knowledge the first practical mobile volumetric video streaming system with three visibility-aware optimizations. ViVo judiciously determines the video content to fetch based on how, what and where a viewer perceives for reducing bandwidth consumption of volumetric video streaming. Our evaluations over real wireless networks (including commercial 5G), mobile devices and users indicate that ViVo can save on average 40% of data usage (up to 80%) with virtually no drop in visual quality.

[1]  GovindanRamesh,et al.  Augmented Vehicular Reality , 2019 .

[2]  Olivier Devillers,et al.  Geometric compression for interactive transmission , 2000, Proceedings Visualization 2000. VIS 2000 (Cat. No.00CH37145).

[3]  Meenakshisundaram Gopi,et al.  A Generic Scheme for Progressive Point Cloud Coding , 2008, IEEE Transactions on Visualization and Computer Graphics.

[4]  Rufael Mekuria,et al.  Emerging MPEG Standards for Point Cloud Compression , 2019, IEEE Journal on Emerging and Selected Topics in Circuits and Systems.

[5]  Reinhard Klein,et al.  Eurographics Symposium on Point-based Graphics (2006) Octree-based Point-cloud Compression , 2022 .

[6]  Andrew Woo,et al.  Fast ray-box intersection , 1990 .

[7]  Feng Li,et al.  Rubiks: Practical 360-Degree Streaming for Smartphones , 2018, MobiSys.

[8]  Reinhard Klein,et al.  Real-time point cloud compression , 2015, 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[9]  Dinesh Manocha,et al.  VolCCD: Fast continuous collision culling between deforming volume meshes , 2011, TOGS.

[10]  Feng Qian,et al.  Flare: Practical Viewport-Adaptive 360-Degree Video Streaming for Mobile Devices , 2018, MobiCom.

[11]  Rufael Mekuria,et al.  Design, Implementation, and Evaluation of a Point Cloud Codec for Tele-Immersive Video , 2017, IEEE Transactions on Circuits and Systems for Video Technology.

[12]  T. V. Geetha,et al.  Indian Logic Ontology based Automatic Query Refinement , 2008 .

[13]  Jihad El-Sana,et al.  Skip Strips: maintaining triangle strips for view-dependent rendering , 1999, Proceedings Visualization '99 (Cat. No.99CB37067).

[14]  Dan Chen.  Lossless Compression of Point-Based 3 D Models , 2022 .

[15]  Jenq-Neng Hwang,et al.  Volumetric Media Streaming for Augmented Reality , 2018, 2018 IEEE Global Communications Conference (GLOBECOM).

[16]  R. Basri,et al.  Direct visibility of point sets , 2007, SIGGRAPH 2007.

[17]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.

[18]  C.-C. Jay Kuo,et al.  Technologies for 3D mesh compression: A survey , 2005, J. Vis. Commun. Image Represent..

[19]  Ruzena Bajcsy,et al.  Multi-camera tele-immersion system with real-time model driven data compression , 2009, The Visual Computer.

[20]  Niloy J. Mitra,et al.  Visibility of noisy point cloud data , 2010, Comput. Graph..

[21]  P. Bekaert,et al.  The Quantized kd-Tree: Efficient Ray Tracing of Compressed Point Clouds , 2006, 2006 IEEE Symposium on Interactive Ray Tracing.

[22]  Ayellet Tal,et al.  On the Visibility of Point Clouds , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[23]  Jan-Michael Frahm,et al.  Structure-from-Motion Revisited , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[24]  A. V. Olgac,et al.  Performance Analysis of Various Activation Functions in Generalized MLP Architectures of Neural Networks , 2011 .

[25]  Touradj Ebrahimi,et al.  Point Cloud Quality Assessment Metric Based on Angular Similarity , 2018, 2018 IEEE International Conference on Multimedia and Expo (ICME).

[26]  Gabriel Taubin,et al.  A Framework for Streaming Geometry in VRML , 1999, IEEE Computer Graphics and Applications.

[27]  Nico Blodow,et al.  Real-time compression of point cloud streams , 2012, 2012 IEEE International Conference on Robotics and Automation.

[28]  Mohammad Hosseini,et al.  Dynamic Adaptive Point Cloud Streaming , 2018, PV@MMSys.

[29]  Vyas Sekar,et al.  Improving fairness, efficiency, and stability in HTTP-based adaptive video streaming with FESTIVE , 2012, CoNEXT '12.

[30]  Alec Wolman,et al.  Demo: Kahawai: high-quality mobile gaming using GPU offload , 2015, MobiSys.

[31]  Jorge Nocedal,et al.  On the limited memory BFGS method for large scale optimization , 1989, Math. Program..

[32]  Céline Hudelot,et al.  3D Mesh Compression , 2015, ACM Comput. Surv..

[33]  Charles T. Loop,et al.  Holoportation: Virtual 3D Teleportation in Real-time , 2016, UIST.

[34]  Feng Qian,et al.  Toward Practical Volumetric Video Streaming on Commodity Smartphones , 2019, HotMobile.

[35]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[36]  Marco Gruteser,et al.  Cutting the Cord: Designing a High-quality Untethered VR System with Low Latency Remote Rendering , 2018, MobiSys.

[37]  Xin Liu,et al.  Shooting a moving target: Motion-prediction-based transmission for 360-degree videos , 2016, 2016 IEEE International Conference on Big Data (Big Data).

[38]  Leif Kobbelt,et al.  A survey of point-based techniques in computer graphics , 2004, Comput. Graph..

[39]  Kalyan Sunkavalli,et al.  Fast Spatially-Varying Indoor Lighting Estimation , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[40]  Alexei A. Efros,et al.  Occlusion-Aware Depth Estimation Using Light-Field Cameras , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[41]  Y. Charlie Hu,et al.  Furion: Engineering High-Quality Immersive Virtual Reality on Today's Mobile Devices , 2017, IEEE Transactions on Mobile Computing.

[42]  Emil Dumic,et al.  Subjective evaluation and objective measures for point clouds — State of the art , 2018, 2018 First International Colloquium on Smart Grid Metrology (SmaGriMet).

[43]  Marc Levoy,et al.  QSplat: a multiresolution point rendering system for large meshes , 2000, SIGGRAPH.

[44]  Marek Kowalski,et al.  Livescan3D: A Fast and Inexpensive 3D Data Acquisition System for Multiple Kinect v2 Sensors , 2015, 2015 International Conference on 3D Vision.