A pipeline for multiparty volumetric video conferencing: transmission of point clouds over low latency DASH

The advent of affordable 3D capture and display hardware is making volumetric videoconferencing feasible. This technology increases the immersion of the participants, breaking the flat restriction of 2D screens, by allowing them to collaborate and interact in shared virtual reality spaces. In this paper we introduce the design and development of an architecture intended for volumetric videoconferencing that provides a highly realistic 3D representation of the participants, based on pointclouds. A pointcloud representation is suitable for real-time applications like video conferencing, due to its low-complexity and because it does not need a time consuming reconstruction process. As transport protocol we selected low latency DASH, due to its popularity and client-based adaptation mechanisms for tiling. This paper presents the architectural design, details the implementation, and provides some referential results. The demo will showcase the system in action, enabling volumetric videoconferencing using pointclouds.

[1]  Cyril Concolato,et al.  Overhead and performance of low latency live streaming using MPEG-DASH , 2014, IISA 2014, The 5th International Conference on Information, Intelligence, Systems and Applications.

[2]  Michele Sanna,et al.  A 3D tele-immersion system based on live captured mesh geometry , 2013, MMSys.

[3]  Petros Daras,et al.  Toward Real-Time and Efficient Compression of Human Time-Varying Meshes , 2014, IEEE Transactions on Circuits and Systems for Video Technology.

[4]  Viswanathan Swaminathan,et al.  Low latency live video streaming using HTTP chunked encoding , 2011, 2011 IEEE 13th International Workshop on Multimedia Signal Processing.

[5]  Alvaro Collet,et al.  High-quality streamable free-viewpoint video , 2015, ACM Trans. Graph..

[6]  Rufael Mekuria,et al.  Network support for social 3-D immersive tele-presence with highly realistic natural and synthetic avatar users , 2015, MMVE@MMSys.

[7]  Oliver Schreer,et al.  An immersive 3D video-conferencing system using shared virtual team user environments , 2002, CVE '02.

[8]  Rufael Mekuria,et al.  Objective and subjective quality assessment of geometry compression of reconstructed 3D humans in a 3D virtual room , 2015, SPIE Optical Engineering + Applications.

[9]  Rufael Mekuria,et al.  Design, Implementation, and Evaluation of a Point Cloud Codec for Tele-Immersive Video , 2017, IEEE Transactions on Circuits and Systems for Video Technology.

[10]  Feng Qian,et al.  Toward Practical Volumetric Video Streaming on Commodity Smartphones , 2019, HotMobile.

[11]  Rufael Mekuria,et al.  Emerging MPEG Standards for Point Cloud Compression , 2019, IEEE Journal on Emerging and Selected Topics in Circuits and Systems.

[12]  Ruzena Bajcsy,et al.  Color-plus-depth level-of-detail in 3D tele-immersive video: a psychophysical approach , 2011, MM '11.