Enabling Geometry Based 3 D Tele-Immersion with Real-Time Mesh Compression and Linear Rateless Coding

3D Tele-immersion enables participants in remote locations to share, in real-time, an activity. It offers users interactive and immersive experiences, but it challenges current networking solutions. Work in the past has mainly focused on the efficient delivery of image-based 3D videos and on the realistic rendering and reconstruction of geometry-based 3D objects. The contribution of this paper is a real-time streaming component for 3D Tele-Immersion with reconstructed geometry. This component includes both a novel fast compression method and a rate-less packet protection scheme specifically designed towards the requirements imposed by real-time transmission of live-reconstructed mesh geometry. Tests on a large dataset show an encoding and decoding speed-up of upto 10 times at comparable compression and quality rates, when compared to the high-end MPEG-4 SC3DMC mesh encoders. The implemented rate-less code ensures complete packet loss protection of the triangle mesh object and a delivery delay within interactive delay bounds. Contrary to most linear fountain codes, the designed codec enables real-time progressive decoding allowing partial decoding each time a packet is received. This approach is compared to a transmission over TCP and heavily outperforms it in packet loss rates and latencies typical in managed WAN and MAN networks. The component has been integrated into a larger environment that includes state of the art 3D reconstruction and rendering modules. This resulted in a prototype that can capture, compress transmit and render triangle mesh geometry in realtime in realistic internet conditions as shown in experiments. Low interactive end-to-end delay and frame rates over 3 times higher compared to alternative methods are achieved.