Immersive 3D Telepresence

Cutting-edge work on 3D telepresence at a multinational research center provides insight into the technology's potential, as well as into its remaining challenges. The first Web extra at http://youtu.be/r4SqJdXkOjQ is a video describing FreeCam, a system capable of generating live free-viewpoint video by simulating the output of a virtual camera moving through a dynamic scene. The second Web extra at http://youtu.be/Dw1glKUKs9A is a video showing a system designed to capture the enhanced 3D structure of a room-sized dynamic scene with commodity depth cameras, such as Microsoft Kinects. The third Web extra at http://youtu.be/G_VPzXRrmIw is a video demonstrating a system that adapts to a wide variety of telepresence scenarios. By combining Kinect-based 3D scanning with optical see-through HMDs, the user can precisely control which parts of the scene are real and which are virtual or remote objects. The fourth Web extra at http://youtu.be/n45N5AHsoCI is a video demonstrating a method based on moving least squares surfaces that robustly and efficiently reconstructs dynamic scenes captured by a set of hybrid color+depth cameras. The reconstruction provides spatiotemporal consistency and seamlessly fuses color and geometric information. The video also illustrates the formulation on a variety of real sequences and demonstrates that it favorably compares to state-of-the-art methods. The fifth Web extra at http://youtu.be/OSl3f2qZzKs is a video demonstrating a 3D acquisition system capable of simultaneously capturing an entire room-sized volume with an array of commodity depth cameras and rendering it from a novel viewpoint in real time. The sixth Web extra at http://youtu.be/zKWByH7evo0 is a video demonstrating a gaze-correction approach based on a single Kinect sensor that preserves both the integrity and expressiveness of the face as well as the fidelity of the scene as a whole, producing nearly artifact-free imagery. The method is suitable for mainstream home video conferencing: it uses inexpensive consumer hardware, achieves real-time performance, and requires just a simple and short setup.