3D video acquisition, representation & editing

3D video is a new kind of media that captures three-dimensional appearance and dynamics of real-world scenes by combining data from multiple input video streams to a consistent model. During playback, it provides the viewer with the possibility of freely choosing the viewpoint of a virtual camera in space and time. In this thesis, we present a 3D video system covering the full processing pipeline from acquisition over data representation, post-processing and editing up to high-quality display. The focus of our research lies on 3D video data representations. We investigate both image-based and geometric models. While the first show advantages in achievable output image quality, the latter provide more flexibility for applications beyond pure playback. In particular, we investigate methods for data streaming and compression, dynamic out-of-core data structures, and interactive 3D video editing. For data acquisition we introduce a scalable system based on multiple sparsely placed 3D video bricks. Each brick captures high-quality depth maps of the scene from its respective viewpoint, using a combination of spacetime stereo vision and structured light projection. Texture images and pattern-augmented views are acquired simultaneously by time-multiplexed projections and synchronized camera exposures. Our image space representation consisting of billboard planes augmented by detailed displacement maps combines the generality of acquired geometry with the regularization properties of purely image-based methods. Being placed in the disparity space of the acquisition cameras, the billboards provide a regular sampling of the scene with a uniform error model. Based on that, we propose a geometry filtering method which generates spatially and temporally coherent models and removes reconstruction noise as well as calibration errors. Rendering is performed using a GPU-accelerated algorithm which generates consistent viewdependent geometry and texture for each individual viewpoint. Point samples are the fundamental primitive of our second representation. By carrying multiple surface attributes such as position and color, they provide a unified model of geometry and appearance of natural scenes. This representation allows for application of various post-processing algorithms for improving noisy input geometry. In particular, we propose a method that effectively removes outliers by enforcing photo consistency with all input views. By augmenting the points by a statistical model of acquisition noise, smooth images of the scene from novel viewpoints can be generated using a probabilistic renderer based on GPU-accelerated EWA volume splatting.