Scalable 3D video of dynamic scenes

In this paper we present a scalable 3D video framework for capturing and rendering dynamic scenes. The acquisition system is based on multiple sparsely placed 3D video bricks, each comprising a projector, two grayscale cameras, and a color camera. Relying on structured light with complementary patterns, texture images and pattern-augmented views of the scene are acquired simultaneously by time-multiplexed projections and synchronized camera exposures. Using space–time stereo on the acquired pattern images, high-quality depth maps are extracted, whose corresponding surface samples are merged into a view-independent, point-based 3D data structure. This representation allows for effective photo-consistency enforcement and outlier removal, leading to a significant decrease of visual artifacts and a high resulting rendering quality using EWA volume splatting. Our framework and its view-independent representation allow for simple and straightforward editing of 3D video. In order to demonstrate its flexibility, we show compositing techniques and spatiotemporal effects.

[1]  Luc Van Gool,et al.  ATTEST: Advanced Three-dimensional Television System Technologies , 2002 .

[2]  Wojciech Matusik,et al.  3D TV: a scalable system for real-time acquisition, transmission, and autostereoscopic display of dynamic scenes , 2004, ACM Trans. Graph..

[3]  Matthias Zwicker,et al.  Surfels: surface elements as rendering primitives , 2000, SIGGRAPH.

[4]  G. Iddan,et al.  3D IMAGING IN THE STUDIO (AND ELSEWHERE...) , 2001 .

[5]  Matthias Zwicker,et al.  Ieee Transactions on Visualization and Computer Graphics Ewa Splatting , 2002 .

[6]  R. Cipolla,et al.  A probabilistic framework for space carving , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.

[7]  In Kyu Park,et al.  Depth image-based representations for static and animated 3D objects , 2002, Proceedings. International Conference on Image Processing.

[8]  Leonard McMillan,et al.  A Real-Time Distributed Light Field Camera , 2002, Rendering Techniques.

[9]  Markus H. Gross,et al.  3D video fragments: dynamic point samples for real-time free-viewpoint video , 2004, Comput. Graph..

[10]  Richard Szeliski,et al.  Boundary Matting for View Synthesis , 2004, CVPR Workshops.

[11]  Matthias Zwicker,et al.  3 Ideal Resampling 3 . 1 Sampling and Aliasing , 2022 .

[12]  Richard Szeliski,et al.  The lumigraph , 1996, SIGGRAPH.

[13]  Takeo Kanade,et al.  Spatio-Temporal View Interpolation , 2002, Rendering Techniques.

[14]  Nelson L. Max,et al.  Image-based rendering of range data with estimated depth uncertainty , 2004, IEEE Computer Graphics and Applications.

[15]  W. P. Cockshott,et al.  Experimental 3-D digital TV studio , 2003 .

[16]  Marcus A. Magnor,et al.  Hardware-Accelerated Dynamic Light Field Rendering , 2002, VMV.

[17]  Markus H. Gross,et al.  Efficient simplification of point-sampled surfaces , 2002, IEEE Visualization, 2002. VIS 2002..

[18]  Marc Alexa,et al.  Point-based computer graphics , 2004, SIGGRAPH '04.

[19]  Markus H. Gross,et al.  3D video recorder , 2002, 10th Pacific Conference on Computer Graphics and Applications, 2002. Proceedings..

[20]  Tim Weyrich,et al.  Post-processing of Scanned 3D Surface Data , 2004, PBG.

[21]  Michael Bosse,et al.  Unstructured lumigraph rendering , 2001, SIGGRAPH.

[22]  Luc Van Gool,et al.  Blue-c: a spatially immersive display and 3D video portal for telepresence , 2003, IPT/EGVE.

[23]  D. Levin,et al.  Mesh-Independent Surface Interpolation , 2004 .

[24]  Marc Levoy,et al.  High performance imaging using large camera arrays , 2005, SIGGRAPH 2005.

[25]  Richard Szeliski,et al.  A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms , 2001, International Journal of Computer Vision.

[26]  Li Zhang,et al.  Spacetime stereo: shape recovery for dynamic scenes , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[27]  Wojciech Matusik,et al.  Polyhedral Visual Hulls for Real-Time Rendering , 2001, Rendering Techniques.

[28]  Markus H. Gross,et al.  Spectral processing of point-sampled geometry , 2001, SIGGRAPH.

[29]  Marc Levoy,et al.  Light field rendering , 1996, SIGGRAPH.

[30]  Richard Szeliski,et al.  Boundary Matting for View Synthesis , 2004, 2004 Conference on Computer Vision and Pattern Recognition Workshop.

[31]  Richard Szeliski,et al.  Layered depth images , 1998, SIGGRAPH.

[32]  Hans-Peter Seidel,et al.  Free-viewpoint video of human actors , 2003, ACM Trans. Graph..

[33]  Steven M. Seitz,et al.  Spacetime faces , 2004, ACM Trans. Graph..

[34]  Markus H. Gross,et al.  Embedding imperceptible patterns into projected images for simultaneous acquisition and display , 2004, Third IEEE and ACM International Symposium on Mixed and Augmented Reality.

[35]  Richard Szeliski,et al.  High-quality video view interpolation using a layered representation , 2004, SIGGRAPH 2004.

[36]  Takeo Kanade,et al.  Virtualized Reality: Constructing Virtual Worlds from Real Scenes , 1997, IEEE Multim..

[37]  Kostas Daniilidis,et al.  View-independent scene acquisition for tele-presence , 2000, Proceedings IEEE and ACM International Symposium on Augmented Reality (ISAR 2000).

[38]  Takeo Kanade,et al.  A multibaseline stereo system with active illumination and real-time image acquisition , 1995, Proceedings of IEEE International Conference on Computer Vision.

[39]  Ramesh Raskar,et al.  Image-based visual hulls , 2000, SIGGRAPH.