Towards On-Line Digital Doubles

We present a modular system for real-time 3D-scanning of human bodies under motion. The high-resolution shape and colour appearance is captured by several scanning units positioned around the object of interest. Each of these units performs a foreground-background segmentation and computes a valid depth-range for the spatially neighbouring units. Multiple depth-ranges are combined in a visual hull representation, which limits the search-range for the 3D-reconstruction. Depth-estimation is based on a hierarchical multi-view-stereo plane sweep approach. Robustness and accuracy is increased by incorporating imperceptible infrared illumination as well as adding local pixel gradient information. All parts of the processing pipeline, involving camera color conversions, segmentation, depth-range computation, visual-hull generation, lossless image compression, network transfer of the infrared and colour images, and the plane sweep algorithm, are implemented on the GPU and highly optimized for speed, allowing scanning times of less than 40 ms per frame. Experimental results demonstrate the applicability of our system to the creation of high-density on-line digital doubles.

[1]  Ruigang Yang,et al.  Multi-resolution real-time stereo on commodity graphics hardware , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[2]  Hans-Peter Seidel,et al.  Hardware-Accelerated Visual Hull Reconstruction and Rendering , 2003, Graphics Interface.

[3]  Luc Van Gool,et al.  Real-time connectivity constrained depth map computation using programmable graphics hardware , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[4]  Adrian Hilton,et al.  Video-rate capture of dynamic face shape and appearance , 2004, Sixth IEEE International Conference on Automatic Face and Gesture Recognition, 2004. Proceedings..

[5]  Adrian Hilton,et al.  Multiple view reconstruction of people , 2004, Proceedings. 2nd International Symposium on 3D Data Processing, Visualization and Transmission, 2004. 3DPVT 2004..

[6]  Luc Van Gool,et al.  GPU-Based Foreground-Background Segmentation using an Extended Colinearity Criterion , 2005 .

[7]  Horst Bischof,et al.  Hierarchical Disparity Estimation with Programmable 3D Hardware , 2004 .

[8]  Edmond Boyer,et al.  Real-Time Capture, Reconstruction and Insertion into Virtual World of Human Actors , 2003, VVG.

[9]  Wojciech Matusik,et al.  Polyhedral Visual Hulls for Real-Time Rendering , 2001, Rendering Techniques.

[10]  Ruigang Yang,et al.  Real‐Time Consensus‐Based Scene Reconstruction Using Commodity Graphics Hardware † , 2003, Comput. Graph. Forum.

[11]  Tomás Svoboda,et al.  A Convenient Multicamera Self-Calibration for Virtual Environments , 2005, Presence: Teleoperators & Virtual Environments.

[12]  Hans-Peter Seidel,et al.  Combining stereo and visual hull information for on-line reconstruction and rendering of dynamic scenes , 2002, 2002 IEEE Workshop on Multimedia Signal Processing..

[13]  Marc Pollefeys,et al.  Multi-view reconstruction using photo-consistency and exact silhouette constraints: a maximum-flow formulation , 2005, Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1.

[14]  Reinhard Koch,et al.  Real-time multi-stereo depth estimation on GPU with approximative discontinuity handling , 2004 .