Increasing Realism and Supporting Content Planning for Dynamic Scenes in a Mixed Reality System incorporating a Time-of-Flight Camera

Mixed reality is the combination of real and virtual scene content. Besides correct alignment of the two modalities and correct occlusion handling the core issues to be tackled are the degree of realism and the ease of use. For a convincing perception correct occlusion handling and shadowing is mandatory. We present a system for mixed reality applications which features enhanced realism through automatic shadow computation and supported placement functions for virtual content. The system relies on the incorporation of a ToF-camera device. This device delivers real-time depth maps of the environment and is used to build a static environment model. It also allows correct handling of mutual occlusion between real and virtual content, shadow computation and enhanced content planning. Other mixed reality systems require expensive studio setups with complex camera tracking installations. Often multi- camera approaches in combination with chroma-keying facilities are used. The presented system is rather inexpensive, compact, mobile, flexible and provides convenient calibration procedures. The standard chroma-keying procedure is replaced by the usage of the environment model. The planning and alignment of virtual content is simplified by automatic extraction of dynamic content in the scene. This is essential for the placement of virtual content. With the presented system appropriate depth maps of the mixed content are available in real-time, which makes the approach suitable for 3DTV productions. The presented paper discusses all key elements of mixed-reality applications based on the presented approach. This includes camera calibration, environment model generation, real-time handling between interacting virtual and real content, shadowing for virtual content and dynamic object tracking for content planning. (10 pages)

[1]  G LoweDavid,et al.  Distinctive Image Features from Scale-Invariant Keypoints , 2004 .

[2]  William E. Lorensen,et al.  Decimation of triangle meshes , 1992, SIGGRAPH.

[3]  Eyal Ofek,et al.  Depth keying , 2003, IS&T/SPIE Electronic Imaging.

[4]  Wolfgang Straßer,et al.  On-the-fly scene acquisition with a handy multi-sensor system , 2008, Int. J. Intell. Syst. Technol. Appl..

[5]  John Billingsley,et al.  Mechatronics and machine vision in practice , 2008 .

[6]  Alice Biber,et al.  Time-of-flight range imaging with a custom solid state image sensor , 1999, Industrial Lasers and Inspection.

[7]  R. Koch,et al.  CALIBRATION OF A PMD-CAMERA USING A PLANAR CALIBRATION PATTERN TOGETHER WITH A MULTI-CAMERA SETUP , 2008 .

[8]  Andreas Kolb,et al.  Lateral and Depth Calibration of PMD-Distance Sensors , 2006, ISVC.

[9]  Matthijs C. Dorst.  Distinctive Image Features from Scale-Invariant Keypoints , 2011 .

[10]  Reinhard Koch,et al.  Realtime Camera Tracking in the MATRIS Project , 2007, SMPTE Motion Imaging Journal.

[11]  Michael Lehmann,et al.  An all-solid-state optical range camera for 3D real-time imaging with sub-centimeter depth resolution (SwissRanger) , 2004, SPIE Optical Systems Design.

[12]  Reinhard Koch,et al.  Calibration of focal length and 3D pose based on the reflectance and depth image of a planar object , 2008, Int. J. Intell. Syst. Technol. Appl..

[13]  Reinhard Koch,et al.  Datastructures for Capturing Dynamic Scenes with a Time-of-Flight Camera , 2009, Dyn3D.

[14]  Reinhard Koch,et al.  Extraction of 3D freeform surfaces as visual landmarks for real-time tracking , 2007, Journal of Real-Time Image Processing.

[15]  Roland Siegwart,et al.  A Flexible Technique for Accurate Omnidirectional Camera Calibration and Structure from Motion , 2006, Fourth IEEE International Conference on Computer Vision Systems (ICVS'06).

[16]  Reinhard Koch,et al.  Monocular Body Pose Estimation by Color Histograms and Point Tracking , 2006, DAGM-Symposium.

[17]  Zhengyou Zhang,et al.  Flexible camera calibration by viewing a plane from unknown orientations , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[18]  Reinhard Koch,et al.  Robust GPU-assisted camera tracking using free-form surface models , 2007, Journal of Real-Time Image Processing.

[19]  Bodo Rosenhahn,et al.  A System for Marker-Less Human Motion Estimation , 2005, DAGM-Symposium.

[20]  Lance Williams,et al.  Casting curved shadows on curved surfaces , 1978, SIGGRAPH.

[21]  James J. Thomas,et al.  Proceedings of the 19th annual conference on Computer graphics and interactive techniques , 1992, SIGGRAPH.

[22]  Reinhard Koch,et al.  Integration of a Time-of-Flight Camera into a Mixed Reality System for Handling Dynamic Scenes, Moving Viewpoints and Occlusions in Real-Time , 2008 .

[23]  R. Reulke,et al.  Remote Sensing and Spatial Information Sciences , 2005 .

[24]  Li Guan,et al.  3D Object Reconstruction with Heterogeneous Sensor Data , 2008 .

[25]  Reinhard Koch,et al.  MixIn3D: 3D Mixed Reality with ToF-Camera , 2009, Dyn3D.

[26]  Carlo Tomasi,et al.  Good features to track , 1994, 1994 Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.

[27]  G. A. Thomas,et al.  A versatile camera position measurement system for virtual reality TV production , 1997 .