Real-time 3D human capture system for mixed-reality art and entertainment

A real-time system for capturing humans in 3D and placing them into a mixed reality environment is presented in this paper. Nine cameras surrounding her capture the subject. Looking through a head-mounted-display with a camera in front pointing at a marker, the user can see the 3D image of this subject overlaid onto a mixed reality scene. The 3D images of the subject viewed from this viewpoint are constructed using a robust and fast shape-from-silhouette algorithm. The paper also presents several techniques to produce good quality and speed up the whole system. The frame rate of our system is around 25 fps using only standard Intel processor-based personal computers. Besides a remote live 3D conferencing and collaborating system, we also describe an application of the system in art and entertainment, named Magic Land, which is a mixed reality environment where captured avatars of human and 3D computer generated virtual animations can form an interactive story and play with each other. This system demonstrates many technologies in human computer interaction: mixed reality, tangible interaction, and 3D communication. The result of the user study not only emphasizes the benefits, but also addresses some issues of these technologies.

[1]  Hirokazu Kato,et al.  Touch-Space: Mixed Reality Game Space Based on Ubiquitous, Tangible, and Social Computing , 2002, Personal and Ubiquitous Computing.

[2]  Thomas W. Malone,et al.  Toward a Theory of Intrinsically Motivating Instruction , 1981, Cogn. Sci..

[3]  Claudia Adams,et al.  The use of computer games as an educational tool: identification of appropriate game types and game elements , 1999, Br. J. Educ. Technol..

[4]  Ronald Azuma,et al.  A Survey of Augmented Reality , 1997, Presence: Teleoperators & Virtual Environments.

[5]  Staffan Björk,et al.  Pirates! Using the Physical World as a Game Board , 2001, INTERACT.

[6]  Wei Liu,et al.  Live three-dimensional content for augmented reality , 2005, IEEE Transactions on Multimedia.

[7]  Hirokazu Kato,et al.  3D live: real time captured content for mixed reality , 2002, Proceedings. International Symposium on Mixed and Augmented Reality.

[8]  Richard Szeliski,et al.  Video mosaics for virtual environments , 1996, IEEE Computer Graphics and Applications.

[9]  Andreas Kunz,et al.  blue-c: a spatially immersive display and 3D video portal for telepresence , 2003, ACM Trans. Graph..

[10]  Adrian David Cheok,et al.  Touchy.Internet: a cybernetics system for human-pet interaction through the Internet , 2003, SIGGRAPH '03.

[11]  Mark R. Stevens,et al.  Methods for Volumetric Reconstruction of Visual Scenes , 2004, International Journal of Computer Vision.

[12]  Steven M. Seitz,et al.  Photorealistic Scene Reconstruction by Voxel Coloring , 1997, International Journal of Computer Vision.

[13]  Wojciech Matusik,et al.  Polyhedral Visual Hulls for Real-Time Rendering , 2001, Rendering Techniques.

[14]  Ronald W. Schafer,et al.  Image-based photo hulls , 2002, Proceedings. First International Symposium on 3D Data Processing Visualization and Transmission.

[15]  Luc Van Gool,et al.  Blue-c: a spatially immersive display and 3D video portal for telepresence , 2003, IPT/EGVE.

[16]  Hirokazu Kato,et al.  MagicCup: a tangible interface for virtual objects manipulation in table-top augmented reality , 2003, 2003 IEEE International Augmented Reality Toolkit Workshop.

[17]  Kazuhiko Sumi,et al.  A robust background subtraction method for changing background , 2000, Proceedings Fifth IEEE Workshop on Applications of Computer Vision.

[18]  Ronald Azuma,et al.  Recent Advances in Augmented Reality , 2001, IEEE Computer Graphics and Applications.

[19]  François X. Sillion,et al.  A Real-Time System for Full Body Interaction with Virtual Worlds , 2004, EGVE.

[20]  Til Aach,et al.  Illumination-Invariant Change Detection Using a Statistical Colinearity Criterion , 2001, DAGM-Symposium.

[21]  Wei Liu,et al.  Human Pacman: a mobile wide-area entertainment system based on physical, social, and ubiquitous computing , 2004, ACE '04.

[22]  Henning Schulzrinne,et al.  RTP: A Transport Protocol for Real-Time Applications , 1996, RFC.

[23]  Hideyuki Tamura,et al.  Mixed Reality: Future Dreams Seen at the Border between Real and Virtual Worlds , 2001, IEEE Computer Graphics and Applications.

[24]  Kiriakos N. Kutulakos,et al.  A Theory of Shape by Space Carving , 2000, International Journal of Computer Vision.