Interactive Theater Experience with 3D Live Captured Actors and Spatial Sound

This chapter introduces a new reality interactive theater that combines the 3D Live human capturing system, spatial sound, augmented reality, human-oriented interaction, and ambient intelligence technologies. Actors/Dancers at different places remotely are captured by the 3D Live system and transmitted to the theater place in real time and rendered in 3D form and real size, thus they can play/dance with the local actors/dancers and virtual characters at the same place in real time. Audience interaction is allowed and handled by an Ambient Intelligence (AI) agent to generate new actions for virtual characters and send meaningful interactions to real actors. The spatial sound system provides the real 3D sound and at the same time generates special sound effects such as controlling the sound direction that only allows people in a certain direction to heard it. All these features combined together bring a totally new theater experience to both actors/dancers and audiences, and extend the grammar of the traditional theater.

[1]  Wojciech Matusik,et al.  Polyhedral Visual Hulls for Real-Time Rendering , 2001, Rendering Techniques.

[2]  Wei Liu,et al.  Magic land: live 3D human capture mixed reality interactive system , 2005, CHI EA '05.

[3]  Flavia Sparacino,et al.  DirectIVE-- choreographing media for interactive virtual environments , 1996 .

[4]  William W. Gaver.  Auditory Icons: Using Sound in Computer Interfaces , 1986, Hum. Comput. Interact..

[5]  Michael Cohen,et al.  Exclude and Include for Audio Sources and Sinks: Analogs of Mute & Solo Are Deafen & Attend , 2000, Presence: Teleoperators & Virtual Environments.

[6]  Gregory D. Abowd,et al.  Context-Awareness in Wearable and Ubiquitous Computing , 1997, SEMWEB.

[7]  Peter Spier,et al.  The history of the theater , 1964 .

[8]  M. Dertouzos.  THE FUTURE OF COMPUTING , 1999 .

[9]  Davide Rocchesso,et al.  Sounding Objects , 2003, IEEE Multim..

[10]  Yu Zhao,et al.  A simplified model for generating 3D realistic sound in the multimedia and virtual reality systems , 2008, Journal of Computer Science and Technology.

[11]  Mahadev Satyanarayanan,et al.  Pervasive computing: vision and challenges , 2001, IEEE Wirel. Commun..

[12]  Michael Cohen,et al.  Audio Narrowcasting and Privacy for Multipresent Avatars on Workstations and Mobile Phones , 2006, IEICE Trans. Inf. Syst..

[13]  Alex Pentland,et al.  Media in performance: Interactive spaces for dance, theater, circus, and museum exhibits , 2000, IBM Syst. J..

[14]  Blair MacIntyre,et al.  Presence and engagement in an interactive drama , 2007, CHI.

[15]  S. Handel.  Listening As Introduction to the Perception of Auditory Events , 1989 .

[16]  Rikk Carey,et al.  The annotated VRML 2.0 reference manual , 1997 .

[17]  Riitta Väänänen,et al.  Parametrization, auralization, and authoring of room acoustics for virtual reality applications , 2003 .