Time-Offset Conversations on a Life-Sized Automultiscopic Projector Array

We present a system for creating and displaying interactive life-sized 3D digital humans based on pre-recorded interviews. We use 30 cameras and an extensive list of questions to record a large set of video responses. Users access videos through a natural conversation interface that mimics face-to-face interaction. Recordings of answers, listening and idle behaviors are linked together to create a persistent visual image of the person throughout the interaction. The interview subjects are rendered using flowed light fields and shown life-size on a special rear-projection screen with an array of 216 video projectors. The display allows multiple users to see different 3D perspectives of the subject in proper relation to their viewpoints, without the need for stereo glasses. The display is effective for interactive conversations since it provides 3D cues such as eye gaze and spatial hand gestures.

[1]  Andrew Jones,et al.  Relighting human locomotion with flowed reflectance fields , 2006, EGSR '06.

[2]  Richard Szeliski,et al.  A Comparison and Evaluation of Multi-View Stereo Reconstruction Algorithms , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[3]  Edmond Boyer,et al.  Holographic and action capture techniques , 2007, SIGGRAPH '07.

[4]  Kallirroi Georgila,et al.  New Dimensions in Testimony: Digitally Preserving a Holocaust Survivor's Interactive Storytelling , 2015, ICIDS.

[5]  Jing Liu,et al.  Interpolating vertical parallax for an autostereoscopic three-dimensional projector array , 2014, J. Electronic Imaging.

[6]  PauschRandy,et al.  Carnegie Mellon's entertainment technology center , 2007 .

[7]  Wojciech Matusik,et al.  3D TV: a scalable system for real-time acquisition, transmission, and autostereoscopic display of dynamic scenes , 2004, ACM Trans. Graph..

[8]  Herbert E. Ives.  The Projection of Parallax Panoramagrams , 1931 .

[9]  Andrew Jones,et al.  Relighting human locomotion with flowed reflectance fields , 2006, EGSR '06.

[10]  Andrew Jones,et al.  Achieving eye contact in a one-to-many 3D video teleconferencing system , 2009, ACM Trans. Graph..

[11]  Alvaro Collet,et al.  High-quality streamable free-viewpoint video , 2015, ACM Trans. Graph..

[12]  Hideshi Yamada,et al.  Rendering for an interactive 360° light field display , 2007, ACM Trans. Graph..

[13]  Anton Leuski,et al.  How to talk to a hologram , 2006, IUI '06.

[14]  Marc Levoy,et al.  Light field rendering , 1996, SIGGRAPH.

[15]  Horst Bischof,et al.  Motion estimation with non-local total variation regularization , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[16]  Anton Leuski,et al.  Ada and Grace: Direct Interaction with Museum Visitors , 2012, IVA.

[17]  Scott Stevens,et al.  Synthetic interviews: the art of creating a “dyad” between humans and machine-based characters , 1998, MULTIMEDIA '98.

[18]  Shoichiro Iwasawa,et al.  3D image quality of 200-inch glasses-free 3D display system , 2012, Electronic Imaging.

[19]  Masahiro Kawakita,et al.  Light-field generation by several screen types for glasses-free tabletop 3D display , 2011, 2011 3DTV Conference: The True Vision - Capture, Transmission and Display of 3D Video (3DTV-CON).

[20]  Anton Leuski,et al.  NPCEditor: Creating Virtual Human Dialogue Using Information Retrieval Techniques , 2011, AI Mag..

[21]  Joakim Gustafson,et al.  The august spoken dialogue system , 1999, EUROSPEECH.

[22]  Wojciech Matusik,et al.  3D TV: a scalable system for real-time acquisition, transmission, and autostereoscopic display of dynamic scenes , 2004, ACM Trans. Graph..

[23]  Anton Leuski,et al.  All Together Now - Introducing the Virtual Human Toolkit , 2013, IVA.

[24]  Anton Leuski,et al.  How Many Utterances Are Needed to Support Time-Offset Interaction? , 2015, FLAIRS Conference.

[25]  Anton Leuski,et al.  Which ASR should I choose for my dialogue system? , 2013, SIGDIAL Conference.

[26]  Anton Leuski,et al.  Ada and Grace: Toward Realistic and Engaging Virtual Museum Guides , 2010, IVA.

[27]  Wilbert J. McKeachie.  Teaching and Learning in the College Classroom. A Review of the Research Literature (1986) and November 1987 Supplement. , 1987 .