A Mixed Reality Telepresence System for Collaborative Space Operation

This paper presents a mixed reality (MR) system that results from the integration of a telepresence system and an application to improve collaborative space exploration. The system combines free viewpoint video with immersive projection technology to support nonverbal communication (NVC), including eye gaze, interpersonal distance, and facial expression. Importantly, these features can be interpreted together as people move around the simulation, maintaining a natural social distance. The application is a simulation of Mars, within which the collaborators must come to agreement over; for example, where the Rover should land and go. The first contribution is the creation of an MR system supporting contextualization of NVC. Two technological contributions are prototyping a technique to subtract a person from a background that may contain physical objects and/or moving images and a lightweight texturing method for multiview rendering, which provides balance in terms of visual and temporal quality. A practical contribution is the demonstration of pragmatic approaches to sharing space between display systems of distinct levels of immersion. A research tool contribution is a system that allows comparison of conventional authored and video-based reconstructed avatars, within an environment that encourages exploration and social interaction. Aspects of system quality, including the communication of facial expression and end-to-end latency are reported.

[1]  M. Slater,et al.  Bystander Responses to a Violent Incident in an Immersive Virtual Environment , 2013, PloS one.

[2]  Oliver Grau.  Studio production system for dynamic 3D content , 2003, Visual Communications and Image Processing.

[3]  Oliver Otto,et al.  Constructing a Gazebo: Supporting Teamwork in a Tightly Coupled, Distributed Task in Virtual Reality , 2003, Presence: Teleoperators & Virtual Environments.

[4]  Shahram Izadi,et al.  3D-board: a whole-body remote collaborative whiteboard , 2014, UIST.

[5]  M. Patterson.  An arousal model of interpersonal intimacy. , 1976 .

[6]  Rob Aspin,et al.  Estimating the Gaze of a Virtuality Human , 2013, IEEE Transactions on Visualization and Computer Graphics.

[7]  Jeremy N. Bailenson,et al.  Equilibrium Theory Revisited: Mutual Gaze and Personal Space in Virtual Environments , 2001, Presence: Teleoperators & Virtual Environments.

[8]  David J. Roberts,et al.  Parallel processing for real-time 3D reconstruction from video streams , 2012, Journal of Real-Time Image Processing.

[9]  Ig-Jae Kim,et al.  Real time 3D avatar for interactive mixed reality , 2004, VRCAI '04.

[10]  Petros Daras,et al.  Real-Time, Full 3-D Reconstruction of Moving Foreground Objects From Multiple Consumer Depth Cameras , 2013, IEEE Transactions on Multimedia.

[11]  Henry Fuchs,et al.  Real-time volumetric 3D capture of room-sized scenes for telepresence , 2012, 2012 3DTV-Conference: The True Vision - Capture, Transmission and Display of 3D Video (3DTV-CON).

[12]  Ferdinand van der Heijden,et al.  Efficient adaptive density estimation per image pixel for the task of background subtraction , 2006, Pattern Recognit. Lett..

[13]  Bruno Raffin,et al.  A 3d data intensive tele-immersive grid , 2010, ACM Multimedia.

[14]  Henry Fuchs,et al.  Encumbrance-free telepresence system with real-time 3D capture and display using commodity depth cameras , 2011, 2011 10th IEEE International Symposium on Mixed and Augmented Reality.

[15]  Keita Higuchi,et al.  ImmerseBoard: Immersive Telepresence Experience using a Digital Whiteboard , 2015, CHI.

[16]  Bernd Hamann,et al.  Spherical Terrain Rendering using the hierarchical HEALPix grid , 2011, VLUDS.

[17]  Edmond Boyer,et al.  Exact polyhedral visual hulls , 2003, BMVC.

[18]  Paul Debevec,et al.  Modeling and Rendering Architecture from Photographs , 1996, SIGGRAPH 1996.

[19]  DuckworthTobias,et al.  Parallel processing for real-time 3D reconstruction from video streams , 2014 .

[20]  G.C. Stockman,et al.  Sensing and recognition of rigid objects using structured light , 1988, IEEE Control Systems Magazine.

[21]  P. Milgram,et al.  A Taxonomy of Mixed Reality Visual Displays , 1994 .

[22]  Qi Tian,et al.  Foreground object detection from videos containing complex background , 2003, MULTIMEDIA '03.

[23]  Robin Wolff,et al.  Communicating Eye-gaze Across a Distance: Comparing an Eye-gaze enabled Immersive Collaborative Virtual Environment, Aligned Video Conferencing, and Being Together , 2009, 2009 IEEE Virtual Reality Conference.

[24]  Robin Wolff,et al.  A collaborative workspace architecture for strengthening collaboration among space scientists , 2015, 2015 IEEE Aerospace Conference.

[25]  Zoran Zivkovic,et al.  Improved adaptive Gaussian mixture model for background subtraction , 2004, Proceedings of the 17th International Conference on Pattern Recognition, 2004. ICPR 2004..

[26]  Henry Fuchs,et al.  Reducing interference between multiple structured light depth sensors using motion , 2012, 2012 IEEE Virtual Reality Workshops (VRW).

[27]  Philip A. Chou,et al.  Viewport: A Distributed, Immersive Teleconferencing System with Infrared Dot Pattern , 2013, IEEE MultiMedia.

[28]  Mel Slater,et al.  An Eye Gaze Model for Dyadic Interaction in an Immersive Virtual Environment: Practice and Experience , 2004, Comput. Graph. Forum.

[29]  Roger Y. Tsai,et al.  A versatile camera calibration technique for high-accuracy 3D machine vision metrology using off-the-shelf TV cameras and lenses , 1987, IEEE J. Robotics Autom..

[30]  H. Fuchs,et al.  A First Look at a Telepresence System with Room-Sized Real-Time 3 D Capture and Life-Sized Tracked Display Wall , 2011 .

[31]  Anita Sellent,et al.  Floating Textures , 2008, Comput. Graph. Forum.

[32]  Rob Aspin,et al.  Synchronization of Images from Multiple Cameras to Reconstruct a Moving Human , 2010, 2010 IEEE/ACM 14th International Symposium on Distributed Simulation and Real Time Applications.

[33]  Bruce G. Baumgart.  A polyhedron representation for computer vision , 1975, AFIPS '75.

[34]  Peter M. Will,et al.  Grid Coding: A Preprocessing Technique for Robot and Machine Vision , 1971, IJCAI.

[35]  Ligang Liu,et al.  Scanning 3D Full Human Bodies Using Kinects , 2012, IEEE Transactions on Visualization and Computer Graphics.

[36]  Radu Horaud,et al.  Time-of-Flight Cameras: Principles, Methods and Applications , 2012 .

[37]  Hélène Laurent,et al.  Review and evaluation of commonly-implemented background subtraction algorithms , 2008, 2008 19th International Conference on Pattern Recognition.

[38]  Ruzena Bajcsy,et al.  High-Quality Visualization for Geographically Distributed 3-D Teleimmersive Applications , 2011, IEEE Transactions on Multimedia.

[39]  Rob Aspin,et al.  withyou—An Experimental End-to-End Telepresence System Using Video-Based Reconstruction , 2015, IEEE Journal of Selected Topics in Signal Processing.

[40]  Rob Aspin,et al.  Projective multi-texturing for integrated real-time 3D reconstruction and rendering of a person , 2011 .

[41]  Radu Horaud,et al.  Time-of-Flight Cameras , 2012, SpringerBriefs in Computer Science.

[42]  Luc Van Gool,et al.  GPU-Based Foreground-Background Segmentation using an Extended Colinearity Criterion , 2005 .

[43]  電子情報通信学会.  IEICE transactions on information and systems , 1992 .

[44]  Takeo Kanade,et al.  Virtual Space Teleconferencing Using a Sea of Cameras , 1994 .

[45]  Jitendra Malik,et al.  Modeling and Rendering Architecture from Photographs: A hybrid geometry- and image-based approach , 1996, SIGGRAPH.

[46]  Greg Welch,et al.  The office of the future: a unified approach to image-based modeling and spatially immersive displays , 1998, SIGGRAPH.

[47]  Paul Ekman,et al.  Facial expression analysis , 2008, Scholarpedia.

[48]  Jean Ponce,et al.  Accurate, Dense, and Robust Multiview Stereopsis , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[49]  Bernd Fröhlich,et al.  Immersive Group-to-Group Telepresence , 2013, IEEE Transactions on Visualization and Computer Graphics.

[50]  Mohan M. Trivedi,et al.  Detecting Moving Shadows: Algorithms and Evaluation , 2003, IEEE Trans. Pattern Anal. Mach. Intell..