Telepresence System based on Simulated Holographic Display

We present a telepresence system based on a custom-made simulated holographic display that produces a full 3D model of the remote participants using commodity depth sensors. Our display is composed of a video projector and a quadrangular pyramid made of acrylic, that allows the user to experience an omnidirectional visualization of a remote person without the need for head-mounted displays. To obtain a precise representation of the participants, we fuse together multiple views extracted using a deep background subtraction method. Our system represents an attempt to democratize high-fidelity 3D telepresence using off-the-shelf components.

[1]  Hojun Lee,et al.  A mixed reality tele-presence platform to exchange emotion and sensory information based on MPEG-V standard , 2017, 2017 IEEE Virtual Reality (VR).

[2]  Charles T. Loop,et al.  Holoportation: Virtual 3D Teleportation in Real-time , 2016, UIST.

[3]  Hasan Sajid,et al.  Background subtraction for static & moving camera , 2015, 2015 IEEE International Conference on Image Processing (ICIP).

[4]  Hyeyoung Yoo.  On Study of the Volumetric Display Techniques In Interactive Media Arts Proceedings , 2014 .

[5]  Huiyu Zhou,et al.  Region-based Mixture of Gaussians modelling for foreground detection in dynamic scenes , 2015, Pattern Recognit..

[6]  Bernd Fröhlich,et al.  Immersive Group-to-Group Telepresence , 2013, IEEE Transactions on Visualization and Computer Graphics.

[7]  D. C. Brown,et al.  Lens distortion for close-range photogrammetry , 1986 .

[8]  Juan R. Terven,et al.  A multiple camera calibration and point cloud fusion tool for Kinect V2 , 2017, Sci. Comput. Program..

[9]  Mohamed Sedky,et al.  Image Processing: Object Segmentation Using Full-Spectrum Matching of Albedo Derived from Colour Images , 2010 .

[10]  P. Blanche,et al.  Holographic three-dimensional telepresence using large-area photorefractive polymer , 2010, Nature.

[11]  Gerhard Rigoll,et al.  Background segmentation with feedback: The Pixel-Based Adaptive Segmenter , 2012, 2012 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops.

[12]  Duane C. Brown,et al.  Close-Range Camera Calibration , 1971 .

[13]  Ramesh Raskar,et al.  Modern approaches to augmented reality: introduction to current approaches , 2006, SIGGRAPH Courses.

[14]  Fatih Murat Porikli,et al.  Changedetection.net: A new change detection benchmark dataset , 2012, 2012 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops.

[15]  Simon J. D. Prince,et al.  Computer Vision: Models, Learning, and Inference , 2012 .

[16]  Henry Fuchs,et al.  General-purpose telepresence with head-worn optical see-through displays and projector-based lighting , 2013, 2013 IEEE Virtual Reality (VR).

[17]  Guillaume-Alexandre Bilodeau,et al.  SuBSENSE: A Universal Change Detection Method With Local Adaptive Sensitivity , 2015, IEEE Transactions on Image Processing.

[18]  Kenneth Levenberg.  A METHOD FOR THE SOLUTION OF CERTAIN NON – LINEAR PROBLEMS IN LEAST SQUARES , 1944 .

[19]  Ermal Dreshaj.  Holosuite : an exploration into interactive holographic telepresence , 2015 .

[20]  Guillaume-Alexandre Bilodeau,et al.  A Self-Adjusting Approach to Change Detection Based on Background Word Consensus , 2015, 2015 IEEE Winter Conference on Applications of Computer Vision.

[21]  Ahmed K. Noor,et al.  Potential of multimodal and multiuser interaction with virtual holography , 2015, Adv. Eng. Softw..

[22]  Rodolfo Romero Herrera,et al.  Projection's Panel of models for touch screen , 2013 .

[23]  D. Marquardt.  An Algorithm for Least-Squares Estimation of Nonlinear Parameters , 1963 .

[24]  Eyal Ofek,et al.  Room2Room: Enabling Life-Size Telepresence in a Projected Augmented Reality Environment , 2016, CSCW.

[25]  Ruigang Yang,et al.  3D Tele-Collaboration Over Internet2 , 2002 .

[26]  Roberto Cipolla,et al.  SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[27]  Gerhard Rigoll,et al.  A deep convolutional neural network for video sequence background subtraction , 2018, Pattern Recognit..

[28]  Dragi Tiro,et al.  The possibility of the hologram pyramid applying in the rapid prototyping , 2015, 2015 4th Mediterranean Conference on Embedded Computing (MECO).

[29]  Henry Fuchs,et al.  Encumbrance-free telepresence system with real-time 3D capture and display using commodity depth cameras , 2011, 2011 10th IEEE International Symposium on Mixed and Augmented Reality.

[30]  Hojun Lee,et al.  A hologram based tele-existence platform for emotional exchange among a group of users in both real and virtual environments , 2016, VRST.

[31]  Juan R. Terven,et al.  Kin2. A Kinect 2 toolbox for MATLAB , 2016, Sci. Comput. Program..

[32]  Rob Fergus,et al.  Predicting Depth, Surface Normals and Semantic Labels with a Common Multi-scale Convolutional Architecture , 2014, 2015 IEEE International Conference on Computer Vision (ICCV).

[33]  W. Eric L. Grimson,et al.  Adaptive background mixture models for real-time tracking , 1999, Proceedings. 1999 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No PR00149).

[34]  Henry Fuchs,et al.  Real-time volumetric 3D capture of room-sized scenes for telepresence , 2012, 2012 3DTV-Conference: The True Vision - Capture, Transmission and Display of 3D Video (3DTV-CON).

[35]  Ju Shen,et al.  An Immersive Telepresence System Using RGB-D Sensors and Head Mounted Display , 2015, 2015 IEEE International Symposium on Multimedia (ISM).