Low-bandwidth 3D visual telepresence system

We present a methodology to develop a low-cost, low-bandwidth visual telepresence system using commodity depth sensors. To obtain a precise representation of the participants, we fuse together multiple views extracted using a deep background subtraction method. We build a proof-of-concept display composed of a video projector and a quadrangular pyramid made of acrylic, to demonstrate the visualization of a remote person without the need for head-mounted displays. Our system represents an attempt to democratize high-fidelity 3D telepresence using off-the-shelf components.

[1]  Juan R. Terven,et al.  Kin2. A Kinect 2 toolbox for MATLAB , 2016, Sci. Comput. Program..

[2]  Juan R. Terven,et al.  A multiple camera calibration and point cloud fusion tool for Kinect V2 , 2017, Sci. Comput. Program..

[3]  Mohamed Sedky,et al.  Image Processing: Object Segmentation Using Full-Spectrum Matching of Albedo Derived from Colour Images , 2010 .

[4]  Guillaume-Alexandre Bilodeau,et al.  A Self-Adjusting Approach to Change Detection Based on Background Word Consensus , 2015, 2015 IEEE Winter Conference on Applications of Computer Vision.

[5]  Rufael Mekuria,et al.  Emerging MPEG Standards for Point Cloud Compression , 2019, IEEE Journal on Emerging and Selected Topics in Circuits and Systems.

[6]  Rob Fergus,et al.  Predicting Depth, Surface Normals and Semantic Labels with a Common Multi-scale Convolutional Architecture , 2014, 2015 IEEE International Conference on Computer Vision (ICCV).

[7]  Cristian Ruz,et al.  2015 IEEE Winter Conference on Applications of Computer Vision, WACV 2015, Waikoloa, HI, USA, January 5-9, 2015 , 2015, WACV.

[8]  Simon J. D. Prince,et al.  Computer Vision: Models, Learning, and Inference , 2012 .

[9]  Hasan Sajid,et al.  Background subtraction for static & moving camera , 2015, 2015 IEEE International Conference on Image Processing (ICIP).

[10]  Roberto Cipolla,et al.  SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[11]  Gerhard Rigoll,et al.  A deep convolutional neural network for video sequence background subtraction , 2018, Pattern Recognit..

[12]  Dragi Tiro,et al.  The possibility of the hologram pyramid applying in the rapid prototyping , 2015, 2015 4th Mediterranean Conference on Embedded Computing (MECO).

[13]  W. Eric L. Grimson,et al.  Adaptive background mixture models for real-time tracking , 1999, Proceedings. 1999 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No PR00149).

[14]  Huiyu Zhou,et al.  Region-based Mixture of Gaussians modelling for foreground detection in dynamic scenes , 2015, Pattern Recognit..

[15]  Fatih Murat Porikli,et al.  Changedetection.net: A new change detection benchmark dataset , 2012, 2012 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops.

[16]  Eyal Ofek,et al.  Room2Room: Enabling Life-Size Telepresence in a Projected Augmented Reality Environment , 2016, CSCW.

[17]  Bernd Fröhlich,et al.  Immersive Group-to-Group Telepresence , 2013, IEEE Transactions on Visualization and Computer Graphics.

[18]  Henry Fuchs,et al.  Encumbrance-free telepresence system with real-time 3D capture and display using commodity depth cameras , 2011, 2011 10th IEEE International Symposium on Mixed and Augmented Reality.

[19]  Duane C. Brown,et al.  Close-Range Camera Calibration , 1971 .

[20]  Irfan Siddavatam,et al.  3D Holographic Projections using Prism and Hand Gesture Recognition , 2015, ICARCSET '15.

[21]  Henry Fuchs,et al.  General-purpose telepresence with head-worn optical see-through displays and projector-based lighting , 2013, 2013 IEEE Virtual Reality (VR).

[22]  Guillaume-Alexandre Bilodeau,et al.  SuBSENSE: A Universal Change Detection Method With Local Adaptive Sensitivity , 2015, IEEE Transactions on Image Processing.

[23]  Ermal Dreshaj.  Holosuite : an exploration into interactive holographic telepresence , 2015 .

[24]  Hojun Lee,et al.  A hologram based tele-existence platform for emotional exchange among a group of users in both real and virtual environments , 2016, VRST.

[25]  Ling Shao,et al.  End-to-end video background subtraction with 3d convolutional neural networks , 2017, Multimedia Tools and Applications.

[26]  Ahmed K. Noor,et al.  Potential of multimodal and multiuser interaction with virtual holography , 2015, Adv. Eng. Softw..

[27]  Rodolfo Romero Herrera,et al.  Projection's Panel of models for touch screen , 2013 .

[28]  Gerhard Rigoll,et al.  Background segmentation with feedback: The Pixel-Based Adaptive Segmenter , 2012, 2012 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops.

[29]  Ruigang Yang,et al.  3D Tele-Collaboration Over Internet2 , 2002 .

[30]  Ju Shen,et al.  An Immersive Telepresence System Using RGB-D Sensors and Head Mounted Display , 2015, 2015 IEEE International Symposium on Multimedia (ISM).

[31]  Charles T. Loop,et al.  Holoportation: Virtual 3D Teleportation in Real-time , 2016, UIST.

[32]  Rufael Mekuria,et al.  MP3DG-PCC, Open Source Software Framework for Implementation and Evaluation of Point Cloud Compression , 2016, ACM Multimedia.

[33]  D. Marquardt.  An Algorithm for Least-Squares Estimation of Nonlinear Parameters , 1963 .

[34]  Hyeyoung Yoo.  On Study of the Volumetric Display Techniques In Interactive Media Arts Proceedings , 2014 .

[35]  P. Blanche,et al.  Holographic three-dimensional telepresence using large-area photorefractive polymer , 2010, Nature.

[36]  Kenneth Levenberg.  A METHOD FOR THE SOLUTION OF CERTAIN NON – LINEAR PROBLEMS IN LEAST SQUARES , 1944 .

[37]  Hojun Lee,et al.  A mixed reality tele-presence platform to exchange emotion and sensory information based on MPEG-V standard , 2017, 2017 IEEE Virtual Reality (VR).

[38]  S. A. Khadar.  Proceedings of the 2015 International Conference on Advanced Research in Computer Science Engineering & Technology (ICARCSET 2015) , 2015 .

[39]  Henry Fuchs,et al.  Real-time volumetric 3D capture of room-sized scenes for telepresence , 2012, 2012 3DTV-Conference: The True Vision - Capture, Transmission and Display of 3D Video (3DTV-CON).

[40]  D. C. Brown,et al.  Lens distortion for close-range photogrammetry , 1986 .

[41]  Simon J. D. Prince,et al.  Computer Vision: Index , 2012 .