Xr360: A Toolkit for Mixed 360 and 3d Productions

Due to the affordability and high quality of the emerging XR devices and platforms, immersive media content has become more accessible to a wide variety of consumers while their technology gains increased interest from content creators. However, the previously mentioned technological advancements are focused more on sensor or rendering-display technologies, while the development of software tools that assist professionals, indies or hobbyists to create immersive content and narratives is left behind. In this work we introduce XR360, a unified toolkit for producing 360° immersive experiences by embedding 3D captured human performances in 360° captured scenes. The presented toolkit comprises a complete solution for content acquisition, 3D asset creation and 3D-360° fusion. It combines a volumetric capturing studio, a performance capture production tool and a 360° fusion application, designed to operate in a cascaded, but disjoint, manner while remaining user-friendly by hiding complex technicalities from its end-users.

[1]  Taehyun Rhee,et al.  MR360: Mixed Reality Rendering for 360° Panoramic Videos , 2017, IEEE Transactions on Visualization and Computer Graphics.

[2]  Petros Daras,et al.  OmniDepth: Dense Depth Estimation for Indoors Spherical Panoramas , 2018, ECCV.

[3]  Peter V. Gehler,et al.  Keep It SMPL: Automatic Estimation of 3D Human Pose and Shape from a Single Image , 2016, ECCV.

[4]  Petros Daras,et al.  Spherical View Synthesis for Self-Supervised 360° Depth Estimation , 2019, 2019 International Conference on 3D Vision (3DV).

[5]  Marc Erich Latoschik,et al.  Immersive Virtual Reality and Gamification Within Procedurally Generated Environments to Increase Motivation During Gait Rehabilitation , 2019, 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR).

[6]  Charles T. Loop,et al.  Holoportation: Virtual 3D Teleportation in Real-time , 2016, UIST.

[7]  Taehyun Rhee,et al.  Real-Time Mixed Reality Rendering for Underwater 360° Videos , 2019, 2019 IEEE International Symposium on Mixed and Augmented Reality (ISMAR).

[8]  Petros Daras,et al.  A Low-Cost, Flexible and Portable Volumetric Capturing System , 2018, 2018 14th International Conference on Signal-Image Technology & Internet-Based Systems (SITIS).

[9]  Vanessa Tan,et al.  VREX: A Framework for Immersive Virtual Reality Experiences , 2018, 2018 IEEE Region Ten Symposium (Tensymp).

[10]  Yifan Wang,et al.  A software application framework for developing immersive virtual reality experiences in health domain , 2017, 2017 IEEE Life Sciences Conference (LSC).

[11]  Markus Funk,et al.  Using virtual reality for prototyping interactive architecture , 2017, MUM.

[12]  Won-Ki Jeong,et al.  DXR: A Toolkit for Building Immersive Data Visualizations , 2019, IEEE Transactions on Visualization and Computer Graphics.

[13]  Petros Daras,et al.  Fast deformable model-based human performance capture and FVV using consumer-grade RGB-D sensors , 2018, Pattern Recognit..

[14]  Ilya Baran,et al.  Automatic rigging and animation of 3D characters , 2007, SIGGRAPH 2007.

[15]  Thomas P. Kersten,et al.  Virtual Reality for Cultural Heritage Monuments - from 3D Data Recording to Immersive Visualisation , 2018, EuroMed.

[16]  Michael M. Kazhdan,et al.  Poisson surface reconstruction , 2006, SGP '06.

[17]  Petros Daras,et al.  Deep Soft Procrustes for Markerless Volumetric Sensor Alignment , 2020, 2020 IEEE Conference on Virtual Reality and 3D User Interfaces (VR).

[18]  Bernhard Preim,et al.  An Interactive Demonstration of Collaborative VR for Laparoscopic Liver Surgery Training , 2019, 2019 IEEE International Conference on Artificial Intelligence and Virtual Reality (AIVR).

[19]  Alvaro Collet,et al.  High-quality streamable free-viewpoint video , 2015, ACM Trans. Graph..

[20]  Anders Grunnet-Jepsen,et al.  Intel RealSense Stereoscopic Depth Cameras , 2017, CVPR 2017.

[21]  Petros Daras,et al.  360° Surface Regression with a Hyper-Sphere Loss , 2019, 2019 International Conference on 3D Vision (3DV).

[22]  Yeo-Jin Yoon,et al.  Interactive and Immersive Learning Using 360° Virtual Reality Contents on Mobile Platforms , 2018, Mob. Inf. Syst..