RealityCheck: Blending Virtual Environments with Situated Physical Reality

Today's virtual reality (VR) systems offer chaperone rendering techniques that prevent the user from colliding with physical objects. Without a detailed geometric model of the physical world, these techniques offer limited possibility for more advanced compositing between the real world and the virtual. We explore this using a realtime 3D reconstruction of the real world that can be combined with a virtual environment. RealityCheck allows users to freely move, manipulate, observe, and communicate with people and objects situated in their physical space without losing the sense of immersion or presence inside their virtual world. We demonstrate RealityCheck with seven existing VR titles, and describe compositing approaches that address the potential conflicts when rendering the real world and a virtual environment together. A study with frequent VR users demonstrate the affordances provided by our system and how it can be used to enhance current VR experiences.

[1]  Anjul Patney,et al.  Towards virtual reality infinite walking , 2018, ACM Trans. Graph..

[2]  Sergio Garrido-Jurado,et al.  Procedurally generated virtual reality from 3D reconstructed physical space , 2016, VRST.

[3]  Ali Farhadi,et al.  You Only Look Once: Unified, Real-Time Object Detection , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[4]  Amitabh Varshney,et al.  Montage4D: interactive seamless fusion of multiview video textures , 2018, I3D.

[5]  Li-Yi Wei,et al.  Mapping virtual and physical reality , 2016, ACM Trans. Graph..

[6]  P. Milgram,et al.  A Taxonomy of Mixed Reality Visual Displays , 1994 .

[7]  J. J. Higgins,et al.  The aligned rank transform for nonparametric factorial analyses using only anova procedures , 2011, CHI.

[8]  Ceenu George,et al.  Like Elephants Do: Sensing Bystanders During HMD Usage , 2017 .

[9]  Daniel Freedman,et al.  Reality Skins: Creating Immersive and Tactile Virtual Environments , 2016, 2016 IEEE International Symposium on Mixed and Augmented Reality (ISMAR).

[10]  Simon J. Julier,et al.  Presence and discernability in conventional and non-photorealistic immersive augmented reality , 2014, 2014 IEEE International Symposium on Mixed and Augmented Reality (ISMAR).

[11]  Pushmeet Kohli,et al.  FLARE: Fast layout for augmented reality applications , 2014, 2014 IEEE International Symposium on Mixed and Augmented Reality (ISMAR).

[12]  Martin Hachet,et al.  One Reality: Augmenting How the Physical World is Experienced by combining Multiple Mixed Reality Modalities , 2017, UIST.

[13]  Erik Reinhard,et al.  Color Transfer between Images , 2001, IEEE Computer Graphics and Applications.

[14]  Pushmeet Kohli,et al.  Fusion4D , 2016, ACM Trans. Graph..

[15]  Roderick Murray-Smith,et al.  A Dose of Reality: Overcoming Usability Challenges in VR Head-Mounted Displays , 2015, CHI Extended Abstracts.

[16]  Larry D. Hostetler,et al.  The estimation of the gradient of a density function, with applications in pattern recognition , 1975, IEEE Trans. Inf. Theory.

[17]  Thomas A. Funkhouser,et al.  Semantic Scene Completion from a Single Depth Image , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[18]  Jun Rekimoto,et al.  JackIn space: designing a seamless transition between first and third person view for effective telepresence collaborations , 2017, AH.

[19]  Adalberto L. Simeone,et al.  Substitutional reality: Towards a research agenda , 2015, 2015 IEEE 1st Workshop on Everyday Virtual Reality (WEVR).

[20]  Andrew D. Wilson.  Fast Lossless Depth Image Compression , 2017, ISS.

[21]  Niels Henze,et al.  Physical Keyboards in Virtual Reality: Analysis of Typing Performance and Effects of Avatar Hands , 2018, CHI.

[22]  Julian Frommel,et al.  ShareVR: Enabling Co-Located Experiences for Virtual Reality between HMD and Non-HMD Users , 2017, CHI.

[23]  Thijs Roumen,et al.  TurkDeck: Physical Virtual Reality Based on People , 2015, UIST.

[24]  Sean Follmer,et al.  A Functional Optimization Based Approach for Continuous 3D Retargeted Touch of Arbitrary, Complex Boundaries in Haptic Virtual Reality , 2018, CHI.

[25]  Carl Gutwin,et al.  The ASPECTA toolkit: affordable full coverage displays , 2016, PerDis.

[26]  Patrick Baudisch,et al.  VirtualSpace - Overloading Physical Space with Multiple Virtual Reality Users , 2018, CHI.

[27]  Daniel J. Wigdor,et al.  Annexing Reality: Enabling Opportunistic Use of Everyday Objects as Tangible Proxies in Augmented Reality , 2016, CHI.

[28]  Galen C. Hunt,et al.  Detours: binary interception of Win32 functions , 1999 .

[29]  Eyal Ofek,et al.  Haptic Retargeting: Dynamic Repurposing of Passive Haptics for Enhanced Virtual Reality Experiences , 2016, CHI.

[30]  Mary C. Whitton,et al.  Improved Redirection with Distractors: A large-scale-real-walking locomotion interface and its effect on navigation in virtual environments , 2010, 2010 IEEE Virtual Reality Conference (VR).

[31]  Blair MacIntyre,et al.  RoomAlive: magical experiences enabled by scalable, adaptive projector-camera units , 2014, UIST.

[32]  Robert J. Teather,et al.  Haptobend: shape-changing passive haptic feedback in virtual reality , 2017, SUI.

[33]  Paul Milgram,et al.  A Taxonomy of Real and Virtual World Display Integration , 1999 .

[34]  Juan D. Tardós,et al.  ORB-SLAM2: An Open-Source SLAM System for Monocular, Stereo, and RGB-D Cameras , 2016, IEEE Transactions on Robotics.

[35]  Hrvoje Benko,et al.  Projected Augmented Reality with the RoomAlive Toolkit , 2016, ISS.

[36]  Y. Benjamini,et al.  Controlling the false discovery rate: a practical and powerful approach to multiple testing , 1995 .

[37]  David Lindlbauer,et al.  Remixed Reality: Manipulating Space and Time in Augmented Reality , 2018, CHI.

[38]  Mel Slater,et al.  Using Presence Questionnaires in Reality , 2000, Presence: Teleoperators & Virtual Environments.

[39]  Seokhee Jeon,et al.  Haptic Augmented Reality: Taxonomy and an Example of Stiffness Modulation , 2009, PRESENCE: Teleoperators and Virtual Environments.

[40]  Jérôme Royan,et al.  Collaborators awareness for user cohabitation in co-located collaborative virtual environments , 2017, VRST.

[41]  Jeffrey M. Richter.  Programming Applications for Microsoft Windows with Cdrom , 1999 .

[42]  Takeo Kanade,et al.  Virtualized Reality: Constructing Virtual Worlds from Real Scenes , 1997, IEEE Multim..

[43]  Vladlen Koltun,et al.  Robust reconstruction of indoor scenes , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[44]  Steve Mann,et al.  Mediated Reality , 1999 .