Enabling Interaction with Arbitrary 2D Applications in Virtual Environments

Virtual environments (VE) provide immersive experiences that allow users to perceive and interact with three dimensional (3D) content. Yet, the 3D applications are often tailored for specific tasks and hardware setups such as a virtual or augmented reality (VR/AR) head mounted display (HMD). This limits the range of available content, because not every standard application (e.g. web browser, calender, office) that is available on a 2D desktop PC can be easily accessed using VR or AR. As commonly used by other systems, we capture arbitrary 2D windows from a desktop PC and display them as a 2D surface in the VE. Furthermore, we provide different techniques that aim to allow easy and fast interaction with the 2D applications in VR. For example, the work bench tool allows to copy and paste subsections of a 2D window. The copied sections are fully interactable and can be moved around and scaled to provide easy access to small menus. With the macro tool, users can create shortcuts to frequently used functions that are executed by cursor presses or hotkeys. The provided tools yield new ways of interacting with 2D interfaces. By allowing the user to easily interact with already available 2D applications, the productivity of VEs increases strongly.

[1]  Henry Sowizral,et al.  Embedding the 2D interaction metaphor in a real 3D virtual environment , 1995, Electronic Imaging.

[2]  Ferran Argelaguet,et al.  Virtual Pads: Decoupling Motor Space and Visual Space for Flexible Manipulation of 2D Windows within VEs , 2007, 2007 IEEE Symposium on 3D User Interfaces.

[3]  Frederick P. Brooks,et al.  Moving objects in space: exploiting proprioception in virtual-environment interaction , 1997, SIGGRAPH.

[4]  Rainer Stiefelhagen,et al.  qVRty: Virtual Keyboard with a Haptic, Real-World Representation , 2018, HCI.

[5]  Tatsuo Nakajima,et al.  VRowser: A Virtual Reality Parallel Web Browser , 2018, HCI.

[6]  Meredith Ringel Morris,et al.  User-defined gestures for surface computing , 2009, CHI.

[7]  Robert W. Lindeman,et al.  Hand-held windows: towards effective 2D interaction in immersive virtual environments , 1999, Proceedings IEEE Virtual Reality (Cat. No. 99CB36316).

[8]  Ronald Azuma,et al.  A Survey of Augmented Reality , 1997, Presence: Teleoperators & Virtual Environments.

[9]  Doug A. Bowman,et al.  An evaluation of techniques for grabbing and manipulating remote objects in immersive virtual environments , 1997, SI3D.

[10]  Ivan Poupyrev,et al.  An Introduction to 3-D User Interface Design , 2001, Presence: Teleoperators & Virtual Environments.

[11]  Holger Regenbrecht,et al.  A tangible AR desktop environment , 2001, Comput. Graph..

[12]  Adrian Heinrich Hoppe,et al.  VirtualTablet: Extending Movable Surfaces with Touch Interaction , 2019, 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR).

[13]  Emad Barsoum,et al.  WebVR: an interactive web browser for virtual environments , 2005, IS&T/SPIE Electronic Imaging.

[14]  Gerhard Leitner,et al.  An evaluation of sticky and force enhanced targets in multi target situations , 2006, NordiCHI '06.

[15]  R. Stiefelhagen,et al.  Applying force fields to black-box guls using computer vision , 2013, 2013 1st IEEE Workshop on User-Centered Computer Vision (UCCV).