Coordinated hybrid virtual environments: Seamless interaction contexts for effective virtual reality

Despite the benefits of presence, immersive virtual reality is still too limiting and ineffective to be widely adopted in people?s everyday lives. One important reason for this is its inability to handle highly diverse 3D interaction tasks, such as object manipulation from different scales, perspectives, reference frames, and dimensions. Inspired by preliminary development of an immersive world builder tool, this paper offers an in-depth presentation of our solution to this problem, using a coordinated, tablet- and HMD-based, hybrid virtual environment (HVE) system. Wearing a non-occlusive HMD, the user is able to view and interact with a tablet mounted on the non-dominant forearm, which provides a multi-touch interaction surface, as well as an exocentric God view of the virtual world. To reduce transition gaps across 3D interaction tasks and interfaces, four coordination mechanisms are proposed, two of which were implemented, and one was evaluated in a user study featuring complex level-editing tasks. Based on subjective ratings, task performance, interview feedback, and video analysis, we found that having multiple seamless Interaction Contexts (ICs) with complementary benefits can lead to good performance and user experience, despite the complexity of learning and using the hybrid system. The results also suggest keeping 3DUI tasks synchronized across the ICs, as this can help users understand their relationships, smoothen within- and between-task IC transitions, and inspire more creative use of different interfaces. Finally, based on these findings, a four-step design process is proposed to aid the process of designing and implementing HVE systems, as well as applying them to appropriate task scenarios. Graphical abstractThis paper proposes using a tablet- and HMD-based, coordinated hybrid virtual environment system to enable effective 3D interaction in immersive virtual reality.Display Omitted HighlightsWe built a novel hybrid virtual environment system for effective interaction in VR.We propose four coordination mechanisms to smoothen context transitions in HVEs.We conducted a user study and proved task synchronization can reduce transition gap.The study shows a complex HVE system can be mastered to benefit interaction in VR.We propose a four-step guideline to aid the design, develop, and application of HVEs.