Magnoramas: Magnifying Dioramas for Precise Annotations in Asymmetric 3D Teleconsultation

When users create hand-drawn annotations in Virtual Reality they often reach their physical limits in terms of precision, especially if the region to be annotated is small. One intuitive solution employs magnification beyond natural scale. However, scaling the whole environment results in wrong assumptions about the coherence between physical and virtual space. In this paper, we introduce Mag-noramas, a novel interaction method for selecting and extracting a region of interest that the user can subsequently scale and transform inside the virtual space. Our technique enhances the user's capabilities to perform supernaturally precise virtual annotations on virtual objects. We explored our technique in a user study within asimplified clinical scenario of a teleconsultation-supported craniectomy procedure that requires accurate annotations on a human head. Teleconsultation was performed asymmetrically between a remote expert in Virtual Reality that collaborated with a local user through Augmented Reality. The remote expert operates inside a reconstructed environment, captured from RGB-D sensors at the local site, and is embodied by an avatar to establish co-presence. The results show that Magnoramas significantly improve the precision of annotations while preserving usability and perceived presence measures compared to the baseline method. By hiding the 3D reconstruction while keeping the Magnorama, users can intentionally choose to lower their perceived social presence and focus on their tasks.

[1]  S. Vandenberg,et al.  Mental Rotations, a Group Test of Three-Dimensional Spatial Visualization , 1978, Perceptual and motor skills.

[2]  Marc Erich Latoschik,et al.  Beyond Replication: Augmenting Social Behaviors in Multi-User Virtual Realities , 2018, 2018 IEEE Conference on Virtual Reality and 3D User Interfaces (VR).

[3]  Randy Pausch,et al.  Virtual reality on a WIM: interactive worlds in miniature , 1995, CHI '95.

[4]  Ferran Argelaguet,et al.  Complexity and Occlusion Management for the World-in-Miniature Metaphor , 2009, Smart Graphics.

[5]  Behrang Keshavarz,et al.  Validating an Efficient Method to Quantify Motion Sickness , 2011, Hum. Factors.

[6]  Mayra Donaji Barrera Machuca,et al.  Smart3DGuides: Making Unconstrained Immersive 3D Drawing More Accurate , 2019, VRST.

[7]  Hannes Kaufmann,et al.  Mutual collision avoidance during walking in real and collaborative virtual environments , 2018, I3D.

[8]  Marc Erich Latoschik,et al.  Construction of the Virtual Embodiment Questionnaire (VEQ) , 2020, IEEE Transactions on Visualization and Computer Graphics.

[9]  Johann Habakuk Israel,et al.  Investigating the Learnability of Immersive Free-Hand Sketching , 2010, SBIM.

[10]  Hannes Kaufmann,et al.  Immersivedeck: a large-scale wireless VR system for multiple users , 2016, 2016 IEEE 9th Workshop on Software Engineering and Architectures for Realtime Interactive Systems (SEARIS).

[11]  Bruce H. Thomas,et al.  Superman vs Giant: A Study on Spatial Perception for a Multi-Scale Mixed Reality Flying Telepresence Interface , 2018, IEEE Transactions on Visualization and Computer Graphics.

[12]  Thierry Duval,et al.  Vishnu: virtual immersive support for HelpiNg users an interaction paradigm for collaborative remote guiding in mixed reality , 2016, 2016 IEEE Third VR International Workshop on Collaborative Virtual Environments (3DCVE).

[13]  Marc Erich Latoschik,et al.  Socially immersive avatar-based communication , 2017, 2017 IEEE Virtual Reality (VR).

[14]  Robert W. Lindeman,et al.  Mini-Me: An Adaptive Avatar for Mixed Reality Remote Collaboration , 2018, CHI.

[15]  Ivan Poupyrev,et al.  The MagicBook: a transitional AR interface , 2001, Comput. Graph..

[16]  Christian Sandor,et al.  Context-Based 3D Grids for Augmented Reality User Interfaces , 2020, UIST.

[17]  C. Moore,et al.  Joint attention : its origins and role in development , 1995 .

[18]  Massimo Bergamasco,et al.  Beaming: An Asymmetric Telepresence System , 2012, IEEE Computer Graphics and Applications.

[19]  Max Mühlhäuser,et al.  VRSketchPen: Unconstrained Haptic Assistance for Sketching in Virtual 3D Environments , 2020, VRST.

[20]  Mel Slater,et al.  The impact of avatar realism and eye gaze control on perceived quality of communication in a shared immersive virtual environment , 2003, CHI '03.

[21]  石原 忍,et al.  The series of plates designed as tests for colour-blindness , 1973 .

[22]  Marc Erich Latoschik,et al.  Not Alone Here?! Scalability and User Experience of Embodied Ambient Crowds in Distributed Social Virtual Reality , 2019, IEEE Transactions on Visualization and Computer Graphics.

[23]  Sandra G. Hart,et al.  Nasa-Task Load Index (NASA-TLX); 20 Years Later , 2006 .

[24]  Rubaiat Habib Kazi,et al.  Experimental Evaluation of Sketching on Surfaces in VR , 2017, CHI.

[25]  S. Hart,et al.  Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research , 1988 .

[26]  Randy F. Pausch,et al.  Voodoo dolls: seamless interaction at multiple scales in virtual environments , 1999, SI3D.

[27]  Marc Erich Latoschik,et al.  A simplified inverse kinematic approach for embodied VR applications , 2016, 2016 IEEE Virtual Reality (VR).

[28]  Ferran Argelaguet,et al.  Multi-scale Manipulation in Indoor Scenes with theWorld in Miniature Metaphor , 2009, EGVE/ICAT/EuroVR.

[29]  Marc Erich Latoschik,et al.  Avatar realism and social interaction quality in virtual reality , 2016, 2016 IEEE Virtual Reality (VR).

[30]  Marc Erich Latoschik,et al.  Technologies for Social Augmentations in User-Embodied Virtual Reality , 2019, VRST.

[31]  Takeo Kanade,et al.  Virtual Space Teleconferencing Using a Sea of Cameras , 1994 .

[32]  Mel Slater,et al.  The Sense of Embodiment in Virtual Reality , 2012, PRESENCE: Teleoperators and Virtual Environments.

[33]  Charles T. Loop,et al.  Holoportation: Virtual 3D Teleportation in Real-time , 2016, UIST.

[34]  H. Fuchs,et al.  A First Look at a Telepresence System with Room-Sized Real-Time 3 D Capture and Life-Sized Tracked Display Wall , 2011 .

[35]  Mayra Donaji Barrera Machuca,et al.  The Effect of Spatial Ability on Immersive 3D Drawing , 2019, Creativity & Cognition.

[36]  James D. Hollan,et al.  Beyond being there , 1992, CHI.

[37]  Tomasz Bednarz,et al.  A systematic review of the current state of collaborative mixed reality technologies: 2013–2018 , 2019, AIMS Electronics and Electrical Engineering.

[38]  Maria V. Sanchez-Vives,et al.  First Person Experience of Body Transfer in Virtual Reality , 2010, PloS one.

[39]  Michael C. Yip,et al.  ARTEMIS: Mixed-Reality Environment for Immersive Surgical Telementoring , 2020, CHI Extended Abstracts.

[40]  Mayra Donaji Barrera Machuca,et al.  Multiplanes: Assisted Freehand VR Sketching , 2018, SUI.

[41]  Doug A. Bowman,et al.  Wayfinding techniques for multiScale virtual environments , 2009, 2009 IEEE Symposium on 3D User Interfaces.

[42]  Mel Slater,et al.  Meeting people vitually: experiments in shared virtual environments , 2002 .

[43]  Steven K. Feiner,et al.  Virtual Replicas for Remote Assistance in Virtual and Augmented Reality , 2015, UIST.

[44]  Fotis Sotiropoulos,et al.  Slice WIM: a multi-surface, multi-touch interface for overview+detail exploration of volume datasets in virtual reality , 2011, SI3D.

[45]  Dirk Heylen,et al.  With a little help from a holographic friend: the OpenIMPRESS mixed reality telepresence toolkit for remote collaboration systems , 2018, VRST.

[46]  Frank Biocca,et al.  The Effect of the Agency and Anthropomorphism on Users' Sense of Telepresence, Copresence, and Social Presence in Virtual Environments , 2003, Presence: Teleoperators & Virtual Environments.

[47]  Doug A. Bowman,et al.  Overcoming World in Miniature Limitations by a Scaled and Scrolling WIM , 2006, 3D User Interfaces (3DUI'06).

[48]  Marc Erich Latoschik,et al.  Hybrid Avatar-Agent Technology – A Conceptual Step Towards Mediated “Social” Virtual Reality and its Respective Challenges , 2015, i-com.

[49]  Christian Geiger,et al.  A Literature Review on Collaboration in Mixed Reality , 2018, REV.

[50]  Mark Billinghurst,et al.  CoVAR: a collaborative virtual and augmented reality system for remote collaboration , 2017, SIGGRAPH ASIA Emerging Technologies.

[51]  Andrew Bluff,et al.  Don’t Panic: Recursive Interactions in a Miniature Metaworld , 2019, VRCAI.

[52]  Eyal Ofek,et al.  Room2Room: Enabling Life-Size Telepresence in a Projected Augmented Reality Environment , 2016, CSCW.

[53]  Naokazu Yokoya,et al.  A collaboration support technique by integrating a shared virtual reality and a shared augmented reality , 1999, IEEE SMC'99 Conference Proceedings. 1999 IEEE International Conference on Systems, Man, and Cybernetics (Cat. No.99CH37028).

[54]  Michael Weinmann,et al.  SLAMCast: Large-Scale, Real-Time 3D Reconstruction and Streaming for Immersive Multi-Client Live Telepresence , 2018, IEEE Transactions on Visualization and Computer Graphics.

[55]  Gary Burnett,et al.  Augmented Mirrors , 2016 .

[56]  Tovi Grossman,et al.  Loki: Facilitating Remote Instruction of Physical Tasks Using Bi-Directional Mixed-Reality Telepresence , 2019, UIST.

[57]  Enrico Rukzio,et al.  VRSketchIn: Exploring the Design Space of Pen and Tablet Interaction for 3D Sketching in Virtual Reality , 2020, CHI.

[58]  J. B. Brooke,et al.  SUS: A 'Quick and Dirty' Usability Scale , 1996 .

[59]  Bernd Fröhlich,et al.  Immersive Group-to-Group Telepresence , 2013, IEEE Transactions on Visualization and Computer Graphics.

[60]  Steven K. Feiner,et al.  3D referencing techniques for physical objects in shared augmented reality , 2012, 2012 IEEE International Symposium on Mixed and Augmented Reality (ISMAR).

[61]  Dieter Schmalstieg,et al.  Indoor navigation with mixed reality world-in-miniature views and sparse localization on mobile devices , 2012, AVI.

[62]  Richard L. Daft,et al.  Organizational information requirements, media richness and structural design , 1986 .

[63]  Marc Levoy,et al.  Zippered polygon meshes from range images , 1994, SIGGRAPH.

[64]  Takuji Narumi,et al.  Virtual Co-Embodiment: Evaluation of the Sense of Agency while Sharing the Control of a Virtual Body among Two Individuals , 2020, IEEE transactions on visualization and computer graphics.

[65]  Boris M. Velichkovsky,et al.  The perception of egocentric distances in virtual environments - A review , 2013, ACM Comput. Surv..

[66]  Arindam Dey,et al.  The Effects of Sharing Awareness Cues in Collaborative Mixed Reality , 2019, Front. Robot. AI.