Evaluating the Combination of Visual Communication Cues for HMD-based Mixed Reality Remote Collaboration

Many researchers have studied various visual communication cues (e.g. pointer, sketching, and hand gesture) in Mixed Reality remote collaboration systems for real-world tasks. However, the effect of combining them has not been so well explored. We studied the effect of these cues in four combinations: hand only, hand + pointer, hand + sketch, and hand + pointer + sketch, with three problem tasks: Lego, Tangram, and Origami. The study results showed that the participants completed the task significantly faster and felt a significantly higher level of usability when the sketch cue is added to the hand gesture cue, but not with adding the pointer cue. Participants also preferred the combinations including hand and sketch cues over the other combinations. However, using additional cues (pointer or sketch) increased the perceived mental effort and did not improve the feeling of co-presence. We discuss the implications of these results and future research directions.

[1]  Tom Rodden,et al.  Turn it this way: grounding collaborative action with remote gestures , 2007, CHI.

[2]  Jun Rekimoto,et al.  JackIn: integrating first-person view with out-of-body vision generation for human-human augmentation , 2014, AH.

[3]  Susan R. Fussell,et al.  Using linguistic features to measure presence in computer-mediated communication , 2006, CHI.

[4]  Susan R. Fussell,et al.  Gestures Over Video Streams to Support Remote Collaboration on Physical Tasks , 2004, Hum. Comput. Interact..

[5]  John C. Tang,et al.  Videodraw: a video interface for collaborative drawing , 1991, TOIS.

[6]  Scott Bateman,et al.  Stabilized Annotations for Mobile Remote Assistance , 2016, CHI.

[7]  Arindam Dey,et al.  Improving Collaboration in Augmented Video Conference using Mutually Shared Gaze , 2017, ICAT-EGVE.

[8]  Jonathan Grudin,et al.  Integration of interpersonal space and shared workspace: ClearBoard design and experiments , 1993, TOIS.

[9]  Tobias Höllerer,et al.  Integrating the physical environment into mobile remote collaboration , 2012, Mobile HCI.

[10]  S. Goldin-Meadow,et al.  The role of gesture in communication and thinking , 1999, Trends in Cognitive Sciences.

[11]  Mark Billinghurst,et al.  Study of augmented gesture communication cues and view sharing in remote collaboration , 2013, 2013 IEEE International Symposium on Mixed and Augmented Reality (ISMAR).

[12]  David S. Kirk,et al.  Comparing remote gesture technologies for supporting collaborative physical tasks , 2006, CHI.

[13]  F.R.H. Zijlstra,et al.  Efficiency in work behaviour: A design approach for modern tools , 1993 .

[14]  Seungwon Kim,et al.  Comparing pointing and drawing for remote collaboration , 2013, ISMAR.

[15]  David A. Forsyth,et al.  BeThere: 3D mobile collaboration with spatial input , 2013, CHI.

[16]  Morten Fjeld,et al.  SEMarbeta: mobile sketch-gesture-video remote support for car drivers , 2013, AH.

[17]  Frank Weichert,et al.  Analysis of the Accuracy and Robustness of the Leap Motion Controller , 2013, Sensors.

[18]  Mark Billinghurst,et al.  Do You See What I See? The Effect of Gaze Tracking on Task Space Remote Collaboration , 2016, IEEE Transactions on Visualization and Computer Graphics.

[19]  Mark Billinghurst,et al.  Using Freeze Frame and Visual Notifications in an Annotation Drawing Interface for Remote Collaboration , 2018, KSII Trans. Internet Inf. Syst..

[20]  J. J. Higgins,et al.  The aligned rank transform for nonparametric factorial analyses using only anova procedures , 2011, CHI.

[21]  Mark Billinghurst,et al.  The Effect of Collaboration Styles and View Independence on Video-Mediated Remote Collaboration , 2018, Computer Supported Cooperative Work (CSCW).

[22]  Weidong Huang,et al.  Gesturing in the Air: Supporting Full Mobility in Remote Collaboration on Physical Tasks , 2013, J. Univers. Comput. Sci..

[23]  Tobias Höllerer,et al.  World-stabilized annotations and virtual scene navigation for remote collaboration , 2014, UIST.

[24]  Henry Been-Lirn Duh,et al.  Augmented 3D hands: a gesture-based mixed reality system for distributed collaboration , 2017, Journal on Multimodal User Interfaces.

[25]  Carl Gutwin,et al.  Implications of We-Awareness to the Design of Distributed Groupware Tools , 2016, Computer Supported Cooperative Work (CSCW).

[26]  Hirokazu Kato,et al.  Marker tracking and HMD calibration for a video-based augmented reality conferencing system , 1999, Proceedings 2nd IEEE and ACM International Workshop on Augmented Reality (IWAR'99).

[27]  J. B. Brooke,et al.  SUS: A 'Quick and Dirty' Usability Scale , 1996 .

[28]  Tom Rodden,et al.  Ways of the Hands , 2005, ECSCW.

[29]  Michael B. McCamy,et al.  Microsaccade and drift dynamics reflect mental fatigue , 2013, The European journal of neuroscience.

[30]  Matt Adcock,et al.  Hand gestures and visual annotation in live 360 panorama-based mixed reality remote collaboration , 2018, OZCHI.

[31]  Wendy Ju,et al.  TEI 2015 chairs' welcome , 2015 .

[32]  Keita Higuchi,et al.  Can Eye Help You?: Effects of Visualizing Eye Fixations on Remote Collaboration Scenarios for Physical Tasks , 2016, CHI.

[33]  Jun Rekimoto,et al.  LiveSphere: Sharing the Surrounding Visual Environment for Immersive Experience in Remote Collaboration , 2015, Tangible and Embedded Interaction.

[34]  Lei Gao,et al.  An oriented point-cloud view for MR remote collaboration , 2016, SIGGRAPH ASIA Mobile Graphics and Interactive Applications.

[35]  Daniel G. Bobrow,et al.  WYSIWIS revised: early experiences with multiuser interfaces , 1987, TOIS.

[36]  Mark Billinghurst,et al.  Improving co-presence with augmented visual communication cues for sharing experience through video conference , 2014, 2014 IEEE International Symposium on Mixed and Augmented Reality (ISMAR).

[37]  Mark Billinghurst,et al.  The Effect of View Independence in a Collaborative AR System , 2015, Computer Supported Cooperative Work (CSCW).

[38]  Philip T. Kortum,et al.  Determining what individual SUS scores mean: adding an adjective rating scale , 2009 .

[39]  Sangtae Ha,et al.  Automatically Freezing Live Video for Annotation during Remote Collaboration , 2015, CHI Extended Abstracts.

[40]  Francisco M. Costela,et al.  Task difficulty in mental arithmetic affects microsaccadic rates and magnitudes , 2014, The European journal of neuroscience.

[41]  Joerg Hauber,et al.  Understanding Remote Collaboration in Video Collaborative Virtual Environments , 2008 .

[42]  Hideaki Kuzuoka,et al.  Can the GestureCam be a Surrogate? , 1995, ECSCW.

[43]  Robert E. Kraut,et al.  Effects of head-mounted and scene-oriented video systems on remote collaboration on physical tasks , 2003, CHI '03.

[44]  Mark Billinghurst,et al.  A User Study on MR Remote Collaboration Using Live 360 Video , 2018, 2018 IEEE International Symposium on Mixed and Augmented Reality (ISMAR).

[45]  Hideaki Kuzuoka,et al.  WACL: supporting telecommunications using - wearable active camera with laser pointer , 2003, Seventh IEEE International Symposium on Wearable Computers, 2003. Proceedings..