Performance effects of multi-sensory displays in virtual teleoperation environments

Multi-sensory displays provide information to users through multiple senses, not only through visuals. They can be designed for the purpose of creating a more-natural interface for users or reducing the cognitive load of a visual-only display. However, because multi-sensory displays are often application-specific, the general advantages of multi-sensory displays over visual-only displays are not yet well understood. Moreover, the optimal amount of information that can be perceived through multi-sensory displays without making them more cognitively demanding than a visual-only displays is also not yet clear. Last, the effects of using redundant feedback across senses on multi-sensory displays have not been fully explored. To shed some light on these issues, this study evaluates the effects of increasing the amount of multi-sensory feedback on an interface, specifically in a virtual teleoperation context. While objective data showed that increasing the number of senses in the interface from two to three led to an improvement in performance, subjective feedback indicated that multi-sensory interfaces with redundant feedback may impose an extra cognitive burden on users.

[1]  J. V. Erp,et al.  Vibrotactile in-vehicle navigation system , 2004 .

[2]  Mica R. Endsley,et al.  Theoretical Underpinnings of Situation Awareness, A Critical Review , 2000 .

[3]  I. Herbst,et al.  Comparing force magnitudes by means of vibro-tactile, auditory, and visual feedback , 2005, IEEE International Workshop on Haptic Audio Visual Environments and their Applications.

[4]  Robert W. Lindeman,et al.  Controller design for a wearable, near-field haptic display , 2003, 11th Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems, 2003. HAPTICS 2003. Proceedings..

[5]  Tapio Lokki,et al.  Comparison of auditory, visual, and audiovisual navigation in a 3D space , 2005, TAP.

[6]  Katherine M. Tsui,et al.  Hand and finger registration for multi-touch joysticks on software-based operator control units , 2011, 2011 IEEE Conference on Technologies for Practical Robot Applications.

[7]  Wendy E. Mackay,et al.  CHI '13 Extended Abstracts on Human Factors in Computing Systems , 2013, CHI 2013.

[8]  Kara A. Latorella,et al.  The Scope and Importance of Human Interruption in Human-Computer Interaction Design , 2002, Hum. Comput. Interact..

[9]  Benoît Bolmont,et al.  Vibrotactile Pattern Recognition: A Portable Compact Tactile Matrix , 2012, IEEE Transactions on Biomedical Engineering.

[10]  Mel Slater,et al.  Using Presence Questionnaires in Reality , 2000, Presence: Teleoperators & Virtual Environments.

[11]  H. Yanco,et al.  Analysis of Human-Robot Interaction for Urban Search and Rescue , 2006 .

[12]  Robert W. Lindeman,et al.  Hear-Through and Mic-Through Augmented Reality: Using Bone Conduction to Display Spatialized Audio , 2007, 2007 6th IEEE and ACM International Symposium on Mixed and Augmented Reality.

[13]  Jean Scholtz,et al.  Common metrics for human-robot interaction , 2006, HRI '06.

[14]  Stéphane Natkin,et al.  The Roles of Spatial Auditory Perception and Cognition in the Accessibility of a Game Map with a First Person View , 2007, Int. J. Intell. Games Simul..

[15]  William R. Provancher,et al.  Mobile Navigation Using Haptic, Audio, and Visual Direction Cues with a Handheld Test Platform , 2012, IEEE Transactions on Haptics.

[16]  Robert S. Kennedy,et al.  Simulator Sickness Questionnaire: An enhanced method for quantifying simulator sickness. , 1993 .

[17]  Frederick P. Brooks,et al.  Moving objects in space: exploiting proprioception in virtual-environment interaction , 1997, SIGGRAPH.

[18]  David B. Kaber,et al.  Investigation of multi-modal interface features for adaptive automation of a human-robot system , 2006, Int. J. Hum. Comput. Stud..

[19]  Sandra G. Hart,et al.  Nasa-Task Load Index (NASA-TLX); 20 Years Later , 2006 .

[20]  Robert W. Lindeman.  Virtual Contact: The Continuum from Purely Visual to Purely Physical , 2003 .

[21]  Brian A. Weiss,et al.  Test arenas and performance metrics for urban search and rescue robots , 2003, Proceedings 2003 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2003) (Cat. No.03CH37453).

[22]  Ivan Poupyrev,et al.  3D User Interfaces: Theory and Practice , 2004 .

[23]  Mark Billinghurst,et al.  The use of sketch maps to measure cognitive maps of virtual environments , 1995, Proceedings Virtual Reality Annual International Symposium '95.

[24]  Michael A. Goodrich,et al.  Ecological Interfaces for Improving Mobile Robot Teleoperation , 2007, IEEE Transactions on Robotics.

[25]  Linda R. Elliott,et al.  Comparing the effects of visual-auditory and visual-tactile feedback on user performance: a meta-analysis , 2006, ICMI '06.

[26]  R. Riener,et al.  Augmented visual, auditory, haptic, and multimodal feedback in motor learning: A review , 2012, Psychonomic Bulletin & Review.

[27]  J. Rinehart,et al.  U . S . Patent , 2006 .

[28]  Martin Pielot,et al.  Tactile Wayfinder: Comparison of Tactile Waypoint Navigation with Commercial Pedestrian Navigation Systems , 2010, Pervasive.

[29]  Masatoshi Ishikawa,et al.  Augmenting spatial awareness with Haptic Radar , 2006, 2006 10th IEEE International Symposium on Wearable Computers.

[30]  Kristopher J. Blom,et al.  Virtual collision notification , 2010, 2010 IEEE Symposium on 3D User Interfaces (3DUI).

[31]  Matthew O. Ward,et al.  Enhancing robot teleoperator situation awareness and performance using vibro-tactile and graphical feedback , 2011, 2011 IEEE Symposium on 3D User Interfaces (3DUI).

[32]  Martin Pielot,et al.  PocketNavigator: studying tactile navigation systems in-situ , 2012, CHI.

[33]  Robert W. Lindeman,et al.  Poster: Comparing vibro-tactile feedback modes for collision proximity feedback in USAR virtual robot teleoperation , 2012, 2012 IEEE Symposium on 3D User Interfaces (3DUI).

[34]  J. Sibert,et al.  Vibrotactile Feedback for Enhanced Control of Urban Search and Rescue Robots , 2007 .

[35]  Jean Scholtz,et al.  Design guidelines for improved human-robot interaction , 2004, CHI EA '04.

[36]  H. Bülthoff,et al.  Merging the senses into a robust percept , 2004, Trends in Cognitive Sciences.

[37]  Jacek Gwizdka.  Using stroop task to assess cognitive load , 2010, ECCE.

[38]  Kazuhiko Kawamura,et al.  Evaluation of an enhanced human-robot interface , 2003, SMC'03 Conference Proceedings. 2003 IEEE International Conference on Systems, Man and Cybernetics. Conference Theme - System Security and Assurance (Cat. No.03CH37483).

[39]  Roope Raisamo,et al.  Orientation Inquiry: A New Haptic Interaction Technique for Non-visual Pedestrian Navigation , 2012, EuroHaptics.