Comparing the effects of visual-auditory and visual-tactile feedback on user performance: a meta-analysis

In a meta-analysis of 43 studies, we examined the effects of multimodal feedback on user performance, comparing visual-auditory and visual-tactile feedback to visual feedback alone. Results indicate that adding an additional modality to visual feedback improves performance overall. Both visual-auditory feedback and visual-tactile feedback provided advantages in reducing reaction times and improving performance scores, but were not effective in reducing error rates. Effects are moderated by task type, workload, and number of tasks. Visual-auditory feedback is most effective when a single task is being performed (g = .87), and under normal workload conditions (g = .71). Visual-tactile feedback is more effective when multiple tasks are begin performed (g = .77) and workload conditions are high (g = .84). Both types of multimodal feedback are effective for target acquisition tasks; but vary in effectiveness for other task types. Implications for practice and research are discussed.

[1]  Joel S. Warm,et al.  Evaluating Adaptive Multisensory Displays for Target Localization in a Flight Task , 2004 .

[2]  John Vavrik,et al.  Effect of a Concurrent Auditory Task on Visual Search Performance in a Driving-Related Image-Flicker Task , 2002, Hum. Factors.

[3]  Adelbert W. Bronkhorst,et al.  Application of a Three-Dimensional Auditory Display in a Flight Task , 1996, Hum. Factors.

[4]  Y C Liu,et al.  Comparative study of the effects of auditory, visual and multimodality displays on drivers' performance in advanced traveller information systems , 2001, Ergonomics.

[5]  Andy Cockburn,et al.  Improving the Acquisition of Small Targets , 2004 .

[6]  L. Hedges,et al.  Statistical Methods for Meta-Analysis , 1987 .

[7]  Timothy L. White,et al.  Effects of Visual and Auditory Cues About Threat Location on Target Acquisition and Attention to Auditory Communications , 2005 .

[8]  Motoyuki Akamatsu,et al.  A multi-modal mouse with tactile and force feedback , 1994, Int. J. Hum. Comput. Stud..

[9]  Arvin Agah,et al.  Multi-Modal Human Interactions with an Intelligent Interface Utilizing Images, Sounds, and Force Feedback , 2001, J. Intell. Robotic Syst..

[10]  Adrian David Cheok,et al.  An experimental study on the role of software synthesized 3D sound in augmented reality environments , 2004, Interact. Comput..

[11]  Ian Oakley,et al.  Putting the feel in ’look and feel‘ , 2000, CHI.

[12]  W. Barfield,et al.  The use of icons, earcons, and commands in the design of an online hierarchical menu , 1991 .

[13]  T Z Strybel,et al.  Aurally Aided Visual Search in the Central Visual Field: Effects of Visual Load and Visual Enhancement of the Target , 1991, Human factors.

[14]  Robert S. Bolia,et al.  Aurally Aided Visual Search in Three-Dimensional Space , 1999, Hum. Factors.

[15]  Stephen A. Brewster,et al.  Correcting menu usability problems with sound , 1999, Behav. Inf. Technol..

[16]  R. Mayer,et al.  A Split-Attention Effect in Multimedia Learning: Evidence for Dual Processing Systems in Working Memory , 1998 .

[17]  Robert W. Lindeman,et al.  Effectiveness of directional vibrotactile cuing on a building-clearing task , 2005, CHI.

[18]  Benjamin A. Clegg,et al.  Interruption Management: The Use of Attention-Directing Tactile Cues , 2005, Hum. Factors.

[19]  Robert W. Lindeman,et al.  Effective Vibrotactile Cueing in a Visual Search Task , 2003, INTERACT.

[20]  Angus H. Rupert,et al.  VIBROTACTILE CUEING AS A MASTER CAUTION SYSTEM FOR VISUAL MONITORING , 2002 .

[21]  David Shinar,et al.  Imperfect In-Vehicle Collision Avoidance Warning Systems Can Aid Drivers , 2004, Hum. Factors.

[22]  Patrick Langdon,et al.  Multiple haptic targets for motion-impaired computer users , 2003, CHI '03.

[23]  Christopher D. Wickens,et al.  Multiple resources and performance prediction , 2002 .

[24]  J. V. Erp,et al.  Vibrotactile in-vehicle navigation system , 2004 .

[25]  Pitu B. Mirchandani.  An Auditory Display in a Dual-Axis Tracking Task , 1972, IEEE Trans. Syst. Man Cybern..

[26]  Jan B F van Erp,et al.  Cross-modal visual and vibrotactile tracking. , 2004, Applied ergonomics.

[27]  Christopher D. Wickens,et al.  Effects of Data-Link Modality and Display Redundancy on Pilot Performance: An Attentional Perspective , 2003 .

[28]  Stephen Brewster,et al.  Using non-speech sound to overcome information overload , 1997 .

[29]  Cristiana Cavina-Pratesi,et al.  Redundant target effect and intersensory facilitation from visual-tactile interactions in simple reaction time , 2002, Experimental Brain Research.

[30]  Marilyn Rose McGee.  A haptically enhanced scrollbar : Force-Feedback as a means of reducing the problems associated with scrolling , 1999 .

[31]  Susan J. Lederman,et al.  Virtual peg-in-hole performance using a 6-DOF magnetic levitation haptic device: comparison with real forces and with visual guidance alone , 2002, Proceedings 10th Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems. HAPTICS 2002.

[32]  Arthur F. Kramer,et al.  Comparison of tactile and visual feedback for a multi-state input mechanism , 1997, Proceedings of the 19th Annual International Conference of the IEEE Engineering in Medicine and Biology Society. 'Magnificent Milestones and Emerging Opportunities in Medical Engineering' (Cat. No.97CH36136).

[33]  Christopher D. Wickens,et al.  Driving and Side Task Performance: The Effects of Display Clutter, Separation, and Modality , 2004, Hum. Factors.

[34]  Ian Oakley,et al.  Tilt to scroll: evaluating a motion based vibrotactile mobile interface , 2005, First Joint Eurohaptics Conference and Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems. World Haptics Conference.

[35]  Gary S. Robinson,et al.  A New Class of Auditory Warning Signals for Complex Systems: Auditory Icons , 1999, Hum. Factors.

[36]  C. Swindells,et al.  TorqueBAR: an ungrounded haptic feedback device , 2003, ICMI '03.

[37]  Stephen Brewster.  Sonically-enhanced drag and drop , 1998 .

[38]  John Chiasson,et al.  Enhanced Situation Awareness in Sea, Air and Land Environments , 2003 .

[39]  Mark W. Lipsey,et al.  Practical Meta-Analysis , 2000 .

[40]  Atsushi Ikeda,et al.  Fundamental study on human interface of narrow road drive assist system based on drivers’ cognitive process , 2003 .

[41]  Alistair Furnell,et al.  Understanding Multisensory Integration for Pilot Spatial Orientation , 2004 .

[42]  Woodrow Barfield,et al.  Visual and Auditory Localization as a Function of Azimuth and Elevation , 1997 .

[43]  Christopher D. Wickens,et al.  Attentional Models of Multitask Pilot Performance Using Advanced Display Technology , 2003, Hum. Factors.

[44]  D. Leutner,et al.  Assessment of cognitive load in multimedia learning using dual-task methodology. , 2002, Experimental psychology.