Effects of modality on virtual button motion and performance

The simple action of pressing a button is a multimodal interaction with an interesting depth of complexity. As the development of computer interfaces supporting 3D tasks progresses, there is a need to understand how users will interact with virtual buttons that generate multimodal feedback. Using a phone number dialing task on a virtual keypad, this study examined the effects of visual, auditory, and haptic feedback combinations on task performance and on the motion of individual button presses. The results suggest that the resistance of haptic feedback alone was not enough to prevent participants from pressing the button farther than necessary. Reinforcing haptic feedback with visual or auditory feedback shortened the depth of the presses significantly. However, the shallower presses that occurred with trimodal feedback may have led participants to release some buttons too early, which may explain an unexpected increase in mistakes where the participant missed digits from the phone number.

[1]  Shumin Zhai,et al.  The performance of touch screen soft buttons , 2009, CHI.

[2]  F. Colavita.  Human sensory dominance , 1974 .

[3]  H S Vitense,et al.  Multimodal feedback: an assessment of performance and mental workload , 2003, Ergonomics.

[4]  C. Davis.  Touch , 1997, The Lancet.

[5]  Stephen A. Brewster,et al.  Crossmodal congruence: the look, feel and sound of touchscreen widgets , 2008, ICMI '08.

[6]  H. McGurk,et al.  Hearing lips and seeing voices , 1976, Nature.

[7]  P. Walker,et al.  Stroop Interference Based on the Multimodal Correlates of Haptic Size and Auditory Pitch , 1985, Perception.

[8]  Tyler Blake,et al.  Feedback and Key Discrimination on Membrane Keypads , 1984 .

[9]  M. Reiner,et al.  Sensory dominance in combinations of audio, visual and haptic stimuli , 2009, Experimental Brain Research.

[10]  C. Spence,et al.  Crossmodal links between vision and touch in covert endogenous spatial attention. , 2000, Journal of experimental psychology. Human perception and performance.

[11]  Sharif Razzaque,et al.  Tactile virtual buttons for mobile devices , 2003, CHI Extended Abstracts.

[12]  Diego Borro,et al.  Influence of multisensory feedback on haptic accessibility tasks , 2006, Virtual Reality.

[13]  Mandayam A. Srinivasan,et al.  The Effect of Auditory Cues on the Haptic Perception of Stiffness in Virtual Environments , 1997, Dynamic Systems and Control.

[14]  Karen M. Cohen.  Membrane Keyboards and Human Performance , 1982 .

[15]  J. Murray,et al.  HANDBOOK OF PSYCHOLOGY , 1951 .

[16]  Christopher D. Wickens,et al.  Multiple resources and performance prediction , 2002 .

[17]  James R. Lewis,et al.  Keys and Keyboards , 1997 .

[18]  John Long.  Effects of Delayed Irregular Feedback on Unskilled and Skilled Keying Performance , 1976 .

[19]  M HERSHENSON,et al.  Reaction time as a measure of intersensory facilitation. , 1962, Journal of experimental psychology.

[20]  Steven van de Par,et al.  Auditory-visual interaction: from fundamental research in cognitive psychology to (possible) applications , 1999, Electronic Imaging.

[21]  Michael J. Fischer,et al.  The String-to-String Correction Problem , 1974, JACM.

[22]  M. A. Srinivassan.  The impact of visual information on the haptic perception of stiffness in virtual environments , 1996 .

[23]  Anatole Lécuyer,et al.  "Boundary of illusion": an experiment of sensory integration with a pseudo-haptic system , 2001, Proceedings IEEE Virtual Reality 2001.

[24]  M. Heller,et al.  Intersensory conflict between vision and touch: The response modality dominates when precise, attention-riveting judgments are required , 1999, Perception & psychophysics.

[25]  Miriam Reiner,et al.  Enhancement of response times to bi- and tri-modal sensory stimuli during active movements , 2008, Experimental Brain Research.

[26]  Thomas H. Massie,et al.  The PHANToM Haptic Interface: A Device for Probing Virtual Objects , 1994 .

[27]  Y. Guiard.  Asymmetric division of labor in human skilled bimanual action: the kinematic chain as a model. , 1987, Journal of motor behavior.

[28]  Linda R. Elliott,et al.  Comparing the effects of visual-auditory and visual-tactile feedback on user performance: a meta-analysis , 2006, ICMI '06.

[29]  Sharon Oviatt,et al.  Multimodal Interfaces , 2008, Encyclopedia of Multimedia.

[30]  Vladimir I. Levenshtein,et al.  Binary codes capable of correcting deletions, insertions, and reversals , 1965 .

[31]  M B Cooper,et al.  The effect of feedback on keying performance. , 1979, Applied ergonomics.