Latency guidelines for touchscreen virtual button feedback

Touchscreens are very widely used, especially in mobile phones. They feature many interaction methods, pressing a virtual button being one of the most popular ones. In addition to an inherent visual feedback, virtual button can provide audio and tactile feedback. Since mobile phones are essentially computers, the processing causes latencies in interaction. However, it has not been known, if the latency is an issue in mobile touchscreen virtual button interaction, and what the latency recommendations for visual, audio and tactile feedback are. The research in this thesis has investigated multimodal latency in mobile touchscreen virtual button interaction. For the first time, an affordable, but accurate tool was built to measure all three feedback latencies in touchscreens. For the first time, simultaneity perception of touch and feedback, as well as the effect of latency on virtual button perceived quality has been studied and thresholds found for both unimodal and bimodal feedback. The results from these studies were combined as latency guidelines for the first time. These guidelines enable interaction designers to establish requirements for mobile phone engineers to optimise the latencies on the right level. The latency measurement tool consisted of a high-speed camera, a microphone and an accelerometer for visual, audio and tactile feedback measurements. It was built with off-the-shelf components and, in addition, it was portable. Therefore, it could be copied at low cost or moved wherever needed. The tool enables touchscreen interaction designers to validate latencies in their experiments, making their results more valuable and accurate. The tool could benefit the touchscreen phone manufacturers, since it enables engineers to validate latencies during development of mobile phones. The tool has been used in mobile phone R&D within Nokia Corporation and for validation of a research device within the University of Glasgow. The guidelines established for unimodal feedback was as follows: visual feedback latency should be between 30 and 85 ms, audio between 20 and 70 ms and tactile between 5 and 50 ms. The guidelines were found to be different for bimodal feedback: visual feedback latency should be 95 and audio 70 ms when the feedback was visual-audio, visual 100 and tactile 55 ms when the feedback was visual-tactile and tactile 25 and audio 100 ms when the feedback was tactile-audio. These guidelines will help engineers and interaction designers to select and optimise latencies to be low enough, but not too low. Designers using these guidelines will make sure that most of the users will both perceive the feedback as simultaneous with their touch and experience high quality virtual buttons. The results from this thesis show that latency has a remarkable effect on touchscreen virtual buttons, and it is a key part of virtual button feedback design. The novel results enable researchers, designers and engineers to master the effect of latencies in research and development. This will lead to more accurate and reliable research results and help mobile phone manufacturers make better products.

[1]  G. Fairbanks,et al.  Effects of delayed auditory feedback upon articulation. , 1958, Journal of speech and hearing research.

[2]  Lorna M. Brown,et al.  Tactile feedback for mobile interactions , 2007, CHI.

[3]  L. Harris,et al.  Space and Time in Perception and Action: Mechanisms of simultaneity constancy , 2010 .

[4]  Ju-Hwan Lee,et al.  The Effect of Multimodal Feedback Presented via a Touch Screen on the Performance of Older Adults , 2009, HAID.

[5]  E. Boring.  A History of Experimental Psychology. , 1930 .

[6]  Sigmund Exner,et al.  Experimentelle Untersuchung der einfachsten psychischen Processe , 1875, Archiv für die gesamte Physiologie des Menschen und der Tiere.

[7]  Bernard D. Adelstein,et al.  Head Tracking Latency in Virtual Environments: Psychophysics and a Model , 2003 .

[8]  Allan D. Shocker,et al.  Managing Brand Equity , 1991 .

[9]  Durand R. Begault,et al.  Sensitivity to haptic-audio asynchrony , 2003, ICMI '03.

[10]  Toshiaki Sugimura,et al.  Active click: tactile feedback for touch panels , 2001, CHI Extended Abstracts.

[11]  Michael I. Hill,et al.  Generalizeability of Latency Detection in a Variety of Virtual Environments , 2004 .

[12]  Laurence R. Harris,et al.  The relative timing of active and passive touch , 2008, Brain Research.

[13]  Caroline Jay,et al.  Delayed visual and haptic feedback in a reciprocal tapping task , 2005, First Joint Eurohaptics Conference and Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems. World Haptics Conference.

[14]  A. J. Yates.  Delayed Auditory Feedback , 1958, Psychological bulletin.

[15]  Seungmoon Choi,et al.  Tactile effect design and evaluation for virtual buttons on a mobile device touchscreen , 2011, Mobile HCI.

[16]  Emilia Koskinen,et al.  Optimizing Tactile Feedback for Virtual Buttons in Mobile Devices , 2008 .

[17]  Stephen A. Brewster,et al.  Crossmodal congruence: the look, feel and sound of touchscreen widgets , 2008, ICMI '08.

[18]  Chang-Franw Lee,et al.  A Study on the Icon Feedback Types of Small Touch Screen for the Elderly , 2009, HCI.

[19]  Roberto Bresin,et al.  IS THE PLAYER MORE INFLUENCED BY THE AUDITORY THAN THE TACTILE FEEDBACK FROM THE INSTRUMENT , 2001 .

[20]  Eve E. Hoggan,et al.  Playing with Tactile Feedback Latency in Touchscreen Interaction: Two Approaches , 2011, INTERACT.

[21]  Paulo V. R. Carvalho,et al.  Human centered design for nuclear power plant control room modernization , 2011 .

[22]  Dave Pape,et al.  Video-Based Measurement of System Latency , 2000 .

[23]  Lalit Wankhade,et al.  Quality Uncertainty and Quality Perception , 2010 .

[24]  Stephen A. Brewster,et al.  Investigating the effectiveness of tactile feedback for mobile touchscreens , 2008, CHI.

[25]  Eve E. Hoggan,et al.  The effect of tactile feedback latency in touchscreen interaction , 2011, 2011 IEEE World Haptics Conference.

[26]  C. Spence,et al.  Multisensory prior entry. , 2001, Journal of experimental psychology. General.

[27]  Robert B. Miller,et al.  Response time in man-computer conversational transactions , 1899, AFIPS Fall Joint Computing Conference.

[28]  Mark E. Nelson,et al.  Interactive MIDI: Real-time Performance Evaluation , 2004 .

[29]  Stephen A. Brewster,et al.  Designing audio and tactile crossmodal icons for mobile devices , 2007, ICMI '07.

[30]  Stephen A. Brewster,et al.  Towards the Temporally Perfect Virtual Button: Touch-Feedback Simultaneity and Perceived Quality in Mobile Touchscreen Press Interactions , 2014, TAP.

[31]  Jean Vroomen,et al.  Adaptation to motor-visual and motor-auditory temporal lags transfer across modalities , 2009, Experimental Brain Research.

[32]  A. F. Dorian.  Dictionary of science and technology : French-English , 1980 .

[33]  Jessica K. Hodgins,et al.  Perceptually motivated guidelines for voice synchronization in film , 2010, TAP.

[34]  Daniel J. Wigdor,et al.  Designing for low-latency direct-touch input , 2012, UIST.

[35]  Massimiliano Di Luca.  New Method to Measure End-to-End Delay of Virtual Reality , 2010, PRESENCE: Teleoperators and Virtual Environments.

[36]  D. Wolfe,et al.  Nonparametric Statistical Methods. , 1974 .

[37]  C. Spence.  Explaining the Colavita visual dominance effect. , 2009, Progress in brain research.

[38]  N. F. Dixon,et al.  The Detection of Auditory Visual Desynchrony , 1980, Perception.

[39]  R. Efron,et al.  Effect of stimulus duration on perceptual onset and offset latencies , 1970 .

[40]  M. Whitton,et al.  Effect of latency on presence in stressful virtual environments , 2003, IEEE Virtual Reality, 2003. Proceedings..

[41]  Glen J. Anderson,et al.  User Perception of Touch Screen Latency , 2011, HCI.

[42]  H S Vitense,et al.  Multimodal feedback: an assessment of performance and mental workload , 2003, Ergonomics.

[43]  Andy Cohen.  A description of MIL-STK-1472 , 1995, SGCH.

[44]  P. Laitinen,et al.  Enabling mobile haptic design: piezoelectric actuator technology properties in hand held devices , 2006, 2006 IEEE International Workshop on Haptic Audio Visual Environments and their Applications (HAVE 2006).

[45]  Amar Chaudhary,et al.  Operating Systems Latency Measurement and Analysis for Sound Synthesis and Processing Applications , 1997, ICMC.

[46]  S.J.V. Nichols.  New Interfaces at the Touch of a Fingertip , 2007 .

[47]  Marieke Rohde,et al.  To Lead and To Lag – Forward and Backward Recalibration of Perceived Visuo-Motor Simultaneity , 2013, Front. Psychology.

[48]  Stephen A. Brewster,et al.  Feedback is... late: measuring multimodal delays in mobile device touchscreen interaction , 2010, ICMI-MLMI '10.

[49]  Macy Ann Valk.  An Experiment to Study Touchscreen “Button” Design , 1985 .

[50]  Matthew Wright,et al.  Audio and Gesture Latency Measurements on Linux and OSX , 2004, ICMC.

[51]  P. R. McGoldrick,et al.  Human interface to a major fusion experiment, a graphics solution , 1979 .

[52]  Hans J. Charwat.  Lexikon der Mensch-Maschine-Kommunikation , 1992 .

[53]  Sharif Razzaque,et al.  Tactile virtual buttons for mobile devices , 2003, CHI Extended Abstracts.

[54]  Ichiro Fujinaga,et al.  Audio Latency Measurements of Desktop Operating Systems , 2001, ICMC.

[55]  Pauli Laitinen,et al.  Designing Haptic Feedback for Touch Display: Experimental Study of Perceived Intensity and Integration of Haptic and Audio , 2006, HAID.

[56]  Daniel J. Wigdor,et al.  How fast is fast enough?: a study of the effects of latency in direct-touch pointing tasks , 2013, CHI.

[57]  Karon E. MacLean,et al.  The perception of cross-modal simultaneity (or ``the Greenwich Observatory Problem'' revisited) , 2001 .

[58]  S. Finney,et al.  Auditory Feedback and Musical Keyboard Performance , 1997 .

[59]  I. Scott MacKenzie,et al.  Lag as a determinant of human performance in interactive systems , 1993, INTERCHI.

[60]  Bernard D. Adelstein,et al.  Factors influencing operator interaction with virtual objects viewed via head-mounted see-through displays: viewing conditions and rendering latency , 1997, Proceedings of IEEE 1997 Annual International Symposium on Virtual Reality.

[61]  S. I. Bennion,et al.  Touch sensitive graphics terminal applied to process control , 1981, COMG.

[62]  Colin Rose,et al.  Mathematical Statistics with Mathematica , 2002 .

[63]  M. Jammer.  Concepts of Simultaneity: From Antiquity to Einstein and Beyond , 2006 .

[64]  Stephen A. Brewster,et al.  Using mobile phones to interact with tabletop computers , 2011, ITS '11.

[65]  Steven C. Seow.  Designing and Engineering Time: The Psychology of Time Perception in Software , 2008 .

[66]  Meera Blattner,et al.  Earcons and Icons: Their Structure and Common Design Principles , 1989, Hum. Comput. Interact..

[67]  Tim Paek,et al.  A practical examination of multimodal feedback and guidance signals for mobile touchscreen keyboards , 2010, Mobile HCI.

[68]  Lorna M. Brown,et al.  Tactons: Structured Tactile Messages for Non-Visual Information Display , 2004, AUIC.

[69]  S. Coren,et al.  In Sensation and perception , 1979 .

[70]  James V. Stone,et al.  When is now? Perception of simultaneity , 2001, Proceedings of the Royal Society of London. Series B: Biological Sciences.

[71]  Anoop Gupta,et al.  In the blink of an eye: investigating latency perception during stylus interaction , 2014, CHI.

[72]  D. Whitaker,et al.  Effect before Cause: Supramodal Recalibration of Sensorimotor Timing , 2009, PloS one.

[73]  Barbara Tillmann,et al.  Thresholds of Auditory-Motor Coupling Measured with a Simple Task in Musicians and Non-Musicians: Was the Sound Simultaneous to the Key Press? , 2014, PloS one.

[74]  D. Poeppel,et al.  Temporal window of integration in auditory-visual speech perception , 2007, Neuropsychologia.

[75]  Anoop Gupta,et al.  How low should we go?: understanding the perception of latency while inking , 2014, Graphics Interface.

[76]  S. Nishida,et al.  Audio–tactile superiority over visuo–tactile and audio–visual combinations in the temporal resolution of synchrony perception , 2009, Experimental Brain Research.

[77]  M. Young,et al.  Discrimination of Changes of Latency during Voluntary Hand Movement of Virtual Objects , 1999 .

[78]  Stephen A. Brewster,et al.  Audio or tactile feedback: which modality when? , 2009, CHI.

[79]  Barbara S. Chaparro,et al.  How Important is Visual Feedback When Using a Touch Screen , 2000 .

[80]  G. A. Miller.  THE PSYCHOLOGICAL REVIEW THE MAGICAL NUMBER SEVEN, PLUS OR MINUS TWO: SOME LIMITS ON OUR CAPACITY FOR PROCESSING INFORMATION 1 , 1956 .

[81]  Sean Enda Power,et al.  Temporal Illusions - Philosophical Considerations , 2010, COST TD0904 International Workshop.

[82]  James L. Wright,et al.  System-Level MIDI Performance Testing , 2001, ICMC.

[83]  Andrew Sears,et al.  Improving Touchscreen Keyboards: Design Issues and a Comparison with Other Devices , 1991, Interact. Comput..

[84]  J Blauert,et al.  M Ami a Taxonomy of Multimodal Interaction in the Human Information Processing System , 2022 .

[85]  Andriy Pavlovych,et al.  The tradeoff between spatial jitter and latency in pointing tasks , 2009, EICS '09.

[86]  M. Kendall,et al.  Kendall's advanced theory of statistics , 1995 .

[87]  Shumin Zhai,et al.  The performance of touch screen soft buttons , 2009, CHI.

[88]  Bernard D. Adelstein,et al.  Sensor spatial distortion, visual latency, and update rate effects on 3D tracking in virtual environments , 1999, Proceedings IEEE Virtual Reality (Cat. No. 99CB36316).

[89]  Ingrid M. L. C. Vogels,et al.  Detection of Temporal Delays in Visual-Haptic Interfaces , 2004, Hum. Factors.

[90]  R. Efron,et al.  THE EFFECT OF STIMULUS INTENSITY ON THE PERCEPTION OF SIMULTANEITY IN RIGHT– AND LEFT–HANDED SUBJECTS , 1963 .

[91]  Ivan Poupyrev,et al.  Tactile interfaces for small touch screens , 2003, UIST '03.

[92]  Chris Shaw,et al.  On temporal-spatial realism in the virtual reality environment , 1991, UIST '91.

[93]  R. Miall,et al.  Adaptation to visual feedback delays in manual tracking: evidence against the Smith Predictor model of human visually guided action , 2006, Experimental Brain Research.

[94]  Durand R. Begault,et al.  Thresholds for Auditory-Tactile Asynchrony , 2005 .

[95]  Anthony Steed.  A simple method for estimating the latency of interactive, real-time graphics simulations , 2008, VRST '08.

[96]  Stephen A. Brewster,et al.  Overcoming the Lack of Screen Space on Mobile Computers , 2002, Personal and Ubiquitous Computing.

[97]  R. A. Chase,et al.  I: Effect of Delayed Auditory Feedback on Speech and Keytapping , 1961 .

[98]  Nic Herndon.  ATTITUDE: A Tidy Touchscreen Interface to a UML Development Environment , 2008 .

[99]  Gary Bishop,et al.  Latency meter: a device for easily monitoring VE delay , 2002 .

[100]  R. Hart.  Does Size Matter ? Exploring the Small Sample Properties of Maximum Likelihood Estimation , 1999 .

[101]  K. Hinckley.  Input technologies and techniques , 2002 .

[102]  Robert J. Teather,et al.  Effects of tracking technology, latency, and spatial jitter on object movement , 2009, 2009 IEEE Symposium on 3D User Interfaces.

[103]  R.H.Y. So,et al.  Sensory Motor Responses in Virtual Environments: Studying the Effects of Image Latencies for Target-directed Hand Movement , 2005, 2005 IEEE Engineering in Medicine and Biology 27th Annual Conference.

[104]  Ben Shneiderman,et al.  The eyes have it: a task by data type taxonomy for information visualizations , 1996, Proceedings 1996 IEEE Symposium on Visual Languages.

[105]  Bernard D. Adelstein,et al.  Discrimination of changes in latency during head movement , 1999, HCI.

[106]  W. Fieger.  The Multivariate Normal Distribution - Y. L Tong. , 1995 .

[107]  Topi Kaaresoja,et al.  Feel-good touch: finding the most pleasant tactile feedback for a mobile touch screen button , 2008, ICMI '08.

[108]  Sebastian Merchel,et al.  Audiotactile Feedback Design for Touch Screens , 2009, HAID.

[109]  Robert S. Allison,et al.  Tolerance of temporal delay in virtual environments , 2001, Proceedings IEEE Virtual Reality 2001.

[110]  C. Spence,et al.  Audio-visual simultaneity judgments , 2005, Perception & psychophysics.

[111]  Roope Raisamo,et al.  Comparison of three designs for haptic button edges on touchscreens , 2010, 2010 IEEE Haptics Symposium.

[112]  Russell B. Millar,et al.  Maximum Likelihood Estimation and Inference: With Examples in R, SAS and ADMB , 2011 .

[113]  Dan Saffer.  Designing gestural interfaces , 2009 .

[114]  C. Davis.  Touch , 1997, The Lancet.

[115]  Colby Leider,et al.  A Low-Cost, Low-Latency Multi-Touch Table with Haptic Feedback for Musical Applications , 2011, NIME.

[116]  Renaud Blanch,et al.  Two touch system latency estimators: high accuracy and low overhead , 2013, ITS.