Combined User Physical, Physiological and Subjective Measures for Assessing User Cost

New technologies are making it possible to provide an enriched view of interaction for researchers using multimodal information. This preliminary study explores the use of multimodal information streams in evaluating user cost. In the study, easy, medium and difficult versions of a game task were used to vary the levels of the cost to user. Multimodal data streams during the three versions were analyzed, including eye tracking, pupil size, hand movement, heart rate variability (HRV) and subjectively reported data. Three findings indicate the potential value of multimodal information in evaluating usability: First, subjective and physiological measures showed significant sensitivity to task difficulty. Second, different user cost levels appeared to correlate with eye movement patterns, especially with a combined eye-hand measure. Third, HRV showed correlations with saccade speed. These results warrant further investigations and take an initial step toward establishing usability evaluation methods based on multimodal information.

[1]  Wenjun Chris Zhang,et al.  Effective attention allocation behavior and its measurement: a preliminary study , 2004, Interact. Comput..

[2]  Gretchen B. Rossman,et al.  Designing qualitative research, 3rd ed. , 1999 .

[3]  C E Izard,et al.  Cardiac rhythmicities and attention in young children. , 1997, Psychophysiology.

[4]  E. Hess,et al.  Pupil Size in Relation to Mental Activity during Simple Problem-Solving , 1964, Science.

[5]  D. Kahneman.  An onset-onset law for one case of apparent motion and metacontrast , 1967 .

[6]  K. J. Vicente,et al.  Spectral Analysis of Sinus Arrhythmia: A Measure of Mental Effort , 1987, Human factors.

[7]  P. Hancock,et al.  Human Mental Workload , 1988 .

[8]  G. Robert J. Hockey,et al.  Level of Operator Control and Changes in Heart Rate Variability during Simulated Flight Maintenance , 1995, Hum. Factors.

[9]  K. Rayner.  Eye movements in reading and information processing: 20 years of research. , 1998, Psychological bulletin.

[10]  M. Turiel,et al.  Power Spectral Analysis of Heart Rate and Arterial Pressure Variabilities as a Marker of Sympatho‐Vagal Interaction in Man and Conscious Dog , 1986, Circulation research.

[11]  Joseph H. Goldberg,et al.  Computer interface evaluation using eye movements: methods and constructs , 1999 .

[12]  W. Levelt,et al.  Pupillary dilation as a measure of attention: a quantitative system analysis , 1993 .

[13]  J. Beatty.  Task-evoked pupillary responses, processing load, and the structure of processing resources. , 1982 .

[14]  Minoru Nakayama,et al.  The act of task difficulty and eye-movement frequency for the 'Oculo-motor indices' , 2002, ETRA.

[15]  Brian P. Bailey,et al.  Categories & Subject Descriptors: H.5.2 [Information , 2022 .

[16]  S. Hébert,et al.  Physiological stress response to video-game playing: the contribution of built-in music. , 2005, Life sciences.

[17]  Rosalind W. Picard.  Affective Computing , 1997 .

[18]  J. Andreassi.  Psychophysiology: Human Behavior and Physiological Response , 1980 .

[19]  L. Stark,et al.  Scanpaths in saccadic eye movements while viewing and recognizing patterns. , 1971, Vision research.

[20]  G. Breithardt,et al.  Heart rate variability: standards of measurement, physiological interpretation and clinical use. Task Force of the European Society of Cardiology and the North American Society of Pacing and Electrophysiology. , 1996 .

[21]  Gm Wilson,et al.  Investigating the impact of audio degradations on users: subjective vs objective assessment methods , 2000 .

[22]  Martin C. Maguire,et al.  Evaluating User-Computer Interaction: A Framework , 1993, Int. J. Man Mach. Stud..

[23]  D. Adam,et al.  Assessment of autonomic function in humans by heart rate spectral analysis. , 1985, The American journal of physiology.

[24]  Tao Lin,et al.  Do physiological data relate to traditional usability indexes? , 2005, OZCHI.

[25]  P. Ekman,et al.  Autonomic nervous system activity distinguishes among emotions. , 1983, Science.

[26]  Gillian May Wilson.  Psychophysiological indicators of the impact of media quality on users , 2001, CHI Extended Abstracts.

[27]  J H Ettema,et al.  Physiological parameters of mental load. , 1971, Ergonomics.

[28]  Catherine C. Marshall,et al.  Designing Qualitative Research , 1996 .

[29]  H. V. van Geijn,et al.  Heart Rate Variability , 1993, Annals of Internal Medicine.

[30]  John L. Sibert,et al.  Heart rate variability: indicator of user state as an aid to human-computer interaction , 1998, CHI.

[31]  Manfred Velden,et al.  The pupillary response to mental overload , 1977 .

[32]  Michael Meehan,et al.  Physiological measures of presence in stressful virtual environments , 2002, SIGGRAPH.

[33]  B. W. Hyndman,et al.  Spectral analysis of sinus arrhythmia during mental loading. , 1975, Ergonomics.

[34]  H. Luczak,et al.  An analysis of heart rate variability. , 1973, Ergonomics.

[35]  J. Beatty,et al.  Pupillary responses in a pitch-discrimination task , 1967 .

[36]  A. W. Siegman,et al.  Nonverbal behavior and communication , 1979 .

[37]  A. Weltman,et al.  Heart period variability of trained and untrained men at rest and during mental challenge. , 1998, Psychophysiology.

[38]  J. Hair.  Multivariate data analysis , 1972 .