Measuring the Quality of Service and Quality of Experience of multimodal human–machine interaction

Quality of Service (QoS) and Quality of Experience (QoE) have to be considered when designing, building and maintaining services involving multimodal human–machine interaction. In order to guide the assessment and evaluation of such services, we first develop a taxonomy of the most relevant QoS and QoE aspects which result from multimodal human–machine interactions. It consists of three layers: (1) The quality factors influencing QoS and QoE related to the user, the system, and the context of use; (2) the QoS interaction performance aspects describing user and system behavior and performance; and (3) the QoE aspects related to the quality perception and judgment processes taking place within the user. For each of these layers, we then provide metrics which are able to capture the QoS and QoE aspects in a quantitative way, either via questionnaires or performance measures. The metrics are meant to guide system evaluation and make it more systematic and comparable.

[1]  Goutam Chakraborty,et al.  The measurement of computer literacy: a comparison of self-appraisal and objective tests , 1994, Int. J. Hum. Comput. Stud..

[2]  Peter Brooks,et al.  User measures of quality of experience: why being objective and quantitative is important , 2010, IEEE Network.

[3]  David E. Kieras,et al.  Using GOMS for user interface design and evaluation: which technique? , 1996, TCHI.

[4]  Peter Caputi,et al.  The development of a measure of subjective computer experience , 2007, Comput. Hum. Behav..

[5]  Virpi Roto,et al.  UX Curve: A method for evaluating long-term user experience , 2011, Interact. Comput..

[6]  Aaron Marcus.  Principles of effective visual communication for graphical user interface design , 1995 .

[7]  Gitte Lindgaard,et al.  Emotional Experiences and Quality Perceptions of Interactive Products , 2007, HCI.

[8]  Niels Ole Bernsen,et al.  Evaluating Conversation with Hans Christian Andersen , 2004, LREC.

[9]  Jakob Nielsen,et al.  Usability engineering , 1997, The Computer Science and Engineering Handbook.

[10]  Robert Schleicher,et al.  The 'Joy-of-Use'-Button: Recording Pleasant Moments While Using a PC , 2009, INTERACT.

[11]  Lyle F. Bachman.  Assessment and Evaluation , 1989, Annual Review of Applied Linguistics.

[12]  Deborah Compeau,et al.  Computer Self-Efficacy: Development of a Measure and Initial Test , 1995, MIS Q..

[13]  J. B. Brooke,et al.  SUS: A 'Quick and Dirty' Usability Scale , 1996 .

[14]  Wayne D. Gray,et al.  Damaged Merchandise? A Review of Experiments That Compare Usability Evaluation Methods , 1998, Hum. Comput. Interact..

[15]  Dafydd Gibbon,et al.  Handbook of Multimodal and Spoken Dialogue Systems , 2000 .

[16]  Niels Ole Bernsen,et al.  Multimodal Usability , 2010, Human-Computer Interaction Series.

[17]  Noam Tractinsky,et al.  Assessing dimensions of perceived visual aesthetics of web sites , 2004 .

[18]  Jonathan Grudin,et al.  Human Computer Interaction: The Year 2000 and Beyond , 1995, HCI.

[19]  Jean-Luc Gauvain,et al.  User evaluation of the MASK kiosk , 1998, Speech Commun..

[20]  Marc Hassenzahl,et al.  Capturing Design Space From a User Perspective: The Repertory Grid Technique Revisited , 2000, Int. J. Hum. Comput. Interact..

[21]  F.R.H. Zijlstra,et al.  Efficiency in work behaviour: A design approach for modern tools , 1993 .

[22]  Edward Nelson,et al.  Syntax and Semantics , 1974 .

[23]  S. Hart,et al.  Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research , 1988 .

[24]  Gijs Huisman,et al.  The development of a graphical emotion measurement instrument using caricatured expressions: the LEMtool , 2010 .

[25]  Sebastian Möller,et al.  Chapter 14 – Evaluation of Multimodal Interfaces for Ambient Intelligence , 2010, AmI 2010.

[26]  Wolfgang Wahlster,et al.  SmartKom: Foundations of Multimodal Dialogue Systems , 2006, SmartKom.

[27]  Siobhan Chapman.  Logic and Conversation , 2005 .

[28]  Alan W. Black,et al.  The Blizzard Challenge 2006 , 2006 .

[29]  Marc Cavazza,et al.  Multimodal and mobile conversational Health and Fitness Companions , 2011, Comput. Speech Lang..

[30]  Jonathan G. Fiscus,et al.  Benchmark Tests for the DARPA Spoken Language Program , 1993, HLT.

[31]  B. Thomas,et al.  Usability Evaluation In Industry , 1996 .

[32]  M. Hassenzahl,et al.  AESTHETICS IN INTERACTIVE PRODUCTS: CORRELATES AND CONSEQUENCES OF BEAUTY , 2008 .

[33]  Kasper Hornbæk,et al.  Meta-analysis of correlations among usability measures , 2007, CHI.

[34]  R. Heinssen,et al.  Assessing computer anxiety: Development and validation of the Computer Anxiety Rating Scale , 1987 .

[35]  Juan Carlos Augusto,et al.  Human-Centric Interfaces for Ambient Intelligence , 2009 .

[36]  Fabian Hermann,et al.  Interaktion mit Informations- und Kommunikationstechnologie: Eine Klassifikation von Benutzertypen , 2008, Mensch & Computer.

[37]  Vicente Moret-Bonillo,et al.  Usability: A Critical Analysis and a Taxonomy , 2009, Int. J. Hum. Comput. Interact..

[38]  Robert C. Williges,et al.  Criteria For Evaluating Usability Evaluation Methods , 2001, Int. J. Hum. Comput. Interact..

[39]  Sarah Diefenbach,et al.  Needs, affect, and interactive products - Facets of user experience , 2010, Interact. Comput..

[40]  Christopher D. Wickens,et al.  Multiple Resources and Mental Workload , 2008, Hum. Factors.

[41]  Ergonomic requirements for office work with visual display terminals ( VDTs ) — Part 11 : Guidance on usability , 1998 .

[42]  Marc Hassenzahl,et al.  Analysis of web sites with the repertory grid technique , 2001, CHI Extended Abstracts.

[43]  Fabian Hermann,et al.  Users Interact Differently: Towards a Usability- Oriented User Taxonomy , 2007, HCI.

[44]  James R. Lewis,et al.  IBM computer usability satisfaction questionnaires: Psychometric evaluation and instructions for use , 1995, Int. J. Hum. Comput. Interact..

[45]  David E. Kieras,et al.  An Overview of the EPIC Architecture for Cognition and Performance With Application to Human-Computer Interaction , 1997, Hum. Comput. Interact..

[46]  Fred D. Davis.  Perceived Usefulness, Perceived Ease of Use, and User Acceptance of Information Technology , 1989, MIS Q..

[47]  M. Bradley,et al.  Measuring emotion: the Self-Assessment Manikin and the Semantic Differential. , 1994, Journal of behavior therapy and experimental psychiatry.

[48]  N. Tractinsky,et al.  What is beautiful is usable , 2000, Interact. Comput..

[49]  Sebastian Möller,et al.  Quality of Telephone-Based Spoken Dialogue Systems , 2005 .

[50]  Mensch & Computer 2010: Interaktive Kulturen, Interdisziplinäre Fachtagung, Duisburg, Germany, September 12-15, 2010 , 2010, Mensch & Computer.

[51]  Ann Blandford,et al.  Conceptualising user hedonic experience , 2004 .

[52]  Tal Oron-Gilad,et al.  'Castling rays' a decision support tool for UAV-switching tasks , 2010, CHI Extended Abstracts.

[53]  Michael Burmester,et al.  Valence method for formative evaluation of user experience , 2010, Conference on Designing Interactive Systems.

[54]  Sarah Diefenbach,et al.  INTUI. Exploring the Facets of Intuitive Interaction , 2010, MuC.

[55]  Sebastian Mller,et al.  Quality of Telephone-Based Spoken Dialogue Systems , 2004 .

[56]  Charles D. Barrett.  Understanding Attitudes and Predicting Social Behavior , 1980 .

[57]  Fred D. Davis.  User Acceptance of Information Technology: System Characteristics, User Perceptions and Behavioral Impacts , 1993, Int. J. Man Mach. Stud..

[58]  Ivo Düntsch,et al.  The IsoMetrics usability inventory: An operationalization of ISO 9241-10 supporting summative and formative evaluation of software systems , 1999, Behav. Inf. Technol..

[59]  Evangelos Karapanos,et al.  On the retrospective assessment of users' experiences over time: memory or actuality? , 2010, CHI Extended Abstracts.

[60]  Cathleen Wharton,et al.  Cognitive Walkthroughs: A Method for Theory-Based Evaluation of User Interfaces , 1992, Int. J. Man Mach. Stud..

[61]  Regan L. Mandryk,et al.  Using psychophysiological techniques to measure user experience with entertainment technologies , 2006, Behav. Inf. Technol..

[62]  G. Borg.  Psychophysical bases of perceived exertion. , 1982, Medicine and science in sports and exercise.

[63]  Nigel Bevan,et al.  Usability is Quality of Use , 1995 .

[64]  Virginie Durin,et al.  Redrawing the Link Between Customer Satisfaction and Speech Quality , 2008 .

[65]  Sebastian Möller,et al.  Parameters describing multimodal interaction - definitions and three usage scenarios , 2010, INTERSPEECH.

[66]  Masahiro Araki,et al.  Spoken, Multilingual and Multimodal Dialogue Systems: Development and Assessment , 2005 .

[67]  Niels Ole Bernsen,et al.  Multimodality in Language and Speech Systems — From Theory to Design Support Tool , 2002 .

[68]  J. Kessler,et al.  DemTect: a new, sensitive cognitive screening test to support the diagnosis of mild cognitive impairment and early dementia , 2004, International journal of geriatric psychiatry.

[69]  P. Hekkert.  Design aesthetics: principles of pleasure in design , 2006 .

[70]  Nigel Bevan,et al.  What is the difference between the purpose of usability and user experience evaluation methods , 2009 .

[71]  Kasper Hornbæk,et al.  Current practice in measuring usability: Challenges to usability studies and research , 2006, Int. J. Hum. Comput. Stud..

[72]  Kent L. Norman,et al.  Development of an instrument measuring user satisfaction of the human-computer interface , 1988, CHI '88.

[73]  Stefan Kopp,et al.  Trading Spaces: How Humans and Humanoids Use Speech and Gesture to Give Directions , 2007 .

[74]  Marilyn A. Walker,et al.  Using Linguistic Cues for the Automatic Recognition of Personality in Conversation and Text , 2007, J. Artif. Intell. Res..

[75]  Khalid Choukri,et al.  Evaluation of multimodal components within CHIL: The evaluation packages and results , 2006, LREC.

[76]  Björn Granström,et al.  Multimodality in Language and Speech Systems , 2002 .

[77]  Marc Hassenzahl,et al.  The Interplay of Beauty, Goodness, and Usability in Interactive Products , 2004, Hum. Comput. Interact..

[78]  Clifford Nass,et al.  Consistency of personality in interactive characters: verbal cues, non-verbal cues, and user characteristics , 2000, Int. J. Hum. Comput. Stud..

[79]  Dafydd Gibbon,et al.  Assessment of interactive systems. , 1998 .

[80]  Johannes Naumann,et al.  Attitudes toward the computer: construct validation of an instrument with scales differentiated by content , 2000 .

[81]  P. Hancock,et al.  Human Mental Workload , 1988 .

[82]  K. Á. T.,et al.  Towards a tool for the Subjective Assessment of Speech System Interfaces (SASSI) , 2000, Natural Language Engineering.

[83]  J. Cacioppo,et al.  Handbook Of Psychophysiology , 2019 .

[84]  Ute Jekosch,et al.  Voice and Speech Quality Perception: Assessment and Evaluation , 2005 .

[85]  Julie A. Jacko,et al.  Interaction design and usability , 2007 .

[86]  Marco Winckler,et al.  Human-Computer Interaction - INTERACT 2009, 12th IFIP TC 13 International Conference, Uppsala, Sweden, August 24-28, 2009, Proceedings, Part I , 2009, INTERACT.

[87]  Ann Blandford,et al.  Four easy pieces for assessing the usability of multimodal interaction: the CARE properties , 1995, INTERACT.

[88]  Thomas P. Moran,et al.  Commentary on "Damaged Merchandise?" , 1998, Hum. Comput. Interact..

[89]  Jörn Hurtienne,et al.  Intuitive Use of User Interfaces: Defining a Vague Concept , 2007, HCI.

[90]  Oscar Mauricio Serrano Jaimes,et al.  EVALUACION DE LA USABILIDAD EN SITIOS WEB, BASADA EN EL ESTANDAR ISO 9241-11 (International Standard (1998) Ergonomic requirements For office work with visual display terminals (VDTs)-Parts II: Guidance on usability , 2012 .

[91]  Dick de Waard,et al.  The measurement of drivers' mental workload , 1996 .

[92]  Pieter Desmet,et al.  Measuring Emotion: Development and Application of an Instrument to Measure Emotional Responses to Products , 2005, Funology.

[93]  Mary Corbett,et al.  SUMI: the Software Usability Measurement Inventory , 1993, Br. J. Educ. Technol..

[94]  Kasper Hornbæk,et al.  Old wine in new bottles or novel challenges: a critical analysis of empirical studies of user experience , 2011, CHI.

[95]  Peter C. Wright,et al.  Funology: from usability to enjoyment , 2005 .

[96]  Michael Burmester,et al.  Hedonic and ergonomic quality aspects determine a software's appeal , 2000, CHI.