A taxonomy of quality of service and Quality of Experience of multimodal human-machine interaction

Quality of Service (QoS) and Quality of Experience (QoE) are not only important for services transmitting multimedia data, but also for services involving multimodal human-machine interaction. In order to guide the assessment and evaluation of such services, we developed a taxonomy of the most relevant QoS and QoE aspects which result from multimodal human-machine interactions. It consists of three layers: (1) The QoS-influencing factors related to the user, the system, and the context of use; (2) the QoS interaction performance aspects describing user and system behavior and performance; and (3) the QoE aspects related to the quality perception and judgment processes taking place inside the user. For each of these layers, we provide metrics which make system evaluation more systematic and comparable.

[1]  Niels Ole Bernsen,et al.  Designing Interactive Speech Systems , 1998, Springer London.

[2]  K. Á. T.,et al.  Towards a tool for the Subjective Assessment of Speech System Interfaces (SASSI) , 2000, Natural Language Engineering.

[3]  Jörn Hurtienne,et al.  Intuitive Use of User Interfaces: Defining a Vague Concept , 2007, HCI.

[4]  Fabian Hermann,et al.  Users Interact Differently: Towards a Usability- Oriented User Taxonomy , 2007, HCI.

[5]  Jean-Luc Gauvain,et al.  User evaluation of the MASK kiosk , 1998, Speech Commun..

[6]  Ute Jekosch,et al.  Voice and Speech Quality Perception: Assessment and Evaluation , 2005 .

[7]  Sebastian Möller,et al.  Quality of Telephone-Based Spoken Dialogue Systems , 2005 .

[8]  Ann Blandford,et al.  Conceptualising user hedonic experience , 2004 .

[9]  Jakob Nielsen,et al.  Usability engineering , 1997, The Computer Science and Engineering Handbook.

[10]  Wolfgang Wahlster,et al.  SmartKom: Foundations of Multimodal Dialogue Systems , 2006, SmartKom.

[11]  Dafydd Gibbon,et al.  Assessment of interactive systems. , 1998 .

[12]  J. G. Hollands,et al.  Engineering Psychology and Human Performance , 1984 .

[13]  Jonathan G. Fiscus,et al.  Benchmark Tests for the DARPA Spoken Language Program , 1993, HLT.

[14]  Siobhan Chapman.  Logic and Conversation , 2005 .

[15]  Alan W. Black,et al.  The Blizzard Challenge 2006 , 2006 .

[16]  Niels Ole Bernsen,et al.  Evaluating Conversation with Hans Christian Andersen , 2004, LREC.

[17]  Sebastian Möller,et al.  Chapter 14 – Evaluation of Multimodal Interfaces for Ambient Intelligence , 2010, AmI 2010.

[18]  John B. Shoven,et al.  I , Edinburgh Medical and Surgical Journal.

[19]  Martin S. Fridson,et al.  Instrumentality , 1995 .

[20]  Niels Ole Bernsen,et al.  Designing interactive speech systems - from first ideas to user testing , 1998 .

[21]  Michael Burmester,et al.  Hedonic and ergonomic quality aspects determine a software's appeal , 2000, CHI.

[22]  Niels Ole Bernsen,et al.  Multimodality in Language and Speech Systems — From Theory to Design Support Tool , 2002 .

[23]  Niels Ole Bernsen,et al.  Evaluation and usability of multimodal spoken language dialogue systems , 2004, Speech Commun..

[24]  Sebastian Mller,et al.  Quality of Telephone-Based Spoken Dialogue Systems , 2004 .

[25]  Regan L. Mandryk,et al.  Using psychophysiological techniques to measure user experience with entertainment technologies , 2006, Behav. Inf. Technol..

[26]  G. Borg.  Psychophysical bases of perceived exertion. , 1982, Medicine and science in sports and exercise.

[27]  Nigel Bevan,et al.  Usability is Quality of Use , 1995 .

[28]  Virginie Durin,et al.  Redrawing the Link Between Customer Satisfaction and Speech Quality , 2008 .

[29]  Caryn Schneider Yaacov,et al.  Instrumentality, Aesthetics, and Symbolism of Office Design , 2005 .

[30]  Michael Burmester,et al.  AttrakDiff: Ein Fragebogen zur Messung wahrgenommener hedonischer und pragmatischer Qualität , 2003, MuC.

[31]  Dick de Waard,et al.  The measurement of drivers' mental workload , 1996 .

[32]  Mary Corbett,et al.  SUMI: the Software Usability Measurement Inventory , 1993, Br. J. Educ. Technol..

[33]  Khalid Choukri,et al.  Evaluation of multimodal components within CHIL: The evaluation packages and results , 2006, LREC.

[34]  Dafydd Gibbon,et al.  Handbook of Multimodal and Spoken Dialogue Systems , 2000 .