Understanding and Measuring User Experience in Conversational Interfaces

Although various methods have been developed to evaluate conversational interfaces, there has been a lack of methods specifically focusing on evaluating user experience. This paper reviews the understandings of user experience (UX) in conversational interfaces literature and examines the six questionnaires commonly used for evaluating conversational systems in order to assess the potential suitability of these questionnaires to measure different UX dimensions in that context. The method to examine the questionnaires involved developing an assessment framework for main UX dimensions with relevant attributes and coding the items in the questionnaires according to the framework. The results show that (i) the understandings of UX notably differed in literature; (ii) four questionnaires included assessment items, in varying extents, to measure hedonic, aesthetic and pragmatic dimensions of UX; (iii) while the dimension of affect was covered by two questionnaires, playfulness, motivation, and frustration dimensions were covered by one questionnaire only. The largest coverage of UX dimensions has been provided by the Subjective Assessment of Speech System Interfaces (SASSI). We recommend using multiple questionnaires to obtain a more complete measurement of user experience or improve the assessment of a particular UX dimension. Varying understandings of UX in conversational interfaces literature. A UX assessment framework with UX dimensions and their relevant attributes. Descriptions of the six main questionnaires for evaluating conversational interfaces. A comparison of the six questionnaires based on their coverage of UX dimensions.

[1]  S. Haynes,et al.  Content validity in psychological assessment: A functional approach to concepts and methods. , 1995 .

[2]  Chris Baber,et al.  Designing habitable dialogues for speech-based interaction with computers , 2001, Int. J. Hum. Comput. Stud..

[3]  Michael J. Albers,et al.  Content and Complexity: Information Design in Technical Communication , 2002 .

[4]  Junho Choi,et al.  Enhancing user experience with conversational agent for movie recommendation: Effects of self-disclosure and reciprocity , 2017, Int. J. Hum. Comput. Stud..

[5]  David Griol,et al.  The Conversational Interface: Talking to Smart Devices , 2016 .

[6]  Ramón López-Cózar,et al.  Enhancement of Conversational Agents By Means of Multimodal Interaction , 2011 .

[7]  James R. Lewis,et al.  Investigating the psychometric properties of the Speech User Interface Service Quality questionnaire , 2015, International Journal of Speech Technology.

[8]  Terry E. Duncan,et al.  Psychometric properties of the Intrinsic Motivation Inventory in a competitive sport setting: a confirmatory factor analysis. , 1989, Research quarterly for exercise and sport.

[9]  Robert Graham,et al.  Towards a tool for the Subjective Assessment of Speech System Interfaces (SASSI) , 2000, Natural Language Engineering.

[10]  A. Syrdal,et al.  Applied speech technology , 1995 .

[11]  Michael Burmester,et al.  AttrakDiff: Ein Fragebogen zur Messung wahrgenommener hedonischer und pragmatischer Qualität , 2003, MuC.

[12]  Marc Hassenzahl,et al.  How motivational orientation influences the evaluation and choice of hedonic and pragmatic interactive products: The role of regulatory focus , 2008, Interact. Comput..

[13]  Christine Kühnel.  Quantifying Quality Aspects of Multimodal Interactive Systems , 2012 .

[14]  Elaine Toms,et al.  What is user engagement? A conceptual framework for defining user engagement with technology , 2008, J. Assoc. Inf. Sci. Technol..

[15]  Marco Winckler,et al.  Human-Computer Interaction - INTERACT 2017 , 2017, Lecture Notes in Computer Science.

[16]  J. Russell,et al.  An approach to environmental psychology , 1974 .

[17]  Ilpo Koskinen,et al.  Co-experience: user experience as interaction , 2005 .

[18]  Vincent Koenig,et al.  User experience: A concept without consensus? Exploring practitioners' perspectives through an international survey , 2015, Comput. Hum. Behav..

[19]  J. Webster,et al.  The Dimensionality and Correlates of Flow in Human-Computer Interactions. , 1993 .

[20]  J. Dewey.  Art as Experience , 1934 .

[21]  Peter C. Wright,et al.  Interdisciplinary criticism: Analysing the experience of riot! a location-sensitive digital narrative , 2006, Behav. Inf. Technol..

[22]  Netta Iivari,et al.  UX Professionals' Definitions of Usability and UX - A Comparison Between Turkey, Finland, Denmark, France and Malaysia , 2017, INTERACT.

[23]  J. B. Brooke,et al.  SUS: A 'Quick and Dirty' Usability Scale , 1996 .

[24]  H. Marsh,et al.  Development and Validation of a Scale to Measure Optimal Experience: The Flow State Scale , 1996 .

[25]  Phil Turner.  A Psychology of User Experience: Involvement, Affect and Aesthetics , 2017 .

[26]  David Griol,et al.  The Conversational Interface , 2016 .

[27]  Ina Wechsung.  How to Evaluate? Development of the MMQQ (MultiModal Quality Questionnaire) , 2014 .

[28]  John C. McCarthy,et al.  Technology as experience , 2004, INTR.

[29]  Noam Tractinsky,et al.  Assessing dimensions of perceived visual aesthetics of web sites , 2004 .

[30]  Marc Hassenzahl,et al.  The Inference of Perceived Usability From Beauty , 2010, Hum. Comput. Interact..

[31]  E. A. Locke,et al.  What Should we do About Motivation Theory? Six Recommendations for the Twenty-First Century , 2004 .

[32]  James R. Lewis,et al.  Expanding the MOS: Development and Psychometric Evaluation of the MOS-R and MOS-X , 2003, Int. J. Speech Technol..

[33]  M. Bradley,et al.  Measuring emotion: the Self-Assessment Manikin and the Semantic Differential. , 1994, Journal of behavior therapy and experimental psychiatry.

[34]  Sebastian Möller,et al.  Measuring the Quality of Service and Quality of Experience of multimodal human–machine interaction , 2012, Journal on Multimodal User Interfaces.

[35]  Niels Ole Bernsen,et al.  Evaluation and usability of multimodal spoken language dialogue systems , 2004, Speech Commun..

[36]  Phil Turner,et al.  A Psychology of User Experience , 2017, Human–Computer Interaction Series.

[37]  Peter C. Wright,et al.  The experience of enchantment in human–computer interaction , 2006, Personal and Ubiquitous Computing.

[38]  Udo Konradt,et al.  Reliability, validity, and sensitivity of a single-item measure of online store usability , 2011, Int. J. Hum. Comput. Stud..

[39]  Marc Hassenzahl,et al.  The Effect of Perceived Hedonic Quality on Product Appealingness , 2001, Int. J. Hum. Comput. Interact..

[40]  M. Glynn,et al.  The Adult Playfulness Scale: An Initial Assessment , 1992 .

[41]  Kasper Hornbæk,et al.  Current practice in measuring usability: Challenges to usability studies and research , 2006, Int. J. Hum. Comput. Stud..

[42]  D. Norman.  Emotional design : why we love (or hate) everyday things , 2004 .