Memo workbench for semi-automated usability testing

In this paper we present a new approach to the automation of usability evaluation for interactive systems. Design ideas or complete systems are modeled as a conditional state machine. Then, user interactions with the system are simulated on the basis of tasks, by first searching for possible solution paths and then generating deviations from these paths under consideration of user groups and system attributes. The approach has been implemented into a workbench which supports the modeling of the system and the evaluation of the simulations. We present first results for the reliability of the approach in modeling interactions with a spoken dialog system.

[1]  Steve Young,et al.  Statistical User Simulation with a Hidden Agenda , 2007, SIGDIAL.

[2]  Fabian Hermann,et al.  Users Interact Differently: Towards a Usability- Oriented User Taxonomy , 2007, HCI.

[3]  Sebastian Möller,et al.  Pragmatic Usage of Linear Regression Models for the Prediction of User Judgments , 2007, SIGDIAL.

[4]  Sebastian Möller,et al.  Analysis of communication failures for spoken dialogue systems , 2007, INTERSPEECH.

[5]  Akinori Ito,et al.  A user simulator based on voiceXML for evaluation of spoken dialog systems , 2006, INTERSPEECH.

[6]  Sebastian Möller,et al.  Memo: towards automatic usability evaluation of spoken dialogue services by user error simulations , 2006, INTERSPEECH.

[7]  Grace Chung,et al.  Developing a Flexible Spoken Dialog System Using Simulation , 2004, ACL.

[8]  Leo Egghe,et al.  The Distribution of N-Grams , 2000, Scientometrics.

[9]  Ernest Friedman Hill,et al.  Jess in Action: Java Rule-Based Systems , 2003 .

[10]  Ramón López-Cózar,et al.  Assessment of dialogue systems by means of a new simulation technique , 2003, Speech Commun..

[11]  Stefan Bordag.  Sentence Co-occurrences as Small-world Graphs: A Solution to Automatic Lexical Disambiguation , 2003, CICLing.

[12]  J. Preece,et al.  The Human-Computer Interaction Handbook , 2003 .

[13]  Marti A. Hearst,et al.  The state of the art in automating usability evaluation of user interfaces , 2001, CSUR.

[14]  Marilyn A. Walker,et al.  PARADISE: A Framework for Evaluating Spoken Dialogue Agents , 1997, ACL.

[15]  Jakob Nielsen,et al.  Usability engineering , 1997, The Computer Science and Engineering Handbook.

[16]  Vijay V. Raghavan,et al.  On modeling of information retrieval concepts in vector spaces , 1987, TODS.