Towards Fine-Grain User-Simulation for Spoken Dialogue Systems

Continuous advances in the field of spoken dialogue systems make the processe of design, implementation and evaluation of these systems more and more com-s plex. To solve problems emerging from this complexity, a technique which has attracted increasing interest during the last decades is based on the automatic generation of dialogues between the system and a user simulator, which is another system that represents human interactions with the dialogue system. This chapter describes the main methodologies and techniques developed to create user simulators, and presents a discussion of their main characteristics and the benefits that they provide for the development, improvement and assessment of this kind of systems. Additionally, we propose a user simulation technique to test the performance of spoken dialogue systems. The technique is based on a novel approach to simulating different levels of user cooperativeness, which allows carrying out a more detailed system assessment. In the experiments we have evaluated a spoken dialogue system designed for the fast food domain. Theevaluation has focused on the performance of the speech recogniser, semantic analyser and dialogue manager of this system. The results show that the technique provides relevant information to obtain a solid evaluation of the system, enabling us to find problems in these modules which cannot be observed taking into account just one cooperativeness level.

[1]  Jakob Nielsen,et al.  Usability engineering , 1997, The Computer Science and Engineering Handbook.

[2]  Stephanie Seneff,et al.  Language model data filtering via user simulation and dialogue resynthesis , 2005, INTERSPEECH.

[3]  Hua Ai,et al.  Assessing Dialog System User Simulation Evaluation Measures Using Human Judges , 2008, ACL.

[4]  Ramón López-Cózar,et al.  A new method for testing dialogue systems based on simulations of real-world conditions , 2002, INTERSPEECH.

[5]  Hua Ai,et al.  Comparing User Simulation Models For Dialog Strategy Learning , 2007, HLT-NAACL.

[6]  Biing-Hwang Juang,et al.  An Overview of Automatic Speech Recognition , 1996 .

[7]  Roberto Pieraccini,et al.  A stochastic model of human-machine interaction for learning dialog strategies , 2000, IEEE Trans. Speech Audio Process..

[8]  Hui Ye,et al.  Agenda-Based User Simulation for Bootstrapping a POMDP Dialogue System , 2007, NAACL.

[9]  Ramón López-Cózar,et al.  A voice activated dialogue system for fast-food restaurant applications , 1997, EUROSPEECH.

[10]  Steve J. Young,et al.  A survey of statistical user simulation techniques for reinforcement-learning of dialogue management strategies , 2006, The Knowledge Engineering Review.

[11]  Kallirroi Georgila,et al.  Quantitative Evaluation of User Simulation Techniques for Spoken Dialogue Systems , 2005, SIGDIAL.

[12]  Oliver Lemon,et al.  Dialogue Policy Learning for Combinations of Noise and User Simulation: Transfer Results , 2007, SIGDIAL.

[13]  Gary Geunbae Lee,et al.  Data-driven user simulation for automated evaluation of spoken dialog systems , 2009, Comput. Speech Lang..

[14]  Oliver Lemon,et al.  DIPPER: Description and Formalisation of an Information-State Update Dialogue System Architecture , 2003, SIGDIAL Workshop.

[15]  Ramón López-Cózar,et al.  Combining language models in the input interface of a spoken dialogue system , 2006, Comput. Speech Lang..

[16]  David Griol,et al.  A statistical approach to spoken dialog systems design and evaluation , 2008, Speech Commun..

[17]  Tatsuya Kawahara,et al.  Flexible Guidance Generation Using User Model in Spoken Dialogue Systems , 2003, ACL.

[18]  Steve Young,et al.  The statistical approach to the design of spoken dialogue systems , 2003 .

[19]  Steve Young,et al.  Statistical User Simulation with a Hidden Agenda , 2007, SIGDIAL.

[20]  Lionel Delphin-Poulat,et al.  From a Wizard of Oz experiment to a real time speech and gesture multimodal interface , 2006, Signal Process..

[21]  E. Levin,et al.  A Stochastic Model of Human Computer Interaction for Learning Dialog Strategies , 1997 .

[22]  Sebastian Mller,et al.  Quality of Telephone-Based Spoken Dialogue Systems , 2004 .

[23]  Ramón López-Cózar,et al.  A Comparison between Dialog Corpora Acquired with Real and Simulated Users , 2009, SIGDIAL Conference.

[24]  Jason D. Williams,et al.  A method for evaluating and comparing user simulations: The Cramér-von Mises divergence , 2007, 2007 IEEE Workshop on Automatic Speech Recognition & Understanding (ASRU).

[25]  Kallirroi Georgila,et al.  User simulation for spoken dialogue systems: learning and evaluation , 2006, INTERSPEECH.

[26]  Steve Young,et al.  Automatic learning of dialogue strategy using dialogue simulation and reinforcement learning , 2002 .

[27]  Lin-Shan Lee,et al.  Computer-aided analysis and design for spoken dialogue systems based on quantitative simulations , 2001, IEEE Trans. Speech Audio Process..

[28]  Kallirroi Georgila,et al.  Learning user simulations for information state update dialogue systems , 2005, INTERSPEECH.

[29]  Encarna Segarra,et al.  User simulation in a stochastic dialog system , 2008, Comput. Speech Lang..

[30]  H. Cuayahuitl,et al.  Human-computer dialogue simulation using hidden Markov models , 2005, IEEE Workshop on Automatic Speech Recognition and Understanding, 2005..

[31]  Grace Chung,et al.  Developing a Flexible Spoken Dialog System Using Simulation , 2004, ACL.

[32]  Roberto Pieraccini,et al.  User modeling for spoken dialogue system evaluation , 1997, 1997 IEEE Workshop on Automatic Speech Recognition and Understanding Proceedings.

[33]  A. Koller,et al.  Speech Acts: An Essay in the Philosophy of Language , 1969 .

[34]  Ramón López-Cózar,et al.  Assessment of dialogue systems by means of a new simulation technique , 2003, Speech Commun..

[35]  Steve Young,et al.  Simulation of human-machine dialogues , 1999 .

[36]  Blair MacIntyre,et al.  Wizard of Oz support throughout an iterative design process , 2005, IEEE Pervasive Computing.

[37]  Olivier Pietquin,et al.  Comparing ASR modeling methods for spoken dialogue simulation and optimal strategy learning , 2005, INTERSPEECH.