A Comparison between Dialog Corpora Acquired with Real and Simulated Users

In this paper, we test the applicability of a stochastic user simulation technique to generate dialogs which are similar to real human-machine spoken interactions. To do so, we present a comparison between two corpora employing a comprehensive set of evaluation measures. The first corpus was acquired from real interactions of users with a spoken dialog system, whereas the second was generated by means of the simulation technique, which decides the next user answer taking into account the previous user turns, the last system answer and the objective of the dialog.

[1]  Thierry Dutoit,et al.  A probabilistic framework for dialog simulation and optimal strategy learning , 2006, IEEE Transactions on Audio, Speech, and Language Processing.

[2]  Diane J. Litman,et al.  Comparing real-real, simulated-simulated, and simulated-real spoken dialogue corpora , 2006 .

[3]  Ramón López-Cózar,et al.  Relations between de-facto criteria in the evaluation of a spoken dialogue system , 2008, Speech Commun..

[4]  Steve J. Young,et al.  Partially observable Markov decision processes for spoken dialog systems , 2007, Comput. Speech Lang..

[5]  Frédéric Béchet,et al.  Conceptual decoding for spoken dialog systems , 2003, INTERSPEECH.

[6]  Steve J. Young,et al.  A survey of statistical user simulation techniques for reinforcement-learning of dialogue management strategies , 2006, The Knowledge Engineering Review.

[7]  Ramón López-Cózar,et al.  Testing the performance of spoken dialogue systems by means of an artificially simulated user , 2006, Artificial Intelligence Review.

[8]  Ramón López-Cózar,et al.  Implementing Modular Dialogue Systems: A Case of Study , 2005 .

[9]  Hua Ai,et al.  Knowledge consistent user simulations for dialog systems , 2007, INTERSPEECH.

[10]  Hua Ai,et al.  Comparing User Simulation Models For Dialog Strategy Learning , 2007, HLT-NAACL.

[11]  Markku Turunen,et al.  Evaluation of a spoken dialogue system with usability tests and long-term pilot studies: similarities and differences , 2006, INTERSPEECH.

[12]  Steve Young,et al.  The statistical approach to the design of spoken dialogue systems , 2003 .

[13]  Oliver Lemon,et al.  Learning multi-goal dialogue strategies using reinforcement learning with reduced state-action spaces , 2006, INTERSPEECH.

[14]  David Griol,et al.  A statistical approach to spoken dialog systems design and evaluation , 2008, Speech Commun..

[15]  Encarna Segarra,et al.  Error handling in a stochastic dialog system through confidence measures , 2005, Speech Commun..

[16]  Kallirroi Georgila,et al.  User simulation for spoken dialogue systems: learning and evaluation , 2006, INTERSPEECH.

[17]  Steve Young,et al.  Automatic learning of dialogue strategy using dialogue simulation and reinforcement learning , 2002 .

[18]  Hua Ai,et al.  Comparing Spoken Dialog Corpora Collected with Recruited Subjects versus Real Users , 2007, SIGDIAL.

[19]  Alex Waibel,et al.  Stochastically-Based Semantic Analysis , 1999 .

[20]  Kallirroi Georgila,et al.  Quantitative Evaluation of User Simulation Techniques for Spoken Dialogue Systems , 2005, SIGDIAL.

[21]  Steve Young,et al.  A data-driven spoken language understanding system , 2003, 2003 IEEE Workshop on Automatic Speech Recognition and Understanding (IEEE Cat. No.03EX721).