A statistical approach to spoken dialog systems design and evaluation

In this paper, we present a statistical approach for the development of a dialog manager and for learning optimal dialog strategies. This methodology is based on a classification procedure that considers all of the previous history of the dialog to select the next system answer. To evaluate the performance of the dialog system, the statistical approach for dialog management has been extended to model the user behavior. The statistical user simulator has been used for the evaluation and improvement of the dialog strategy. Both the user model and the system model are automatically learned from a training corpus that is labeled in terms of dialog acts. New measures have been defined to evaluate the performance of the dialog system. Using these measures, we evaluate both the quality of the simulated dialogs and the improvement of the new dialog strategy that is obtained with the interaction of the two modules. This methodology has been applied to develop a dialog manager within the framework of the DIHANA project, whose goal is the design and development of a dialog system to access a railway information system using spontaneous speech in Spanish. We propose the use of corpus-based methodologies to develop the main modules in the dialog system.

[1]  Encarna Segarra,et al.  Extracting Semantic Information Through Automatic Learning Techniques , 2002, Int. J. Pattern Recognit. Artif. Intell..

[2]  Emilio Sanchis Arnal,et al.  Uniclass and Multiclass Connectionist Classification of Dialogue Acts , 2003, CIARP.

[3]  Deborah L. McGuinness,et al.  The Role of Frame-Based Representation on the Semantic Web , 2001 .

[4]  Marilyn A. Walker,et al.  Reinforcement Learning for Spoken Dialogue Systems , 1999, NIPS.

[5]  Kallirroi Georgila,et al.  User simulation for spoken dialogue systems: learning and evaluation , 2006, INTERSPEECH.

[6]  Steve Young,et al.  Automatic learning of dialogue strategy using dialogue simulation and reinforcement learning , 2002 .

[7]  Steve Young,et al.  Statistical User Simulation with a Hidden Agenda , 2007, SIGDIAL.

[8]  Steve Young,et al.  Simulation of human-machine dialogues , 1999 .

[9]  M. Ferguson,et al.  Automatic Evaluation , 2009 .

[10]  Joelle Pineau,et al.  Spoken Dialogue Management Using Probabilistic Reasoning , 2000, ACL.

[11]  Alex Waibel,et al.  Stochastically-Based Semantic Analysis , 1999 .

[12]  Kallirroi Georgila,et al.  Quantitative Evaluation of User Simulation Techniques for Spoken Dialogue Systems , 2005, SIGDIAL.

[13]  Arne Jönsson,et al.  Talking to a Computer Is Not like Talking to Your Best Friend , 1988, SCAI.

[14]  Margaret King,et al.  Evaluation of natural language processing systems , 1991 .

[15]  Steve J. Young,et al.  A survey of statistical user simulation techniques for reinforcement-learning of dialogue management strategies , 2006, The Knowledge Engineering Review.

[16]  Steve Young,et al.  The statistical approach to the design of spoken dialogue systems , 2003 .

[17]  Thierry Dutoit,et al.  A probabilistic framework for dialog simulation and optimal strategy learning , 2006, IEEE Transactions on Audio, Speech, and Language Processing.

[18]  Kallirroi Georgila,et al.  Learning user simulations for information state update dialogue systems , 2005, INTERSPEECH.

[19]  Oliver Lemon,et al.  Learning multi-goal dialogue strategies using reinforcement learning with reduced state-action spaces , 2006, INTERSPEECH.

[20]  Patrick Henry Winston,et al.  The psychology of computer vision , 1976, Pattern Recognit..

[21]  I. Lane,et al.  Cooperative Dialogue Planning with User and Situation Models via Example-based Training , 2004 .

[22]  Richard Fikes,et al.  The role of frame-based representation in reasoning , 1985, CACM.

[23]  Roberto Pieraccini,et al.  A stochastic model of human-machine interaction for learning dialog strategies , 2000, IEEE Trans. Speech Audio Process..

[24]  Hui Ye,et al.  Agenda-Based User Simulation for Bootstrapping a POMDP Dialogue System , 2007, NAACL.

[25]  Steve Young,et al.  A data-driven spoken language understanding system , 2003, 2003 IEEE Workshop on Automatic Speech Recognition and Understanding (IEEE Cat. No.03EX721).

[26]  Marvin Minsky,et al.  A framework for representing knowledge" in the psychology of computer vision , 1975 .

[27]  Niels Ole Bernsen,et al.  Usability Issues in Spoken Language Dialogue Systems , 2001 .

[28]  Steve J. Young,et al.  Partially observable Markov decision processes for spoken dialog systems , 2007, Comput. Speech Lang..

[29]  Nigel Gilbert,et al.  Simulating speech systems , 1991 .

[30]  Frédéric Béchet,et al.  Conceptual decoding for spoken dialog systems , 2003, INTERSPEECH.

[31]  Niels Ole Bernsen,et al.  Usability issues in spoken dialogue systems , 2000, Natural Language Engineering.

[32]  Marilyn A. Walker,et al.  Evaluating spoken dialogue agents with PARADISE: Two case studies , 1998, Comput. Speech Lang..

[33]  Kallirroi Georgila,et al.  EVALUATING EFFECTIVENESS AND PORTABILITY OF REINFORCEMENT LEARNED DIALOGUE STRATEGIES WITH REAL USERS: THE TALK TOWNINFO EVALUATION , 2006, 2006 IEEE Spoken Language Technology Workshop.

[34]  Oliver Lemon,et al.  DIPPER: Description and Formalisation of an Information-State Update Dialogue System Architecture , 2003, SIGDIAL Workshop.

[35]  Roberto Pieraccini,et al.  User modeling for spoken dialogue system evaluation , 1997, 1997 IEEE Workshop on Automatic Speech Recognition and Understanding Proceedings.

[36]  Encarna Segarra,et al.  Development of a stochastic dialog manager driven by semantics , 2003, INTERSPEECH.

[37]  Christopher M. Bishop,et al.  Neural networks for pattern recognition , 1995 .

[38]  Encarna Segarra,et al.  The Incorporation of Confidence Measures to Language Understanding , 2003, TSD.

[39]  Niels Ole Bernsen,et al.  Evaluation and usability of multimodal spoken language dialogue systems , 2004, Speech Commun..

[40]  Hui Ye,et al.  The Hidden Information State Approach to Dialog Management , 2007, 2007 IEEE International Conference on Acoustics, Speech and Signal Processing - ICASSP '07.

[41]  Marvin Minsky,et al.  A framework for representing knowledge , 1974 .

[42]  E. Levin,et al.  A Stochastic Model of Human Computer Interaction for Learning Dialog Strategies , 1997 .

[43]  Oliver Lemon,et al.  Cluster-based user simulations for learning dialogue strategies , 2006, INTERSPEECH.

[44]  Emilio Sanchis,et al.  Managing Unseen Situations in a Stochastic Dialog Model , 2006 .

[45]  Konrad Scheffler,et al.  Probabilistic simulation of human-machine dialogues , 2000, 2000 IEEE International Conference on Acoustics, Speech, and Signal Processing. Proceedings (Cat. No.00CH37100).

[46]  Olivier Pietquin,et al.  Comparing ASR modeling methods for spoken dialogue simulation and optimal strategy learning , 2005, INTERSPEECH.

[47]  Geoffrey E. Hinton,et al.  Learning internal representations by error propagation , 1986 .

[48]  David Griol,et al.  A stochastic approach for dialog management based on neural networks , 2006, INTERSPEECH.

[49]  J. Schatztnann,et al.  Effects of the user model on simulation-based learning of dialogue strategies , 2005, IEEE Workshop on Automatic Speech Recognition and Understanding, 2005..

[50]  Oliver Lemon,et al.  Dialogue Policy Learning for Combinations of Noise and User Simulation: Transfer Results , 2007, SIGDIAL.

[51]  Lynette Hirschman,et al.  Comparing Several Aspects of Human-Computer and Human-Human Dialogues , 2001, SIGDIAL Workshop.

[52]  Roberto Pieraccini,et al.  Concept-based spontaneous speech understanding system , 1995, EUROSPEECH.

[53]  Eric Horvitz,et al.  Conversation as Action Under Uncertainty , 2000, UAI.

[54]  H. Cuayahuitl,et al.  Human-computer dialogue simulation using hidden Markov models , 2005, IEEE Workshop on Automatic Speech Recognition and Understanding, 2005..

[55]  L.F. Hurtado,et al.  A stochastic approach to dialog management , 2005, IEEE Workshop on Automatic Speech Recognition and Understanding, 2005..

[56]  Eduardo Lleida,et al.  Design and acquisition of a telephone spontaneous speech dialogue corpus in Spanish: DIHANA , 2006, LREC.

[57]  Roberto Pieraccini,et al.  The use of belief networks for mixed-initiative dialog modeling , 2000, IEEE Trans. Speech Audio Process..

[58]  Emilio Sanchis,et al.  A dialog system for the DIHANA project , 2006 .