Setting Up User Action Probabilities in User Simulations for Dialog System Development

User simulations are shown to be useful in spoken dialog system development. Since most current user simulations deploy probability models to mimic human user behaviors, how to set up user action probabilities in these models is a key problem to solve. One generally used approach is to estimate these probabilities from human user data. However, when building a new dialog system, usually no data or only a small amount of data is available. In this study, we compare estimating user probabilities from a small user data set versus handcrafting the probabilities. We discuss the pros and cons of both solutions for different dialog system development tasks.

[1]  Ramón López-Cózar,et al.  Assessment of dialogue systems by means of a new simulation technique , 2003, Speech Commun..

[2]  Tim Paek,et al.  Reinforcement Learning for Spoken Dialogue Systems: Comparing Strengths and Weaknesses for Practical Deployment , 2006 .

[3]  Kenneth R. Koedinger,et al.  Learning Factors Analysis - A General Method for Cognitive Model Evaluation and Improvement , 2006, Intelligent Tutoring Systems.

[4]  Scotty D. Craig,et al.  Affect and learning: An exploratory look into the role of affect in learning with AutoTutor , 2004 .

[5]  Oliver Lemon,et al.  Cluster-based user simulations for learning dialogue strategies , 2006, INTERSPEECH.

[6]  Joel R. Tetreault,et al.  A Reinforcement Learning approach to evaluating state representations in spoken dialogue systems , 2008, Speech Commun..

[7]  Steve J. Young,et al.  A survey of statistical user simulation techniques for reinforcement-learning of dialogue management strategies , 2006, The Knowledge Engineering Review.

[8]  Hui Ye,et al.  Agenda-Based User Simulation for Bootstrapping a POMDP Dialogue System , 2007, NAACL.

[9]  Diane J. Litman,et al.  ITSPOKE: An Intelligent Tutoring Spoken Dialogue System , 2004, NAACL.

[10]  Oliver Lemon,et al.  Dialogue Policy Learning for Combinations of Noise and User Simulation: Transfer Results , 2007, SIGDIAL.

[11]  Simulating the Behaviour of Older versus Younger Users when Interacting with Spoken Dialogue Systems , 2008, ACL.

[12]  Steve J. Young,et al.  Partially observable Markov decision processes for spoken dialog systems , 2007, Comput. Speech Lang..

[13]  Steve Young,et al.  Statistical User Simulation with a Hidden Agenda , 2007, SIGDIAL.

[14]  Hua Ai,et al.  Knowledge consistent user simulations for dialog systems , 2007, INTERSPEECH.

[15]  Kallirroi Georgila,et al.  Learning user simulations for information state update dialogue systems , 2005, INTERSPEECH.

[16]  James A. Larson,et al.  Special Issue on "Evaluating new methods and models for advanced speech-based interactive systems" , 2008, Speech Commun..

[17]  Oliver Lemon,et al.  User simulations for online adaptation and knowledge-alignment in troubleshooting dialogue systems , 2008 .

[18]  Carolyn Penstein Rosé,et al.  The Architecture of Why2-Atlas: A Coach for Qualitative Physics Essay Writing , 2002, Intelligent Tutoring Systems.

[19]  Hua Ai,et al.  Comparing User Simulation Models For Dialog Strategy Learning , 2007, HLT-NAACL.

[20]  Harris Wu,et al.  Evaluating Web-based Question Answering Systems , 2002, LREC.

[21]  Kallirroi Georgila,et al.  Quantitative Evaluation of User Simulation Techniques for Spoken Dialogue Systems , 2005, SIGDIAL.

[22]  Diane J. Litman,et al.  Comparing real-real, simulated-simulated, and simulated-real spoken dialogue corpora , 2006 .