Measuring Heterogeneous User Behaviors During the Interaction with Dialog Systems

In this paper, we describe a technique to develop simulated user agents that are able to interact with dialog systems. By means of these agents, it is possible not only to automatically evaluate the overall operation of the dialog system, but also to assess the impact of the user responses on the decisions that are selected by the system. The selection of the user responses by the simulated user agent are based on a statistical model that is automatically learned from a dialog corpus. The complete history of the interaction is considered to carry out this selection. The paper describes the application of this technique to evaluate a practical dialog system providing tourist information and services.

[1]  Lin-Shan Lee,et al.  Computer-aided analysis and design for spoken dialogue systems based on quantitative simulations , 2001, IEEE Trans. Speech Audio Process..

[2]  Steve J. Young,et al.  Error simulation for training statistical dialogue systems , 2007, 2007 IEEE Workshop on Automatic Speech Recognition & Understanding (ASRU).

[3]  Encarna Segarra,et al.  The Incorporation of Confidence Measures to Language Understanding , 2003, TSD.

[4]  Steve J. Young,et al.  Partially observable Markov decision processes for spoken dialog systems , 2007, Comput. Speech Lang..

[5]  Ramón López-Cózar,et al.  A domain-independent statistical methodology for dialog management in spoken dialog systems , 2014, Comput. Speech Lang..

[6]  Klaus-Peter Engelbrecht.  Estimating Spoken Dialog System Quality with User Models , 2012 .

[7]  John Mourjopoulos,et al.  Automatic speech recognition performance in different room acoustic environments with and without dereverberation preprocessing , 2013, Comput. Speech Lang..

[8]  Steve J. Young,et al.  A survey of statistical user simulation techniques for reinforcement-learning of dialogue management strategies , 2006, The Knowledge Engineering Review.

[9]  David Griol,et al.  Modeling Users Emotional State for an Enhanced Human-Machine Interaction , 2015, HAIS.

[10]  Christopher M. Bishop,et al.  Neural networks for pattern recognition , 1995 .

[11]  Feng Gao,et al.  Spoken language understanding using weakly supervised learning , 2010, Comput. Speech Lang..

[12]  Oliver Lemon,et al.  Learning what to say and how to say it: Joint optimisation of spoken dialogue management and natural language generation , 2011, Comput. Speech Lang..

[13]  Sebastian Möller,et al.  Memo: towards automatic usability evaluation of spoken dialogue services by user error simulations , 2006, INTERSPEECH.

[14]  Grace Chung,et al.  Developing a Flexible Spoken Dialog System Using Simulation , 2004, ACL.

[15]  Ramón López-Cózar,et al.  Assessment of dialogue systems by means of a new simulation technique , 2003, Speech Commun..