Spoken Dialogue Systems

Abstract Considerable progress has been made in recent years in the development of dialogue systems that support robust and efficient human-machine interaction using spoken language. Spoken dialogu...

[1]  Jeremy H. Wright,et al.  Using Natural Language Processing and Discourse Features to Identify Understanding Errors in a Spoken Dialogue System , 2000 .

[2]  Jean Carletta,et al.  Assessing Agreement on Classification Tasks: The Kappa Statistic , 1996, CL.

[3]  Encarna Segarra,et al.  Error handling in a stochastic dialog system through confidence measures , 2005, Speech Commun..

[4]  Constantine Stephanidis,et al.  The Universal Access Handbook , 2009 .

[5]  John J. Godfrey,et al.  SWITCHBOARD: telephone speech corpus for research and development , 1992, [Proceedings] ICASSP-92: 1992 IEEE International Conference on Acoustics, Speech, and Signal Processing.

[6]  Masahiro Araki,et al.  Automatic Evaluation Environment for Spoken Dialogue Systems , 1996, ECAI Workshop on Dialogue Processing in Spoken Language Systems.

[7]  Ramón López-Cózar,et al.  Testing the performance of spoken dialogue systems by means of an artificially simulated user , 2006, Artificial Intelligence Review.

[8]  Oliver Lemon,et al.  Author manuscript, published in "European Conference on Speech Communication and Technologies (Interspeech'07), Anvers: Belgium (2007)" Machine Learning for Spoken Dialogue Systems , 2022 .

[9]  Alexander I. Rudnicky,et al.  Modeling the cost of misunderstanding errors in the CMU Communicator dialog system , 2001, IEEE Workshop on Automatic Speech Recognition and Understanding, 2001. ASRU '01..

[10]  Michael F. McTear,et al.  Handling errors and determining confirmation strategies - An object-based approach , 2003, Speech Commun..

[11]  Encarna Segarra,et al.  User simulation in a stochastic dialog system , 2008, Comput. Speech Lang..

[12]  Wayne H. Ward,et al.  Recent Improvements in the CMU Spoken Language Understanding System , 1994, HLT.

[13]  David Griol,et al.  A stochastic approach for dialog management based on neural networks , 2006, INTERSPEECH.

[14]  Martha E. Pollack,et al.  Introducing the Tileworld: Experimentally Evaluating Agent Architectures , 1990, AAAI.

[15]  Markus Chimani,et al.  DiamondHelp: A Collaborative Task Guidance Framework for Complex Devices , 2005, AAAI.

[16]  Markku Turunen,et al.  Adaptive Dialogue Systems - Interaction with Interact , 2002, SIGDIAL Workshop.

[17]  Sebastian Möller.  A new Taxonomy for the Quality of Telephone Services Based on Spoken Dialogue Systems , 2002, SIGDIAL Workshop.

[18]  M. Pickering,et al.  Toward a mechanistic psychology of dialogue , 2004, Behavioral and Brain Sciences.

[19]  Julia Hirschberg,et al.  Predicting Automatic Speech Recognition Performance Using Prosodic Cues , 2000, ANLP.

[20]  Diane J. Litman,et al.  Speech recognition performance and learning in spoken dialogue tutoring , 2005, INTERSPEECH.

[21]  J. Gabriel Amores,et al.  A MULTIMODAL ARCHITECTURE FOR HOME CONTROL BY DISABLED USERS , 2006, 2006 IEEE Spoken Language Technology Workshop.

[22]  K. Colby.  Artificial paranoia; a computer simulation of paranoid processes , 1975 .

[23]  Gabriel Skantze,et al.  GALATEA: A Discourse Modeller Supporting Concept-Level Error Handling in Spoken Dialogue Systems , 2005, SIGDIAL.

[24]  H. H. Clark,et al.  Referring as a collaborative process , 1986, Cognition.

[25]  Lynette Hirschman,et al.  Overview of evaluation in speech and natural language processing , 1997 .

[26]  Kristiina Jokinen,et al.  Adaptivity and Response Generation in a Spoken Dialogue System , 2003 .

[27]  Julia Hirschberg,et al.  Error handling in spoken dialogue systems , 2005, Speech Commun..

[28]  Tim Paek,et al.  Reinforcement Learning for Spoken Dialogue Systems: Comparing Strengths and Weaknesses for Practical Deployment , 2006 .

[29]  Karen E. Lochbaum,et al.  Using collaborative plans to model the intentional structure of discourse , 1995 .

[30]  Tatsuya Kawahara,et al.  Flexible Mixed-Initiative Dialogue Management using Concept-Level Confidence Measures of Speech Recognizer Output , 2000, COLING.

[31]  Stanley Peters,et al.  CHAT to Your Destination , 2007, SIGDIAL.

[32]  S. Young,et al.  Scaling POMDPs for Spoken Dialog Management , 2007, IEEE Transactions on Audio, Speech, and Language Processing.

[33]  S. Singh,et al.  Optimizing Dialogue Management with Reinforcement Learning: Experiments with the NJFun System , 2011, J. Artif. Intell. Res..

[34]  Steve Young,et al.  Automatic learning of dialogue strategy using dialogue simulation and reinforcement learning , 2002 .

[35]  Sun-Yuan Kung,et al.  Environment adaptation for robust speaker verification by cascading maximum likelihood linear regression and reinforced learning , 2007, Comput. Speech Lang..

[36]  Hua Ai,et al.  Comparing Spoken Dialog Corpora Collected with Recruited Subjects versus Real Users , 2007, SIGDIAL.

[37]  Gunnar Evermann,et al.  Large vocabulary decoding and confidence estimation using word posterior probabilities , 2000, 2000 IEEE International Conference on Acoustics, Speech, and Signal Processing. Proceedings (Cat. No.00CH37100).

[38]  Alexander I. Rudnicky,et al.  Sorry and I Didn’t Catch That! - An Investigation of Non-understanding Errors and Recovery Strategies , 2005, SIGDIAL.

[39]  Sebastian Möller.  Towards generic quality prediction models for spoken dialogue systems - a case study , 2005, INTERSPEECH.

[40]  Joëlle Coutaz,et al.  A generic platform for addressing the multimodal challenge , 1995, CHI '95.

[41]  Joseph Weizenbaum,et al.  and Machine , 1977 .

[42]  Ralf Schlüter,et al.  Using word probabilities as confidence measures , 1998, Proceedings of the 1998 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP '98 (Cat. No.98CH36181).

[43]  R. Cole,et al.  Survey of the State of the Art in Human Language Technology , 2010 .

[44]  Ronald A. Cole,et al.  TOOLS FOR RESEARCH AND EDUCATION IN SPEECH SCIENCE , 1999 .

[45]  Morena Danieli,et al.  Metrics for Evaluating Dialogue Strategies in a Spoken Language System , 1996, ArXiv.

[46]  Marilyn A. Walker,et al.  Evaluating Response Strategies in a Web-Based Spoken Dialogue Agent , 1998, ACL.

[47]  David G. Stork,et al.  Hal's Legacy: 2001's Computer as Dream and Reality , 1996 .

[48]  Michael Johnston,et al.  Unification-based Multimodal Parsing , 1998, ACL.

[49]  Anton Leuski,et al.  A Virtual Human for Tactical Questioning , 2007, SIGDIAL 2007.

[50]  Douglas B. Terry,et al.  Using collaborative filtering to weave an information tapestry , 1992, CACM.

[51]  Ramón López-Cózar,et al.  ASR post-correction for spoken dialogue systems based on semantic, syntactic, lexical and contextual information , 2008, Speech Commun..

[52]  Marilyn A. Walker,et al.  PARADISE: A Framework for Evaluating Spoken Dialogue Agents , 1997, ACL.

[53]  David R. Traum,et al.  Cooperation, dialogue and ethics , 2000, Int. J. Hum. Comput. Stud..

[54]  Kristiina Jokinen,et al.  Adaptation and user expertise modelling in AthosMail , 2006, Universal Access in the Information Society.

[55]  David Traum,et al.  Computational Models of Grounding in Collaborative Systems , 1999 .

[56]  Kristiina Jokinen,et al.  User expectations and real experience on a multimodal interactive system , 2006, INTERSPEECH.

[57]  Staffan Larsson,et al.  Information state and dialogue management in the TRINDI dialogue move engine toolkit , 2000, Natural Language Engineering.

[58]  Susan T. Dumais,et al.  Personalized information delivery: an analysis of information filtering methods , 1992, CACM.

[59]  Nicole Beringer,et al.  PROMISE - A Procedure for Multimodal Interactive System Evaluation , 2002 .

[60]  Andreas Stolcke,et al.  Dialogue act modeling for automatic tagging and recognition of conversational speech , 2000, CL.

[61]  Victor Zue,et al.  GALAXY-II: a reference architecture for conversational system development , 1998, ICSLP.

[62]  Ronnie W. Smith,et al.  Effective Spoken Natural Language Dialog Requires Variable Initiative Behavior: An Empirical Study , 1993 .

[63]  Roberto Pieraccini,et al.  A stochastic model of human-machine interaction for learning dialog strategies , 2000, IEEE Trans. Speech Audio Process..

[64]  Hui Ye,et al.  Agenda-Based User Simulation for Bootstrapping a POMDP Dialogue System , 2007, NAACL.

[65]  Steve J. Young,et al.  Partially observable Markov decision processes for spoken dialog systems , 2007, Comput. Speech Lang..

[66]  Günther Görz,et al.  Towards understanding spontaneous speech: word accuracy vs. concept accuracy , 1996, Proceeding of Fourth International Conference on Spoken Language Processing. ICSLP '96.

[67]  Nigel Gilbert,et al.  Simulating speech systems , 1991 .

[68]  James H. Martin,et al.  Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition , 2000 .

[69]  Gökhan Tür,et al.  The AT&T spoken language understanding system , 2006, IEEE Transactions on Audio, Speech, and Language Processing.

[70]  Nicole Yankelovich,et al.  How do users know what to say? , 1996, INTR.

[71]  Joakim Gustafson,et al.  Providing Computer Game Characters with Conversational Abilities , 2005, IVA.

[72]  Emiel Krahmer,et al.  The dual of denial: Two uses of disconfirmations in dialogue and their prosodic correlates , 2002, Speech Commun..

[73]  Marilyn A. Walker,et al.  The Effect of Resource Limits and Task Complexity on Collaborative Planning in Dialogue , 1995, Artif. Intell..

[74]  David Traum,et al.  CONVERSATIONAL AGENCY: THE TRAINS-93 DIALOGUE MANAGER , 2007 .

[75]  Victor Zue,et al.  Data collection and performance evaluation of spoken dialogue systems: the MIT experience , 2000, INTERSPEECH.

[76]  Philip R. Cohen,et al.  Intentions in Communication. , 1992 .

[77]  Alexander I. Rudnicky,et al.  Task and domain specific modelling in the Carnegie Mellon communicator system , 2000, INTERSPEECH.

[78]  J. Carletta,et al.  Risk-taking and recovery in task-oriented dialogue , 1996 .

[79]  Gregory A. Sanders,et al.  DARPA communicator dialog travel planning systems: the june 2000 data collection , 2001, INTERSPEECH.

[80]  Michael F. McTear,et al.  Implementing advanced spoken dialogue management in Java , 2005, Sci. Comput. Program..

[81]  Niels Ole Bernsen,et al.  Evaluating Conversation with Hans Christian Andersen , 2004, LREC.

[82]  Stephanie Seneff,et al.  Dialogue Management in the Mercury Flight Reservation System , 2000 .

[83]  H. Grice.  Logic and conversation , 1975 .

[84]  Kristiina Jokinen,et al.  Modality fusion in a route navigation system , 2006 .

[85]  Wolfgang Wahlster,et al.  SmartKom: Foundations of Multimodal Dialogue Systems , 2006, SmartKom.

[86]  Oliver Lemon,et al.  DIPPER: Description and Formalisation of an Information-State Update Dialogue System Architecture , 2003, SIGDIAL Workshop.

[87]  Eric K. Ringger,et al.  Error correction via a post-processor for continuous speech recognition , 1996, 1996 IEEE International Conference on Acoustics, Speech, and Signal Processing Conference Proceedings.

[88]  Lou Boves,et al.  Dialogue management in the dutch ARISE train timetable information system , 1999, EUROSPEECH.

[89]  Salim Roukos,et al.  Bleu: a Method for Automatic Evaluation of Machine Translation , 2002, ACL.

[90]  Pablo A. Haya,et al.  A Plug and Play Spoken Dialogue Interface for Smart Environments , 2004, CICLing.

[91]  Mark Weiser.  The computer for the 21st century , 1991 .

[92]  Ute Jekosch,et al.  Voice and Speech Quality Perception: Assessment and Evaluation , 2005 .

[93]  Candace L. Sidner.  Building Spoken Language Collaborative Interface Agents , 2004 .

[94]  Ramón López-Cózar,et al.  Assessment of dialogue systems by means of a new simulation technique , 2003, Speech Commun..

[95]  Gareth M. James,et al.  Challenges For Spoken Dialogue Systems , 1999 .

[96]  Niels Ole Bernsen,et al.  A Methodology for Evaluating Spoken Language Dialogue Systems and Their Components , 2000, LREC.

[97]  R. Power.  The organisation of purposeful dialogues , 1979 .

[98]  Jason Williams,et al.  Applying POMDPs to Dialog Systems in the Troubleshooting Domain , 2007, HLT-NAACL 2007.

[99]  Alexander I. Rudnicky,et al.  Stochastic Language Generation for Spoken Dialogue Systems , 2000 .

[100]  Clifford Nass,et al.  Computers that care: investigating the effects of orientation of emotion exhibited by an embodied computer agent , 2005, Int. J. Hum. Comput. Stud..

[101]  Sven Behnke,et al.  Fritz - A Humanoid Communication Robot , 2007, RO-MAN 2007 - The 16th IEEE International Symposium on Robot and Human Interactive Communication.

[102]  C. Sidner,et al.  Plans for Discourse , 1988 .

[103]  Eija Kaasinen,et al.  User acceptance of location-aware mobile guides based on seven field studies , 2005, Behav. Inf. Technol..

[104]  Alexander I. Rudnicky,et al.  Creating natural dialogs in the carnegie mellon communicator system , 1999, EUROSPEECH.

[105]  Oliver Lemon,et al.  multithreaded context for robust conversational interfaces: Context-sensitive speech recognition and interpretation of corrective fragments , 2004, TCHI.

[106]  Stephanie Seneff,et al.  TINA: A Natural Language System for Spoken Language Applications , 1992, Comput. Linguistics.

[107]  Alexander I. Rudnicky,et al.  A schema based approach to dialog control , 1998, ICSLP.

[108]  Johanna D. Moore,et al.  A Reactive Approach to Explanation , 1989, IJCAI.

[109]  James F. Allen,et al.  Towards Conversational Human-Computer Interaction , 2000 .

[110]  Marilyn A. Walker,et al.  Learning to Predict Problematic Situations in a Spoken Dialogue System: Experiments with How May I Help You? , 2000, ANLP.

[111]  Oliver Lemon,et al.  A Corpus Collection and Annotation Framework for Learning Multimodal Clarification Strategies , 2005, SIGDIAL.

[112]  Michael F. McTear,et al.  Book Review , 2005, Computational Linguistics.

[113]  Alexander I. Rudnicky,et al.  Ravenclaw: dialog management using hierarchical task decomposition and an expectation agenda , 2003, INTERSPEECH.

[114]  Roger K. Moore,et al.  Handbook of Multimodal and Spoken Dialogue Systems: Resources, Terminology and Product Evaluation , 2000 .

[115]  J. Cassell,et al.  Embodied conversational agents , 2000 .

[116]  John H. L. Hansen,et al.  University of Colorado Dialogue Systems for Travel and Navigation , 2001, HLT.

[117]  Shimei Pan,et al.  Designing and Evaluating an Adaptive Spoken Dialogue System , 2002, User Modeling and User-Adapted Interaction.

[118]  Timothy W. Bickmore,et al.  Establishing and maintaining long-term human-computer relationships , 2005, TCHI.

[119]  Hinrich Schütze,et al.  Book Reviews: Foundations of Statistical Natural Language Processing , 1999, CL.

[120]  Gabriel Skantze,et al.  Exploring human error recovery strategies: Implications for spoken dialogue systems , 2005, Speech Communication.

[121]  J. Forgas.  Mood and judgment: the affect infusion model (AIM). , 1995, Psychological bulletin.

[122]  Hector J. Levesque,et al.  Rational interaction as the basis for communication , 2003 .

[123]  Marilyn A. Walker,et al.  Automatic Detection of Poor Speech Recognition at the Dialogue Level , 1999, ACL.

[124]  Graeme Hirst,et al.  Evaluating WordNet-based Measures of Lexical Semantic Relatedness , 2006, CL.

[125]  Niels Ole Bernsen,et al.  Designing Co-operativity in Spoken Human-Machine Dialogues , 1997 .

[126]  Laurel Fais.  Conversation as collaboration: Some syntactic evidence , 1994, Speech Communication.

[127]  Lou Boves,et al.  Incorporating confidence measures in the Dutch train timetable information system developed in the ARISE project , 1999, 1999 IEEE International Conference on Acoustics, Speech, and Signal Processing. Proceedings. ICASSP99 (Cat. No.99CH36258).

[128]  Michael F. McTear,et al.  Evaluating the DI@l-log system on a cohort of elderly, diabetic patients: results from a preliminary study , 2005, INTERSPEECH.

[129]  Ben Shneiderman,et al.  Designing the User Interface: Strategies for Effective Human-Computer Interaction , 1998 .

[130]  Marilyn A. Walker,et al.  An Application of Reinforcement Learning to Dialogue Strategy Selection in a Spoken Dialogue System for Email , 2000, J. Artif. Intell. Res..

[131]  Ali Choumane,et al.  A model for multimodal representation and processing for reference resolution , 2007, WMISI '07.

[132]  Adam Cheyer,et al.  Multimodal Maps: An Agent-Based Approach , 1995, Multimodal Human-Computer Communication.

[133]  Kristina Höök,et al.  Steps to take before intelligent user interfaces become real , 2000, Interact. Comput..

[134]  Anthony Jameson,et al.  Pros and Cons of Controllability: An Empirical Study , 2002, AH.

[135]  Dong Yu,et al.  An introduction to voice search , 2008, IEEE Signal Processing Magazine.

[136]  Johanna D. Moore,et al.  Information Presentation in Spoken Dialogue Systems , 2006, EACL.

[137]  Roberto Pieraccini,et al.  Technical Support Dialog Systems:Issues, Problems, and Solutions , 2007, HLT-NAACL 2007.

[138]  Chung Hee Hwang,et al.  The TRAINS project: a case study in building a conversational planning agent , 1994, J. Exp. Theor. Artif. Intell..

[139]  Scott McGlashan,et al.  Units of dialogue management: an example , 1996, Proceeding of Fourth International Conference on Spoken Language Processing. ICSLP '96.

[140]  Kristiina Jokinen,et al.  Distributed Dialogue Management in a Blackboard Architecture , 2003 .

[141]  Robert Graham,et al.  Towards a tool for the Subjective Assessment of Speech System Interfaces (SASSI) , 2000, Natural Language Engineering.

[142]  Alexander I. Rudnicky,et al.  Sorry, I Didn’t Catch That! , 2008 .

[143]  P. J. Price,et al.  Evaluation of Spoken Language Systems: the ATIS Domain , 1990, HLT.

[144]  Philippe Bretier,et al.  ARTIMIS: Natural Dialogue Meets Rational Agency , 1997, IJCAI.

[145]  Barbara Di Eugenio,et al.  Squibs and Discussions: The Kappa Statistic: A Second Look , 2004, CL.

[146]  Steve J. Young,et al.  A survey of statistical user simulation techniques for reinforcement-learning of dialogue management strategies , 2006, The Knowledge Engineering Review.

[147]  Giuseppe Riccardi,et al.  How may I help you? , 1997, Speech Commun..

[148]  Alexander I. Rudnicky,et al.  Olympus: an open-source framework for conversational spoken language interface research , 2007, HLT-NAACL 2007.

[149]  K. Chang,et al.  Embodiment in conversational interfaces: Rea , 1999, CHI '99.

[150]  Patti Price,et al.  The DARPA 1000-word resource management database for continuous speech recognition , 1988, ICASSP-88., International Conference on Acoustics, Speech, and Signal Processing.

[151]  Jason D. Williams,et al.  The best of both worlds: unifying conventional dialog systems and POMDPs , 2008, INTERSPEECH.

[152]  Hui Ye,et al.  Training a real-world POMDP-based Dialog System , 2007, HLT-NAACL 2007.

[153]  Gabriel Skantze.  Error Handling in Spoken Dialogue Systems : Managing Uncertainty, Grounding and Miscommunication , 2007 .

[154]  Kristiina Jokinen,et al.  User Expertise Modeling and Adaptivity in a Speech-Based E-Mail System , 2004, ACL.

[155]  Jakob Nielsen,et al.  Usability inspection methods , 1994, CHI 95 Conference Companion.

[156]  Sebastian Möller,et al.  Memo: towards automatic usability evaluation of spoken dialogue services by user error simulations , 2006, INTERSPEECH.

[157]  Emiel Krahmer,et al.  Problem spotting in human-machine interaction , 1999, EUROSPEECH.

[158]  Emiel Krahmer,et al.  Error Detection in Spoken Human-Machine Interaction , 2001, Int. J. Speech Technol..

[159]  Laila Dybkjær,et al.  Recent trends in discourse and dialogue , 2008 .

[160]  R. Bashirullah,et al.  Technology and Signal Processing for Brain-Machine Interfaces , 2008, IEEE Signal Processing Magazine.

[161]  Marilyn A. Walker,et al.  Evaluating spoken dialogue agents with PARADISE: Two case studies , 1998, Comput. Speech Lang..

[162]  David A. van Leeuwen,et al.  Multilingual large vocabulary speech recognition: the European SQALE project , 1997, Comput. Speech Lang..

[163]  Matthew Purver,et al.  Robust interpretation in dialogue by combining confidence scores with contextual features , 2006, INTERSPEECH.

[164]  Kallirroi Georgila,et al.  Hybrid reinforcement/supervised learning for dialogue policies from COMMUNICATOR data , 2005 .

[165]  Donald A. Norman,et al.  Emotion & design: attractive things work better , 2002, INTR.

[166]  Jeremy H. Wright,et al.  Automatically Training a Problematic Dialogue Predictor for a Spoken Dialogue System , 2011, J. Artif. Intell. Res..

[167]  Toni Giorgino,et al.  Automated Spoken Dialog System for Hypertensive Patient Home Management , 2004 .

[168]  Alexander I. Rudnicky,et al.  Error awareness and recovery in conversational spoken language interfaces , 2007 .

[169]  Philip R. Cohen,et al.  Persistence, Intention, and Commitment , 2003 .

[170]  Joseph Polifroni,et al.  A form-based dialogue manager for spoken language applications , 1996, Proceeding of Fourth International Conference on Spoken Language Processing. ICSLP '96.

[171]  Gerd Gigerenzer,et al.  How do we tell an association from a rule , 1996 .

[172]  Volker Steinbiss,et al.  The Philips automatic train timetable information system , 1995, Speech Commun..

[173]  David P. Morgan,et al.  How to build a speech recognition application : a style guide for telephony dialogues , 2001 .

[174]  Sebastian Mller,et al.  Quality of Telephone-Based Spoken Dialogue Systems , 2004 .

[175]  Colin Potts,et al.  Design of Everyday Things , 1988 .

[176]  Joseph Polifroni,et al.  Integrating recognition confidence scoring with language understanding and dialogue modeling , 2000, INTERSPEECH.

[177]  Eric Horvitz,et al.  Conversation as Action Under Uncertainty , 2000, UAI.

[178]  Candace L. Sidner,et al.  Lessons Learned in Building Spoken Language Collaborative Interface Agents , 2000 .

[179]  Jennifer Balogh,et al.  Voice User Interface Design , 2004 .

[180]  E. Levin,et al.  Evaluation of Spoken Dialogue Technology for Real-Time Health Data Collection , 2006, Journal of medical Internet research.

[181]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[182]  Michael F. McTear,et al.  Promoting extension and reuse in a spoken dialog manager: An evaluation of the queen's communicator , 2007, TSLP.

[183]  Lynette Hirschman,et al.  Comparing Several Aspects of Human-Computer and Human-Human Dialogues , 2001, SIGDIAL Workshop.

[184]  James F. Allen,et al.  An architecture for more realistic conversational systems , 2001, IUI '01.

[185]  Norbert Reithinger,et al.  Insights into the Dialogue Processing of VERBMOBIL , 1997, ANLP.

[186]  Melita Hajdinjak,et al.  The PARADISE Evaluation Framework: Issues and Findings , 2006, Computational Linguistics.

[187]  Kallirroi Georgila,et al.  Learning user simulations for information state update dialogue systems , 2005, INTERSPEECH.

[188]  Ian McGraw,et al.  The WAMI toolkit for developing, deploying, and evaluating web-accessible multimodal interfaces , 2008, ICMI '08.

[189]  David R. Traum,et al.  Discourse Obligations in Dialogue Processing , 1994, ACL.