A Multi-lingual Evaluation of the vAssist Spoken Dialog System. Comparing Disco and RavenClaw

vAssist (Voice Controlled Assistive Care and Communication Services for the Home) is a European project for which several research institutes and companies have been working on the development of adapted spoken interfaces to support home care and communication services. This paper describes the spoken dialog system that has been built. Its natural language understanding module includes a novel reference resolver and it introduces a new hierarchical paradigm to model dialog tasks. The user-centered approach applied to the whole development process led to the setup of several experiment sessions with real users. Multilingual experiments carried out in Austria, France and Spain are described along with their analyses and results in terms of both system performance and user experience. An additional experimental comparison of the RavenClaw and Disco-LFF dialog managers built into the vAssist spoken dialog system highlighted similar performance and user acceptance.

[1]  Marc Schröder,et al.  The German Text-to-Speech Synthesis System MARY: A Tool for Research, Development and Teaching , 2003, Int. J. Speech Technol..

[2]  Volker Steinbiss,et al.  The Philips automatic train timetable information system , 1995, Speech Commun..

[3]  Steve J. Young,et al.  Reinforcement learning for parameter estimation in statistical spoken dialogue systems , 2012, Comput. Speech Lang..

[4]  Maxine Eskénazi,et al.  Let's go public! taking a spoken dialog system to the real world , 2005, INTERSPEECH.

[5]  Roberto Pieraccini,et al.  Using Markov decision process for learning dialogue strategies , 1998, Proceedings of the 1998 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP '98 (Cat. No.98CH36181).

[6]  Viswanath Venkatesh,et al.  Technology Acceptance Model 3 and a Research Agenda on Interventions , 2008, Decis. Sci..

[7]  Staffan Larsson,et al.  Information state and dialogue management in the TRINDI dialogue move engine toolkit , 2000, Natural Language Engineering.

[8]  William F. Ross.  Highlights of the 1974 Lake Arrowhead Workshop , 1975, Computer.

[9]  Alexander I. Rudnicky,et al.  The RavenClaw dialog management framework: Architecture and systems , 2009, Comput. Speech Lang..

[10]  Milica Gasic,et al.  Transformation-based learning for semantic parsing , 2009, INTERSPEECH.

[11]  David Griol,et al.  A statistical approach to spoken dialog systems design and evaluation , 2008, Speech Commun..

[12]  G. Chollet,et al.  vAssist : Le Majordome des personnes dépendantes , 2011 .

[13]  Alexander I. Rudnicky,et al.  Olympus: an open-source framework for conversational spoken language interface research , 2007, HLT-NAACL 2007.

[14]  M. Inés Torres.  Stochastic Bi-Languages to model Dialogs , 2013, FSMNLP.

[15]  J. B. Brooke,et al.  SUS: A 'Quick and Dirty' Usability Scale , 1996 .

[16]  Fabrizio Ghigi,et al.  Decision Making Strategies for Finite-State Bi-automaton in Dialog Management , 2015, Natural Language Dialog Systems and Intelligent Assistants.

[17]  Milica Gasic,et al.  POMDP-Based Statistical Spoken Dialog Systems: A Review , 2013, Proceedings of the IEEE.

[18]  Steve J. Young,et al.  Partially observable Markov decision processes for spoken dialog systems , 2007, Comput. Speech Lang..

[19]  K. Á. T.,et al.  Towards a tool for the Subjective Assessment of Speech System Interfaces (SASSI) , 2000, Natural Language Engineering.

[20]  Joseph Weizenbaum,et al.  and Machine , 1977 .

[21]  Stephen Young.  Probabilistic methods in spoken–dialogue systems , 2000, Philosophical Transactions of the Royal Society of London. Series A: Mathematical, Physical and Engineering Sciences.

[22]  Eric Brill,et al.  Transformation-Based Error-Driven Learning and Natural Language Processing: A Case Study in Part-of-Speech Tagging , 1995, CL.

[23]  Victor Zue,et al.  JUPlTER: a telephone-based conversational interface for weather information , 2000, IEEE Trans. Speech Audio Process..

[24]  Pierre Lison,et al.  A hybrid approach to dialogue management based on probabilistic rules , 2015, Comput. Speech Lang..

[25]  Thomas S. Tullis,et al.  A Comparison of Methods for Eliciting Post-Task Subjective Ratings in Usability Testing , 2006 .

[26]  Gary Geunbae Lee,et al.  A Situation-Based Dialogue Management using Dialogue Examples , 2006, 2006 IEEE International Conference on Acoustics Speech and Signal Processing Proceedings.

[27]  Pierrick Milhorat,et al.  An open-source framework for supporting the design and implementation of natural-language spoken dialog systems. (Une plate-forme ouverte pour la conception et l'implémentation de systèmes de dialogue vocaux en langage naturel) , 2014 .

[28]  Oliver Lemon,et al.  Evaluation of a hierarchical reinforcement learning spoken dialogue system , 2010, Comput. Speech Lang..

[29]  Charles Rich,et al.  Building Task-Based User Interfaces with ANSI/CEA-2018 , 2009, Computer.

[30]  Oliver Lemon,et al.  Parallel Computing and Practical Constraints when applying the Standard POMDP Belief Update Formalism to Spoken Dialogue Management , 2011 .