Evaluation of a domain independent approach to natural language processing for game-like user interfaces

Many researchers that develop full software applications in the broad field of natural language processing (NLP) typically implement their single system's components from scratch. While there is nothing wrong with such a methodology from an operational perspective, it typically results in a waste of time. Furthermore, it leads to a substantial diversion of the researchers' efforts from more conceptual and theoretical aspects that could be geared towards the advancement of the state-of-the-art in the field. These main drawbacks call for an implementation approach allowing components' reusability across domains and applications. In that respect, this paper presents an evaluation of a domain independent approach to natural language understanding (NLU) that we have been implementing over the last several years. We have successfully tested and used our approach in three different natural language interfaces to game-like applications, each with its own conversational domains.

[1]  Lynette Hirschman.  Session 3: Human Language Evaluation , 1994, HLT.

[2]  Alexander I. Rudnicky,et al.  Multi-Site Data Collection and Evaluation in Spoken Language Understanding , 1993, HLT.

[3]  Lynette Hirschman,et al.  The cost of errors in a spoken language system , 1993, EUROSPEECH.

[4]  Niels Ole Bernsen,et al.  Structured interview-based evaluation of spoken multimodal conversation with h.c. andersen , 2004, INTERSPEECH.

[5]  Santiago Ontañón,et al.  Drama Management Evaluation for Interactive Fiction Games , 2007, AAAI Fall Symposium: Intelligent Narrative Technologies.

[6]  Andrea Corradini,et al.  Handling out of domain topics by a conversational character , 2008, DIMEA.

[7]  Nicoletta Calzolari,et al.  ELRA – European Language Resources Association-Background, Recent Developments and Future Perspectives , 2005, Lang. Resour. Evaluation.

[8]  Victor Zue,et al.  Experiments in Evaluating Interactive Spoken Language Systems , 1992, HLT.

[9]  Chris Mellish,et al.  Evaluation in the context of natural language generation , 1998, Comput. Speech Lang..

[10]  Christopher Cieri.  Multiple Annotations of Reusable Data Resources: Corpora for Topic Detection and Tracking , 2000 .

[11]  Andrea Corradini,et al.  Conversational Characters that Support Interactive Play and Learning for Children , 2009 .

[12]  Santiago Ontañón,et al.  Developing a Drama Management Architecture for Interactive Fiction Games , 2008, ICIDS.

[13]  Elizabeth Shriberg,et al.  Subject-Based Evaluation Measures for Interactive Spoken Language Systems , 1992, HLT.

[14]  Lynette Hirschman,et al.  The Evolution of evaluation: Lessons from the Message Understanding Conferences , 1998, Comput. Speech Lang..

[15]  Andrea Corradini,et al.  Interacting with an Animated Conversational Agent in a 3 D Graphical Setting , 2022 .