Development and Evaluation of NL interfaces in a Small Shop

The standard development of a dialogue system today involves the following steps: corpus collection and analysis, system development guided by corpus analysis, and finally, rigorous evaluation. Often, evaluation may involve more than one version of the system, for example when it is desirable to show the effect of system parameters that differ from one version to another. In this paper, we discuss the difficulties that small research groups face in pursuing the development of dialogue systems. The primary difficulties are the lack of adequate resources and the excessive amount of time it takes to see the systems through to a meaningful evaluation. As a case in point, we discuss our development and evaluation of a natural language generation component to improve the feedback provided by an interactive tutoring system. Our goal has been to use relatively inexpensive text structuring techniques to make aggregate content more fluent and comprehensible.

[1]  Michael White,et al.  EXEMPLARS: A Practical, Extensible Framework For Dynamic Text Generation , 1998, INLG.

[2]  Niels Ole Bernsen,et al.  Annotating Communication Problems Using the MATE Workbench , 2000, LREC.

[3]  Elizabeth Shriberg,et al.  Switchboard SWBD-DAMSL shallow-discourse-function annotation coders manual , 1997 .

[4]  Barbara Di Eugenio,et al.  The COCONUT project: Dialogue Annotation Manual , 1998 .

[5]  Xiaorong Huang,et al.  Paraphrasing and Aggregating Argumentative Texts Using Text Structure , 1996, INLG.

[6]  William C. Mann,et al.  Rhetorical Structure Theory: Toward a functional theory of text organization , 1988 .

[7]  Barbara Di Eugenio,et al.  The binomial cumulative distribution function, or, is my system better than yours? , 2002, LREC.

[8]  Scott Weinstein,et al.  Centering: A Framework for Modeling the Local Coherence of Discourse , 1995, CL.

[9]  Stuart C. Shapiro.  SNePS: a logic for natural language understanding and commonsense reasoning , 2000 .

[10]  Johanna D. Moore,et al.  Generating descriptions of complex activities , 1997 .

[11]  Cécile Paris,et al.  Tailoring Object Descriptions to a User's Level of Expertise , 1988, Comput. Linguistics.

[12]  Marilyn A. Walker,et al.  Towards Automatic Generation of Natural Language Generation Systems , 2002, COLING.

[13]  L. Lamel,et al.  Multi-layer Dialogue Annotation for Automated Multilingual Customer Service , 2003 .

[14]  S. Siegel,et al.  Nonparametric Statistics for the Behavioral Sciences. , 1957 .

[15]  James Shaw,et al.  Segregatory Coordination and Ellipsis in Text Generation , 1998, ACL.

[16]  Johanna D. Moore,et al.  An Empirical Study of the Influence of Argument Conciseness on Argument Effectiveness , 2000, ACL.

[17]  Douglas M. Towne.  Approximate Reasoning Techniques for Intelligent Diagnostic Instruction , 1997 .

[18]  Barbara Di Eugenio,et al.  The DIAG experiments: Natural Language Generation for Intelligent Tutoring Systems , 2002, INLG.

[19]  Beatrice Santorini,et al.  Building a Large Annotated Corpus of English: The Penn Treebank , 1993, CL.

[20]  Benoit Lavoie,et al.  A Fast and Portable Realizer for Text Generation Systems , 1997, ANLP.