Evaluating Text Quality: Judging Output Texts Without a Clear Source

We consider how far two attributes of text quality commonly used in MT evaluation -- intelligibility and fidelity -- apply within NLG. While the former appears to transfer directly, the latter needs to be completely re-interpreted. We make a crucial distinction between the needs of symbolic authors and those of end-readers. We describe a form of textual feedback, based on a controlled language used for specifying software requirements that appears well suited to authors' needs, and an approach for incrementally improving the fidelity of this feedback text to the content model.

[1]  Norbert E. Fuchs,et al.  Attempto Controlled English (ACE) , 1996, ArXiv.

[2]  Srinivas Bangalore,et al.  Exploiting a Probabilistic Hierarchical Model for Generation , 2000, COLING.

[3]  Srinivas Bangalore,et al.  Evaluation Metrics for Generation , 2000, INLG.

[4]  Ivana Kruijff-Korbayová,et al.  Multilinguality in a Text Generation System For Three Slavic Languages , 2000, COLING.

[5]  Marian Petre,et al.  Why looking isn't always seeing: readership skills and graphical programming , 1995, CACM.

[6]  Richard Power,et al.  Generation as a Solution to Its Own Problem , 1998, INLG.

[7]  Cécile Paris,et al.  A Support Tool for Writing Multilingual Instructions , 1995, IJCAI.

[8]  Daniel S. Paiva,et al.  In search of a reference architecture for NLG systemsLynne , 1999 .

[9]  Jun'ichi Tsujii,et al.  The Japanese Government Project for Machine Translation , 1985, Comput. Linguistics.

[10]  Richard Power,et al.  Multilingual Authoring Using Feedback Texts , 1998, COLING-ACL.

[11]  Jade Goldstein-Stewart,et al.  Interactive graphic design using automatic presentation knowledge , 1994, CHI '94.

[12]  Tanya Korelsky,et al.  Bilingual Generation of Job from Quasi-Conceptual Descriptions Forms , 1994, ANLP.

[13]  Johanna D. Moore,et al.  Describing Complex Charts in Natural Language: A Caption Generation System , 1998, CL.

[14]  Gordon Davis,et al.  Effects of conceptual data modeling formalisms on user validation and analyst modeling of information requirements , 1990 .

[15]  John Lehrberger,et al.  Machine Translation: Linguistic characteristics of MT systems and general methodology of evaluation , 1988 .

[16]  David E. Caldwell,et al.  Bilingual Generation of Job Descriptions from Quasi-Conceptual Forms * , 1998 .

[17]  Norbert E. Fuchs,et al.  Attempto Controlled English (ACE)Language ManualVersion 3.0 , 1999 .

[18]  John B. Carroll.  An experiment in evaluating the quality of translations , 1966, Mech. Transl. Comput. Linguistics.

[19]  National Academy of Sciences National Research Council , 1961 .

[20]  Srinivas Bangalore,et al.  Automatic Acquisition of Hierarchical Transduction Models for Machine Translation , 1998, COLING-ACL.

[21]  Doug Arnold,et al.  Machine Translation: An Introductory Guide , 1994 .