Evaluating an Information Extraction System

Many natural language researchers are now turning their attention to a relatively new task orientation known as information extraction. Information extraction systems are predicated on an I/O orientation that makes it possible to conduct formal evaluations and meaningful cross-system comparisons. This article presents the challenge of information extraction and shows how information extraction systems are currently being evaluated. We describe a specific system developed at the University of Massachusetts, identify key research issues of general interest, and conclude with some observations about the role of performance evaluations as a stimulus for basic research.

[1]  Beth Sundheim,et al.  A Performance Evaluation of Text-Analysis Technologies , 1991, AI Mag..

[2]  Wendy G. Lehnert,et al.  Symbolic/Subsymbolic Sentence Analysi: Exploiting the Best of Two Worlds , 1988 .

[3]  Claire Cardie,et al.  University of Massachusetts: MUC-3 test results and analysis , 1991, MUC.

[4]  Claire Cardie,et al.  University of Massachusetts: Description of the CIRCUS System as Used for MUC-4 , 1992, MUC.

[5]  E. Riloff,et al.  Automated dictionary construction for information extraction from text , 1993, Proceedings of 9th IEEE Conference on Artificial Intelligence for Applications.

[6]  Beth Sundheim,et al.  Overview of the Fourth Message Understanding Evaluation and Conference , 1992, MUC.

[7]  Claire Cardie.  Using Cognitive Biases to Guide Feature Set Selection , 1992 .

[8]  Claire Cardie,et al.  A Cognitively Plausible Approach to Understanding Complex Syntax , 1991, AAAI.

[9]  Ellen Riloff,et al.  Automatically Constructing a Dictionary for Information Extraction Tasks , 1993, AAAI.

[10]  Claire Cardie,et al.  Learning to Disambiguate Relative Pronouns , 1992, AAAI.

[11]  Ellen Riloff.  Using cases to represent context for text classification , 1993, CIKM '93.

[12]  Claire Cardie,et al.  University of Massachusetts: Description of the CIRCUS System as Used for MUC-3 , 1991, MUC.

[13]  Ellen Riloff,et al.  Applying Statistical Methods to Small Corpora: Benefitting from a Limited Domain* , 1992 .

[14]  Claire Cardie,et al.  University of Massachusetts: MUC-4 test results and analysis , 1992, MUC.

[15]  Claire Cardie,et al.  Corpus-Based Acquisition of Relative Pronoun Disambiguation Heuristics , 1992, ACL.

[16]  Ellen Riloff,et al.  Classifying Texts Using Relevancy Signatures , 1992, AAAI.

[17]  Lynette Hirschman,et al.  Evaluating Message Understanding Systems: An Analysis of the Third Message Understanding Conference (MUC-3) , 1993, CL.

[18]  Claire Cardie,et al.  The CIRCUS System as Used in MUC-3 , 1991 .