Design of the MUC-6 evaluation

The sixth in a series of "Message Understanding Conferences", which are designed to promote and evaluate research in information extraction, was held last fall. MUC-6 introduced several innovations over prior MUCs, most notably in the range of different tasks for which evaluations were conducted. We describe the development of the "message understanding" task over the course of the prior MUCs, some of the motivations for the new format, and the steps which led up to the formal evaluation.