Metrics for Evaluation of Ontology-based Information Extraction

The evaluation of the quality of ontological classification is an important part of semantic web technology. Because this area is under constant development, it requires improvement and standardisation. This paper discusses existing evaluation metrics, and proposes a new method for evaluating the ontology population task, which is general enough to be used in a variety of situations, yet more precise than many current metrics. The paper further describes our first eorts in operationalising the evaluation procedure, including the creation of a semantically annotated corpus that will function as a test bed for the proposed evaluation mechanism, and comparison of dierent evaluation metrics. We conclude that for ontology-based evaluation, a more complex mechanism than is traditionally used is preferable. This mechanism aims to drive a benchmarking assessment tool for the current state-of-the-art of ontology population, and to set a standard for best practice for future evaluation of human language technology for the semantic web.

[1]  Yoram Singer,et al.  Large margin hierarchical classification , 2004, ICML.

[2]  Jonathan G. Fiscus,et al.  NIST's 1998 topic detection and tracking evaluation (TDT2) , 1999, EUROSPEECH.

[3]  Nello Cristianini,et al.  An Introduction to Support Vector Machines and Other Kernel-based Learning Methods , 2000 .

[4]  Atanas Kiryakov,et al.  KIM - Semantic Annotation Platform , 2003, SEMWEB.

[5]  Eneko Agirre,et al.  Word Sense Disambiguation using Conceptual Density , 1996, COLING.

[6]  Diana Maynard,et al.  D2.1.4 Specication of a methodology, general criteria, and benchmark suites for benchmarking ontology tools , 2005 .

[7]  Steffen Staab,et al.  S-CREAM: Semiautomatic CREAtion of Metadata , 2002, SAAKM@ECAI.

[8]  Sophia Ananiadou,et al.  Identifying Terms by their Family and Friends , 2000, COLING.

[9]  Nancy Chinchor,et al.  MUC-4 evaluation metrics , 1992, MUC.

[10]  Fredrik Olsson,et al.  Notions of Correctness when Evaluating Protein Name Taggers , 2002, COLING.

[11]  Kalina Bontcheva,et al.  Using Uneven Margins SVM and Perceptron for Information Extraction , 2005, CoNLL.

[12]  Udo Hahn,et al.  Towards Text Knowledge Engineering , 1998, AAAI/IAAI.

[13]  Kalina Bontcheva,et al.  Hierarchical, perceptron-like learning for ontology-based information extraction , 2007, WWW '07.

[14]  Ralph Grishman,et al.  Message Understanding Conference- 6: A Brief History , 1996, COLING.

[15]  Kalina Bontcheva,et al.  SVM Based Learning System for Information Extraction , 2004, Deterministic and Statistical Methods in Machine Learning.

[16]  Rémi Eyraud,et al.  Proceedings of CoNLL , 2006 .

[17]  Maghi King.  Living up to standards , 2003 .

[18]  Atanas Kiryakov,et al.  Semantic Annotation, Indexing, and Retrieval , 2003, SEMWEB.

[19]  Steffen Staab,et al.  Automatic Acquisition of Taxonomies from Text: FCA meets NLP , 2003 .

[20]  L. R. Rasmussen,et al.  In information retrieval: data structures and algorithms , 1992 .