Evaluating Ontology Evaluation

Before we can give industry recommendations for incorporating ontology technology into their IT systems, we must consider two types of evaluation: content evaluation and ontology technology evaluation. Evaluating content is a must for preventing applications from using inconsistent, incorrect, or redundant ontologies. It’s unwise to publish an ontology that one or more software applications will use without first evaluating it. A wellevaluated ontology won’t guarantee the absence of problems, but it will make its use safer. Similarly, evaluating ontology technology will ease its integration with other software environments, ensuring a correct technology transfer from the academic to the industrial world. In this contribution, I explore both evaluation dimensions to try to answer the following questions: • How were widely used ontologies (including Cyc, WordNet and EuroWordNet, Standard Upper Ontology, and the DAML+OIL library) evaluated during development or once they were implemented in an ontology language? • How robust are ontology evaluation methods? What type of ontology components do they evaluate? Are they independent of the language used to implement the ontologies? • How do ontology development platforms perform content evaluation? How mature are the evaluation tools incorporated on such platforms? Which types of errors do these tools detect? • What are the criteria used for evaluating ontology tools? What are the results?