Metrics and methods for comparative ontology evaluation

While progress has been made toward describing the need for ontology evaluation and offering proposals concerning what properties to measure and how, work remains to develop ontology evaluation as a rigorous discipline. Ontologies as information artifacts have a variety of aspects that can inform their evaluation, both in terms of what is evaluated and the metrics used. Ontology evaluation as a discipline requires (1) having a systematic account of the different aspects of ontologies and the properties relevant to those aspects, (2) critically developing methods for examining those properties, (3) developing comparative metrics that allow ontology engineers to compare the effects of various modeling choices and allow users to compare the merits of existing ontologies, and (4) charting possible pitfalls of evaluation. This paper considers various properties of ontologies that have been proposed and organizes these properties according to different aspects of ontologies. To begin bringing previous work together and to illustrate where pitfalls and potential solutions might enter into a rigorous evaluation, I offer a more in depth (though still partial) analysis of evaluating the correctness of ontologies. I conclude with a discussion of next steps in systematizing ontology evaluation.