Quality Assurance of Medical Ontologies

OBJECTIVE To review the literature concerning the quality assurance of medical ontologies. METHODS scholar.google.com was searched using the search strings (+ontology +"quality assurance") and (+ontology +"evaluation/evaluating"). Relevant publications were selected by manual review. Other work already familiar to the author, or suggested by other researchers contacted by the author, were included. The papers were analysed for common themes. RESULTS Four broad properties of an ontology were identified that may be quality-assured: philosophical validity, compliance with meta-ontological commitments, 'content correctness', and fitness for purpose. Each published methodology addressed only a subset of these properties. 'Content' may be divided into domain knowledge content, and metadata describing either the provenance of domain knowledge content, or relationships between it and lexical information (e.g. for display and retrieval). 'Correctness' (whether of domain knowledge content or metadata) may also be further subdivided into truth, completeness, parsimony and internal consistency. CONCLUSIONS Understanding of how to assure the quality of ontologies, or evaluate their fitness for specific purposes, is improving but remains poor. A combination of methodologies is required, but tools to support a comprehensive quality assurance programme remain lacking. Perfect quality of an ontology is not provable and may not be desirable: an ontology compliant with all current philosophical theories, following necessary ontological commitments, and with entirely 'correct' content, may be too complex to be directly usable or useful. The extent to which an ontology's fitness for purpose is predicted or influenced by its other properties remains to be determined. Field studies of ontologies in use, including interrater effects, are required.

[1]  N. D. de Keizer,et al.  Methods for Evaluation of Medical Terminological Systems , 2005, Methods of Information in Medicine.

[2]  A. Gómez-Pérez,et al.  Evaluation of ontologies , 2001, Int. J. Intell. Syst..

[3]  John R. Josephson,et al.  What Are They? Why Do We Need Them? , 1999 .

[4]  Carole A. Goble,et al.  A Methodology to Migrate the Gene Ontology to a Description Logic Environment Using DAML+OIL , 2002, Pacific Symposium on Biocomputing.

[5]  Victoria Warmington,et al.  Randomised crossover trial comparing the performance of Clinical Terms Version 3 and Read Codes 5 byte set coding schemes in general practice , 2003, BMJ : British Medical Journal.

[6]  Carole Goble,et al.  The Montagues and the Capulets , 2004, Comparative and functional genomics.

[7]  Asunción Gómez-Pérez,et al.  From Knowledge Based Systems to Knowledge Sharing Technology: Evaluation and Assessment , 1994 .

[8]  Thomas R. Gruber,et al.  Toward principles for the design of ontologies used for knowledge sharing? , 1995, Int. J. Hum. Comput. Stud..

[9]  Olivier Bodenreider,et al.  Investigating subsumption in DL-based terminologies: A Case Study in SNOMED CT , 2004, KR-MED.

[10]  A. Gomez-Perez,et al.  Some ideas and examples to evaluate ontologies , 1995, Proceedings the 11th Conference on Artificial Intelligence for Applications.

[11]  Franz Baader,et al.  Restricted Role-value-maps in a Description Logic with Existential Restrictions and Terminological Cycles , 2003, Description Logics.

[12]  Kent A. Spackman,et al.  Role grouping as an extension to the description logic of Ontylog, motivated by concept modeling in SNOMED , 2002, AMIA.

[13]  Ronald Cornet,et al.  Two DL-based Methods for Auditing Medical Terminological Systems , 2005, AMIA.

[14]  Diego Calvanese,et al.  The Description Logic Handbook: Theory, Implementation, and Applications , 2003, Description Logic Handbook.

[15]  Colin Price,et al.  Application of Technology: Read Code Quality Assurance: From Simple Syntax to Semantic Stability , 1998, J. Am. Medical Informatics Assoc..

[16]  Peter F. Patel-Schneider,et al.  A Semantics and Complete Algorithm for Subsumption in the CLASSIC Description Logic , 1993, J. Artif. Intell. Res..

[17]  Cornelius Rosse,et al.  A formal theory for spatial representation and reasoning in biomedical ontologies , 2006, Artif. Intell. Medicine.

[18]  Alan L. Rector,et al.  Validating clinical terminology structures: integration and cross-validation of Read Thesaurus and GALEN , 1998, AMIA.

[19]  Maureen Donnelly.  Containment Relations in Anatomical Ontologies , 2005, AMIA.

[20]  Kent A. Spackman,et al.  Examining SNOMED from the perspective of formal ontological principles: Some preliminary analysis and observations , 2004, KR-MED.

[21]  Balakrishnan Chandrasekaran,et al.  What are ontologies, and why do we need them? , 1999, IEEE Intell. Syst..

[22]  Nicola Guarino,et al.  Sweetening Ontologies with DOLCE , 2002, EKAW.

[23]  C Payne,et al.  Read Codes Version 3: A User Led Terminology , 1995, Methods of Information in Medicine.

[24]  Kent A. Spackman,et al.  SNOMED RT: a reference terminology for health care , 1997, AMIA.

[25]  Colin Price,et al.  Semantic quality through semantic definition: refining the Read Codes through internal consistency , 1997, AMIA.

[26]  Alan L. Rector,et al.  Feasibility and Reliability of Computerised Review of Long Term Prescribing , 2002 .

[27]  Mike Uschold,et al.  Building Ontologies: Towards a Unified Methodology , 1996 .

[28]  Michael Gruninger,et al.  Methodology for the Design and Evaluation of Ontologies , 1995, IJCAI 1995.

[29]  Barry Smith,et al.  Biodynamic ontology: applying BFO in the biomedical domain. , 2004, Studies in health technology and informatics.

[30]  Kent A. Spackman,et al.  A "lexically-suggested logical closure" metric for medical terminology maturity , 1998, AMIA.

[31]  Werner Ceusters,et al.  Ontological Theory for Ontological Engineering: Biomedical Systems Information Integration , 2004, KR.

[32]  J. Cimino.  Desiderata for Controlled Medical Vocabularies in the Twenty-First Century , 1998, Methods of Information in Medicine.

[33]  Shari Lawrence Pfleeger,et al.  Software Quality: The Elusive Target , 1996, IEEE Softw..

[34]  Werner Ceusters,et al.  Some Ontology Engineering Processes and Their Supporting Technologies , 2002, EKAW.

[35]  Philip J. B. Brown,et al.  Does size matter?-Evaluation of value added content of two decades of successive coding schemes in secondary care , 2001, AMIA.

[36]  Yorick Wilks,et al.  Data Driven Ontology Evaluation , 2004, LREC.

[37]  Nicola Guarino,et al.  Evaluating ontological decisions with OntoClean , 2002, CACM.

[38]  Werner Ceusters,et al.  Ontology-Based Error Detection in SNOMED-CT® , 2004, MedInfo.

[39]  Steffen Staab,et al.  Towards a Quantitative, Platform-Independent Analysis of Knowledge Systems , 2004, KR.

[40]  Angela Swallow,et al.  Automated quality checks on repeat prescribing. , 2003, The British journal of general practice : the journal of the Royal College of General Practitioners.

[41]  Chris Welty,et al.  Evaluating Ontological Analysis , 2003 .

[42]  C. Chute,et al.  The content coverage of clinical classifications. For The Computer-Based Patient Record Institute's Work Group on Codes & Structures. , 1996, Journal of the American Medical Informatics Association : JAMIA.

[43]  York Sure-Vetter,et al.  A survey on methodologies for developing, maintaining, integrating, evaluating and reengineering ontologies , 2002 .

[44]  Christopher G. Chute,et al.  The Content Coverage of Clinical Classifications , 1996 .

[45]  Asunción Gómez-Pérez,et al.  The Integration of OntoClean in WebODE , 2002, EON.

[46]  Kent A. Spackman,et al.  Normal forms for description logic expressions of clinical concepts in SNOMED RT , 2001, AMIA.

[47]  Alan L. Rector,et al.  GALEN Ten Years On: Tasks and Supporting Tools , 2001, MedInfo.

[48]  Asunción Gómez-Pérez,et al.  Criteria to Verify Knowledge Sharing Technology , 1995 .

[49]  Werner Ceusters,et al.  Ontological Theory for Ontological Engineering , 2004, KR 2004.

[50]  Mariana Casella dos Santos,et al.  Philosophical Scrutiny for Run-Time Support of Application Ontology Development , 2004 .

[51]  Michael Bada,et al.  GOAT: The Gene Ontology Annotation Tool , 2003 .

[52]  Alexander Borgida,et al.  On the Relative Expressiveness of Description Logics and Predicate Logics , 1996, Artif. Intell..