A quantitative assessment of a methodology for collaborative specification and evaluation of clinical guidelines

We introduce a three-phase, nine-step methodology for specification of clinical guidelines (GLs) by expert physicians, clinical editors, and knowledge engineers and for quantitative evaluation of the specification's quality. We applied this methodology to a particular framework for incremental GL structuring (mark-up) and to GLs in three clinical domains. A gold-standard mark-up was created, including 196 plans and subplans, and 326 instances of ontological knowledge roles (KRs). A completeness measure of the acquired knowledge revealed that 97% of the plans and 91% of the KR instances of the GLs were recreated by the clinical editors. A correctness measure often revealed high variability within clinical editor pairs structuring each GL, but for all GLs and clinical editors the specification quality was significantly higher than random (p<0.01). Procedural KRs were more difficult to mark-up than declarative KRs. We conclude that given an ontology-specific consensus, clinical editors with mark-up training can structure GL knowledge with high completeness, whereas the main demand for correct structuring is training in the ontology's semantics.

[1]  Yuval Shahar,et al.  Synthesis of Research: EON: A Component-Based Approach to Automation of Protocol-Directed Therapy , 1996, J. Am. Medical Informatics Assoc..

[2]  Neill Jones,et al.  Achieving Reuse of Computable Guideline Systems , 2001, MedInfo.

[3]  Richard N. Shiffman,et al.  An implementation framework for GEM encoded guidelines , 2001, AMIA.

[4]  Giordano Lanzola,et al.  Flexible guideline-based patient careflow systems , 2001, Artif. Intell. Medicine.

[5]  Raymond H. Myers,et al.  Probability and Statistics for Engineers and Scientists. , 1973 .

[6]  Yuval Shahar,et al.  Runtime application of Hybrid-Asbru clinical guidelines , 2007, J. Biomed. Informatics.

[7]  Omolola Ogunyemi,et al.  GLIF3: a representation format for sharable computer-interpretable clinical practice guidelines , 2004, J. Biomed. Informatics.

[8]  Vimla L. Patel,et al.  Research Paper: Representing Clinical Guidelines in GLIF: Individual and Collaborative Expertise , 1998, J. Am. Medical Informatics Assoc..

[9]  Silvia Miksch,et al.  Bridging the Gap Between Informal and Formal Guideline Representations , 2006, ECAI.

[10]  Aziz A. Boxwala,et al.  Representation primitives, process models and patient data in computer-interpretable clinical practice guidelines: : A literature review of guideline representation models , 2002, Int. J. Medical Informatics.

[11]  John Fox,et al.  Application of Information Technology: The Syntax and Semantics of the PROforma Guideline Modeling Language , 2003, J. Am. Medical Informatics Assoc..

[12]  Alessio Bottrighi,et al.  Supporting physicians in taking decisions in clinical guidelines: the GLARE "what if" facility , 2002, AMIA.

[13]  Vojtech Svátek,et al.  Mark-up based analysis of narrative guidelines with the Stepper tool , 2004, CGP.

[14]  L. Ohno-Machado.  Journal of Biomedical Informatics , 2001 .

[15]  J. Grimshaw,et al.  Is evidence‐based implementation of evidence‐based care possible? , 2004, The Medical journal of Australia.

[16]  John Fox,et al.  Comparing computer-interpretable guideline models: a case-study approach. , 2003, Journal of the American Medical Informatics Association : JAMIA.

[17]  Yuval Shahar,et al.  AsbruView: Visualization of Time-Oriented, Skeletal Plans , 1998, AIPS.

[18]  G. Burstein,et al.  Sexually transmitted diseases treatment guidelines , 2003, Current opinion in pediatrics.

[19]  Yuval Shahar,et al.  A Graphical Framework for Specification of Clinical Guidelines at Multiple Representation Levels , 2005, AMIA.

[20]  W. W. Daniel.  Applied Nonparametric Statistics , 1979 .

[21]  J M Grimshaw,et al.  Effect of clinical guidelines on medical practice: a systematic review of rigorous evaluations. , 1994, Lancet.

[22]  Aziz A. Boxwala,et al.  Analysis of the Process of Encoding Guidelines: An Evaluation of GLIF3 , 2002 .

[23]  Arie Hasman,et al.  Approaches for creating computer-interpretable guidelines that facilitate decision support , 2004, Artif. Intell. Medicine.

[24]  Yuval Shahar,et al.  A framework for a distributed, hybrid, multiple-ontology clinical-guideline library, and automated guideline-support tools , 2004, J. Biomed. Informatics.

[25]  Vojtech Svátek,et al.  Step-by-step mark-up of medical guideline documents , 2003, Int. J. Medical Informatics.

[26]  Silvia Miksch,et al.  Facilitating Knowledge Maintenance of Clinical Guidelines and Protocols , 2004, MedInfo.

[27]  Robert A. Greenes,et al.  Research Paper: The GuideLine Interchange Format: A Model for Representing Guidelines , 1998, J. Am. Medical Informatics Assoc..

[28]  Natalya F. Noy,et al.  Knowledge-Acquisition Interfaces for Domain Experts: An Empirical Evaluation of Protégé-2000 , 2000 .

[29]  Richard N. Shiffman,et al.  A preliminary evaluation of guideline content mark-up using GEM-an XML guideline elements model , 2000, AMIA.

[30]  Arie Hasman,et al.  Experiences with the Development, Implementation and Evaluation of Automated Decision Support Systems , 2004, MedInfo.

[31]  Yuval Shahar,et al.  The Importance of Creating an Ontology-Specific Consensus Before a Markup-Based Specification of Clinical Guidelines , 2006 .

[32]  Yuval Shahar,et al.  The Asgaard project: a task-specific framework for the application and critiquing of time-oriented clinical guidelines , 1998, Artif. Intell. Medicine.

[33]  Richard N. Shiffman,et al.  Model Formulation: GEM: A Proposal for a More Comprehensive Guideline Document Model Using XML , 2000, J. Am. Medical Informatics Assoc..

[34]  P Reeves,et al.  Telematics for clinical guidelines: a conceptual modelling approach. , 1997, Studies in health technology and informatics.

[35]  Paolo Terenziani,et al.  Suppor ting Physicians in Taking Decisions in Clinical Guidelines : the GLARE “ What if ” , 2022 .

[36]  Stanley M. Huff,et al.  Modeling Guidelines for Integration into Clinical Workflow , 2004, MedInfo.

[37]  Samson W. Tu,et al.  Protégé-2000: An Open-Source Ontology-Development and Knowledge-Acquisition Environment: AMIA 2003 Open Source Expo , 2003, AMIA.

[38]  S. Skonetzki,et al.  Evolution of the HELEN Representation for Managing Clinical Practice Guidelines , 2005 .

[39]  V. Patel,et al.  Analysis of the process of encoding guidelines: a comparison of GLIF2 and GLIF3. , 2002, Methods of information in medicine.

[40]  Casimir A. Kulikowski,et al.  HGML: a hypertext guideline markup language , 2000, AMIA.

[41]  Randolph A. Miller,et al.  The anatomy of decision support during inpatient care provider order entry (CPOE): Empirical observations from a decade of CPOE experience at Vanderbilt , 2005, J. Biomed. Informatics.

[42]  Yuval Shahar,et al.  Vaidurya - A Concept-Based, Context-Sensitive Search Engine For Clinical Guidelines , 2004, MedInfo.

[43]  Vimla L. Patel,et al.  Interpreting procedures from descriptive guidelines , 2006, J. Biomed. Informatics.

[44]  Nigel Shadbolt,et al.  The experimental evaluation of knowledge acquisition techniques and methods: history, problems and new directions , 1999, Int. J. Hum. Comput. Stud..

[45]  Richard Hellman,et al.  American Association of Clinical Endocrinologists medical guidelines for clinical practice for the evaluation and treatment of hyperthyroidism and hypothyroidism. , 2002, Endocrine practice : official journal of the American College of Endocrinology and the American Association of Clinical Endocrinologists.

[46]  J. B. Brooke,et al.  SUS: A 'Quick and Dirty' Usability Scale , 1996 .