Validation of a knowledge based reminder system for diagnostic test ordering in general practice

We describe the validation of a real-time automated reminder system that assists General Practitioners (GP) in appropriate test ordering. We compared the comments of human experts with the comments of the reminder system using a retrospective random selection of 253 request forms. A panel of three expert physicians judged the requested tests independently based on their interpretations of the practice guidelines. The majority assessment of the physicians was compared with the assessment of the reminder system. In case the system's output differed from the majority assessment the written practice guidelines were consulted. On average 1.75 reminders were produced per form. In total 32 of the 442 given reminders (7%) were given incorrectly. The amount of information and the level of detail (the specificity of the terms) in which the GP describes the patients' medical status are crucial for the reminder system to react correctly.

[1]  J. Marc Overhage,et al.  Case Report: Computerizing Guidelines to Improve Care and Patient Outcomes: The Example of Heart Failure , 1995, J. Am. Medical Informatics Assoc..

[2]  J A Knottnerus,et al.  Effects of routine individual feedback over nine years on general practitioners' requests for tests , 1996, BMJ.

[3]  M Peters,et al.  Effects of a computerised protocol management system on ordering of clinical tests. , 1994, Quality in health care : QHC.

[4]  William DuMouchel,et al.  A meta-analysis of 16 randomized controlled trials to evaluate computer-based clinical reminder systems for preventive care in the ambulatory setting. , 1996, Journal of the American Medical Informatics Association : JAMIA.

[5]  P Pop,et al.  A diagnostic centre for general practitioners: results of individual feedback on diagnostic actions. , 1989, The Journal of the Royal College of General Practitioners.

[6]  Richard N. Shiffman,et al.  Model Formulation: Representation of Clinical Practice Guidelines in Conventional and Augmented Decision Tables , 1997, J. Am. Medical Informatics Assoc..

[7]  B. Bentsen.  International classification of primary care. , 1986, Scandinavian journal of primary health care.

[8]  Arie Hasman,et al.  A test ordering system with automated reminders for primary care based on practice guidelines , 2000, Int. J. Medical Informatics.

[9]  R. Goldman,et al.  Improving peer review: alternatives to unstructured judgments by a single reviewer. , 1996, The Joint Commission journal on quality improvement.

[10]  J P Kassirer,et al.  Our stubborn quest for diagnostic certainty. A cause of excessive testing. , 1989, The New England journal of medicine.

[11]  M. Fieschi.  Towards validation of expert systems as medical decision aids. , 1990, International journal of bio-medical computing.

[12]  Beswick Kb,et al.  Decision support in the United Kingdom for general practice: past, present, and future. , 1995 .

[13]  J A Blom.  Temporal logics and real time expert systems. , 1996, Computer methods and programs in biomedicine.

[14]  Joyce A. Mitchell,et al.  The clinical value of computerized information services. A review of 98 randomized clinical trials. , 1996, Archives of family medicine.

[15]  R L Kane,et al.  Peer review of the quality of care. Reliability and sources of variability for outcome and process assessments. , 1997, JAMA.

[16]  Jonathan M. Teich,et al.  Using information systems to measure and improve quality , 1999, Int. J. Medical Informatics.

[17]  M M Ohayon,et al.  Validation of expert systems: examples and considerations. , 1995, Medinfo. MEDINFO.

[18]  D. K. Donker.  Assessment and evaluation of information technologies in medicine , 1996 .

[19]  J Studnicki,et al.  A feedback system for reducing excessive laboratory tests. , 1993, Archives of pathology & laboratory medicine.

[20]  Julie M. Fiskio,et al.  Automated evidence-based critiquing of orders for abdominal radiographs: impact on utilization and appropriateness. , 1997, Journal of the American Medical Informatics Association : JAMIA.

[21]  R. Jorde,et al.  Improvement in clinical work through feedback: intervention study , 1999, BMJ.

[22]  R Tomlins,et al.  Randomised controlled trial of effect of feedback on general practitioners' prescribing in Australia , 1999, BMJ.

[23]  J van der Lei,et al.  Design of a Decision Support System for Test Ordering in General Practice: Choices and Decisions to Make , 1999, Methods of Information in Medicine.

[24]  M. Mugford,et al.  Effects of feedback of information on clinical practice: a review. , 1991, BMJ.

[25]  P. Pop,et al.  Randomised controlled trial of routine individual feedback to improve rationality and reduce numbers of test requests , 1995, The Lancet.

[26]  R. Goldman,et al.  The reliability of peer assessments of quality of care. , 1992, JAMA.

[27]  Jonathan M. Teich,et al.  Reminders for Redundant Tests: Results of a Randomized Controlled Trial , 1995 .

[28]  F. Fowkes,et al.  Audit Reviewed: Does Feedback on Performance Change Clinical Behaviour? , 1985, Journal of the Royal College of Physicians of London.

[29]  C J McDonald,et al.  Computer predictions of abnormal test results. Effects on outpatient testing. , 1988, JAMA.

[30]  E. M. S. J. Van Gennip,et al.  Assessment and evaluation of information technogies in medicine , 1995 .

[31]  C. McDonald,et al.  Reminders to physicians from an introspective computer medical record. A two-year randomized trial. , 1984, Annals of internal medicine.

[32]  R. Haynes,et al.  Effects of Computer-based Clinical Decision Support Systems on Clinician Performance and Patient Outcome: A Critical Appraisal of Research , 1994, Annals of Internal Medicine.

[33]  J A Knottnerus,et al.  Does feedback improve the quality of cervical smears? A randomized controlled trial. , 1993, The British journal of general practice : the journal of the Royal College of General Practitioners.