Effects of computerized clinical decision support systems on practitioner performance and patient outcomes: a systematic review.

CONTEXT Developers of health care software have attributed improvements in patient care to these applications. As with any health care intervention, such claims require confirmation in clinical trials. OBJECTIVES To review controlled trials assessing the effects of computerized clinical decision support systems (CDSSs) and to identify study characteristics predicting benefit. DATA SOURCES We updated our earlier reviews by searching the MEDLINE, EMBASE, Cochrane Library, Inspec, and ISI databases and consulting reference lists through September 2004. Authors of 64 primary studies confirmed data or provided additional information. STUDY SELECTION We included randomized and nonrandomized controlled trials that evaluated the effect of a CDSS compared with care provided without a CDSS on practitioner performance or patient outcomes. DATA EXTRACTION Teams of 2 reviewers independently abstracted data on methods, setting, CDSS and patient characteristics, and outcomes. DATA SYNTHESIS One hundred studies met our inclusion criteria. The number and methodologic quality of studies improved over time. The CDSS improved practitioner performance in 62 (64%) of the 97 studies assessing this outcome, including 4 (40%) of 10 diagnostic systems, 16 (76%) of 21 reminder systems, 23 (62%) of 37 disease management systems, and 19 (66%) of 29 drug-dosing or prescribing systems. Fifty-two trials assessed 1 or more patient outcomes, of which 7 trials (13%) reported improvements. Improved practitioner performance was associated with CDSSs that automatically prompted users compared with requiring users to activate the system (success in 73% of trials vs 47%; P = .02) and studies in which the authors also developed the CDSS software compared with studies in which the authors were not the developers (74% success vs 28%; respectively, P = .001). CONCLUSIONS Many CDSSs improve practitioner performance. To date, the effects on patient outcomes remain understudied and, when studied, inconsistent.

[1]  Jacob Cohen,et al.  The Equivalence of Weighted Kappa and the Intraclass Correlation Coefficient as Measures of Reliability , 1973 .

[2]  L. Sheiner,et al.  Computer-assisted digoxin therapy. , 1973, The New England journal of medicine.

[3]  F. Oski,et al.  Impact of a system of computer-assisted diagnosis. Initial evaluation of the hospitalized patient. , 1975, American journal of diseases of children.

[4]  C. Mc Donald,et al.  Use of a computer to detect and respond to clinical events: its effect on clinician behavior. , 1976, Annals of internal medicine.

[5]  C. McDonald.  Protocol-based computer reminders, the quality of care and the non-perfectability of man. , 1976, The New England journal of medicine.

[6]  S. Oparil,et al.  Treatment of hypertension by computer and physician-a prospective controlled study. , 1977, Journal of chronic diseases.

[7]  J L Rogers,et al.  The Impact of a Computerized Medical Record Summary System on Incidence and Length of Hospitalization , 1979, Medical care.

[8]  C J McDonald,et al.  Physician response to computer reminders. , 1980, JAMA.

[9]  D W Young.  Improving the consistency with which investigations are requested. , 1981, Medical informatics = Medecine et informatique.

[10]  P M Wortman,et al.  Medical Information Systems: Assessing Impact in the Areas of Hypertension, Obesity and Renal Disease , 1982, Medical care.

[11]  T J O'Leary,et al.  Evaluation of a computer‐assisted method for individualized anticoagulation: Retrospective and prospective studies with a pharmacodynamic model , 1982, Clinical pharmacology and therapeutics.

[12]  G. Barnett,et al.  A Computer-Based Monitoring System for Follow-Up of Elevated Blood Pressure , 1983, Medical care.

[13]  R W Jelliffe,et al.  Clinical studies with computer-assisted initial lidocaine therapy. , 1984, Archives of internal medicine.

[14]  C. McDonald,et al.  Reminders to physicians from an introspective computer medical record. A two-year randomized trial. , 1984, Annals of internal medicine.

[15]  O. Haring,et al.  Changes in patient attitudes following the implementation of a medical information system. , 1984, QRB. Quality review bulletin.

[16]  R B D'Agostino,et al.  A predictive instrument to improve coronary-care-unit admission practices in acute ischemic heart disease. A prospective multicenter clinical trial. , 1984, The New England journal of medicine.

[17]  T A Pryor,et al.  Application of a computerized medical decision-making process to the problem of digoxin intoxication. , 1984, Journal of the American College of Cardiology.

[18]  J. McNeil,et al.  A randomized controlled clinical trial of pharmacokinetic theophylline dosing. , 1986, The American review of respiratory disease.

[19]  C J McDonald,et al.  Delayed Feedback of Physician Performance Versus Immediate Reminders to Perform Preventive Care: Effects on Physician Compliance , 1986, Medical care.

[20]  H. Covvey,et al.  Randomised controlled trial of computer assisted management of hypertension in primary care. , 1986, British medical journal.

[21]  G Brownbridge,et al.  An interactive computerized protocol for the management of hypertension: effects on the general practitioner's clinical behaviour. , 1986, The Journal of the Royal College of General Practitioners.

[22]  B. Carter,et al.  Evaluation of three dosage-prediction methods for initial in-hospital stabilization of warfarin therapy. , 1987, Clinical pharmacy.

[23]  R. Haynes,et al.  Computer-aided quality assurance. A critical appraisal. , 1987, Archives of internal medicine.

[24]  M. Daschbach,et al.  Initiation of warfarin therapy: comparison of physician dosing with computer-assisted dosing. , 1987, Journal of general internal medicine.

[25]  C J McDonald,et al.  Computer predictions of abnormal test results. Effects on outpatient testing. , 1988, JAMA.

[26]  E. Begg,et al.  Individualised aminoglycoside dosage based on pharmacokinetic analysis is superior to dosage based on physician intuition at achieving target plasma drug concentrations. , 1989, British journal of clinical pharmacology.

[27]  S. Mcphee,et al.  Promoting cancer screening. A randomized, controlled trial of three interventions. , 1989 .

[28]  C. Chambers,et al.  Microcomputer-generated reminders. Improving the compliance of primary care physicians with mammography screening guidelines. , 1989, The Journal of family practice.

[29]  J P Ornato,et al.  Computer-assisted optimization of aminophylline therapy in the emergency department. , 1989, The American journal of emergency medicine.

[30]  D. Spiegelhalter,et al.  Evaluating medical expert systems: what to test and how? , 1990, Medical informatics = Medecine et informatique.

[31]  William M. Tierney,et al.  Postgraduate Medical Education , 2008 .

[32]  K Canfield,et al.  Evaluation of UNIS: Urological Nursing Information Systems. , 1991, Proceedings. Symposium on Computer Applications in Medical Care.

[33]  P. Rust,et al.  Computer-generated physician and patient reminders. Tools to improve population adherence to selected preventive services. , 1991, The Journal of family practice.

[34]  J E Rodnick,et al.  Promoting cancer prevention activities by primary care physicians. Results of a randomized, controlled trial. , 1991, JAMA.

[35]  W. Rosser,et al.  Use of reminders for preventive procedures in family medicine. , 1991, CMAJ : Canadian Medical Association journal = journal de l'Association medicale canadienne.

[36]  M E Burton,et al.  A controlled trial of the cost benefit of computerized bayesian aminoglycoside administration , 1991, Clinical pharmacology and therapeutics.

[37]  D. Mungall,et al.  Outpatient management of warfarin therapy: comparison of computer-predicted dosage adjustment to skilled professional care. , 1991, Therapeutic drug monitoring.

[38]  D J Balaban,et al.  The Effect Of Microcomputer-Generated Reminders On Influenza Vaccination Rates In A University-Based Family Practice Center , 1991, The Journal of the American Board of Family Medicine.

[39]  C J McDonald,et al.  Effects of computer reminders for influenza vaccination on morbidity during influenza epidemics. , 1992, M.D. computing : computers in medical practice.

[40]  D J Spiegelhalter,et al.  How does computer-aided diagnosis improve the management of acute abdominal pain? , 1992, Annals of the Royal College of Surgeons of England.

[41]  W Berger,et al.  Clinical application of two computerized diabetes management systems: comparison with the log-book method. , 1992, Diabetes research.

[42]  L. Poller,et al.  Prospective comparative study of computer programs used for management of warfarin. , 1993, Journal of clinical pathology.

[43]  T G Tape,et al.  Computerized medical records and preventive health care: success depends on many factors. , 1993, The American journal of medicine.

[44]  C. McDonald,et al.  Physician inpatient order writing on microcomputer workstations. Effects on resource utilization. , 1993, JAMA.

[45]  P. Casner,et al.  A randomized controlled trial of computerized pharmacokinetic theophylline dosing versus empiric physician dosing , 1993, Clinical pharmacology and therapeutics.

[46]  R. C. Turner,et al.  Patient-carried card prompts vs computer-generated prompts to remind private practice physicians to perform health maintenance measures. , 1994, Archives of internal medicine.

[47]  P. Gimotty,et al.  Promoting Screening Mammography in Inner-City Settings: A Randomized Controlled Trial of Computerized Reminders as a Component of a Program to Facilitate Mammography , 1994, Medical care.

[48]  R. Haynes,et al.  Effects of Computer-based Clinical Decision Support Systems on Clinician Performance and Patient Outcome: A Critical Appraisal of Research , 1994, Annals of Internal Medicine.

[49]  W J Hall,et al.  Computer-based vs manual health maintenance tracking. A controlled trial. , 1994, Archives of family medicine.

[50]  D. Mungall,et al.  A prospective randomized comparison of the accuracy of computer‐assisted versus GUSTO nomogram‐directed heparin therapy , 1994, Clinical pharmacology and therapeutics.

[51]  C Safran,et al.  Guidelines for management of HIV infection with computer-based patient's record , 1995, The Lancet.

[52]  M J Lincoln,et al.  A computer-generated reminder system improves physician compliance with diabetes preventive care guidelines. , 1995, Proceedings. Symposium on Computer Applications in Medical Care.

[53]  P D Clayton,et al.  Computer-generated informational messages directed to physicians: effect on length of hospital stay. , 1995, Journal of the American Medical Informatics Association : JAMIA.

[54]  R M Gardner,et al.  Factors impacting the success of computerized preadmission screening. , 1995, Proceedings. Symposium on Computer Applications in Medical Care.

[55]  C P Bradley,et al.  Evaluation of computerized decision support for oral anticoagulation management based in primary care. , 1996, The British journal of general practice : the journal of the Royal College of General Practitioners.

[56]  Peter Dews,et al.  The effect of patient and physician reminders on use of screening mammography in a health maintenance organization: Results of a randomized controlled trial , 1996, Cancer.

[57]  C J McDonald,et al.  Computer reminders to implement preventive care guidelines for hospitalized patients. , 1996, Archives of internal medicine.

[58]  Charles Young,et al.  A randomized controlled trial of a computer-based physician workstation in an outpatient setting: implementation barriers to outcome evaluation. , 1996, Journal of the American Medical Informatics Association : JAMIA.

[59]  Y Reisman,et al.  Computer-based clinical decision aids. A review of methods and assessment of systems. , 1996, Medical informatics = Medecine et informatique.

[60]  G. Lewis,et al.  Computerized assessment of common mental disorders in primary care: effect on clinical outcome. , 1996, Family practice.

[61]  J. Marc Overhage,et al.  Research Paper: A Randomized Trial of "Corollary Orders" to Prevent Errors of Omission , 1997, J. Am. Medical Informatics Assoc..

[62]  M. Leaning,et al.  Comparison of oral anticoagulant control by a nurse-practitioner using a computer decision-support system with that by clinicians. , 1997, Clinical and laboratory haematology.

[63]  P. Gimotty,et al.  Promoting screening mammography in inner-city settings. The sustained effectiveness of computerized reminders in a randomized controlled trial. , 1997, Medical care.

[64]  W. Hammond,et al.  Computerized decision support based on a clinical practice guideline improves compliance with care standards. , 1997, The American journal of medicine.

[65]  M. Leaning,et al.  Evaluation of a decision support system for initiation and control of oral anticoagulation in a randomised trial , 1997, BMJ.

[66]  M. Abrahamowicz,et al.  Can computerized risk profiles help patients improve their coronary risk? The results of the Coronary Health Assessment Study (CHAS). , 1998, Preventive medicine.

[67]  W. Ageno,et al.  A randomized comparison of a computer-based dosing program with a manual system to monitor oral anticoagulant therapy. , 1998, Thrombosis research.

[68]  J. Jespersen,et al.  Multicentre randomised study of computerised anticoagulant dosage , 1998, The Lancet.

[69]  J. Wyatt,et al.  On‐screen computer reminders: effects on professional practice and health care outcomes , 1998 .

[70]  R. Haynes,et al.  Effects of computer-based clinical decision support systems on physician performance and patient outcomes: a systematic review. , 1998, JAMA.

[71]  J. Holmen,et al.  Implementing clinical guidelines in the treatment of hypertension in general practice. , 1998, Blood pressure.

[72]  J P Ornato,et al.  Use of the Acute Cardiac Ischemia Time-Insensitive Predictive Instrument (ACI-TIPI) To Assist with Triage of Patients with Chest Pain or Other Symptoms Suggestive of Acute Cardiac Ischemia: A Multicenter, Controlled Clinical Trial , 1998, Annals of Internal Medicine.

[73]  W M Tierney,et al.  Effectiveness of Computer-Generated Reminders for Increasing Discussions about Advance Directives and Completion of Advance Directive Forms , 1998, Annals of Internal Medicine.

[74]  R W Sanson-Fisher,et al.  Randomized controlled trial of a computer strategy to increase general practitioner preventive care. , 1999, Preventive medicine.

[75]  Alan H. Morris,et al.  Efficacy of computerized decision support for mechanical ventilation: results of a prospective multi-center randomized trial , 1999, AMIA.

[76]  Susan M. Gorden,et al.  Research Paper: Measuring the Effects of Reminders for Outpatient Influenza Immunizations at the Point of Clinical Opportunity , 1999, J. Am. Medical Informatics Assoc..

[77]  Richard N. Shiffman,et al.  Review: Computer-based Guideline Implementation Systems: A Systematic Review of Functionality and Effectiveness , 1999, J. Am. Medical Informatics Assoc..

[78]  D. Bates,et al.  Improving response to critical laboratory results with automation: results of a randomized controlled trial. , 1999, Journal of the American Medical Informatics Association : JAMIA.

[79]  Bradley N. Doebbeling,et al.  Randomized study of online vaccine reminders in adult primary care , 1999, AMIA.

[80]  G J Kuperman,et al.  A randomized trial of a computer-based intervention to reduce utilization of redundant laboratory tests. , 1999, The American journal of medicine.

[81]  J. Holmen,et al.  Implementing clinical guidelines in the treatment of hypertension in general practice. Evaluation of patient outcome related to implementation of a computer-based clinical decision support system. , 1999, Scandinavian journal of primary health care.

[82]  W G Henderson,et al.  Improving residents' compliance with standards of ambulatory care: results from the VA Cooperative Study on Computerized Reminders. , 2000, JAMA.

[83]  Tom Fahey,et al.  Evaluation of computer based clinical decision support system and risk chart for management of hypertension in primary care: randomised controlled trial , 2000, BMJ : British Medical Journal.

[84]  F D Hobbs,et al.  Oral anticoagulation management in primary care with the use of computerized decision support and near-patient testing: a randomized, controlled trial. , 2000, Archives of internal medicine.

[85]  T. Payne.  Computer decision support systems. , 2000, Chest.

[86]  D. Cannon,et al.  A comparison of the effects of computer and manual reminders on compliance with a mental health clinical practice guideline. , 2000, Journal of the American Medical Informatics Association : JAMIA.

[87]  J. Holmen,et al.  IMPLEMENTING CLINICAL GUIDELINES IN THE TREATMENT OF DIABETES MELLITUS IN GENERAL PRACTICE , 2000, International Journal of Technology Assessment in Health Care.

[88]  P S Gibbons,et al.  Enabling the diagnosis of occult psychiatric illness in the emergency department: a randomized, controlled trial of the computerized, self-administered PRIME-MD diagnostic system. , 2001, Annals of emergency medicine.

[89]  M. Moia,et al.  Effect of computer-aided management on the quality of treatment in anticoagulated patients: a prospective, randomized, multicenter trial of APROAT (Automated PRogram for Oral Anticoagulant Treatment). , 2001, Haematologica.

[90]  Bonnie Kaplan,et al.  Clinical decision support systems for the practice of evidence-based medicine. , 2001, Journal of the American Medical Informatics Association : JAMIA.

[91]  C. McDonald,et al.  A computerized reminder system to increase the use of preventive care for hospitalized patients. , 2001, The New England journal of medicine.

[92]  R G Neville,et al.  Lessons from a randomized controlled trial designed to evaluate computer decision support software to improve the management of asthma , 2001, Medical informatics and the Internet in medicine.

[93]  Bonnie Kaplan,et al.  Evaluating informatics applications - clinical decision support systems literature review , 2001, Int. J. Medical Informatics.

[94]  F. Sullivan,et al.  A descriptive feast but an evaluative famine: systematic review of published articles on primary care computing during 1980-97 , 2001, BMJ : British Medical Journal.

[95]  T D East,et al.  Computerized decision support for mechanical ventilation of trauma induced ARDS: results of a randomized clinical trial. , 2001, The Journal of trauma.

[96]  F P Rivara,et al.  A randomized controlled trial of point-of-care evidence to improve the antibiotic prescribing practices for otitis media in children. , 2001, Pediatrics.

[97]  Margaret Maglione,et al.  Interventions That Increase Use of Adult Immunization and Cancer Screening Services , 2002, Annals of Internal Medicine.

[98]  John Fox,et al.  Clinical decision support systems: a discussion of quality, safety and legal liability issues , 2003, AMIA.

[99]  Victor M Montori,et al.  The impact of planned care and a diabetes electronic management system on community-based diabetes care: the Mayo Health System Diabetes Translation Project. , 2002, Diabetes care.

[100]  A Bogusevicius,et al.  Prospective randomised trial of computer-aided diagnosis and contrast radiography in acute small bowel obstruction. , 2002, The European journal of surgery = Acta chirurgica.

[101]  J. Parinaud,et al.  A computerized decision support system for ovarian stimulation by gonadotropins. , 2002, Fertility and sterility.

[102]  Silvia Miksch,et al.  Development and evaluation of VIE-PNN, a knowledge-based system for calculating the parenteral nutrition of newborn infants , 2002, Artif. Intell. Medicine.

[103]  George Hripcsak,et al.  Review Paper: Design and Analysis of Controlled Trials in Naturally Clustered Environments: Implications for Medical Informatics , 2002, J. Am. Medical Informatics Assoc..

[104]  David Parkin,et al.  Effect of computerised evidence based guidelines on management of asthma and angina in adults in primary care: cluster randomised controlled trial , 2002, BMJ : British Medical Journal.

[105]  J. Griffith,et al.  Use of the Electrocardiograph-Based Thrombolytic Predictive Instrument To Assist Thrombolytic and Reperfusion Therapy for Acute Myocardial Infarction: A Multicenter, Randomized, Controlled, Clinical Effectiveness Trial , 2002, Annals of Internal Medicine.

[106]  Paul A. Heidenreich,et al.  Improving Guideline Adherence: A Randomized Trial Evaluating Strategies to Increase &bgr;-Blocker Use in Heart Failure , 2003, Circulation.

[107]  Alessandro Filippi,et al.  Effects of an automated electronic reminder in changing the antiplatelet drug-prescribing behavior among Italian general practitioners in diabetic patients: an intervention trial. , 2003, Diabetes care.

[108]  D. Bates,et al.  Effects of computerized physician order entry and clinical decision support systems on medication safety: a systematic review. , 2003, Archives of internal medicine.

[109]  P. Gimotty,et al.  The effect of adding Pap smear information to a mammography reminder system in an HMO: results of randomized controlled trial. , 2003, Preventive medicine.

[110]  R. Macwalter,et al.  Cluster-randomized, controlled trial of computer-based decision support for selecting long-term anti-thrombotic therapy after acute ischaemic stroke. , 2003, QJM : monthly journal of the Association of Physicians.

[111]  Anne Holbrook,et al.  What Factors Determine The Success of Clinical Decision Support Systems? , 2003, AMIA.

[112]  J. Hanley,et al.  The medical office of the 21st century (MOXXI): effectiveness of computerized decision-making support in reducing inappropriate prescribing in primary care. , 2003, CMAJ : Canadian Medical Association journal = journal de l'Association medicale canadienne.

[113]  David M Nathan,et al.  A controlled trial of web-based diabetes disease management: the MGH diabetes primary care improvement project. , 2003, Diabetes care.

[114]  Kensaku Kawamoto,et al.  Clinical Decision Support Provided within Physician Order Entry Systems: A Systematic Review of Features Effective for Changing Clinician Behavior , 2003, AMIA.

[115]  Daniel H Solomon,et al.  Self-management education programs in chronic disease: a systematic review and methodological critique of the literature. , 2004, Archives of internal medicine.

[116]  V. Hasselblad,et al.  Does disease management improve clinical and economic outcomes in patients with chronic diseases? A systematic review. , 2004, The American journal of medicine.

[117]  J M Grimshaw,et al.  Effectiveness and efficiency of guideline dissemination and implementation strategies , 2004, International Journal of Technology Assessment in Health Care.

[118]  C. McDonald,et al.  Failure of Computerized Treatment Suggestions to Improve Health Outcomes of Outpatients with Uncomplicated Hypertension: Results of a Randomized Controlled Trial , 2004, Pharmacotherapy.

[119]  Ida Sim,et al.  Characteristics of Outpatient Clinical Decision Support Systems: A Taxonomic Description , 2004, MedInfo.

[120]  James C. Thomas,et al.  The effect on cost of medical care for patients treated with an automated clinical audit system , 1983, Journal of Medical Systems.

[121]  E. Begg,et al.  A prospective randomised trial comparing individualised pharmacokinetic dosage prediction for aminoglycosides with prediction based on estimated creatinine clearance in critically ill patients , 2004, Intensive Care Medicine.

[122]  H. Halkin,et al.  Computer assisted design of a theophylline dosing regimen in acute bronchospasm: Serum concentrations and clinical outcome , 2005, European Journal of Clinical Pharmacology.

[123]  J. Stockman.  Effects of Computerized Clinical Decision Support Systems on Practitioner Performance and Patient Outcomes: A Systematic Review , 2006 .

[124]  MPH Ralph A. Rossi MD,et al.  A computerized intervention to decrease the use of calcium channel blockers in hypertension , 2007, Journal of General Internal Medicine.

[125]  Henry J. Lowe,et al.  A randomized trial using computerized decision support to improve treatment of major depression in primary care , 2002, Journal of General Internal Medicine.

[126]  C. McDonald,et al.  Effects of computerized guidelines for managing heart disease in primary care , 2003, Journal of General Internal Medicine.

[127]  B. Turner,et al.  A controlled trial to improve delivery of preventive care , 1989, Journal of General Internal Medicine.

[128]  Stephan D. Fihn,et al.  A computerized intervention to improve timing of outpatient follow-up , 1994, Journal of General Internal Medicine.

[129]  J. M. McCoy,et al.  Improving patient quality of life with feedback to physicians about functional status , 1995, Journal of General Internal Medicine.