Is medical informatics a mature science? A review of measurement practice in outcome studies of clinical systems

OBJECTIVE To determine the extent of explicit attention to formal issues of 'measurement' in studies examining outcomes of the deployment of clinical information systems. METHODS A structured literature review identified 27 published studies reporting quantitative outcomes including attitudes, clinician behavior, quality of care, and cost of care. These studies were analyzed for types of outcome reported, evidence of 'reuse' of measurement methodology, and evidence of formal study of the reliability and validity of these measures. RESULTS The 27 studies meeting the inclusion criteria were published between 1976 and 2002. Several of the studies addressed multiple outcome types. Nine examined clinician attitudes; 22 examined health care behaviors; 15 examined patient health status/quality of care; and four examined economic indicators. There were eight examples of reuse of measurement methods, five of which represented reuse within a single research group. Reliability indices were reported in three studies. There were no reported validity indices. CONCLUSION Based on this sample of studies, specific attention to issues of measurement is sparse in outcome studies of deployed clinical information systems. As such, medical informatics does not appear to be on a par with more mature sciences in its approaches to measurement of key outcome variables.

[1]  D. Bates,et al.  Effects of computerized physician order entry on prescribing practices. , 2000, Archives of internal medicine.

[2]  Kent L. Norman,et al.  Development of an instrument measuring user satisfaction of the human-computer interface , 1988, CHI '88.

[3]  Henry J. Lowe,et al.  The electronic medical record. A randomized trial of its impact on primary care physicians' initial management of major depression [corrected]. , 2001, Archives of internal medicine.

[4]  J H van Bemmel,et al.  Assessment of Decision Support for Blood Test Ordering in Primary Care , 2001, Annals of Internal Medicine.

[5]  R. Raschke,et al.  A computer alert system to prevent injury from adverse drug events: development and evaluation in a community teaching hospital. , 1998, JAMA.

[6]  D D Schlager.  A comprehensive patient care system for the family practice , 1983, Journal of medical systems.

[7]  C. McDonald,et al.  Physician inpatient order writing on microcomputer workstations. Effects on resource utilization. , 1993, JAMA.

[8]  Joseph L. Kannry,et al.  Research Paper: Physician Satisfaction with Two Order Entry Systems , 2001, J. Am. Medical Informatics Assoc..

[9]  D. Bates,et al.  Effect of computerized physician order entry and a team intervention on prevention of serious medication errors. , 1998, JAMA.

[10]  T G Tape,et al.  Computerized medical records and preventive health care: success depends on many factors. , 1993, The American journal of medicine.

[11]  R. Haux,et al.  A Randomized Evaluation of a Computer-Based Nursing Documentation System , 2001, Methods of Information in Medicine.

[12]  G E Kissling,et al.  Benefits of computerized outcome analysis in diabetes management. , 2000, Endocrine practice : official journal of the American College of Endocrinology and the American Association of Clinical Endocrinologists.

[13]  C. McDonald.  Protocol-based computer reminders, the quality of care and the non-perfectability of man. , 1976, The New England journal of medicine.

[14]  KAVEH G. SHOJANIA,et al.  Research Paper: Reducing Vancomycin Use Utilizing a Computer Guideline: Results of a Randomized Controlled Trial , 1998, J. Am. Medical Informatics Assoc..

[15]  Jonathan M. Teich,et al.  The impact of computerized physician order entry on medication error prevention. , 1999, Journal of the American Medical Informatics Association : JAMIA.

[16]  R. Haynes,et al.  Effects of Computer-Based Clinical Decision Support Systems on Physician Performance and Patient Outcomes , 1998 .

[17]  Charles P. Friedman,et al.  Development and initial validation of an instrument to measure physicians' use of, knowledge about, and attitudes toward computers. , 1998, Journal of the American Medical Informatics Association : JAMIA.

[18]  J. H. van Bemmel,et al.  Comparison of computer-aided and human review of general practitioners' management of hypertension , 1991, The Lancet.

[19]  M. Musen,et al.  Handbook of Medical Informatics , 2002 .

[20]  C. McDonald,et al.  A computerized reminder system to increase the use of preventive care for hospitalized patients. , 2001, The New England journal of medicine.

[21]  Charles P. Friedman,et al.  Evaluation Methods in Medical Informatics , 1997, Computers and Medicine.

[22]  Alan L. Rector,et al.  MEDICAL INFORMATICS , 1990, The Lancet.

[23]  R. Haynes,et al.  Effects of computer-based clinical decision support systems on physician performance and patient outcomes: a systematic review. , 1998, JAMA.

[24]  T. Clemmer,et al.  A computer-assisted management program for antibiotics and other antiinfective agents. , 1998, The New England journal of medicine.

[25]  Tom Fahey,et al.  Evaluation of computer based clinical decision support system and risk chart for management of hypertension in primary care: randomised controlled trial , 2000, BMJ : British Medical Journal.

[26]  J. Marc Overhage,et al.  Research Paper: Controlled Trial of Direct Physician Order Entry: Effects on Physicians' Time Utilization in Ambulatory Primary Care Internal Medicine Practices , 2001, J. Am. Medical Informatics Assoc..

[27]  M Sleutel,et al.  As good as it gets? Going online with a clinical information system. , 1999, Computers in nursing.

[28]  J. Holmen,et al.  IMPLEMENTING CLINICAL GUIDELINES IN THE TREATMENT OF DIABETES MELLITUS IN GENERAL PRACTICE , 2000, International Journal of Technology Assessment in Health Care.

[29]  Julie M. Fiskio,et al.  Research Paper: Improving Response to Critical Laboratory Results with Automation: Results of a Randomized Controlled Trial , 1999, J. Am. Medical Informatics Assoc..

[30]  Lawrence M. Fagan,et al.  Medical informatics: computer applications in health care and biomedicine (Health informatics) , 2003 .

[31]  Jonathan M. Teich,et al.  Improving allergy alerting in a computerized physician order entry system , 2000, AMIA.

[32]  David C Classen,et al.  Evaluation of a Computer-Assisted Antibiotic-Dose Monitor , 1999, The Annals of pharmacotherapy.

[33]  A Haines,et al.  Evaluation of General Practice Computer Templates , 1999, Methods of Information in Medicine.

[34]  C. McDonald,et al.  Reminders to physicians from an introspective computer medical record. A two-year randomized trial. , 1984, Annals of internal medicine.

[35]  V. Fuchs,et al.  The new technology assessment. , 1990, The New England journal of medicine.

[36]  C. McDonald,et al.  A randomized, controlled trial of clinical information shared from another institution. , 2002, Annals of emergency medicine.

[37]  J M Dean,et al.  Development and impact of a computerized pediatric antiinfective decision support program. , 2001, Pediatrics.