Effects of Computer-based Clinical Decision Support Systems on Clinician Performance and Patient Outcome: A Critical Appraisal of Research

The application of artificial intelligence and other computing and information science techniques to the field of health care has resulted in the development of computer-based clinical decision support systems (CDSSs), sometimes called, generically, expert systems. Although no consensus has been achieved on the definition of a CDSS, Wyatt and Spiegelhalter [1] have defined medical decision aids as active knowledge systems which use two or more items of patient data to generate case-specific advice, thus capturing the main attributes of these systems in a simple statement. Much has been written about the theoretical and technical aspects of CDSSs and their reliability, validity, and acceptability. Wyatt and Spiegelhalter [2] described a systematic approach to laboratory and field testing of CDSSs, suggesting that the final stages should include evaluation of effects on the health care process and on patient outcomes. This overview focuses on studies of the final and most clinically important stages of evaluation and examines controlled trials designed to measure the effects of CDSSs on clinician performance and patient outcomes. Methods Study Identification Previously published reviews of CDSSs were identified using a MEDLINE search from January 1983 through February 1992 and through a manual search of textbooks and conference proceedings in the areas of artificial intelligence and computer applications in medicine. Original studies were identified by an all-language MEDLINE search of articles published from 1974 (the year of publication of the first article to evaluate the effect of a CDSS on clinician performance [3]) to February 1992 (search terms available on request). Studies were also identified through an update of a previous review on computer-aided quality assurance [4]; through an EMBASE (Excerpta Medica) search for the same time period; through an INSPEC (International Information Service for the Physics and Engineering Communities) search; through review of citations in the articles from electronic searches and a search forward on three citations [5-7], one each from the areas of dose determination, diagnosis, and quality assurance, using SCISEARCH; through articles on related topics collected by the Health Information Research Unit of McMaster University, including a regularly updated bibliography of studies of continuing education [8]; and by scanning the Proceedings of the Symposium on Computer Applications in Medical Care, 1989 through 1991. After a set of relevant publications was selected for inclusion in the overview, a list of their titles was sent to corresponding authors and experts in medical informatics with a request for information about any additional published or unpublished studies. Study Selection We selected studies for review if 1) the population of interest was composed of clinicians [physicians, nurses, dentists, and so on] in practice or training; 2) the intervention was a computer-based CDSS evaluated in a clinical setting; 3) the outcomes assessed were clinician performance, a measure of the process of care, or patient outcomes, including any aspect of patient well-being; and 4) the type of evidence was limited to prospective studies with a contemporaneous control group where patient care with a CDSS was compared with patient care without one. Crossover studies were included. A CDSS was defined as follows: computer software using a knowledge base designed for use by a clinician involved in patient care as a direct aid to clinical decision making. Characteristics of an individual patient were matched to information in the knowledge base. Patient-specific information in the form of assessments (management options or probabilities) or recommendations were presented to the clinician. Lists of citations, with abstracts and indexing terms if available, were assessed by two of the authors, one with a background in expert system development and one with experience in critical appraisal of applied medical research. Each rater indicated whether the citation was potentially relevant (that is, appeared to meet the selection criteria, applied loosely at this stage), was clearly not relevant, or gave insufficient information to make a judgment. The latter usually lacked abstracts; the full texts of these articles were examined by one of the raters in the library and rated as potentially relevant or definitely irrelevant. A copy was made of the full article for citations judged to be potentially relevant by either rater and then rated independently by both raters according to the four criteria listed above. To be included in the overview, a study was required to meet all the selection criteria. A third rater independently performed the same detailed relevance rating if the original raters disagreed or were uncertain about the study's relevance to the overview. In these cases, a study was included if two of the three raters deemed it relevant. Study Evaluation Two reviewers independently rated the studies selected for the overview on each of five potential sources of bias (see the Appendix). Disagreements were handled in the same way as those encountered during study selection. Additional information about study design was solicited from authors if necessary. Options under each item were assigned a score of 2, 1, or 0 points. Scores were summed for the five items to achieve an overall score; 10 was the highest possible score. The first rating option under each item indicated that appropriate steps were taken to minimize bias in the study results; the second option indicated that although efforts were made to reduce bias, they may not have been adequate; and the third option indicated a study in which a definite potential for bias existed or in which insufficient information was given to assure the reader that no bias was present. Item three (unit of allocation) was a measure of potential contamination specific to evaluations in which interventions are applied to clinicians and end points are measured for patients. When individual patients are allocated to intervention and control groups, clinicians may treat patients with or without the aid of the CDSS. Because only a portion of the patients in a given study may have been treated with the aid of a CDSS, knowledge gained from the CDSS may be applied to control patients, leading to an underestimate of the system's effect. In some cases, the CDSS may be accessible during assessment of control patients and could be used in their care. Similar problems can occur when individual clinicians are the unit of allocation. The presence of a CDSS or of colleagues who are using a CDSS in the clinic may influence the treatment given by control clinicians. Item five, concerning follow-up rates, was not applicable for several studies of computer-aided quality assurance in which all patients in the practice were randomly assigned to be evaluated with the aid of a CDSS or to be control patients but in which data were generated only for those coming to the clinic for a visit during the study period. These studies were scored on the first four criteria and the total was prorated so that the maximum possible score was 10 points. Data Extraction Data concerning setting, clinicians and patients, interventions, and outcomes were extracted from each article by one investigator using a structured data collection form. Data extraction was verified by a second rater, and corrections were made where necessary. A letter was sent to the corresponding author for each study to request missing data and verification of the description of the study setting, subjects, CDSS, and the main outcome to be included in the overview. Analysis We began the overview expecting that studies would come from several areas of health care and would evaluate a number of CDSSs developed for different purposes. They would be assessed in several settings using many types of outcomes: continuous, discrete, long-term, and short-term, with some having more direct clinical relevance than others. A single measure of effect, in our judgment, would not be the best expression of our current knowledge of the effects of CDSSs on clinicians and patients, and a formal meta-analysis could be misleading for most applications. We decided, instead, to present summaries of the effect of each type of system for several clinician and patient outcomes separately so that system developers and consumers could focus on the type of system and outcome that would be relevant to their practice. An effect size was calculated for each study reporting a continuous outcome variable by dividing the difference of the means for the intervention and control groups by the pooled standard deviation. The effect size indicated the number of standard deviations by which the CDSS and control groups differed. For dichotomous outcome variables, odds ratios were calculated. An odds ratio whose 95% CI did not include 1 indicated a significant beneficial effect of the CDSS. An odds ratio less than 1 favored the CDSS when the outcome variable of interest was the number of adverse effects, and an odds ratio of greater than 1 favored the CDSS when the outcome variable measured health or clinician performance. Differences in the findings for studies of the same type of CDSS were assessed statistically for heterogeneity using the Homogeneity Q test [9] or the Breslow-Day test [10]. Two crossover studies were treated as parallel designs because the risk for carryover effects from one study period to the next was considered minimal. Results Retrieval of Previous Reviews and Original Studies We identified seven previous reviews that examined experimental evidence of the effects of CDSSs on the process or outcome of care [4, 11-16]. Only one [4] was a systematic overview, and this was restricted to the evidence available by 1986 concerning computer-aided quality assurance, which is only one of several types of CDSSs available. To find reports of orig