Quantitative evaluation of clinical software, exemplified by decision support systems.

To ensure the correctness of publicity material ('truth in labelling') and to inform their licensing decisions, agencies certifying or regulating any clinical computer system will need information about the system's structure, performance and likely impact on users and the environment in which they work. This information must be reliable and complete, so it needs to be collected in a structured, rigorous evaluation programme. Clinical decision support systems are generally more complex and their effects less easy to predict than most other clinical software, so pose the greatest challenge to evaluators. They are therefore the focus of this paper.

[1]  J. Wyatt.  Commentary: Telemedicine trials—clinical pull or technology push? , 1996 .

[2]  H A Heathfield,et al.  Philosophies for the Design and Development of Clinical Decision-Support Systems , 1993, Methods of Information in Medicine.

[3]  J. Marc Overhage,et al.  A plea for controlled trials in medical informatics , 1994, J. Am. Medical Informatics Assoc..

[4]  G Tomson,et al.  Randomization by group in studying the effect of drug information in primary care. , 1992, International journal of epidemiology.

[5]  J. Wyatt.  Medical informatics, artefacts or science? , 1996, Methods of information in medicine.

[6]  J. Wyatt,et al.  Clinical data systems, part 3: development and evaluation , 1994, The Lancet.

[7]  H. Sox,et al.  Clinical prediction rules. Applications and methodological standards. , 1985, The New England journal of medicine.

[8]  A Donabedian,et al.  The quality of medical care. , 1978, Science.

[9]  Jonathan Mant,et al.  Detecting differences in quality of care: the sensitivity of measures of process and outcome in treating acute myocardial infarction , 1995, BMJ.

[10]  I. Adams,et al.  Computer aided diagnosis of acute abdominal pain: a multicentre study. , 1986, British medical journal.

[11]  D. Spiegelhalter,et al.  Evaluating medical expert systems: what to test and how? , 1990, Medical informatics = Medecine et informatique.

[12]  F. Mosteller,et al.  A comparison of results of meta-analyses of randomized control trials and recommendations of clinical experts. Treatments for myocardial infarction. , 1992, JAMA.

[13]  E H Shortliffe,et al.  Improvements in data collection through physician use of a computer-based chemotherapy treatment consultant. , 1985, Journal of clinical oncology : official journal of the American Society of Clinical Oncology.

[14]  R. Peto,et al.  Large‐Scale Randomized Evidence: Large, Simple Trials and Overviews of Trials , 1993, Journal of clinical epidemiology.

[15]  P A Emerson,et al.  An audit of doctor's management of patients with chest pain in the accident and emergency department. , 1989, The Quarterly journal of medicine.

[16]  T C Chalmers,et al.  Getting to grips with Archie Cochrane's agenda. , 1992, BMJ.

[17]  H A Heathfield,et al.  The Road to Professionalism in Medical Informatics: A Proposal for Debate , 1995, Methods of Information in Medicine.

[18]  C. McDonald,et al.  Physician inpatient order writing on microcomputer workstations. Effects on resource utilization. , 1993, JAMA.

[19]  P L Miller,et al.  Exploring subjective vs. objective issues in the validation of computer-based critiquing advice. , 1990, Computer methods and programs in biomedicine.

[20]  Charles P. Friedman,et al.  Evaluation Methods in Medical Informatics , 1997, Computers and Medicine.

[21]  J. Wyatt,et al.  INFORMATION FOR CLINICIANS Use and sources of medical knowledge , 1991, The Lancet.

[22]  J C Wyatt,et al.  Safety and Decision Support in Oncology , 1994, Methods of Information in Medicine.

[23]  R. Haynes,et al.  Effects of Computer-based Clinical Decision Support Systems on Clinician Performance and Patient Outcome: A Critical Appraisal of Research , 1994, Annals of Internal Medicine.

[24]  Jeremy Wyatt,et al.  Decision Aids and the Law , 1989, The Lancet.

[25]  Lawrence M. Fagan,et al.  Use of a Domain Model to Drive an Interactive Knowledge-Editing Tool , 1987, Int. J. Man Mach. Stud..

[26]  G F Cooper,et al.  An evaluation of explanations of probabilistic inference. , 1993, Computers and biomedical research, an international journal.

[27]  Allen Newell,et al.  The Knowledge Level , 1989, Artif. Intell..

[28]  J. Wyatt,et al.  Computer-based knowledge systems , 1991, The Lancet.