Impact of a Search Engine on Clinical Decisions Under Time and System Effectiveness Constraints: Research Protocol

Background Many clinical questions arise during patient encounters that clinicians are unable to answer. An evidence-based medicine approach expects that clinicians will seek and apply the best available evidence to answer clinical questions. One commonly used source of such evidence is scientific literature, such as that available through MEDLINE and PubMed. Clinicians report that 2 key reasons why they do not use search systems to answer questions is that it takes too much time and that they do not expect to find a definitive answer. So, the question remains about how effectively scientific literature search systems support time-pressured clinicians in making better clinical decisions. The results of this study are important because they can help clinicians and health care organizations to better assess their needs with respect to clinical decision support (CDS) systems and evidence sources. The results and data captured will contribute a significant data collection to inform the design of future CDS systems to better meet the needs of time-pressured, practicing clinicians. Objective The purpose of this study is to understand the impact of using a scientific medical literature search system on clinical decision making. Furthermore, to understand the impact of realistic time pressures on clinicians, we vary the search time available to find clinical answers. Finally, we assess the impact of improvements in search system effectiveness on the same clinical decisions. Methods In this study, 96 practicing clinicians and final year medical students are presented with 16 clinical questions which they must answer without access to any external resource. The same questions are then represented to the clinicians; however, in this part of the study, the clinicians can use a scientific literature search engine to find evidence to support their answers. The time pressures of practicing clinicians are simulated by limiting answer time to one of 3, 6, or 9 min per question. The correct answer rate is reported both before and after search to assess the impact of the search system and the time constraint. In addition, 2 search systems that use the same user interface, but which vary widely in their search effectiveness, are employed so that the impact of changes in search system effectiveness on clinical decision making can also be assessed. Results Recruiting began for the study in June 2018. As of the April 4, 2019, there were 69 participants enrolled. The study is expected to close by May 30, 2019, with results to be published in July. Conclusions All data collected in this study will be made available at the University of Queensland’s UQ eSpace public data repository. International Registered Report Identifier (IRRID) DERR1-10.2196/12803

[1]  K. Ramos,et al.  Real-time information-seeking behavior of residency physicians. , 2003, Family medicine.

[2]  H. C. Coumou,et al.  How do primary care physicians seek answers to clinical questions? A literature review. , 2006, Journal of the Medical Library Association : JMLA.

[3]  Peter Wright The harassed decision maker: Time pressures, distractions, and the use of evidence. , 1974 .

[4]  K. Pridham,et al.  Sources of maternal confidence and uncertainty and perceptions of problem-solving competence. , 1988, Journal of advanced nursing.

[5]  D. Sackett,et al.  Finding and applying evidence during clinical rounds: the "evidence cart". , 1998, JAMA.

[6]  H W Ferguson BIOPHYSICAL PROFILE SCORING: The Fetal Apgar , 1988, The American journal of nursing.

[7]  W. Hersh,et al.  Factors associated with successful answering of clinical questions using an information retrieval system. , 2002, Bulletin of the Medical Library Association.

[8]  James Allan,et al.  When will information retrieval be "good enough"? , 2005, SIGIR '05.

[9]  Johanna I Westbrook,et al.  The Impact of an Online Evidence System on Confidence in Decision Making in a Controlled Setting , 2005, Medical decision making : an international journal of the Society for Medical Decision Making.

[10]  L. Insabato,et al.  Fine‐needle aspiration cytology of angiolymphoid hyperplasia with eosinophilia: A case report with electron microscopy and immunohistochemistry , 1989, Diagnostic cytopathology.

[11]  E. Baker Valvoplasty, angioplasty and embolotherapy in congenital heart disease. , 1986, International journal of cardiology.

[12]  O. Svenson,et al.  Judgment and Decision Making Under Time Pressure , 1993 .

[13]  Falk Scholer,et al.  User performance versus precision measures for simple search tasks , 2006, SIGIR.

[14]  Andrew Turpin,et al.  Do batch and user evaluations give the same results? , 2000, SIGIR '00.

[15]  G H Guyatt,et al.  Transferring evidence from research into practice: 1. The role of clinical care research evidence in clinical decisions , 1996, Evidence Based Medicine.

[16]  Johanna I. Westbrook,et al.  Do online information retrieval systems help experienced clinicians answer clinical questions? , 2005, Journal of the American Medical Informatics Association : JAMIA.

[17]  Ellen M. Voorhees,et al.  Overview of the TREC 2014 Clinical Decision Support Track , 2014, TREC.

[18]  James J. Cimino,et al.  A Survey from a Large Academic Medical Center , 2015, Applied Clinical Informatics.

[19]  James Allan,et al.  When Will Information Retrieval Be "Good Enough"? User Effectiveness As a Function of Retrieval Accuracy , 2005 .

[20]  G. Diamond,et al.  The clinical importance of the inferolateral trunk of the internal carotid artery. , 1991, Neurosurgery.

[21]  Boryung Ju,et al.  Proceedings of the 76th ASIS&T Annual Meeting: Beyond the Cloud: Rethinking Information Boundaries , 2013 .

[22]  Leif Azzopardi,et al.  Impacts of Time Constraints and System Delays on User Experience , 2016, CHIIR.

[23]  Nick Sevdalis,et al.  The influence of time pressure on adherence to guidelines in primary care: an experimental study , 2013, BMJ Open.

[24]  J. Rhodin Perfusion and superfusion fixation effects on rat mesentery microvascular beds. Intravital and electron microscope analyses. , 1986, Journal of submicroscopic cytology.

[25]  C. J. Clark,et al.  Assessment of work performance in asthma for determination of cardiorespiratory fitness and training capacity. , 1988, Thorax.

[26]  Lisa D. Ordóñez,et al.  Decisions under Time Pressure: How Time Constraint Affects Risky Decision Making , 1997 .

[27]  Emine Yilmaz,et al.  A statistical method for system evaluation using incomplete judgments , 2006, SIGIR.

[28]  P. Bacchetti,et al.  Sample size calculations in clinical research. , 2002, Anesthesiology.

[29]  Charles P. Friedman,et al.  Research Paper: Factors Associated with Success in Searching MEDLINE and Applying Evidence to Answer Clinical Questions , 2002, J. Am. Medical Informatics Assoc..

[30]  C. Hicks,et al.  Evidence of serum antibodies to Neisseria gonorrhoeae before gonococcal infection. , 1987, The Journal of infectious diseases.

[31]  Chang Liu,et al.  The impacts of time constraint on users' search strategy during search process , 2016, ASIST.

[32]  Arjen Hoogendam,et al.  Answers to Questions Posed During Daily Patient Care Are More Likely to Be Answered by UpToDate Than PubMed , 2008, Journal of medical Internet research.

[33]  Ola Svenson,et al.  Framing and Time Pressure in Decision Making , 1993 .

[34]  K. Dunn,et al.  Examining the role of MEDLINE as a patient care information resource: an analysis of data from the Value of Libraries study , 2017, Journal of the Medical Library Association : JMLA.

[35]  B. Haynes Of studies, syntheses, synopses, summaries, and systems: the “5S” evolution of information services for evidence-based healthcare decisions , 2007, Evidence-based medicine.

[36]  B. Haynes Of studies, syntheses, synopses, summaries, and systems: the "5S" evolution of information services for evidence-based healthcare decisions. , 2006, Evidence-based nursing.

[37]  John W. Payne,et al.  Adapting to Time Constraints , 1993 .

[38]  M. Ebell,et al.  Analysis of questions asked by family doctors regarding patient care , 1999, BMJ.

[39]  U. Baandrup,et al.  Clinical and non-invasive assessment of anthracycline cardiotoxicity: perspectives on myocardial protection. , 1986, International journal of clinical pharmacology research.

[40]  J. Lederberg Medical science, infectious disease, and the unity of humankind. , 1988, JAMA.

[41]  P. Gorman,et al.  A taxonomy of generic clinical questions: classification study , 2000, BMJ : British Medical Journal.

[42]  Alistair Moffat,et al.  Users versus models: what observation tells us about effectiveness metrics , 2013, CIKM.

[43]  Joemon M. Jose,et al.  How users assess Web pages for information seeking , 2005, J. Assoc. Inf. Sci. Technol..

[44]  E. Spitzer,et al.  Comparative in vitro activity of LY121019 and amphotericin B against clinical isolates of Candida species. , 1988, European journal of clinical microbiology & infectious diseases : official publication of the European Society of Clinical Microbiology.

[45]  David A. Cook,et al.  Features of Effective Medical Knowledge Resources to Support Point of Care Learning: A Focus Group Study , 2013, PloS one.

[46]  Illhoi Yoo,et al.  Analysis of PubMed User Sessions Using a Full-Day PubMed Query Log: A Comparison of Experienced and Nonexperienced PubMed Users , 2015, JMIR medical informatics.

[47]  Alexander Kotov,et al.  WSU-IR at TREC 2015 Clinical Decision Support Track: Joint Weighting of Explicit and Latent Medical Query Concepts from Diverse Sources , 2015, TREC.

[48]  Robert G. Capra,et al.  Time pressure, user satisfaction and task difficulty , 2013, ASIST.

[49]  Douglas B. Fridsma,et al.  Research Paper: Effectiveness of Clinician-selected Electronic Information Resources for Answering Primary Care Physicians' Information Needs , 2006, J. Am. Medical Informatics Assoc..