Measuring agreement between decision support reminders: the cloud vs. the local expert

BackgroundA cloud-based clinical decision support system (CDSS) was implemented to remotely provide evidence-based guideline reminders in support of preventative health. Following implementation, we measured the agreement between preventive care reminders generated by an existing, local CDSS and the new, cloud-based CDSS operating on the same patient visit data.MethodsElectronic health record data for the same set of patients seen in primary care were sent to both the cloud-based web service and local CDSS. The clinical reminders returned by both services were captured for analysis. Cohen’s Kappa coefficient was calculated to compare the two sets of reminders. Kappa statistics were further adjusted for prevalence and bias due to the potential effects of bias in the CDS logic and prevalence in the relative small sample of patients.ResultsThe cloud-based CDSS generated 965 clinical reminders for 405 patient visits over 3 months. The local CDSS returned 889 reminders for the same patient visit data. When adjusted for prevalence and bias, observed agreement varied by reminder from 0.33 (95% CI 0.24 – 0.42) to 0.99 (95% CI 0.97 – 1.00) and demonstrated almost perfect agreement for 7 of the 11 reminders.ConclusionsPreventive care reminders delivered by two disparate CDS systems show substantial agreement. Subtle differences in rule logic and terminology mapping appear to account for much of the discordance. Cloud-based CDSS therefore show promise, opening the door for future development and implementation in support of health care providers with limited resources for knowledge management of complex logic and rules.

[1]  Andrew Kakabadse,et al.  Reviewing the knowledge management literature: towards a taxonomy , 2003, J. Knowl. Manag..

[2]  D. Berwick,et al.  The triple aim: care, health, and cost. , 2008, Health affairs.

[3]  D. Blumenthal,et al.  Information technology comes to medicine. , 2007, The New England journal of medicine.

[4]  Paul R Dexter,et al.  Inpatient computer-based standing orders vs physician reminders to increase influenza and pneumococcal vaccination rates: a randomized trial. , 2004, JAMA.

[5]  P. Shekelle,et al.  Systematic Review: Impact of Health Information Technology on Quality, Efficiency, and Costs of Medical Care , 2006, Annals of Internal Medicine.

[6]  Sowmya R. Rao,et al.  Use of electronic health records in U.S. hospitals. , 2009, The New England journal of medicine.

[7]  Adam Wright,et al.  Challenges in creating an enterprise clinical rules service. , 2008, AMIA ... Annual Symposium proceedings. AMIA Symposium.

[8]  Shaun J. Grannis,et al.  Chapter 5 – Regenstrief Medical Informatics: Experiences with Clinical Decision Support Systems , 2014 .

[9]  V. Hasselblad,et al.  Effect of Clinical Decision-Support Systems , 2012, Annals of Internal Medicine.

[10]  Omolola Ogunyemi,et al.  Review Paper: The InterMed Approach to Sharable Computer-interpretable Guidelines: A Review , 2004, J. Am. Medical Informatics Assoc..

[11]  Roger B. Davis,et al.  Physicians' decisions to override computerized drug alerts in primary care. , 2003, Archives of internal medicine.

[12]  Robert Tibshirani,et al.  An Introduction to the Bootstrap CHAPMAN & HALL/CRC , 1993 .

[13]  P. Basch.  Quality of health care delivered to adults in the United States. , 2003, New England Journal of Medicine.

[14]  D. Bates,et al.  Improving safety with information technology. , 2003, The New England journal of medicine.

[15]  J. Osheroff,et al.  Clinical Decision Support Systems : State of the Art , 2009 .

[16]  J. Marc Overhage,et al.  Case Report: Computerizing Guidelines to Improve Care and Patient Outcomes: The Example of Heart Failure , 1995, J. Am. Medical Informatics Assoc..

[17]  C. McDonald,et al.  A computerized reminder system to increase the use of preventive care for hospitalized patients. , 2001, The New England journal of medicine.

[18]  J. R. Landis,et al.  The measurement of observer agreement for categorical data. , 1977, Biometrics.

[19]  Maulik S Joshi,et al.  Progress toward meaningful use: hospitals' adoption of electronic health records. , 2011, The American journal of managed care.

[20]  Shobha Phansalkar,et al.  Best Practices in Clinical Decision Support , 2010, Applied Clinical Informatics.

[21]  J. Carlin,et al.  Bias, prevalence and kappa. , 1993, Journal of clinical epidemiology.

[22]  Tonya Hongsermeier,et al.  Creating shareable decision support services: an interdisciplinary challenge. , 2010, AMIA ... Annual Symposium proceedings. AMIA Symposium.

[23]  Tonya Hongsermeier,et al.  A pilot study of distributed knowledge management and clinical decision support in the cloud , 2013, Artif. Intell. Medicine.

[24]  Adam Wright,et al.  Comparison of clinical knowledge management capabilities of commercially-available and leading internally-developed electronic health records , 2011, BMC Medical Informatics Decis. Mak..

[25]  尚弘 島影.  National Institute of Standards and Technologyにおける超伝導研究及び生活 , 2001 .

[26]  Tonya Hongsermeier,et al.  A legal framework to enable sharing of Clinical Decision Support knowledge and services across institutional boundaries. , 2011, AMIA ... Annual Symposium proceedings. AMIA Symposium.

[27]  C. McDonald,et al.  Regenstrief Medical Informatics: Experiences with Clinical Decision Support Systems , 2007 .

[28]  Michael J. Earl,et al.  Knowledge Management Strategies: Toward a Taxonomy , 2001, J. Manag. Inf. Syst..

[29]  Adam Wright,et al.  Identifying Best Practices for Clinical Decision Support and Knowledge Management in the Field , 2010, MedInfo.

[30]  J. Marc Overhage,et al.  Computerized Reminders for Syphilis Screening in an Urban Emergency Department , 2003, AMIA.

[31]  David W. Bates,et al.  Standard practices for computerized clinical decision support in community hospitals: a national survey , 2012, J. Am. Medical Informatics Assoc..

[32]  Aziz A. Boxwala,et al.  Demonstrating Cloud-based Clinical Decision Support at Scale: The Clinical Decision Support Consortium , 2013, MedInfo.

[33]  C. DesRoches,et al.  A progress report on electronic health records in U.S. hospitals. , 2010, Health affairs.

[34]  Hhs Centers for Medicare Medicare Services.  Medicare and Medicaid programs; electronic health record incentive program. Final rule. , 2010, Federal register.

[35]  Adam Wright,et al.  The state of the art in clinical knowledge management: An inventory of tools and techniques , 2010, Int. J. Medical Informatics.

[36]  W M Tierney,et al.  Effectiveness of Computer-Generated Reminders for Increasing Discussions about Advance Directives and Completion of Advance Directive Forms , 1998, Annals of Internal Medicine.

[37]  David W. Bates,et al.  Governance for clinical decision support: case studies and recommended practices from leading institutions , 2011, J. Am. Medical Informatics Assoc..

[38]  David Parkin,et al.  Effect of computerised evidence based guidelines on management of asthma and angina in adults in primary care: cluster randomised controlled trial , 2002, BMJ : British Medical Journal.

[39]  Jihoon Kim,et al.  A multi-layered framework for disseminating knowledge for computer-based decision support , 2011, J. Am. Medical Informatics Assoc..

[40]  Blackford Middleton,et al.  The Clinical Decision Support Consortium , 2009, MIE.

[41]  Brian E. Dixon,et al.  Knowledge sharing using codification and collaboration technologies to improve health care: lessons from the public sector , 2009 .

[42]  Jennie Harvell,et al.  Hospitals ineligible for federal meaningful-use incentives have dismally low rates of adoption of electronic health records. , 2012, Health affairs.

[43]  Farzad Mostashari,et al.  Adoption of electronic health records grows rapidly, but fewer than half of US hospitals had at least a basic system in 2012. , 2013, Health affairs.

[44]  Diane L. Seger,et al.  Application of Information Technology: Improving Acceptance of Computerized Prescribing Alerts in Ambulatory Care , 2006, J. Am. Medical Informatics Assoc..

[45]  Adam Wright,et al.  Using a Service Oriented Architecture Approach to Clinical Decision Support: Performance Results from Two CDS Consortium Demonstrations , 2012, AMIA.