Automated analysis of ambulatory surgery patient experience comments using artificial intelligence for quality improvement: A patient centered approach

Abstract Background Quality improvement in healthcare is limited by both the quality and quantity of data available in the electronic health records or self reported by clinicians. Appropriate and timely reporting can help identify opportunities for quality improvement and deployment of mitigation steps but many of these process remain manual. Negative patient experience comments can potentially be used in identifying opportunities for quality improvement. Artificial intelligence (AI) techniques, like natural language processing (NLP) including sentiment analysis and topic modeling can be used to characterize patient experience comments in near real time, thus making the process timely, efficient and measurable. Methods We analyzed 15,453 de-identified patient experience comments from the Press Ganey™ survey of adult patients undergoing outpatient surgeries with anesthesia at Cleveland Clinic from 01/01/2012-05/03/2016. We used open source NLP including sentiment analysis and topical modeling to analyze post-discharge patient experience survey comments and feedback verbatim. For sentiment analysis we used an open source sentiment analyzer called VADER. For topic modeling we used Latent Dirichlet Allocation (LDA) algorithm. Results Sentiment analysis of patient comments using VADER was highly accurate with F1 score ranging from 0.83 to 0.84 for positive comments and 0.71–0.73 for negative comments compared to clinician's assessment. Two clinicians reviewed 1955 random comments as positive or negative with good agreement, with Cohen's k test score of k = 0.92 (p  Conclusion We conclude that artificial intelligence can help in near real-time analysis of patient experience surveys. Commonly used open source algorithms can possibly be utilized in healthcare for quality improvement negating the need for significant development resources with the opportunity to generalize and scale.

[1]  A. Kurz,et al.  Identify and monitor clinical variation using machine intelligence: a pilot in colorectal surgery , 2018, Journal of Clinical Monitoring and Computing.

[2]  Michael I. Jordan,et al.  Latent Dirichlet Allocation , 2001, J. Mach. Learn. Res..

[3]  Manabu Torii,et al.  Feasibility of Real-Time Satisfaction Surveys Through Automated Analysis of Patients' Unstructured Comments and Sentiments , 2012, Quality management in health care.

[4]  Yang Yu,et al.  User-Generated Content : Using Sentiment Analysis Technique to Study Hotel Service Quality , 2012 .

[5]  Kai Zheng,et al.  How Do General-Purpose Sentiment Analyzers Perform when Applied to Health-Related Online Social Media Data? , 2019, MedInfo.

[6]  Maria Uriyo,et al.  Using sentiment analysis to review patient satisfaction data located on the internet. , 2015, Journal of health organization and management.

[7]  Bing Liu,et al.  Sentiment Analysis and Subjectivity , 2010, Handbook of Natural Language Processing.

[8]  Ann C. Haas,et al.  How event reporting by US hospitals has changed from 2005 to 2009 , 2011, BMJ quality & safety.

[9]  M. Makary,et al.  Medical error—the third leading cause of death in the US , 2016, British Medical Journal.

[10]  J. Cywinski,et al.  Impacting Perioperative Quality and Patient Safety Using Artificial Intelligence , 2020 .

[11]  Using Electronic Health Records to Measure Quality Improvement Efforts: Findings from a Large Practice Facilitation Initiative. , 2019, Joint Commission journal on quality and patient safety.

[12]  Trevor A Sheldon,et al.  Sensitivity of routine system for reporting patient safety incidents in an NHS hospital: retrospective patient case note review , 2006, BMJ : British Medical Journal.

[13]  R. Dutton,et al.  Factors Affecting Patient Satisfaction With Their Anesthesiologist: An Analysis of 51,676 Surveys From a Large Multihospital Practice. , 2019, Anesthesia and analgesia.

[14]  D. Ring,et al.  Negative Patient-Experience Comments After Total Shoulder Arthroplasty , 2019, The Journal of bone and joint surgery. American volume.

[15]  J. Battles,et al.  Adverse-event-reporting practices by US hospitals: results of a national survey , 2008, Quality & Safety in Health Care.

[16]  Shaowen Yao,et al.  An overview of topic modeling and its current applications in bioinformatics , 2016, SpringerPlus.

[17]  Steven Bird,et al.  NLTK: The Natural Language Toolkit , 2002, ACL.

[18]  Charles Elkan,et al.  Optimal Thresholding of Classifiers to Maximize F1 Measure , 2014, ECML/PKDD.

[19]  Andreas Holzinger,et al.  Use of Sentiment Analysis for Capturing Patient Experience From Free-Text Comments Posted Online , 2013, Journal of medical Internet research.

[20]  Yogesh Kumar Meena,et al.  YouTube Video Ranking by Aspect-Based Sentiment Analysis on User Feedback , 2019 .

[21]  Rakib Hossain,et al.  Comparative Sentiment Analysis using Difference Types of Machine Learning Algorithm , 2019, 2019 8th International Conference System Modeling and Advancement in Research Trends (SMART).