An evaluation of existing text de-identification tools for use with patient progress notes from Australian general practice

INTRODUCTION Digitized patient progress notes from general practice represent a significant resource for clinical and public health research but cannot feasibly and ethically be used for these purposes without automated de-identification. Internationally, several open-source natural language processing tools have been developed, however, given wide variations in clinical documentation practices, these cannot be utilized without appropriate review. We evaluated the performance of four de-identification tools and assessed their suitability for customization to Australian general practice progress notes. METHODS Four tools were selected: three rule-based (HMS Scrubber, MIT De-id, Philter) and one machine learning (MIST). 300 patient progress notes from three general practice clinics were manually annotated with personally identifying information. We conducted a pairwise comparison between the manual annotations and patient identifiers automatically detected by each tool, measuring recall (sensitivity), precision (positive predictive value), f1-score (harmonic mean of precision and recall), and f2-score (weighs recall 2x higher than precision). Error analysis was also conducted to better understand each tool's structure and performance. RESULTS Manual annotation detected 701 identifiers in seven categories. The rule-based tools detected identifiers in six categories and MIST in three. Philter achieved the highest aggregate recall (67%) and the highest recall for NAME (87%). HMS Scrubber achieved the highest recall for DATE (94%) and all tools performed poorly on LOCATION. MIST achieved the highest precision for NAME and DATE while also achieving similar recall to the rule-based tools for DATE and highest recall for LOCATION. Philter had the lowest aggregate precision (37%), however preliminary adjustments of its rules and dictionaries showed a substantial reduction in false positives. CONCLUSION Existing off-the-shelf solutions for automated de-identification of clinical text are not immediately suitable for our context without modification. Philter is the most promising candidate due to its high recall and flexibility however will require extensive revising of its pattern matching rules and dictionaries.

[1]  C. Tam,et al.  A framework for de-identification of free-text data in electronic medical records enabling secondary use. , 2022, Australian health review : a publication of the Australian Hospital Association.

[2]  N. Jamaludin,et al.  A Comparative Study on Part-of-Speech Taggers’ Performance on Examination Questions Classification According to Bloom’s Taxonomy , 2022, Journal of Physics: Conference Series.

[3]  J. Jonnagaddala,et al.  The OpenDeID corpus for patient de-identification , 2021, Scientific Reports.

[4]  Bradley Malin,et al.  Building a best-in-class automated de-identification tool for electronic health records through ensemble learning , 2021, Patterns.

[5]  Anthony N. Nguyen,et al.  De-identifying Australian hospital discharge summaries: An end-to-end framework using ensemble of deep learning models , 2021, J. Biomed. Informatics.

[6]  A. Butte,et al.  Protected Health Information filter (Philter): accurately and securely de-identifying free-text clinical notes , 2020, npj Digital Medicine.

[7]  Jessica Bell,et al.  Gathering data for decisions: best practice use of primary care electronic records for research , 2019, The Medical journal of Australia.

[8]  Jillian Oderkirk,et al.  Readiness of electronic health record systems to contribute to national health information and research , 2017 .

[9]  Donia Scott,et al.  Extracting information from the text of electronic medical records to improve case detection: a systematic review , 2016, J. Am. Medical Informatics Assoc..

[10]  Özlem Uzuner,et al.  Annotating longitudinal clinical narratives for de-identification: The 2014 i2b2/UTHealth corpus , 2015, J. Biomed. Informatics.

[11]  Michael Klompas,et al.  Uses of electronic health records for public health surveillance to advance public health. , 2015, Annual review of public health.

[12]  S. Muller,et al.  Electronic medical records: the way forward for primary care research? , 2014, Family practice.

[13]  Peter J. Haug,et al.  Improving performance of natural language processing part-of-speech tagging on clinical narratives through domain adaptation , 2013, J. Am. Medical Informatics Assoc..

[14]  S. Meystre,et al.  Evaluating current automatic de-identification methods with Veteran’s health administration clinical documents , 2012, BMC Medical Research Methodology.

[15]  Rob Koeling,et al.  Automatically estimating the incidence of symptoms recorded in GP free text notes , 2011, MIXHS '11.

[16]  Hua Xu,et al.  Data from clinical notes: a perspective on the tension between structure and flexible documentation , 2011, J. Am. Medical Informatics Assoc..

[17]  Lynette Hirschman,et al.  The MITRE Identification Scrubber Toolkit: Design, training, and assessment , 2010, Int. J. Medical Informatics.

[18]  Peter Szolovits,et al.  Automated de-identification of free-text medical records , 2008, BMC Medical Informatics Decis. Mak..

[19]  Peter Szolovits,et al.  Evaluating the state-of-the-art in automatic de-identification. , 2007, Journal of the American Medical Informatics Association : JAMIA.

[20]  Ulysses J. Balis,et al.  Development and evaluation of an open source software tool for deidentification of pathology reports , 2006, BMC Medical Informatics Decis. Mak..

[21]  Hercules Dalianis,et al.  Clinical Text Mining , 2018, Springer International Publishing.