Zero-Shot Information Extraction for Clinical Meta-Analysis using Large Language Models

Meta-analysis of randomized clinical trials (RCTs) plays a crucial role in evidence-based medicine but can be labor-intensive and error-prone. This study explores the use of large language models to enhance the efficiency of aggregating results from randomized clinical trials (RCTs) at scale. We perform a detailed comparison of the performance of these models in zero-shot prompt-based information extraction from a diverse set of RCTs to traditional manual annotation methods. We analyze the results for two different meta-analyses aimed at drug repurposing in cancer therapy pharmacovigilience in chronic myeloid leukemia. Our findings reveal that the best model for the two demonstrated tasks, ChatGPT can generally extract correct information and identify when the desired information is missing from an article. We additionally conduct a systematic error analysis, documenting the prevalence of diverse error types encountered during the process of prompt-based information extraction.

[1]  Jason Alan Fries,et al.  The Shaky Foundations of Clinical Foundation Models: A Survey of Large Language Models and Foundation Models for EMRs , 2023, ArXiv.

[2]  A. Perotte,et al.  EvidenceMap: a three-level knowledge representation for medical evidence computation and comprehension , 2023, J. Am. Medical Informatics Assoc..

[3]  Henning Müller,et al.  Not so weak PICO: leveraging weak supervision for participants, interventions, and outcomes recognition for systematic review automation , 2023, JAMIA open.

[4]  Cassie S. Mitchell,et al.  CCS Explorer: Relevance Prediction, Extractive Summarization, and Named Entity Recognition from Clinical Cohort Studies , 2022, 2022 IEEE International Conference on Big Data (Big Data).

[5]  J. Podichetty,et al.  How can natural language processing help model informed drug development?: a review , 2022, JAMIA open.

[6]  J. Leskovec,et al.  LinkBERT: Pretraining Language Models with Document Links , 2022, ACL.

[7]  Ryan J. Lowe,et al.  Training language models to follow instructions with human feedback , 2022, NeurIPS.

[8]  Hao Cheng,et al.  Fine-tuning large neural language models for biomedical natural language processing , 2021, Patterns.

[9]  Cassie S. Mitchell,et al.  Meta-Analysis of Gastrointestinal Adverse Events from Tyrosine Kinase Inhibitors for Chronic Myeloid Leukemia , 2021, Cancers.

[10]  Valentina R Minciacchi,et al.  Chronic Myeloid Leukemia: A Model Disease of the Past, Present and Future , 2021, Cells.

[11]  Allan Hanbury,et al.  Effective Crowd-Annotation of Participants, Interventions, and Outcomes in the Text of Clinical Trial Reports , 2020, FINDINGS.

[12]  Byron C. Wallace,et al.  Generating (Factual?) Narrative Summaries of RCTs: Experiments with Neural Multi-Document Summarization , 2020, AMIA ... Annual Symposium proceedings. AMIA Symposium.

[13]  Kush R. Varshney,et al.  A Natural Language Processing System for Extracting Evidence of Drug Repurposing from Scientific Publications , 2020, AAAI.

[14]  Junyi Jessy Li,et al.  A Corpus with Multi-Level Annotations of Patients, Interventions and Outcomes to Support Language Processing for Medical Literature , 2018, ACL.

[15]  H. Kantarjian,et al.  Chronic myeloid leukemia: 2018 update on diagnosis, therapy and monitoring , 2018, American journal of hematology.

[16]  Andrew W. Brown,et al.  Analysis of the time and workers needed to conduct systematic reviews of medical interventions using data from the PROSPERO registry , 2017, BMJ Open.

[17]  Dalia Al-Karawi,et al.  Bright light therapy for nonseasonal depression: Meta-analysis of clinical trials. , 2016, Journal of affective disorders.

[18]  Kara van de Graaf.  Manifest , 2014, Migratory Sound.

[19]  P. Sedgwick.  Meta-analyses: heterogeneity and subgroup analysis , 2013 .

[20]  D. Moher,et al.  The nuts and bolts of PROSPERO: an international prospective register of systematic reviews , 2012, Systematic Reviews.

[21]  Chin-Yew Lin,et al.  ROUGE: A Package for Automatic Evaluation of Summaries , 2004, ACL 2004.

[22]  Á. Atallah,et al.  Hormonal cryptorchidism therapy: systematic review with metanalysis of randomized clinical trials , 2004, Pediatric Surgery International.

[23]  R. Sigal,et al.  Effects of exercise on glycemic control and body mass in type 2 diabetes mellitus: a meta‐analysis of controlled clinical trials , 2002, JAMA.

[24]  David R. Jones,et al.  Methods for Exploring Heterogeneity in Meta-Analysis , 2001 .

[25]  L. Walker,et al.  Enteral nutritional supplementation with key nutrients in patients with critical illness and cancer: a meta-analysis of randomized controlled clinical trials. , 1999, Annals of surgery.

[26]  Lucy Lu Wang,et al.  Overview of MSLR2022: A Shared Task on Multi-document Summarization for Literature Reviews , 2022, SDP.

[27]  Vijay K. Shanker,et al.  BioM-Transformers: Building Large Biomedical Language Models with BERT, ALBERT and ELECTRA , 2021, BIONLP.

[28]  S. Gopalakrishnan,et al.  Systematic Reviews and Meta-analysis: Understanding the Best Evidence in Primary Healthcare , 2013, Journal of family medicine and primary care.

[29]  K R Abrams,et al.  Methods for exploring heterogeneity in meta-analysis. , 2001, Evaluation & the health professions.