Semi-Supervised Exaggeration Detection of Health Science Press Releases

Public trust in science depends on honest and factual communication of scientific papers. However, recent studies have demonstrated a tendency of news media to misrepresent scientific papers by exaggerating their findings. Given this, we present a formalization of and study into the problem of exaggeration detection in science communication. While there are an abundance of scientific papers and popular media articles written about them, very rarely do the articles include a direct link to the original paper, making data collection challenging. We address this by curating a set of labeled press release/abstract pairs from existing expert annotated studies on exaggeration in press releases of scientific papers suitable for benchmarking the performance of machine learning models on the task. Using limited data from this and previous studies on exaggeration detection in science, we introduce MT-PET, a multi-task version of Pattern Exploiting Training (PET), which leverages knowledge from complementary clozestyle QA tasks to improve few-shot learning. We demonstrate that MT-PET outperforms PET and supervised learning both when data is limited, as well as when there is an abundance of data for the main task.1

[1]  Kalina Bontcheva,et al.  Stance Detection with Bidirectional Conditional Encoding , 2016, EMNLP.

[2]  Andreas Vlachos,et al.  FEVER: a Large-scale Dataset for Fact Extraction and VERification , 2018, NAACL.

[3]  Helmut Schmid,et al.  Automatically Identifying Words That Can Serve as Labels for Few-Shot Text Classification , 2020, COLING.

[4]  Steven Woloshin,et al.  Press Releases by Academic Medical Centers: Not So Academic? , 2009, Annals of Internal Medicine.

[5]  Colin Raffel,et al.  Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer , 2019, J. Mach. Learn. Res..

[6]  Horst Po¨ttker.  News and its communicative quality: the inverted pyramid—when and why did it appear? , 2003 .

[7]  Steven Woloshin,et al.  Press releases: translating research into news. , 2002, JAMA.

[8]  Bei Yu,et al.  An NLP Analysis of Exaggerated Claims in Science News , 2017, NLPmJ@EMNLP.

[9]  Isabelle Augenstein,et al.  Fact Check-Worthiness Detection as Positive Unlabelled Learning , 2020, EMNLP.

[10]  Lysandre Debut,et al.  HuggingFace's Transformers: State-of-the-art Natural Language Processing , 2019, ArXiv.

[11]  Petroc Sumner,et al.  The association between exaggeration in health related science news and academic press releases: retrospective observational study , 2014, BMJ : British Medical Journal.

[12]  Alec Radford,et al.  Improving Language Understanding by Generative Pre-Training , 2018 .

[13]  Isabelle Augenstein,et al.  CiteWorth: Cite-Worthiness Detection for Improved Scientific Document Understanding , 2021, FINDINGS.

[14]  Bei Yu,et al.  Measuring Correlation-to-Causation Exaggeration in Press Releases , 2020, COLING.

[15]  Preslav Nakov,et al.  Few-Shot Cross-Lingual Stance Detection with Sentiment-Based Pre-Training , 2021, AAAI.

[16]  Lucy Lu Wang,et al.  MSˆ2: Multi-Document Summarization of Medical Studies , 2021, EMNLP.

[17]  Mark Chen,et al.  Language Models are Few-Shot Learners , 2020, NeurIPS.

[18]  Marin Soljacic,et al.  We Can Explain Your Research in Layman's Terms: Towards Automating Science Journalism at Scale , 2021, AAAI.

[19]  Alberto Barrón-Cedeño,et al.  The CLEF-2021 CheckThat! Lab on Detecting Check-Worthy Claims, Previously Fact-Checked Claims, and Fake News , 2021, ECIR.

[20]  Bei Yu,et al.  Detecting Causal Language Use in Science Findings , 2019, EMNLP.

[21]  Daniel King,et al.  ScispaCy: Fast and Robust Models for Biomedical Natural Language Processing , 2019, BioNLP@ACL.

[22]  Timo Schick,et al.  Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference , 2020, EACL.

[23]  Matthias Hagen,et al.  Crowdsourcing a Large Corpus of Clickbait on Twitter , 2018, COLING.

[24]  Lewis Bott,et al.  The association between exaggeration in health-related science news and academic press releases: a replication study , 2019, Wellcome open research.

[25]  Bryan Catanzaro,et al.  Zero-shot Text Classification With Generative Language Models , 2019, ArXiv.

[26]  Steven Schockaert,et al.  Inducing Relational Knowledge from BERT , 2019, AAAI.

[27]  Isabelle Augenstein,et al.  Cross-Domain Label-Adaptive Stance Detection , 2021, EMNLP.

[28]  Preslav Nakov,et al.  Integrating Stance Detection and Fact Checking in a Unified Corpus , 2018, NAACL.

[29]  D. Mark,et al.  Selling Science: How the Press Covers Science and Technology , 1995 .

[30]  Hannaneh Hajishirzi,et al.  Fact or Fiction: Verifying Scientific Claims , 2020, EMNLP.

[31]  Hinrich Schutze,et al.  It’s Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners , 2020, NAACL.

[32]  Chin-Yew Lin,et al.  ROUGE: A Package for Automatic Evaluation of Summaries , 2004, ACL 2004.

[33]  Christian Hansen,et al.  MultiFC: A Real-World Multi-Domain Dataset for Evidence-Based Fact Checking of Claims , 2019, EMNLP.