Semi-Supervised Exaggeration Detection of Health Science Press Releases

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10824–10836 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 10824 Semi-Supervised Exaggeration Detection of Health Science Press Releases Dustin Wright and Isabelle Augenstein Dept. of Computer Science University of Copenhagen Denmark {dw|augenstein}@di.ku.dk Abstract Public trust in science depends on honest and factual communication of scientific pa- pers. However, recent studies have demon- strated a tendency of news media to misrep- resent scientific papers by exaggerating their findings. Given this, we present a formaliza- tion of and study into the problem of exag- geration detection in science communication. While there are an abundance of scientific pa- pers and popular media articles written about them, very rarely do the articles include a di- rect link to the original paper, making data col- lection challenging. We address this by cu- rating a set of labeled press release/abstract pairs from existing expert annotated studies on exaggeration in press releases of scientific papers suitable for benchmarking the perfor- mance of machine learning models on the task. Using limited data from this and previous stud- ies on exaggeration detection in science, we in- troduce MT-PET, a multi-task version of Pat- tern Exploiting Training (PET), which lever- ages knowledge from complementary cloze- style QA tasks to improve few-shot learning. We demonstrate that MT-PET outperforms PET and supervised learning both when data is limited, as well as when there is an abundance of data for the main task.1 1 Introduction Factual and honest science communication is important for maintaining public trust in sci- ence (Nelkin, 1987; Moore, 2006), and the “domi- nant link between academia and the media” are press releases about scientific articles (Sumner et al., 2014). However, multiple studies have demonstrated that press releases have a signifi- cant tendency to sensationalize their associated scientific articles (Sumner et al., 2014; Bratton et al., 2019; Woloshin et al., 2009; Woloshin and 1The code and data are available on- line at https://github.com/copenlu/ scientific-exaggeration-detection Figure 1: Scientific exaggeration detection is the prob- lem of identifying when a news article reporting on a scientific finding has exaggerated the claims made in the original paper. In this work, we are concerned with predicting exaggeration of the main finding of a scien- tific abstract as reported by a press release. Schwartz, 2002). In this paper, we explore how natural language processing can help identify exag- gerations of scientific papers in press releases. While Sumner et al. (2014) and Bratton et al. (2019) performed manual analyses to understand the prevalence of exaggeration in press releases of scientific papers from a variety of sources, recent work has attempted to expand this using methods from NLP (Yu et al., 2019, 2020; Li et al., 2017). These works focus on the problem of automatically detecting the difference in the strength of causal claims made in scientific articles and press releases. They accomplish this by first building datasets of main claims taken from PubMed abstracts and (un- related) press releases from EurekAlert2 labeled for their …

Content truncated to 500 words