Modeling Information Change in Science Communication with Semantically Matched Paraphrases

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1783 - 1807 December 7-11, 2022 ©2022 Association for Computational Linguistics Modeling Information Change in Science Communication with Semantically Matched Paraphrases Dustin Wright♭∗ Jiaxin Pei♯∗ David Jurgens♯ Isabelle Augenstein♭ ♭Dept. of Computer Science, University of Copenhagen, Denmark ♯School of Information, University of Michigan, Ann Arbor, MI, USA {dw,augenstein}@di.ku.dk {pedropei,jurgens}@umich.edu Abstract Whether the media faithfully communicate sci- entific information has long been a core issue to the science community. Automatically identify- ing paraphrased scientific findings could enable large-scale tracking and analysis of information changes in the science communication process, but this requires systems to understand the sim- ilarity between scientific information across multiple domains. To this end, we present the SCIENTIFIC PARAPHRASE AND INFOR- MATION CHANGE DATASET (SPICED), the first paraphrase dataset of scientific findings annotated for degree of information change. SPICED contains 6,000 scientific finding pairs extracted from news stories, social media dis- cussions, and full texts of original papers. We demonstrate that SPICED poses a challenging task and that models trained on SPICED im- prove downstream performance on evidence re- trieval for fact checking of real-world scientific claims. Finally, we show that models trained on SPICED can reveal large-scale trends in the degrees to which people and organizations faithfully communicate new scientific findings. Data, code, and pre-trained models are avail- able at http://www.copenlu.com/publication/ 2022_emnlp_wright/. 1 Introduction Science communication disseminates scholarly in- formation to audiences outside the research com- munity, such as the public and policymakers (National Academies of Sciences, Engineering, and Medicine, 2017). This process usually in- volves translating highly technical language to non- technical, less-formal language that is engaging and easily understandable for lay people (Salita, 2015). The public relies on the media to learn about new scientific findings, and media portray- als of science affect people’s trust in science while at the same time influencing their future actions ∗denotes equal contribution Text Increasing dietary magnesium intake is associated with a reduced risk of stroke, heart failure, diabetes, and all-cause mortality. The study findings suggest that increased consumption of magnesium-rich foods may have health benefits. #Magnesium saves lives https://t.co/K0M6QdjWcc https://t.co/FUtWZ8jADM Figure 1: We are interested in measuring the informa- tion similarity of statements about scientific findings between different sources, including scientific papers, news, and tweets, shown here with real examples. The finding in this figure comes from Fang et al. (2016) and the news quote is from this Reuters story. (Gustafson and Rice, 2019; Fischhoff, 2012; Kuru et al., 2021). However, not all scientific communi- cation accurately conveys the original information, as shown in Figure 1. Identifying cases where sci- entific information has changed is a critical but challenging task due to the complex translating and paraphrasing done by effective communicators. Our work introduces a new task of measuring sci- entific information change, and through developing new data and models aims to address the gap in studying faithful scientific communication. Though efforts exist to track and flag when pop- ular media misrepresent science,1 the sheer …

Content truncated to 500 words