Measuring Sentence-Level and Aspect-Level (Un)certainty in Science Communications

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 9959–10011 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 9959 Measuring Sentence-Level and Aspect-Level (Un)certainty in Science Communications Jiaxin Pei School of Information University of Michigan pedropei@umich.edu David Jurgens School of Information University of Michigan jurgens@umich.edu Abstract Certainty and uncertainty are fundamental to science communication. Hedges have widely been used as proxies for uncertainty. How- ever, certainty is a complex construct, with au- thors expressing not only the degree but the type and aspects of uncertainty in order to give the reader a certain impression of what is known. Here, we introduce a new study of certainty that models both the level and the aspects of certainty in scientific findings. Us- ing a new dataset of 2167 annotated scientific findings, we demonstrate that hedges alone account for only a partial explanation of cer- tainty. We show that both the overall certainty and individual aspects can be predicted with pre-trained language models, providing a more complete picture of the author’s intended com- munication. Downstream analyses on 431K scientific findings from news and scientific abstracts demonstrate that modeling sentence- level and aspect-level certainty is meaningful for areas like science communication. Both the model and datasets used in this paper are released at https://blablablab.si. umich.edu/projects/certainty/ 1 Introduction Expressing certainty about what is known is a nec- essary characteristic of scientific work as science involves producing knowledge about what was pre- viously unknown (Friedman et al., 1999; Smith- son, 2012). Given the natural aversion to uncer- tainty, existing studies have found that presenting uncertainty in science communications influences people’s perception of scientific findings and trust in science (Gustafson and Rice, 2019; Fischhoff, 2012; Van Der Bles et al., 2020). Therefore, un- derstanding how journalists and scientists commu- nicate certainty and uncertainty is critical for un- derstanding the current ecosystem of science jour- nalism and further provides better guidance for uncertainty communication (National Academies of Sciences, Engineering, and Medicine, 2017). Arctic sea ice is definitely declining[Probability ] at a rate of 13.3[Number] percent per decade[Extent] and we believe that[Psychological] the society may need to take action[Suggestion] to control global warming. Arctic sea ice is declining[Probability ] … at a rate of 13.3[Number] percent per decade[Extent] and we believe that[Psychological] the society may need to take action[Suggestion] to control global warming. Arctic sea ice could be declining[Probability ] … at a rate of 13.3[Number] percent per decade[Extent] and we believe that[Psychological] the society may need to take action[Suggestion] to control global warming. Arctic sea ice might be declining[Probability ] at a rate of 13.3[Number ] percent per decade[Extent ] and we believe that[Framing ] the society may need to take action[Suggestion ] to control global warming. Uncertain Certain Sentence-level Certainty Figure 1: Certainty is a multi-dimensional construct. The certainty of a scientific finding can be perceived holistically at the sentence level from its description. However, scientific findings may involve multiple as- pects that may each be described as certain (↑) or un- certain …

Content truncated to 500 words