Detecting Causal Language Use in Science Findings

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 4664–4674, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 4664 Detecting Causal Language Use in Science Findings Bei Yu1, Yingya Li1, and Jun Wang2 1School of Information Studies, Syracuse University 2Independent Researcher {byu, yli48}@syr.edu, junwang4@gmail.com Abstract Causal interpretation of correlational findings from observational studies has been a major type of misinformation in science communica- tion. Prior studies on identifying inappropriate use of causal language relied on manual con- tent analysis, which is not scalable for exam- ining a large volume of science publications. In this study, we first annotated a corpus of over 3,000 PubMed research conclusion sen- tences, then developed a BERT-based predic- tion model that classifies conclusion sentences into “no relationship”, “correlational”, “condi- tional causal”, and “direct causal” categories, achieving an accuracy of 0.90 and a macro- F1 of 0.88. We then applied the prediction model to measure the causal language use in the research conclusions of about 38,000 ob- servational studies in PubMed. The prediction result shows that 21.7% studies used direct causal language exclusively in their conclu- sions, and 32.4% used some direct causal lan- guage. We also found that the ratio of causal language use differs among authors from dif- ferent countries, challenging the notion of a shared consensus on causal language use in the global science community. Our prediction model could also be used to help identify the inappropriate use of causal language in science publications. 1 Introduction Establishing causality is one of the most impor- tant goals and concerns of science. Therefore, the language that describes causal relationships plays a crucial role in communicating science find- ings among scientists and with the general public (Kleinberg and Hripcsak, 2011). However, scien- tists and journalists have been found to inappropri- ately use causal language in research publications and news articles. Specifically, causal interpreta- tion of correlational findings from observational studies has been a major type of misinformation in science communication (Cofield et al., 2010; Sumner et al., 2014; Chiu et al., 2017; Boutron and Ravaud, 2018). In scientific research, observa- tional studies (non-intervention) are designed for testing association/correlation between variables, while intervention studies, such as clinical trials, are for testing causal relations (Buhse et al., 2018). Misinterpreting correlations as causations can lead to serious consequences, such as wrong medical decisions (Buhse et al., 2018) or harmful mis- perception of certain groups of people in society (Richardson et al., 2014). The misinterpretations are often attributed to humans’ inherent tendency to conflate correlation and causation (Bleske-Rechek et al., 2015), and journalists’ lack of scientific training or their pur- suit of sensational effect (Fahnestock, 1998; Sum- ner et al., 2014). However, misinterpreting corre- lational findings as causal claims has been found not only in news stories and press releases writ- ten by journalists (Sumner et al., 2014), but also in research papers published by scientists. For exam- ple, Robinson et al. (2007) …

Content truncated to 500 words