Generating Scientific Claims for Zero-Shot Scientific Fact Checking

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 2448 - 2460 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Generating Scientific Claims for Zero-Shot Scientific Fact Checking Dustin Wright♮∗ David Wadden♯ Kyle Lo♭ Bailey Kuehl♭ Arman Cohan♭ Isabelle Augenstein♮ Lucy Lu Wang♭ ♮Dept. of Computer Science, University of Copenhagen, Denmark ♯University of Washington, Seattle, WA, USA ♭Allen Institute for Artificial Intelligence, Seattle, WA, USA {dw,augenstein}@di.ku.dk dwadden@cs.washington.edu {kylel,baileyk,armanc,lucyw}@allenai.org Abstract Automated scientific fact checking is difficult due to the complexity of scientific language and a lack of significant amounts of training data, as annotation requires domain expertise. To address this challenge, we propose scien- tific claim generation, the task of generating one or more atomic and verifiable claims from scientific sentences, and demonstrate its use- fulness in zero-shot fact checking for biomedi- cal claims. We propose CLAIMGEN-BART, a new supervised method for generating claims supported by the literature, as well as KBIN, a novel method for generating claim nega- tions. Additionally, we adapt an existing unsupervised entity-centric method of claim generation to biomedical claims, which we call CLAIMGEN-ENTITY. Experiments on zero-shot fact checking demonstrate that both CLAIMGEN-ENTITY and CLAIMGEN-BART, coupled with KBIN, achieve up to 90% perfor- mance of fully supervised models trained on manually annotated claims and evidence. A rigorous evaluation study demonstrates signifi- cant improvement in generated claim and nega- tion quality over existing baselines.1 1 Introduction Scientific documents contain complex assertions about scientific processes, making it difficult to au- tomate important tasks such as claim extraction and scientific fact checking. Additionally, the collec- tion of manually annotated labels to train models on tasks with scientific data is time consuming and expensive due to the need for domain exper- tise (Collins et al., 2017; Augenstein and Søgaard, 2017; Lehman et al., 2019; Wadden et al., 2020; DeYoung et al., 2021). As such, methods which re- quire less manual annotation are especially useful in this domain. This work addresses this challenge ∗Work completed while an intern at AI2 1Code and data available at: https://github.com/ allenai/scientific-claim-generation (1) ALS is the most common adult motor neuron disease with an incidence of 2 per 100,000 and prevalence of 5.4 per 100,000 individuals. (2) Current treatment options are based on symptom management and respiratory support with the only approved medications in widespread use, Riluzole and Edaravone, providing only modest benefits and only in some patients. Current treatment options for ALS are based on symptom management and respiratory support Riluzole is an approved ALS medication in widespread use Edaravone is an approved ALS medication in widespread use Riluzole and Edaravone are the only approved ALS medications in widespread use Riluzole provides modest benefits in only some ALS patients Edaravone provides modest benefits in only some ALS patients Figure 1: A complex excerpt from Mejzini et al. (2019) (top) and the set of valid claims that can be generated from the bolded sentence (c1-c6). by exploring how automatic generation of scien- tific claims can assist with dataset creation …

Content truncated to 500 words