Multi2Claim: Generating Scientific Claims from Multi-Choice Questions for Scientific Fact-Checking

Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 2652–2664 May 2-6, 2023 ©2023 Association for Computational Linguistics Multi2Claim: Generating Scientific Claims from Multi-Choice Questions for Scientific Fact-Checking Ne¸set Özkan TAN Trung Nguyen Joshua Bensemann Alex Yuxuan Peng Qiming Bao Yang Chen Mark Gahegan Michael Witbrock The School of Computer Science University of Auckland Auckland, New Zealand {neset.tan, trung.nguyen, josh.bensemann, alex.peng}@auckland.ac.nz {qiming.bao, yang.chen, m.gahegan, m.witbrock}@auckland.ac.nz Abstract In automated scientific fact-checking, machine learning models are trained to verify scientific claims given evidence. A major bottleneck of this task is the availability of large-scale train- ing datasets on different domains, due to the required domain expertise for data annotation. However, multiple-choice question-answering datasets are readily available across many dif- ferent domains, thanks to the modern online ed- ucation and assessment systems. As one of the first steps towards addressing the fact-checking dataset scarcity problem in scientific domains, we propose a pipeline for automatically con- verting multiple-choice questions into fact- checking data, which we call Multi2Claim. By applying the proposed pipeline, we gen- erated two large-scale datasets for scientific- fact-checking: Med-Fact and Gsci-Fact for the medical and general science domains, re- spectively. These two datasets are among the first examples of large-scale scientific-fact- checking datasets. We developed baseline mod- els for the verdict prediction task using each dataset. Additionally, we demonstrated that the datasets could be used to improve per- formance measured by weighted F1 on ex- isting fact-checking datasets such as SciFact, HEALTHVER, COVID-Fact, and CLIMATE- FEVER. In some cases, the improvement in performance was up to a 26% increase. The generated datasets are publicly available1. 1 Introduction Learning to verify the claims in scientific papers and “science releases” (media announcements of scientific findings) is a difficult task for both artifi- cial intelligence (AI) systems and humans. How- ever, this task is crucial because learning to separate verified facts from speculation or falsehoods has im- portant consequences. Success at this task can help the reader understand scientific topics and promote science. Conversely, failure at this task leads to the 1https://github.com/taneset/Multi2Claim. Figure 1: Examples of “supported” (green), “refuted” (red), and “not-enough-info” (yellow) types of claims from the Gsci-Fact dataset which is generated from the original multi-choice question (grey). spread of misinformation and exaggeration, which can cause distortion in scientific communication and undermine public confidence in science. Unfor- tunately, several studies have revealed that science releases and scientific articles can contain signifi- cant exaggeration and misinformation (Woloshin et al., 2009), (West and Bergstrom, 2021), (Sum- ner et al., 2014), (Woloshin and Schwartz, 2002). This misleading information can directly impact people’s lives, as was the case for media releases concerning the COVID-19 pandemic (Roozenbeek et al., 2020). In 2014, some estimates claimed that 40% of the press releases contained exaggerated advice and 33% contained exaggerated claims in science-related news (Sumner et al., 2014). Con- sidering that this problem has not disappeared over the last eight years and has possibly been exasper- ated by the continued growth of social …

Content truncated to 500 words