Adapting Open Domain Fact Extraction and Verification to COVID-FACT through In-Domain Language Modeling

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2395–2400 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 2395 Adapting Open Domain Fact Extraction and Verification to COVID-FACT through In-Domain Language Modeling Zhenghao Liu1,2, Chenyan Xiong5, Zhuyun Dai6, Si Sun4, Maosong Sun1,3, Zhiyuan Liu1,3 1Department of Computer Science and Technology, Tsinghua University, Beijing, China Institute for Artificial Intelligence, Tsinghua University, Beijing, China Beijing National Research Center for Information Science and Technology 2State Key Lab on Intelligent Technology and Systems, Tsinghua University, Beijing, China 3Beijing Academy of Artificial Intelligence 4Department of Electronic Engineering, Tsinghua University, Beijing, China 5Microsoft Research, Redmond, USA 6Carnegie Mellon University, USA Abstract With the epidemic of COVID-19, verifying the scientifically false online information, such as fake news and maliciously fabricated state- ments, has become crucial. However, the lack of training data in the scientific domain lim- its the performance of fact verification models. This paper proposes an in-domain language modeling method for fact extraction and verifi- cation systems. We come up with SciKGAT to combine the advantages of open-domain literature search, state-of-the-art fact verifica- tion systems and in-domain medical knowl- edge through language modeling. Our experi- ments on SCIFACT, a dataset of expert-written scientific fact verification, show that SciKGAT achieves 30% absolute improvement on preci- sion. Our analyses show that such improve- ment thrives from our in-domain language model by picking up more related evidence pieces and accurate fact verification. Our codes and data are released via Github1. 1 Introduction Online contents with false information, such as lies, rumors and conspiracy theories, have been grow- ing significantly and spreading widely during the COVID-19 epidemic. An automatic fact-checking system is urgently needed to check these scientific claims, which can avoid undesired consequences. Automatic fact-checking has drawn lots of attention from NLP community. Researchers mainly focus on stopping misinformation transmission through videos and texts (Cinelli et al., 2020; Hossain et al., 2020; Li et al., 2020; Serrano et al., 2020). The scientific fact verification task (Wadden et al., 2020) is come up to deal with COVID-FACT with high-quality articles of spanning domains from basic science to clinical medicine. Neverthe- less, the small-scale training data of SCIFACT may 1https://github.com/thunlp/KernelGAT limit the performance of COVID-FACT checking. The state-of-the-art model (Wadden et al., 2020) achieves only 46.6% precision of fact verification, which is hard to be trusted for users. This paper presents the Scientific KGAT (SciK- GAT) to deal with low-resource COVID-FACT ver- ification. SciKGAT employs the in-domain lan- guage model in the fact extraction and verification pipeline (Thorne et al., 2018; Wadden et al., 2020) to adapt fact-checking into COVID domain. The in-domain language model transfers COVID do- main knowledge into pre-trained language models with continuous training and learns medical token semantics towards COVID with mask language model based training. The state-of-the-art fact ver- ification model KGAT (Liu et al., 2020; Ye et al., 2020) is also used in SciKGAT for multi-evidence reasoning in the fact verification module. Our experiments show that the in-domain lan- guage modelings achieve …

Content truncated to 500 words