PASTA: Table-Operations Aware Fact Verification via Sentence-Table Cloze Pre-training

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 4971 - 4983 December 7-11, 2022 ©2022 Association for Computational Linguistics PASTA: Table-Operations Aware Fact Verification via Sentence-Table Cloze Pre-training Zihui Gu1, Ju Fan1, Nan Tang2, Preslav Nakov3, Xiaoman Zhao1∗, Xiaoyong Du1 1Renmin University of China, Beijing, China, 2Qatar Computing Research Institute, HBKU, Doha, Qatar 3Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, UAE 1{guzh,fanj,xiaomanzhao,duyong}@ruc.edu.cn, 2ntang@hbku.edu.qa, 3preslav.nakov@mbzuai.ac.ae Abstract Fact verification has attracted a lot of research attention recently, e.g., in journalism, market- ing, and policymaking, as misinformation and disinformation online can sway one’s opinion and affect one’s actions. While fact-checking is a hard task in general, in many cases, false statements can be easily debunked based on analytics over tables with reliable informa- tion. Hence, table-based fact verification has recently emerged as an important and growing research area. Yet, progress has been limited due to the lack of datasets that can be used to pre-train language models (LMs) to be aware of common table operations, such as aggregating a column or comparing tuples. To bridge this gap, in this paper we introduce PASTA, a novel state-of-the-art framework for table-based fact verification via pre-training with synthesized sentence–table cloze questions. In particular, we design six types of common sentence–table cloze tasks, including Filter, Aggregation, Superlative, Comparative, Ordinal, and Unique, based on which we synthesize a large corpus consisting of 1.2 million sentence–table pairs from WikiTables. PASTA uses a recent pre-trained LM, DeBERTaV3, and further pre- trains it on our corpus. Our experimental re- sults show that PASTA achieves new state-of- the-art performance on two table-based fact verification benchmarks: TabFact and SEM- TAB-FACTS. In particular, on the complex set of TabFact, which contains multiple operations, PASTA largely outperforms the previous state of the art by 4.7 points (85.6% vs. 80.9%), and the gap between PASTA and human perfor- mance on the small TabFact test set is narrowed to just 1.5 points (90.6% vs. 92.1%). 1 1 Introduction Fact verification, which checks the factuality of a statement, is crucial for journalism (Shu et al., ∗Xiaoman Zhao is the corresponding author. 1The pre-trained model, the pre-training corpus, and the source code are released at https://github.com/ruc-datalab/PASTA 2017), and is increasingly being applied in other fields (Ott et al., 2011; Yoon et al., 2019). Accord- ing to Duke Reporters’ Lab, there are 300+ active certified fact-checking organizations worldwide.2 Automatic and explainable approaches, a.k.a. reference-based approaches, are widely used to as- sist fact-checkers. They verify the input statement against a trusted source, such as relevant passages from Wikipedia (Popat et al., 2017; Thorne et al., 2018; Shaar et al., 2020). Recently, table-based fact verification has been extensively studied (Chen et al., 2020a; Zhong et al., 2020; Eisenschlos et al., 2020) due to the wide availability of tabular data. Evidently, performing fact verification over ta- bles requires the ability to reason about table- based operations, such as aggregating the values in a column or comparing tuples. For example, for the statement S1 in Figure 1, …

Content truncated to 500 words