Joint Verification and Reranking for Open Fact Checking Over Tables

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 6787–6799 August 1–6, 2021. ©2021 Association for Computational Linguistics 6787 Joint Verification and Reranking for Open Fact Checking Over Tables Michael Schlichtkrull1∗, Vladimir Karpukhin2, Barlas O˘guz2, Mike Lewis2, Wen-tau Yih2, Sebastian Riedel2,3 1University of Cambridge, 2Facebook AI Research, 3University College London michael.schlichtkrull@cst.cam.ac.uk, {vladk, barlaso, mikelewis, scottyih, sriedel}@fb.com Abstract Structured information is an important knowl- edge source for automatic verification of factual claims. Nevertheless, the majority of existing research into this task has fo- cused on textual data, and the few recent in- quiries into structured data have been for the closed-domain setting where appropriate evi- dence for each claim is assumed to have al- ready been retrieved. In this paper, we in- vestigate verification over structured data in the open-domain setting, introducing a joint reranking-and-verification model which fuses evidence documents in the verification compo- nent. Our open-domain model achieves perfor- mance comparable to the closed-domain state- of-the-art on the TabFact dataset, and demon- strates performance gains from the inclusion of multiple tables as well as a significant im- provement over a heuristic retrieval baseline. 1 Introduction Verifying whether a given fact coheres with a trusted body of knowledge is a fundamental prob- lem in NLP, with important applications to auto- mated fact checking (Vlachos and Riedel, 2014) and other tasks in computational journalism (Cohen et al., 2011; Flew et al., 2012). Despite extensive investigation of the problem under different con- ditions including entailment and natural language inference (Dagan et al., 2005; Bowman et al., 2015) as well as claim verification (Vlachos and Riedel, 2014; Alhindi et al., 2018; Thorne and Vlachos, 2018), relatively little attention has been devoted to the setting where the trusted body of evidence is structured in nature — that is, where it consists of tabular or graph-structured data. Recently, two datasets were introduced for claim verification over tables (Chen et al., 2020b; Gupta et al., 2020). In both datasets, claims can be verified ∗Work done while interning with Facebook AI Research. The Daily Express and the Sunday Mirror are owned by the same company. True Title Established Parent Company Daily Mail 1896 DMGT Mail on Sunday 1982 DMGT ... ... ... Daily Express 1900 Reach Sunday Mirror 1915 Reach Sunday People 1881 Reach Title 2019 Election party support Daily Mail Conservative Party Mail on Sunday Conservative Party ... ... The Sun Conservative Party Daily Mirror Labour Party Sunday Mirror Labour Party Figure 1: Example query to be evaluated against two re- trieved tables. Named entities represent a strong base- line for retrieval, but ultimately a more complex model is required to distinguish highly similar tables. given a single associated table. While highly useful for the development of models, this closed setting is not reflective of real-world fact checking tasks where it is usually not known which table to consult for evidence. Realistic systems must first retrieve evidence from a large data source. That …

Content truncated to 500 words