DeSePtion: Dual Sequence Prediction and Adversarial Examples for Improved Fact-Checking

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8593–8606 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 8593 DeSePtion: Dual Sequence Prediction and Adversarial Examples for Improved Fact-Checking Christopher Hidey,1∗Tuhin Chakrabarty,1 Tariq Alhindi,1 Siddharth Varia,1 Kriste Krstovski,1,3 Mona Diab,4,5∗and Smaranda Muresan1,2 1Department of Computer Science, Columbia University 2Data Science Institute, Columbia University 3Columbia Business School, Columbia University 4Facebook AI 5Department of Computer Science, George Washington University {chidey, tariq, smara}@cs.columbia.edu, mtdiab@email.gwu.edu {tuhin.chakrabarty, sv2504, kriste.krstovski}@columbia.edu Abstract The increased focus on misinformation has spurred development of data and systems for detecting the veracity of a claim as well as retrieving authoritative evidence. The Fact Extraction and VERification (FEVER) dataset provides such a resource for evaluating end- to-end fact-checking, requiring retrieval of ev- idence from Wikipedia to validate a veracity prediction. We show that current systems for FEVER are vulnerable to three categories of realistic challenges for fact-checking – multi- ple propositions, temporal reasoning, and am- biguity and lexical variation – and introduce a resource with these types of claims. Then we present a system designed to be resilient to these “attacks” using multiple pointer net- works for document selection and jointly mod- eling a sequence of evidence sentences and ve- racity relation predictions. We find that in han- dling these attacks we obtain state-of-the-art results on FEVER, largely due to improved ev- idence retrieval. 1 Introduction The growing presence of biased, one-sided, and often altered discourse, is posing a challenge to our media platforms from newswire to social media (Vosoughi et al., 2018). To overcome this challenge, fact-checking has emerged as a necessary part of journalism, where experts examine ”check-worthy” claims (Hassan et al., 2017) published by others for their “shades” of truth (e.g., FactCheck.org or Poli- tiFact). However, this process is time-consuming, and thus building computational models for auto- matic fact-checking has become an active area of research (Graves, 2018). Advances were made pos- sible by new open source datasets and shared tasks: the Fact Extraction and Verification Shared Task (FEVER) 1.0 and 2.0 (Thorne et al., 2018; Thorne ∗Work completed in part at Amazon Claim: Murda Beatz′s real name is Marshall Mathers. Evidence: [Murda Beatz] Shane Lee Lindstrom (born February 11, 1994), known professionally as Murda Beatz, is a Canadian hip hop record producer and song- writer from Fort Erie, Ontario. Label: REFUTES Figure 1: Example from FEVER 1.0 Dataset and Vlachos, 2019), SemEval 2019 Shared Task 8: Fact-Checking in Community Forums (Mihaylova et al., 2019), and LIAR(+) datasets with claims from PolitiFact (Wang, 2017; Alhindi et al., 2018). The FEVER 1.0 shared task dataset (Thorne et al., 2018) has enabled the development of end- to-end fact-checking systems, requiring document retrieval and evidence sentence extraction to cor- roborate a veracity relation prediction (supports, refutes, not enough info). An example is given in Figure 1. Since the claims in FEVER 1.0 were man- ually written using information from Wikipedia, the dataset may lack linguistic challenges that oc- cur in verifying naturally occurring check-worthy claims, such …

Content truncated to 500 words