ProoFVer: Natural Logic Theorem Proving for Fact Verification

ProoFVer: Natural Logic Theorem Proving for Fact Verification Amrith Krishna Department of Computer Science, University of Cambridge, United Kingdom ak2329@cam.ac.uk Sebastian Riedel Meta AI and University College London, United Kingdom sriedel@fb.com Andreas Vlachos Department of Computer Science, University of Cambridge, United Kingdom av308@cam.ac.uk Abstract Fact verification systems typically rely on neu- ral network classifiers for veracity prediction, which lack explainability. This paper proposes ProoFVer, which uses a seq2seq model to gen- erate natural logic-based inferences as proofs. These proofs consist of lexical mutations be- tween spans in the claim and the evidence retrieved, each marked with a natural logic operator. Claim veracity is determined solely based on the sequence of these operators. Hence, these proofs are faithful explanations, and this makes ProoFVer faithful by construc- tion. Currently, ProoFVer has the highest label accuracy and the second best score in the FEVER leaderboard. Furthermore, it improves by 13.21% points over the next best model on a dataset with counterfactual instances, demonstrating its robustness. As explanations, the proofs show better overlap with human rationales than attention-based highlights and the proofs help humans predict model de- cisions correctly more often than using the evidence directly.1 1 Introduction Fact verification systems typically comprise an evidence retrieval model followed by a textual en- tailment classifier (Thorne et al., 2018b). Recent high-performing fact verification systems (Zhong et al., 2020; Ye et al., 2020) use neural models for textual entailment whose reasoning is opaque to humans despite advances in interpretablity (Han et al., 2020). On the other hand, proof sys- tems like NaturalLI (Angeli and Manning, 2014) provide transparency in their decision making for entailment tasks, by using explicit proofs in the form of natural logic. However, the accuracy of such approaches often does not match that of neural models (Abzianidze, 2017a). 1Find our code and data at https://github.com /krishnamrith12/ProoFVer. Justifying decisions is central to fact verifica- tion (Uscinski and Butler, 2013). While models such as those developed for FEVER (Thorne et al., 2018b) typically substantiate their decisions by presenting the evidence as is, more recent proposals use the evidence to generate explana- tions. Here, models highlight salient parts of the evidence (Popat et al., 2018; Wu et al., 2020), generate summaries (Kotonya and Toni, 2020b; Atanasova et al., 2020), correct factual errors (Thorne and Vlachos, 2021b; Schuster et al., 2021), answer claim-related questions (Fan et al., 2020), or perform rule discovery (Ahmadi et al., 2019; Gad-Elrab et al., 2019). An explanation is faith- ful only if it reflects the information that is used for decision making (Lipton, 2018; Jacovi and Goldberg, 2020), which these systems do not guarantee. A possible exception here would be the rule discovery models, although, their per- formance often suffers due to limited knowledge base coverage and/or the noise in rule extraction from text (Kotonya and Toni, 2020a; Pezeshkpour et al., 2020). Faithful explanations are useful as mechanisms to dispute, debug, or advise (Jacovi and Goldberg, 2021), which may aid a news agency for advice, a user to dispute decisions, and a …

Content truncated to 500 words