Natural Logic-guided Autoregressive Multi-hop Document Retrieval for Fact Verification

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 6123 - 6135 December 7-11, 2022 ©2022 Association for Computational Linguistics Natural Logic-guided Autoregressive Multi-hop Document Retrieval for Fact Verification Rami Aly University of Cambridge Department of Computer Science and Technology rami.aly@cl.cam.ac.uk Andreas Vlachos University of Cambridge Department of Computer Science and Technology andreas.vlachos@cl.cam.ac.uk Abstract A key component of fact verification is the evi- dence retrieval, often from multiple documents. Recent approaches use dense representations and condition the retrieval of each document on the previously retrieved ones. The latter step is performed over all the documents in the collection, requiring storing their dense repre- sentations in an index, thus incurring a high memory footprint. An alternative paradigm is retrieve-and-rerank, where documents are retrieved using methods such as BM25, their sentences are reranked, and further documents are retrieved conditioned on these sentences, reducing the memory requirements. However, such approaches can be brittle as they rely on heuristics and assume hyperlinks between docu- ments. We propose a novel retrieve-and-rerank method for multi-hop retrieval, that consists of a retriever that jointly scores documents in the knowledge source and sentences from pre- viously retrieved documents using an autore- gressive formulation and is guided by a proof system based on natural logic that dynamically terminates the retrieval process if the evidence is deemed sufficient. This method is compet- itive with current state-of-the-art methods on FEVER, HoVer and FEVEROUS-S, while us- ing 5 to 10 times less memory than competing systems. Evaluation on an adversarial dataset indicates improved stability of our approach compared to commonly deployed threshold- based methods. Finally, the proof system helps humans predict model decisions correctly more often than using the evidence alone. 1 Introduction With the growing volume of potentially misleading and false claims (Graves, 2018), automated fact verification (Hardalov et al., 2022; Guo et al., 2022) is of growing interest. A key component of open- domain fact verification systems is the retrieval of relevant documents from a knowledge base (KB) which provide the necessary evidence supporting Claim: The 66th Primetime Emmy Awards was hosted by an Iraqi comedian born in 1973. Evidence Documents: 66th Primetime Emmy Awards The 66th Primetime Emmy Awards honored the best in U.S. prime time television programming from June 1, 2013 until May 31, 2014, as chosen by the Academy of Television Arts & Sciences. Comedian and Late Night host Seth Meyers hosted the ceremony for the first time. Seth Meyers Seth Adam Meyers (born December 28 , 1973) is an American comedian , writer , producer , political commentator , actor , media critic , and television host. He hosts Late Night with Seth Meyers, a late-night talk show on NBC. Prior to that, he was a cast member and head writer for NBC’s Saturday Night Live (2001–2014). Verdict: Refuted Figure 1: A FEVER example where multiple documents are required for verification (relevant evidence in red). or refuting a claim. Evidence retrieval accuracy correlates strongly with fact-checking accuracy, as observed in a recent …

Content truncated to 500 words