Improving Evidence Retrieval with Claim-Evidence Entailment

Proceedings of Recent Advances in Natural Language Processing, pages 1553–1558 Sep 1–3, 2021. https://doi.org/10.26615/978-954-452-072-4_174 1553 Improving Evidence Retrieval with Claim-Evidence Entailment Fan Yang∗ Amazon Alexa Cambridge, MA, USA fyaamz@amazon.com Eduard Dragut Temple University Philadelphia, PA, USA edragut@temple.edu Arjun Mukherjee University of Houston Houston, TX, USA arjun4787@gmail.com Abstract Claim verification is challenging because it re- quires first to find textual evidence and then apply claim-evidence entailment to verify a claim. Previous works evaluate the entailment step based on the retrieved evidence, whereas we hypothesize that the entailment prediction can provide useful signals for evidence re- trieval, in the sense that if a sentence supports or refutes a claim, the sentence must be rele- vant. We propose a novel model that uses the entailment score to express the relevancy. Our experiments verify that leveraging entailment prediction improves ranking multiple pieces of evidence. 1 Introduction Claim verification verifies the credibility of a tex- tual claim by inferring relevant and reliable textual evidence. An example in this space is FEVER, which regards Wiki pages as potential evidence and creates claims by crowdsourcing (Thorne et al., 2018). They propose a three-step pipeline: (i) doc- ument-level evidence retrieval; (ii) sentence-level evidence retrieval; (iii) claim-evidence entailment. Some works follow the pipeline and propose new models to improve claim verification (Yoneda et al., 2018; Nie et al., 2019; Hanselowski et al., 2018; Zhou et al., 2019), while other works combine the second and the third step and leverage all possi- ble sentences for claim verification (Yin and Roth, 2018; Ma et al., 2019). We refer to the former as the pipeline framework and the latter as the multi-task framework. The pipeline framework restricts a few sentences, and therefore it may not cover relevant evidence. The multi-task framework includes all possible sentences, where irrelevant ones may bring over- whelming noise and hurt claim verification. We ∗The paper was done before the author joined Ama- zon.com Inc. Claim: Andy Roddick lost 5 Master Series between 2002 and 2010. Candidate: Roddick was ranked in the top 10 for nine consecutive years between 2002 and 2010, and won five Masters Series in that period. Label: REFUTE Candidate: Roddick’s hard-court record in 2003 included his first Masters Series titles coming at Canada and Cincinnati and his only Grand Slam title. Label: NOT ENOUGH INFO Candidate: Federer won his first Master Se- ries event at the 2002 Hamburg Masters on clay, over Marat Safin. Label: NOT ENOUGH INFO Table 1: The entailment result can imply whether the candidate is relevant or not. argue that previous works focus on improving in- dividual components but neglect to examine how those components connect. For example, will the entailment improve the retrieval? We hypothesize that claim-evidence entailment can provide useful signals for evidence retrieval: if a sentence supports or refutes a claim, the sentence must be relevant. As in Table 1, the first candidate (actual evidence) shares more words and longer phrases than the other candidates. In contrast, the other two candidates may be relevant to the claim …

Content truncated to 500 words