HoVer: A Dataset for Many-Hop Fact Extraction And Claim Verification

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3441–3460 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 3441 HOVER: A Dataset for Many-Hop Fact Extraction And Claim Verification Yichen Jiang†∗ Shikha Bordia‡∗ Zheng Zhong‡ Charles Dognin‡ Maneesh Singh‡ Mohit Bansal† †UNC Chapel Hill ‡Verisk {shikha.bordia, zheng.zhong, charles.dognin, msingh}@verisk.com {yichenj, mbansal}@cs.unc.edu Abstract We introduce HOVER (HOppy VERification), a dataset for many-hop evidence extraction and fact verification. It challenges models to extract facts from several Wikipedia arti- cles that are relevant to a claim and classify whether the claim is SUPPORTED or NOT- SUPPORTED by the facts. In HOVER, the claims require evidence to be extracted from as many as four English Wikipedia articles and embody reasoning graphs of diverse shapes. Moreover, most of the 3/4-hop claims are writ- ten in multiple sentences, which adds to the complexity of understanding long-range de- pendency relations such as coreference. We show that the performance of an existing state- of-the-art semantic-matching model degrades significantly on our dataset as the number of reasoning hops increases, hence demonstrat- ing the necessity of many-hop reasoning to achieve strong results. We hope that the in- troduction of this challenging dataset and the accompanying evaluation task will encourage research in many-hop fact retrieval and infor- mation verification.1 1 Introduction The proliferation of social media platforms and digital content has been accompanied by a rise in deliberate disinformation and hoaxes, leading to polarized opinions among masses. With the in- creasing number of inexact statements, there is a large interest in a fact-checking system that can ver- ify claims based on automatically retrieved facts and evidence. FEVER (Thorne et al., 2018) is an open-domain fact extraction and verification dataset closely related to this real-world application. However, more than 87% of the claims in FEVER require information from a single Wikipedia article, ∗Equal contribution. 1We make HoVer dataset publicly available at https://hover-nlp.github.io while real-world “claims” might refer to informa- tion from multiple sources. QA datasets like HOT- POTQA (Yang et al., 2018) and QAngaroo (Welbl et al., 2018) represent the first efforts to challenge models to reason with information from three doc- uments at most. However, Chen and Durrett (2019) and Min et al. (2019) show that single-hop models can achieve good results in these multi-hop datasets. Moreover, most models were also shown to degrade in adversarial evaluation (Perez et al., 2020), where word-matching reasoning shortcuts are suppressed by extra adversarial documents (Jiang and Bansal, 2019). In the HOTPOTQA open-domain setting, the two supporting documents can be accurately retrieved by a neural model exploiting a single hy- perlink (Nie et al., 2019b; Asai et al., 2020). Hence, while providing very useful starting points for the community, FEVER is mostly re- stricted to a single-hop setting and existing multi- hop QA datasets are limited by the number of rea- soning steps and the word overlapping between the question and all evidence. An ideal multi-hop ex- ample should have at least one piece of evidence (supporting document) that cannot be retrieved …

Content truncated to 500 words