FaVIQ: FAct Verification from Information-seeking Questions

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 5154 - 5166 May 22-27, 2022 c⃝2022 Association for Computational Linguistics FAVIQ: FAct Verification from Information-seeking Questions Jungsoo Park1∗ Sewon Min2∗ Jaewoo Kang1 Luke Zettlemoyer2 Hannaneh Hajishirzi2,3 1Korea University 2University of Washington 3Allen Institute of AI {jungsoo_park,kangj}@korea.ac.kr {sewon,lsz,hannaneh}@cs.washington.edu Abstract Despite significant interest in developing gen- eral purpose fact checking models, it is chal- lenging to construct a large-scale fact verifi- cation dataset with realistic real-world claims. Existing claims are either authored by crowd- workers, thereby introducing subtle biases that are difficult to control for, or manually verified by professional fact checkers, causing them to be expensive and limited in scale. In this paper, we construct a large-scale challenging fact ver- ification dataset called FAVIQ, consisting of 188k claims derived from an existing corpus of ambiguous information-seeking questions. The ambiguities in the questions enable auto- matically constructing true and false claims that reflect user confusions (e.g., the year of the movie being filmed vs. being released). Claims in FAVIQ are verified to be natural, contain little lexical bias, and require a com- plete understanding of the evidence for verifi- cation. Our experiments show that the state- of-the-art models are far from solving our new task. Moreover, training on our data helps in professional fact-checking, outperforming models trained on the widely used dataset FEVER or in-domain data by up to 17% abso- lute. Altogether, our data will serve as a chal- lenging benchmark for natural language under- standing and support future progress in profes- sional fact checking.1 1 Introduction Fact verification, the task of verifying the factuality of the natural language claim, is an important NLP application (Cohen et al., 2011) and has also been used to evaluate the amount of external knowledge a model has learned (Petroni et al., 2021). How- ever, it is challenging to construct fact verification data with claims that contain realistic and implicit misinformation. Crowdsourced claims from prior work such as FEVER (Thorne et al., 2018a) are ∗Equal Contribution 1Data available at https://faviq.github.io. Disambiguations: Does created mean released? → answer: 2001 Does created mean filmed? → answer: 2000 Claim: The first ‘fast and furious’ film was filmed in 2001. Evidence: The Fast and the Furious is a 2001 action film (...) filming ended in October 2000. FaVIQ ✔ ✔ QA Data Label (verdict): refute Question: When was the first ‘fast and furious’ created? Figure 1: An example of a refute claim on FAVIQ, constructed using ambiguity in the information-seeking question, e.g., through a crossover of the year of the film being released and being filmed. written with minimal edits to reference sentences, leading to strong lexical biases such as the overuse of explicit negation and unrealistic misinformation that is less likely to occur in real life (Schuster et al., 2019). On the other hand, data constructed by professional fact-checkers are expensive and are typically small-scale (Hanselowski et al., 2019). In this paper, we show it is possible to use information-seeking …

Content truncated to 500 words