WatClaimCheck: A new Dataset for Claim Entailment and Inference

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 1293 - 1304 May 22-27, 2022 c⃝2022 Association for Computational Linguistics WatClaimCheck: A new Dataset for Claim Entailment and Inference Kashif Khan Computer Science University of Waterloo Ontario, Canada k2@iwrk.com Ruizhe Wang Computer Science University of Waterloo Ontario, Canada r322wang@uwaterloo.ca Pascal Poupart Computer Science University of Waterloo Vector Institute Ontario, Canada ppoupart@uwaterloo.ca Abstract We contribute a new dataset1 for the task of automated fact checking and an evaluation of state of the art algorithms. The dataset in- cludes claims (from speeches, interviews, so- cial media and news articles), review articles published by professional fact checkers and premise articles used by those professional fact checkers to support their review and verify the veracity of the claims. An important chal- lenge in the use of premise articles is the iden- tification of relevant passages that will help to infer the veracity of a claim. We show that transferring a dense passage retrieval model trained with review articles improves the re- trieval quality of passages in premise articles. We report results for the prediction of claim veracity by inference from premise articles. 1 Introduction The rise of social media has led to a democratiza- tion of news, but it has also amplified issues related to fake news and misinformation. To that effect, many fact checking organizations (e.g., Politifact, Snopes, AFP Fact Check, Alt News, FactCheck.org, Africa Check, etc.) have emerged around the globe. They investigate debatable claims made by author- ities, politicians, celebrities and the public. For each claim, they publish a review article with links to sources that support a verdict (e.g., true, partly true/false, false) about the veracity of the claim. Those reviews debunk false claims and mitigate the spread of misinformation. We consider a key NLP challenge in the context of automated fact check- ing: claim inference from premise articles. Note that determining the veracity of a claim without additional information is nearly impossible since claims are selected by professional fact checkers 1Code and form to request the dataset are available at https://github.com/nxii/WatClaimCheck. The WatClaimCheck dataset is available upon request for non- commercial research purposes only under the fair dealing exception of the Canada Copyright Act. in part because their veracity is far from obvious and also because of their degree of controversy. To that effect, professional fact checkers invest a fair amount of time to research each claim by finding relevant sources and publishing a review article that explains their verdict of the claim. Hence there is a natural entailment problem, whereby anyone who reads a review article should be able to arrive at the same verdict as the professional fact checker regarding the claim. Unlike many entailment tasks that consist of short text (e.g., pairs of utterances) that may be artificially generated or extracted, this is a natural and challenging entailment task that involves an entire document (review article) with an utterance (claim) that requires a certain degree of reading comprehension. …

Content truncated to 500 words