“Who said it, and Why?” Provenance for Natural Language Claims

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4416–4426 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 4416 “Who said it, and Why?” Provenance for Natural Language Claims Yi Zhang Zachary G. Ives Department of Computer and Information Science University of Pennsylvania {yizhang5, zives, danroth}@cis.upenn.edu Dan Roth Abstract In an era where generating content and pub- lishing it is so easy, we are bombarded with information and are exposed to all kinds of claims, some of which do not always rank high on the truth scale. This paper suggests that the key to a longer-term, holistic, and systematic approach to navigating this information pol- lution is capturing the provenance of claims. To do that, we develop a formal definition of provenance graph for a given natural language claim, aiming to understand where the claim may come from and how it has evolved. To construct the graph, we model provenance in- ference, formulated mainly as an information extraction task and addressed via a textual en- tailment model. We evaluate our approach using two benchmark datasets, showing ini- tial success in capturing the notion of prove- nance and its effectiveness on the application of claim verification. 1 Introduction Never before have humans been able to generate and disseminate content so easily, leading to a con- tamination of information supply with irrelevant, redundant, unsolicited, and often low-value infor- mation (Orman, 1984). While significant atten- tion has been devoted recently to identifying false claims, the age of “information pollution” we live in calls for the development of additional impor- tant insights. At the heart of these insights is the need to determine the provenance of claims — who first made a given claim, and how an original claim developed and changed over time (and potentially across contributors). Consider the following claim: “Facebook soon plans to charge fees to users of the social net- work.”1 As shown in Figure 1, a typical modern 1https://www.snopes.com/fact-check/ facebook-implementing-user-fees/ https://www.welivesecurity.com … are feverishly sharing a "news report" claiming that from November 1st you'll be paying $2.99 every month to access the site … https://www.snopes.com/ … "It is official: Facebook will start charging due to the new profile changes." … … A hoax about Facebook charging fees for using their site is circulating the internet once again … https://wtkr.com/ Facebook soon plans to charge fees to users of the social network. Query Claim: textually entailed textually entailed contradicted 𝒇 Real/Fake/ Undetermined Evidence1 Evidence2 Evidence3 Label: Figure 1: A typical claim verification pipeline of our running example. It starts with searching for evidence, and feed it to a textual entailment model f to decide the claim’s veracity. claim verification pipeline starts with searching for existing evidence for the given query claim, and then leverages textual entailment models to de- termine the veracity of the claim relative to the evidence (Thorne et al., 2018). However, sites such as snopes.com and other fact-checking web- sites will not only provide their conclusion about the veracity of the …

Content truncated to 500 words