NewsClaims: A New Benchmark for Claim Detection from News with Attribute Knowledge

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 6002 - 6018 December 7-11, 2022 ©2022 Association for Computational Linguistics NEWSCLAIMS: A New Benchmark for Claim Detection from News with Attribute Knowledge Revanth Gangi Reddy1, Sai Chetan Chinthakindi1, Zhenhailong Wang1, Yi R. Fung1, Kathryn Conger2, Ahmed Elsayed2, Martha Palmer2, Preslav Nakov3, Eduard Hovy4,5, Kevin Small6, Heng Ji1 1UIUC 2CU Boulder 3MBZUAI 4DARPA 5CMU 6Amazon {revanth3,hengji}@illinois.edu Abstract Claim detection and verification are crucial for news understanding and have emerged as promising technologies for mitigating misinfor- mation and disinformation in the news. How- ever, most existing work has focused on claim sentence analysis while overlooking additional crucial attributes (e.g., the claimer and the main object associated with the claim). In this work, we present NEWSCLAIMS, a new benchmark for attribute-aware claim detection in the news domain. We extend the claim de- tection problem to include extraction of addi- tional attributes related to each claim and re- lease 889 claims annotated over 143 news arti- cles. NEWSCLAIMS1 aims to benchmark claim detection systems in emerging scenarios, com- prising unseen topics with little or no training data. To this end, we see that zero-shot and prompt-based baselines show promising perfor- mance on this benchmark, while still consider- ably behind human performance. 1 Introduction The internet era has ushered in an explosion of online content creation, resulting in increased con- cerns regarding misinformation in news, online debates, and social media. A key element of iden- tifying misinformation is detecting the claims and the arguments that have been presented. In this regard, news articles are particularly interesting as they contain claims in various formats: from ar- guments by journalists to reported statements by prominent public figures. Check-worthiness estimation aims to decide if a piece of text is worth fact-checking, i.e., whether it contains an important verifiable factual claim (Has- san et al., 2017a). Most current approaches (Jara- dat et al., 2018; Shaar et al., 2021) largely ignore relevant attributes of the claim (e.g., the claimer and the primary object associated with the claim). 1The code and data have been made publicly available here: https://github.com/blender-nlp/NewsClaims Figure 1: A news article containing a claim regarding the origin of COVID-19 with the claim sentence in ital- ics, the claim span in red, and the claimer in blue. Also shown are the claimer stance and the claim object. Moreover, current claim detection tasks mainly identify claims in debates (Gencheva et al., 2017), speeches (Atanasova et al., 2019a), and social me- dia (Nakov et al., 2022), where the claim source (i.e., the claimer) is known. News articles, on the other hand, have more com- plex arguments, requiring a deeper understanding of what each claim is about and identifying where it comes from. Thus, here we introduce the notion of claim object, which we define as an entity that identifies what is being claimed with respect to the topic of the claim. Figure 1 shows a claim about the origin of COVID-19, suggesting that the virus came from …

Content truncated to 500 words