CrowdChecked: Detecting Previously Fact-Checked Claims in Social Media

Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 266–285 November 20–23, 2022. ©2022 Association for Computational Linguistics 266 CrowdChecked: Detecting Previously Fact-Checked Claims in Social Media Momchil Hardalov1 Anton Chernyavskiy2 Ivan Koychev1 Dmitry Ilvovsky2 Preslav Nakov3 1Sofia University “St. Kliment Ohridski”, Bulgaria 2HSE University, Russia 3Mohamed bin Zayed University of Artificial Intelligence, UAE {hardalov, koychev}@fmi.uni-sofia.bg {acherniavskii, dilvovsky}@hse.ru preslav.nakov@mbzuai.ac.ae Abstract While there has been substantial progress in developing systems to automate fact-checking, they still lack credibility in the eyes of the users. Thus, an interesting approach has emerged: to perform automatic fact-checking by verifying whether an input claim has been previously fact-checked by professional fact-checkers and to return back an article that explains their de- cision. This is a sensible approach as people trust manual fact-checking, and as many claims are repeated multiple times. Yet, a major issue when building such systems is the small num- ber of known tweet–verifying article pairs avail- able for training. Here, we aim to bridge this gap by making use of crowd fact-checking, i.e., mining claims in social media for which users have responded with a link to a fact-checking article. In particular, we mine a large-scale collection of 330,000 tweets paired with a cor- responding fact-checking article. We further propose an end-to-end framework to learn from this noisy data based on modified self-adaptive training, in a distant supervision scenario. Our experiments on the CLEF’21 CheckThat! test set show improvements over the state of the art by two points absolute. Our code and datasets are available at https://github.com/mhardalov/ crowdchecked-claims 1 Introduction The massive spread of disinformation online, es- pecially in social media, was counter-acted by ma- jor efforts to limit the impact of false information not only by journalists and fact-checking orga- nizations but also by governments, private com- panies, researchers, and ordinary Internet users. This includes building systems for automatic fact- checking (Zubiaga et al., 2016; Derczynski et al., 2017; Nakov et al., 2021a; Gu et al., 2022; Guo et al., 2022; Hardalov et al., 2022), fake news (Fer- reira and Vlachos, 2016; Nguyen et al., 2022), and fake news website detection (Baly et al., 2020; Ste- fanov et al., 2020; Panayotov et al., 2022). Figure 1: Crowd fact-checking thread on Twitter. The first tweet (Post w/ claim) makes the claim that Iver- mectin causes sterility in men, which then receives replies. A (crowd) fact-checker replies with a link to a verifying article from a fact-checking website. We pair the article with the tweet that made this claim (the first post ✓), as it is irrelevant (✗) to the other replies. Unfortunately, fully automatic systems still lack credibility, and thus it was proposed to focus on detecting previously fact-checked claims instead: Given a user comment, detect whether the claim it makes was previously fact-checked with respect to a collection of verified claims and their cor- responding articles (see Table 1). This task …

Content truncated to 500 words