Claim Matching Beyond English to Scale Global Fact-Checking

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 4504–4517 August 1–6, 2021. ©2021 Association for Computational Linguistics 4504 Claim Matching Beyond English to Scale Global Fact-Checking Ashkan Kazemi University of Michigan and Meedan ashkank@umich.edu Kiran Garimella MIT garimell@mit.edu Devin Gaffney Meedan devin@meedan.com Scott A. Hale University of Oxford and Meedan scott@meedan.com Abstract Manual fact-checking does not scale well to serve the needs of the internet. This issue is further compounded in non-English contexts. In this paper, we discuss claim matching as a possible solution to scale fact-checking. We define claim matching as the task of iden- tifying pairs of textual messages containing claims that can be served with one fact-check. We construct a novel dataset of WhatsApp tipline and public group messages alongside fact-checked claims that are first annotated for containing “claim-like statements” and then matched with potentially similar items and annotated for claim matching. Our dataset contains content in high-resource (English, Hindi) and lower-resource (Bengali, Malay- alam, Tamil) languages. We train our own em- bedding model using knowledge distillation and a high-quality “teacher” model in order to address the imbalance in embedding quality between the low- and high-resource languages in our dataset. We provide evaluations on the performance of our solution and compare with baselines and existing state-of-the-art multilin- gual embedding models, namely LASER and LaBSE. We demonstrate that our performance exceeds LASER and LaBSE in all settings. We release our annotated datasets1, codebooks, and trained embedding model2 to allow for fur- ther research. 1 Introduction Human fact-checking is high-quality but time- consuming. Given the effort that goes into fact- checking a piece of content, it is desirable that a fact-check be easily matched with any content to which it applies. It is also necessary for fact- checkers to prioritize content for fact-checking 1https://doi.org/10.5281/zenodo. 4890949 2https://huggingface.co/meedan/ indian-xlm-r since there is not enough time to fact-check every- thing. In practice, there are many factors that affect whether a message is ‘fact-check worthy’ (Kon- stantinovskiy et al., 2020; Hassan et al., 2017), but one important factor is prevalence. Fact-checkers often want to check claims that currently have high viewership and avoid fact-checking ‘fringe’ claims as a fact-check could bring more attention to the claims—an understudied process known as ampli- fication (Phillips, 2018; Wardle, 2018). While the number of exact duplicates and shares of a message can be used as a proxy for popularity, discovering and grouping together multiple messages making the same claims in different ways can give a more accurate view of prevalence. Such algorithms are also important for serving relevant fact-checks via ‘misinformation tiplines’ on WhatsApp and other platforms (Wardle et al., 2019; Meedan, 2019; Ma- gall´on Rosa, 2019). Identifying pairs of textual messages containing claims that can be served with one fact-check is a potential solution to these issues. The ability to group claim-matched textual content in different languages would enable fact-checking organiza- tions around the globe to prioritize and scale up their …

Content truncated to 500 words