Stanceosaurus: Classifying Stance Towards Multicultural Misinformation

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 2132 - 2151 December 7-11, 2022 ©2022 Association for Computational Linguistics STANCEOSAURUS: Classifying Stance Towards Multicultural Misinformation Jonathan Zheng, Ashutosh Baheti, Tarek Naous, Wei Xu, Alan Ritter School of Interactive Computing Georgia Institute of Technology {jonathanqzheng, abaheti3, tareknaous}@gatech.edu; {wei.xu, alan.ritter}@cc.gatech.edu Abstract We present Stanceosaurus, a new corpus of 28,033 tweets in English, Hindi, and Arabic annotated with stance towards 251 misinfor- mation claims. As far as we are aware, it is the largest corpus annotated with stance to- wards misinformation claims. The claims in Stanceosaurus originate from 15 fact-checking sources that cover diverse geographical re- gions and cultures. Unlike existing stance datasets, we introduce a more fine-grained 5- class labeling strategy with additional subcat- egories to distinguish implicit stance. Pre- trained transformer-based stance classifiers that are fine-tuned on our corpus show good generalization on unseen claims and regional claims from countries outside the training data. Cross-lingual experiments demonstrate Stanceosaurus’ capability of training multi- lingual models, achieving 53.1 F1 on Hindi and 50.4 F1 on Arabic without any target- language fine-tuning. Finally, we show how a domain adaptation method can be used to improve performance on Stanceosaurus using additional RumourEval-2019 data. We make Stanceosaurus publicly available to the re- search community and hope it will encourage further work on misinformation identification across languages and cultures.1 1 Introduction The prevalence of misinformation on online social media has become an increasingly severe societal problem. A key language technology, which has the potential to help content moderators identify rapidly-spreading misinformation, is the automatic identification of both affective and epistemic stance (Jaffe, 2009; Zuczkowski et al., 2017) towards false claims. Progress on the problem of stance identi- fication has largely been driven by the availability of annotated corpora, such as RumourEval (Der- czynski et al., 2017; Gorrell et al., 2019). However, 1Our code and data are available at https://tinyurl. com/stanceosaurus ���������� Translation: Out of 16 priests of Tirupati temple, 128 kg gold, 150 crores cash, 77 crores diamonds were found from Income Tax raid at the house of one priest. �������� Translation: These are the priests of our country who have immense wealth. ����������� ����������������� �������� Figure 1: Example Hindi and English tweets in Stanceosaurus with stance towards the claim “Raid at Tirupati temple priest’s house, 128 kg gold found”. existing corpora mostly focus on misinformation spreading within western countries. In this paper, we present Stanceosaurus, a di- verse and high-quality corpus that builds on the best design choices made in previous misinfor- mation corpora, including RumourEval-2019 and CovidLies. Stanceosaurus covers more diverse top- ics, geographic regions, and cultures than prior work. It includes 28,033 tweets in English, Hindi, and Arabic that are manually annotated for stance (see Figure 1) towards 251 misinformation claims, collected from 15 independent fact-checking web- sites that cover India, Singapore, Australia, New Zealand, Canada, the United States, Europe, and the Arab World (the Levantine, Gulf, Northwest African regions, and Egypt). To the best of our knowledge, Stanceosaurus …

Content truncated to 500 words