The ComMA Dataset V0.2: Annotating Aggression and Bias in Multilingual Social Media Discourse

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 4149–4161 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 4149 The ComMA Dataset V0.2: Annotating Aggression and Bias in Multilingual Social Media Discourse Ritesh Kumar1, Shyam Ratan1, Siddharth Singh1, Enakshi Nandi2, Laishram Niranjana Devi2, Akash Bhagat3, Yogesh Dawer1 Bornini Lahiri3, Akanksha Bansal2, Atul Kr. Ojha4,2 1Dr Bhimrao Ambedkar University, 2Panlingua Language Processing LLP, 3Indian Institute of Technology-Kharagpur 4DSI, National University of Ireland Galway New Delhi, Agra, Kharagpur, Galway comma.kmi@gmail.com Abstract In this paper, we discuss the development of a multilingual dataset annotated with a hierarchical, fine-grained tagset marking different types of aggression and the “context” in which they occur. The context, here, is defined by the conversational thread in which a specific comment occurs and also the “type” of discursive role that the comment is performing with respect to the previous comment. The initial dataset, being discussed here consists of a total 59,152 annotated comments in four languages - Meitei, Bangla, Hindi, and Indian English - collected from various social media platforms such as YouTube, Facebook, Twitter and Telegram. As is usual on social media websites, a large number of these comments are multilingual, mostly code-mixed with English. The paper gives a detailed description of the tagset being used for annotation and also the process of developing a multi-label, fine-grained tagset that has been used for marking comments with aggression and bias of various kinds including sexism (called gender bias in the tagset), religious intolerance (called communal bias in the tagset), class/caste bias and ethnic/racial bias. We also define and discuss the tags that have been used for marking the different discursive role being performed through the comments, such as attack, defend, etc. Finally we present a basic statistical analysis of the dataset. The dataset is being incrementally made publicly available on the project website . Keywords: aggression, bias, Meitei, Bangla, Hindi, Tagset 1. Introduction Aggression, bias, polarisation and hate are now com- monplace phenomena on all kinds of social media plat- forms. And so are the research efforts to automati- cally identify these and process them in some mean- ingful way so as to reduce their harmful impact on so- cial communication and society, in general. Two recent systematic surveys (Vidgen and Derczynski, 2020) and (Poletto et al., 2021) have shown that over 60 datasets annotated with different aspects of hateful and abusive speech have been developed in various languages of the world in just around half a decade. The wide range of interrelated phenomena that has been used for annotating the datasets include broad dimensions such as abusive language (Nobata et al., 2016); (Waseem et al., 2017), toxic language ((Kol- hatkar et al., 2020); (Kaggle, 2020)), aggressive lan- guage ((Haddad et al., 2019); (Kumar et al., 2018); (Bhattacharya et al., 2020)), offensive language ((Chen et al., 2012); (Mubarak et al., 2017); (Nascimento et al., 2019); (de Pelle and Moreira, 2016); (Sch¨afer and Burtenshaw, 2019); (Zampieri et al., 2019a), (Zampieri et …

Content truncated to 500 words