BD-SHS: A Benchmark Dataset for Learning to Detect Online Bangla Hate Speech in Different Social Contexts

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 5153–5162 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 5153 BD-SHS: A Benchmark Dataset for Learning to Detect Online Bangla Hate Speech in Different Social Contexts Nauros Romim1, Mosahed Ahmed2, Md. Saiful Islam3 Arnab Sen Sharma4, Hriteshwar Talukder5, Mohammad Ruhul Amin6 Shahjalal University of Science and Technology1, 2, 3, 4 University of Alberta, Canada3 University of Boston, USA5 Fordham University, USA 6 naurosromim@gmail.com1, mosahed32@student.sust.edu2, saiful-cse@sust.edu3 arnab-cse@sust.edu4 hriteshwar-eee@sust.edu5 mamin17@fordham.edu6 Abstract Social media platforms and online streaming services have spawned a new breed of Hate Speech (HS). Due to the massive amount of user-generated content on these sites, modern machine learning techniques are found to be feasible and cost-effective to tackle this problem. However, linguistically diverse datasets covering different social contexts in which offensive language is typically used are required to train generalizable models. In this paper, we identify the shortcomings of existing Bangla HS datasets and introduce a large manually labeled dataset BD-SHS that includes HS in different social contexts. The labeling criteria were prepared following a hierarchical annotation process, which is the first of its kind in Bangla HS to the best of our knowledge. The dataset includes more than 50,200 offensive comments crawled from online social networking sites and is at least 60% larger than any existing Bangla HS datasets. We present the benchmark result of our dataset by training different NLP models resulting in the best one achieving an F1-score of 91.0%. In our experiments, we found that a word embedding trained exclusively using 1.47 million comments from social media and streaming sites consistently resulted in better modeling of HS detection in comparison to other pre-trained embeddings. Our dataset and all accompanying codes is publicly available at github.com/naurosromim/hate-speech-dataset-for-Bengali-social-media Keywords: Offensive Language in Social Context, Bangla Benchmark Dataset, Modeling Hate Speech 1. Introduction Social media and online streaming platforms are gradually becoming an integral part of peoples’ daily lives. These platforms have democratized in- formation and allowed people a greater freedom of speech, sometimes even offering anonymity. How- ever, as a result, these sites have become a new frontier for spreading misinformation and HS. But the anonymous and global nature of these sites makes it difficult for law enforcement agencies to regulate them (Gagliardone et al., 2015). Identify- ing HS and curtailing its spread is crucial to safe- guard human rights and prevent it from marginaliz- ing individuals and groups of people based on their race, gender, ethnicity, or other affiliations. And, due to the massive amount of user-generated data on these kinds of sites, the only feasible solution to effectively tackle this challenge is by utilizing modern natural language processing and machine learning models. However, the lack of formal lan- guage syntax, spelling mistakes, and use of various swear and non-standard acronyms in the comment sections of social media and online streaming sites makes this task harder (Nobata et al., 2016). Re- searchers have started working in this very chal- …

Content truncated to 500 words