A Turkish Hate Speech Dataset and Detection System

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 4177–4185 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 4177 A Turkish Hate Speech Dataset and Detection System Fatih Beyhan1,2, Buse C¸ arık1,2, ˙Inanc¸ Arın1,2, Ays¸ecan Terzio˘glu3, Berrin Yanıko˘glu1,2, Reyyan Yeniterzi1,2 1Faculty of Engineering and Natural Sciences, Sabanci University, Istanbul, Turkey 34956 2Center of Excellence in Data Analytics (VERIM), Sabanci University, Istanbul, Turkey 34956 3Faculty of Arts and Social Sciences, Sabanci University, Istanbul, Turkey 34956 {fatihbeyhan,busecarik,inanc,aysecan,berrin,reyyan}@sabanciuniv.edu Abstract Social media posts containing hate speech are reproduced and redistributed at an accelerated pace, reaching greater audiences at a higher speed. We present a machine learning system for automatic detection of hate speech in Turkish, along with a hate speech dataset consisting of tweets collected in two separate domains. We first adopted a definition for hate speech that is in line with our goals and amenable to easy annotation; then designed the annotation schema for annotating the collected tweets. The Istanbul Convention dataset consists of tweets posted following the withdrawal of Turkey from the Istanbul Convention. The Refugees dataset was created by collecting tweets about immigrants by filtering based on commonly used keywords related to immigrants. Finally, we have developed a hate speech detection system using the transformer architecture (BERTurk), to be used as a baseline for the collected dataset. The binary classification accuracy is 77% when the system is evaluated using 5-fold cross validation on the Istanbul Convention dataset and 71% for the Refugee dataset. We also tested a regression model with 0.66 and 0.83 RMSE on a scale of [0-4], for the Istanbul Convention and Refugees datasets. Keywords: Hate speech detection, Deep learning, Turkish 1. Introduction Definition. Hate speech refers to discourse that targets a specific group based on race, gender, religion, sexual ori- entation, etc., and indicates some level of hatred towards them. In fact, there is no fully agreed convention on what constitutes hate speech (Davidson et al., 2017; Poletto et al., 2017), however, there have been various attempts to define it. In a commonly accepted definition, hate speech is de- scribed as “speech that targets disadvantaged social groups in a manner that is potentially harmful to them” (Jacobs et al., 1998; Walker, 1994). Hate speech comes in a spectrum, ranging from mild stereotyping to open calls to violence toward the target group, partially explaining the difficulty in coming up with a description and annotating collected samples. Some works in the literature use a binary categorization (hate speech or not), while others use a finer level of categoriza- tion (Davidson et al., 2017; Poletto et al., 2017). Further- more, what constitutes hate speech is subjective to some level and may require background knowledge on the topic to detect. Aim. Negative discourses including hate speech are repro- duced and redistributed on social media and internet at an accelerated pace, allowing access at a higher speed and to greater audiences in number. Online hate speech does not only take place in unofficial or …

Content truncated to 500 words