Investigating Multilingual Abusive Language Detection: A Cautionary Tale

Proceedings of Recent Advances in Natural Language Processing, pages 1151–1160, Varna, Bulgaria, Sep 2–4, 2019. https://doi.org/10.26615/978-954-452-056-4_132 1151 Investigating Multilingual Abusive Language Detection: A Cautionary Tale Kenneth Steimel, Daniel Dakota, Yue Chen, Sandra K¨ubler Indiana University {ksteimel,ddakota,yc59,skuebler}@indiana.edu Abstract Abusive language detection has received much attention in the last years, and re- cent approaches perform the task in a num- ber of different languages. We investigate which factors have an effect on multilin- gual settings, focusing on the compatibil- ity of data and annotations. In the current paper, we focus on English and German. Our findings show large differences in per- formance between the two languages. We find that the best performance is achieved by different classification algo- rithms. Sampling to address class imbal- ance issues is detrimental for German and beneficial for English. The only similarity that we find is that neither data set shows clear topics when we compare the results of topic modeling to the gold standard. Based on our findings, we can conclude that a multilingual optimization of classi- fiers is not possible even in settings where comparable data sets are used. 1 Introduction The last decade has seen a massive increase in user generated content in social media. While most people are interested in connecting with family and friends and in exchanging experiences, there is an increasing number of posts that cross the line from sharing negative opinions to becoming abusive. Since the data is too massive for man- ual filtering, automated methods to detect abusive language reliably are required. This has created a novel research area under the titles of abusive lan- guage detection, hate speech detection, flame or cyberbullying detection. While most of the work on abusive language detection has focused on English (Schmidt and Wiegand, 2017; Park and Fung, 2017; Lee et al., 2018), there is some work on other languages, and first attempts have also been made to de- velop methods that work across different lan- guages (Fehn Unsv˚ag and Gamb¨ack, 2018). Our interest also focuses on multilingual abu- sive language detection. However, before we en- gage in a full scale investigation of which methods work well across multiple languages, we need to know more about which factors have an effect on multilingual settings, including but not restricted to the compatibility of data and annotations, dif- ferences between languages, and topic effects. In the current paper, we focus on two languages, En- glish and German, where we have access to similar data (see section 3 for more information). We ap- proach the following questions as we investigate an approach across the two languages: 1. Do classifiers behave similarly across the two languages? I.e., can we establish the best classifier on one language and then use it suc- cessfully for the second language? 2. Can we determine which types of features are necessary for a classifier? Are the types of features and the number of features compara- ble across the two languages? 3. The data sets are skewed towards non-abusive language, and research in …

Content truncated to 500 words