Stop PropagHate at SemEval-2019 Tasks 5 and 6: Are abusive language classification results reproducible?

Proceedings of the 13th International Workshop on Semantic Evaluation (SemEval-2019), pages 745–752 Minneapolis, Minnesota, USA, June 6–7, 2019. ©2019 Association for Computational Linguistics 745 Stop PropagHate at SemEval-2019 Tasks 5 and 6: Are abusive language classification results reproducible? Paula Fortuna1,2 Juan Soler-Company2 S´ergio Nunes1,3 (1) INESC TEC and (3) FEUP, University of Porto Rua Dr. Roberto Frias s/n, 4200-465 Porto, Portugal paula.fortuna@fe.up.pt, sergio.nunes@fe.up.pt (2) Pompeu Fabra University Carrer de Roc Boronat 138, 08018 Barcelona, Spain juan.soler@upf.edu Abstract This paper summarizes the participation of Stop PropagHate team at SemEval 2019. Our approach is based on replicating one of the most relevant works on the literature, using word embeddings and LSTM. After circum- venting some of the problems of the original code, we found poor results when applying it to the HatEval contest (F1=0.45). We think this is due mainly to inconsistencies in the data of this contest. Finally, for the OffensEval the classifier performed well (F1=0.74), proving to have a better performance for offense de- tection than for hate speech. 1 Introduction In the last few years, several evaluation tasks in the context of hate speech detection and catego- rization have been created. Some of these tasks include e.g., EVALITA (Bosco et al., 2018) and TRAC-1 (Kumar et al., 2018). These type of initiatives promote the development of different but comparable solutions for the same problem, within a short period of time, which is an inter- esting contribution for a research field. In this pa- per, we describe the participation of team “Stop PropagHate” in the HatEval and OffensEval tasks of SemEval 2019. The main goal of both tasks is to improve the classification of Hate Speech and Offensive Lan- guage. Some of the works in the literature achieve a very competitive performance, e.g. Badjatiya et al. (2017) obtain an F1 score of 0.93 when us- ing deep learning for classifying hate speech in one of the most commonly used baseline datasets (e.g. Waseem (2016)). In this context, we have a specific objective with our approach: we aim to reproduce a state-of-the-art classifier as described in the literature of this topic. We choose to reproduce the study by Badjatiya et al. (2017), not only because of the good perfor- mance of the developed models, but also because in this work the authors published their code. Con- sidering the amount of parameters available for definition and tuning in a machine learning classi- fication pipeline, a precise and extensive definition of an experiment’s parameters is not simple and is hardly ever provided. Thus, having the code of the experiment is the best way to understand not only which steps were conducted, but also how those steps were indeed executed. This is a highly cited paper, which can be regarded as an indicator of its relevance in the area. In this paper, we describe our journey in the process of replication and the results achieved when applying this classifier in both shared tasks. The paper is structured as follows: Section 2 briefly reviews the …

Content truncated to 500 words