SemEval-2019 Task 6: Identifying and Categorizing Offensive Language in Social Media (OffensEval)

Proceedings of the 13th International Workshop on Semantic Evaluation (SemEval-2019), pages 75–86 Minneapolis, Minnesota, USA, June 6–7, 2019. ©2019 Association for Computational Linguistics 75 SemEval-2019 Task 6: Identifying and Categorizing Offensive Language in Social Media (OffensEval) Marcos Zampieri,1 Shervin Malmasi,2 Preslav Nakov,3 Sara Rosenthal,4 Noura Farra,5 Ritesh Kumar6 1University of Wolverhampton, UK, 2Amazon Research, USA 3Qatar Computing Research Institute, HBKU, Qatar 4IBM Research, USA, 5Columbia University, USA, 6Bhim Rao Ambedkar University, India m.zampieri@wlv.ac.uk Abstract We present the results and the main findings of SemEval-2019 Task 6 on Identifying and Cate- gorizing Offensive Language in Social Media (OffensEval). The task was based on a new dataset, the Offensive Language Identification Dataset (OLID), which contains over 14,000 English tweets. It featured three sub-tasks. In sub-task A, the goal was to discriminate be- tween offensive and non-offensive posts. In sub-task B, the focus was on the type of of- fensive content in the post. Finally, in sub-task C, systems had to detect the target of the offen- sive posts. OffensEval attracted a large num- ber of participants and it was one of the most popular tasks in SemEval-2019. In total, about 800 teams signed up to participate in the task, and 115 of them submitted results, which we present and analyze in this report. 1 Introduction Recent years have seen the proliferation of offen- sive language in social media platforms such as Facebook and Twitter. As manual filtering is very time consuming, and as it can cause post-traumatic stress disorder-like symptoms to human annota- tors, there have been many research efforts aim- ing at automating the process. The task is usually modeled as a supervised classification problem, where systems are trained on posts annotated with respect to the presence of some form of abusive or offensive content. Examples of offensive con- tent studied in previous work include hate speech (Davidson et al., 2017; Malmasi and Zampieri, 2017, 2018), cyberbulling (Dinakar et al., 2011), and aggression (Kumar et al., 2018). Moreover, given the multitude of terms and definitions used in the literature, some recent studies have investi- gated the common aspects of different abusive lan- guage detection sub-tasks (Waseem et al., 2017; Wiegand et al., 2018). Interestingly, none of this previous work has stud- ied both the type and the target of the offensive language, which is our approach here. Our task, OffensEval1, uses the Offensive Language Identi- fication Dataset (OLID)2 (Zampieri et al., 2019), which we created specifically for this task. OLID is annotated following a hierarchical three-level annotation schema that takes both the target and the type of offensive content into account. Thus, it can relate to phenomena captured by previous datasets such as the one by Davidson et al. (2017). Hate speech, for example, is commonly under- stood as an insult targeted at a group, whereas cy- berbulling is typically targeted at an individual. We defined three sub-tasks, corresponding to the three levels in our annotation schema:3 Sub-task A: Offensive language identification (104 participating teams) Sub-task B: Automatic categorization of offense …

Content truncated to 500 words