ConvAI at SemEval-2019 Task 6: Offensive Language Identification and Categorization with Perspective and BERT

Proceedings of the 13th International Workshop on Semantic Evaluation (SemEval-2019), pages 571–576 Minneapolis, Minnesota, USA, June 6–7, 2019. ©2019 Association for Computational Linguistics 571 ConvAI at SemEval-2019 Task 6: Offensive Language Identification and Categorization with Perspective and BERT John Pavlopoulos Ion Androutsopoulos Department of Informatics Athens University of Economics and Business, Greece annis,ion@aueb.gr Nithum Thain Lucas Dixon Jigsaw nthain,ldixon@google.com Abstract This paper presents the application of two strong baseline systems for toxicity detection and evaluates their performance in identify- ing and categorizing offensive language in so- cial media. Perspective is an API, that serves multiple machine learning models for the im- provement of conversations online, as well as a toxicity detection system, trained on a wide variety of comments from platforms across the Internet. BERT is a recently popular language representation model, fine tuned per task and achieving state of the art performance in mul- tiple NLP tasks. Perspective performed bet- ter than BERT in detecting toxicity, but BERT was much better in categorizing the offensive type. Both baselines were ranked surpris- ingly high in the SEMEVAL-2019 OFFENSE- VAL competition, Perspective in detecting an offensive post (12th) and BERT in categoriz- ing it (11th). The main contribution of this pa- per is the assessment of two strong baselines for the identification (Perspective) and the cat- egorization (BERT) of offensive language with little or no additional training data. 1 Introduction Offensive language detection refers to computa- tional approaches for detecting abusive language, such as threats, insults, calumniation, discrimina- tion, swearing (Pavlopoulos et al., 2017b), which could be targeted (at an individual or group) or not (Waseem et al., 2017). These computational ap- proaches are often used by moderators who face an increasing volume of abusive content and would like assistance in managing it efficiently.1 Although offensive language detection is not a new task (Dinakar et al., 2011; Dadvar et al., 2013; Kwok and Wang, 2013; Burnap and Williams, 2015; Tulkens et al., 2016), the creation of large 1See, for example, https://goo.gl/VQNDNX. corpora (Wulczyn et al., 2017), along with recent advances in pre-training text representations (De- vlin et al., 2018) allow for much more efficient ap- proaches. Furthermore, while new competitions and corpora are being introduced (Zampieri et al., 2019a),2 there is a need for strong baselines to as- sess the performance of more complex systems. This paper assesses two systems for the detection and categorization of offensive language, which require few or no task-specific annotated training instances. The first baseline is a Convolutional Neural Network (CNN) for toxicity detection, trained on millions of user comments from different on- line publishers, which is made publicly available through the Perspective API.3 This model re- quires no extra training or fine tuning and can be directly applied to score unseen posts. The second strong baseline is the recently popular Bidirec- tional Encoder Representations from Transform- ers (BERT), a pre-trained model that has been reported to achieve state of the art performance in multiple NLP tasks with limited fine-tuning on task-specific training data (Devlin et …

Content truncated to 500 words