GoldenWind at SemEval-2021 Task 5: Orthrus - An Ensemble Approach to Identify Toxicity

Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021), pages 860–864 Bangkok, Thailand (online), August 5–6, 2021. ©2021 Association for Computational Linguistics 860 GoldenWind at SemEval-2021 Task 5: Orthrus – An Ensemble Approach to Identify Toxicity Marco Palomino∗ Dawid Grad† School of Engineering, Computing and Mathematics University of Plymouth Plymouth, Devon, PL4 8AA, United Kingdom marco.palomino@plymouth.ac.uk∗ {dawid.grad†,james.bedwell‡}@students.plymouth.ac.uk James Bedwell‡ Abstract Many new developments to detect and miti- gate toxicity are currently being evaluated. We are particularly interested in the correlation be- tween toxicity and the emotions expressed in online posts. While toxicity may be disguised by amending the wording of posts, emotions will not. Therefore, we describe here an en- semble method to identify toxicity and clas- sify the emotions expressed on a corpus of annotated posts published by Task 5 of Se- mEval 2021—our analysis shows that the ma- jority of such posts express anger, sadness and fear. Our method to identify toxicity combines a lexicon-based approach, which on its own achieves an F1 score of 61.07%, with a su- pervised learning approach, which on its own achieves an F1 score of 60%. When both meth- ods are combined, the ensemble achieves an F1 score of 66.37%. 1 Introduction Healthy conversations are only possible when peo- ple feel safe from abuse and do not resort to using violent language. Regrettably, violent and inflam- matory language is becoming increasingly common online. Indeed, the rhetoric of violence recently em- ployed on social media has persuaded platforms, such as Twitter, to create new policies to prevent the use of threatening language (Twitter, Inc., 2021). A jargon word, cyberbullying, has been coined lately to refer to the use of electronic communication to send or post messages of an intimidating or threat- ening nature (Zaheri et al., 2020). Along with cyberbullying, other forms of verbal abuse employed on social media, such as online harassment and hate speech, are now being collec- tively referred to as toxicity in language (Mohan et al., 2017). We are interested in developing algo- rithms to recognise toxicity and measure its impact on the sentiment expressed. Most of the data available to investigate toxic- ity classify whole comments or documents (Wul- czyn et al., 2017; Borkan et al., 2019), and do not identify “spans”—that is, the precise word se- quences that make a text toxic. Given how impor- tant such spans are for the implementation of semi- automated moderation, we have participated on Task 5 (Toxic Spans Detection) of the International Workshop on Semantic Evaluation (SemEval) 2021 (Pavlopoulos et al., 2021). Thus far, we have con- centrated on the combination of two approaches: a lexicon-based approach and a supervised learning approach to identify toxic spans. Although the identification of toxic spans in on- line posts can be aided by a suitable lexicon of toxic words, such words can easily be concealed through minor changes—for instance, “fck urself” is a toxic span that would evade detection based on basic lists of profane words. However, emotions are harder to conceal. Hence, …

Content truncated to 500 words