IIITG-ADBU at SemEval-2020 Task 12: Comparison of BERT and BiLSTM in Detecting Offensive Language

Proceedings of the 14th International Workshop on Semantic Evaluation, pages 1562–1568 Barcelona, Spain (Online), December 12, 2020. 1562 IIITG-ADBU at SemEval-2020 Task 12: Comparison of BERT and BiLSTM in Detecting Offensive Language Arup Baruah1, Kaushik Amar Das1, Ferdous Ahmed Barbhuiya1, and Kuntal Dey2∗ 1IIIT Guwahati, India 2Accenture Technology Labs, Bangalore arup.baruah@gmail.com, kaushikamardas@gmail.com, ferdous@iiitg.ac.in, kuntal.dey@accenture.com Abstract Task 12 of SemEval 2020 consisted of 3 subtasks, namely offensive language identification (Subtask A), categorization of offense type (Subtask B), and offense target identification (Subtask C). This paper presents the results our classifiers obtained for the English language in the 3 subtasks. The classifiers used by us were BERT and BiLSTM. On the test set, our BERT classifier obtained a macro F1 score of 0.90707 for subtask A, and 0.65279 for subtask B. The BiLSTM classifier obtained a macro F1 score of 0.57565 for subtask C. The paper also performs an analysis of the errors made by our classifiers. We conjecture that the presence of a few misleading instances in the dataset is affecting the performance of the classifiers. Our analysis also discusses the need for temporal context and world knowledge to determine the offensiveness of a few comments. 1 Introduction Detecting offensive language in social media is gaining a lot of importance. It is becoming commonplace to encounter offensive content in social media. This type of content if left unattended has the potential of creating a lot of damage to society. As such, research has been directed at developing automated systems for the detection and removal of offensive posts. OffensEval 2020 is a shared task organized as part of the International Workshop on Semantic Evalu- ation 2020 (SemEval 2020) (Zampieri et al., 2020) to develop systems for detecting offensive content. OffensEval was also organized as part of SemEval 2019 (Zampieri et al., 2019). OffensEval 2020 had 3 subtasks: (1) Subtask A required participants to determine if a given tweet is offensive or not, (2) for offensive tweets, Subtask B required determining whether the offense is targeted or untargeted, and (3) if the tweet is a targeted offensive tweet, Subtask C required determining if it targets an individual, group, or other. This task was held for Arabic, Danish, English, Greek, and Turkish language. We participated in all three subtasks for the English language. In this study, we used the Bidirectional Encoder Representations from Transformers (BERT) (Devlin et al., 2019) and Bidirectional Long Short- Term Memory (BiLSTM) (Hochreiter and Schmidhuber, 1997; Schuster and Paliwal, 1997) classifiers. 2 Related Work The history of automatic offensive language detection goes back to Spertus (1997), where a rule based method was used to detect flames in comments received through the feedback forms of web pages. This topic has gained much research attention in recent times. Apart from offensive language, research has been performed to detect hate speech, abusive language, cyberbully, profanity, insults, and aggressiveness. Fortuna and Nunes (2018) defines these related terms. Waseem and Hovy (2016) used a logistic regression classifier to detect racist and sexist tweets. Logistic regression …

Content truncated to 500 words