UoB at SemEval-2020 Task 12: Boosting BERT with Corpus Level Information

Proceedings of the 14th International Workshop on Semantic Evaluation, pages 2216–2221 Barcelona, Spain (Online), December 12, 2020. 2216 UoB at SemEval-2020 Task 12: Boosting BERT with Corpus Level Information. Wah Meng Lim University of Birmingham United Kingdom wahmengsch@gmail.com Harish Tayyar Madabushi University of Birmingham United Kingdom H.TayyarMadabushi.1@bham.ac.uk Abstract Pre-trained language model word representation, such as BERT, have been extremely successful in several Natural Language Processing tasks significantly improving on the state-of-the-art. This can largely be attributed to their ability to better capture semantic information contained within a sentence. Several tasks, however, can benefit from information available at a corpus level, such as Term Frequency-Inverse Document Frequency (TF-IDF). In this work we test the effectiveness of integrating this information with BERT on the task of identifying abuse on social media and show that integrating this information with BERT does indeed significantly improve performance. We participate in Sub-Task A (abuse detection) wherein we achieve a score within two points of the top performing team and in Sub-Task B (target detection) wherein we are ranked 4 of the 44 participating teams. 1 Introduction Offensive language is pervasive in social media in this day and age. Offensive language is so common and it is often used as emphasis instead of its semantic meaning, because of this, it can be hard to identify truly offensive content. Individuals frequently take advantage of the perceived anonymity of computer-mediated communication, using this to engage in behaviour that many of them would not consider in real life. Online communities, social media platforms, and technology companies have been investing heavily in ways to cope with offensive language to prevent abusive behavior in social media. One of the most effective strategies for tackling this problem is to use computational methods to identify offense, aggression, and hate speech in user-generated content (e.g. posts, comments, microblogs, etc.). The SemEval 2020 task on abuse detection (Zampieri et al., 2020) aims to study both the target and the type of offensive language that has not been covered by previous works on various offenses such as hate speech detection and cyberbullying. In this paper, we focus on the first two sub-tasks: Sub-task A, that focuses on offensive language or profanity detection, a binary classification problem wherein the objective is to determine if a tweet is offensive or not. Sub-task B, which focuses on the identification of target presence, also a binary classification problem, wherein we are required to determine if a tweet is targeted at someone or something or not. 1.1 Boosting Pre-trained Representations Recent Natural Language Processing (NLP) systems have focused on the use of deep learning methods that take word embeddings as input. While these methods have been extremely successful on several tasks, we believe that information pertaining to the importance of individual words available at a corpus level might not be effectively captures by models that use pre-trained embeddings, especially given the small number of training epochs (usually 3) used. We hypothesise that deep learning models, especially those that use pre-trained embeddings and …

Content truncated to 500 words