UniParma at SemEval-2021 Task 5: Toxic Spans Detection Using CharacterBERT and Bag-of-Words Model

Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021), pages 220–224 Bangkok, Thailand (online), August 5–6, 2021. ©2021 Association for Computational Linguistics 220 UniParma at SemEval-2021 Task 5: Toxic Spans Detection Using CharacterBERT and Bag-of-Words Model Akbar Karimi Leonardo Rossi University of Parma {akbar.karimi, leonardo.rossi, andrea.prati}@unipr.it Andrea Prati Abstract With the ever-increasing availability of digital information, toxic content is also on the rise. Therefore, the detection of this type of lan- guage is of paramount importance. We tackle this problem utilizing a combination of a state- of-the-art pre-trained language model (Charac- terBERT) and a traditional bag-of-words tech- nique. Since the content is full of toxic words that have not been written according to their dictionary spelling, attendance to individual characters is crucial. Therefore, we use Char- acterBERT to extract features based on the word characters. It consists of a Character- CNN module that learns character embeddings from the context. These are, then, fed into the well-known BERT architecture. The bag- of-words method, on the other hand, further improves upon that by making sure that some frequently used toxic words get labeled accord- ingly. With a ∼4 percent difference from the first team, our system ranked 36th in the com- petition. The code is available for further re- search and reproduction of the results1. 1 Introduction The user generated digital content is increasing rapidly every second of the day. This can include some toxic language whose detection can be diffi- cult due to the complexities of human languages. We address this problem by participating in Se- mEval Workshop 2021 Task 5 (Pavlopoulos et al., 2021). In many cases, the data, which are considered to be toxic, contain words that have not been written in their standard forms. There might also be a lot of misspelling or letter replacements. In addition, usually the words that are considered to be the most offensive are bleeped which makes them difficult to be recognized if we use a model which learns the 1https://github.com/IMPLabUniPr/ UniParma-at-semeval-2021-task-5 content representation based on the words. Apart from word related issues, the context also plays a crucial role in the meaning that a word conveys since words in different contexts can have various meanings. Therefore, in order to cope with these issues, we opt for a recently pre-trained language model which has been trained on character level. Char- acterBERT (El Boukkouri et al., 2020) is a deep neural network model that has been pre-trained on Wikipedia and OpenWebText (Gokaslan and Cohen) corpora using the BERT architecture (De- vlin et al., 2019) with an addition of a character- aware Convolutional Neural Network (CNN) (Kim et al., 2016; Peters et al., 2018). BERT-based mod- els have now become pervasive in many different natural language processing tasks such as read- ing comprehension (Xu et al., 2019), named entity recognition (Liang et al., 2020), sentiment analysis (Karimi et al., 2020), and language understanding (Jiao et al., 2020) as well as similar ones to toxic language detection such as propaganda detection (Jurkiewicz et al., 2020). …

Content truncated to 500 words