UHH-LT at SemEval-2020 Task 12: Fine-Tuning of Pre-Trained Transformer Networks for Offensive Language Detection

Proceedings of the 14th International Workshop on Semantic Evaluation, pages 1638–1644 Barcelona, Spain (Online), December 12, 2020. 1638 UHH-LT at SemEval-2020 Task 12: Fine-Tuning of Pre-Trained Transformer Networks for Offensive Language Detection Gregor Wiedemann Seid Muhie Yimam Language Technology Group Department of Informatics University of Hamburg, Germany {gwiedemann, yimam, biemann}@informatik.uni-hamburg.de Chris Biemann Abstract Fine-tuning of pre-trained transformer networks such as BERT yield state-of-the-art results for text classification tasks. Typically, fine-tuning is performed on task-specific training datasets in a supervised manner. One can also fine-tune in unsupervised manner beforehand by further pre- training the masked language modeling (MLM) task. Hereby, in-domain data for unsupervised MLM resembling the actual classification target dataset allows for domain adaptation of the model. In this paper, we compare current pre-trained transformer networks with and without MLM fine-tuning on their performance for offensive language detection. Our MLM fine-tuned RoBERTa-based classifier officially ranks 1st in the SemEval 2020 Shared Task 12 for the English language. Further experiments with the ALBERT model even surpass this result. 1 Offensive Language Detection The automatic detection of hate-speech, cyber-bullying, or aggressive and offensive language became a vividly studied task in natural language processing (NLP) in recent years. The offensive language detection (OLD) shared Task 6 of 2019’s International Workshop on Semantic Evaluation (SemEval) (Zampieri et al., 2019b) attracted submissions from more than 100 teams. The Offensive Language Iden- tification Dataset (OLID) used in this shared task comprises three hierarchical classification sub-tasks: A) offensive language detection, B) categorization of offensive language, and C) offensive language tar- get identification (Zampieri et al., 2019a). For Task A, 14,100 Twitter tweets were manually labeled as either offensive (OFF) or not offensive (NOT). Task B distinguishes the 4,640 offensive tweets from task A into targeted (TIN) or general, untargeted (UNT) offensive language. Task C, finally, separates targeted insults into the three categories: groups (GRP), individuals (IND), and others (OTH). OffensEval 2020, the SemEval 2020 Offensive Language Detection Shared Task, does not provide an own manually labeled training set for the English language (Zampieri et al., 2020). Instead, a large ‘weakly labeled’ dataset was published by the organizers, containing roughly nine million tweets. Each tweet has been automatically classified by an ensemble of five different supervised classifiers trained on the OLID dataset. The weakly labeled dataset contains the raw tweet (with user mentions replaced by a special ‘USER’ token) along with the average label probability and the variance of the five classifier predictions. Since there is no way that such weak labels themselves carry more useful information to a machine learning system than the original dataset on which the five classifiers were trained, we decided not to use any of the weakly labeled information. Instead, for our classification systems, we rely on the 2019 OLID dataset only. However, the OffensEval 2020 dataset is an ample source to build models using unsupervised learning, particularly for domain-adaptation of a pre-trained language model such as BERT (Devlin et al., 2019) or its successors which are based on the transformer neural network …

Content truncated to 500 words