PRHLT-UPV at SemEval-2020 Task 12: BERT for Multilingual Offensive Language Detection

Proceedings of the 14th International Workshop on Semantic Evaluation, pages 1605–1614 Barcelona, Spain (Online), December 12, 2020. 1605 PRHLT-UPV at SemEval-2020 Task 12: BERT for Multilingual Offensive Language Detection Gretel Liz De la Pe˜na Sarrac´en and Paolo Rosso Universitat Polit`ecnica de Val`encia, Val`encia, Spain gredela@posgrado.upv.es prosso@dsic.upv.es Abstract The present paper describes the system submitted by the PRHLT-UPV team for the task 12 of SemEval-2020: OffensEval 2020. The official title of the task is Multilingual Offensive Language Identification in Social Media, and aims to identify offensive language in texts. The languages included in the task are English, Arabic, Danish, Greek and Turkish. We propose a model based on the BERT architecture for the analysis of texts in English. The approach leverages knowledge within a pre-trained model and performs fine-tuning for the particular task. In the analysis of the other languages the Multilingual BERT is used, which has been pre-trained for a large number of languages. In the experiments, the proposed method for English texts is compared with other approaches to analyze the relevance of the architecture used. Furthermore, simple models for the other languages are evaluated to compare them with the proposed one. The experimental results show that the model based on BERT outperforms other approaches. The main contribution of this work lies in this study, despite not obtaining the first positions in most cases of the competition ranking. 1 Introduction BERT, the Bidirectional Encoder Representations for Transformers (Devlin et al., 2019), is a model producing context representations that leverage on language model pre-training. It is based on transformers (Vaswani et al., 2017), which are models that process words in relation to all other words in a sentence, rather than word by word in order. That is, as opposed to directional models, which read the text input sequentially (forward and/or backward), the transformer reads the entire sequence of words at once. This characteristic allows the model to learn the context of a word based on all of its surroundings. Hence, BERT models can consider the full context of a word by looking at the words that come before and after it. This is very useful to understand the intent behind sentences. Therefore, unlike other kinds of models, with BERT models it is possible to effectively capture the general meaning of a sentence by detecting relevant words and their relationship to others. BERT is a deep, bidirectional, unsupervised language representation, pre-trained using only plain text corpus. The pre-training has been performed by using two strategies on a large corpus of unlabelled text which includes the entire Wikipedia and a book corpus. One training strategy is masked language model, where the model attempts to predict the original value of some masked words in a sequence, based on the context provided by the non-masked words in the sequence. The other training strategy is next sentence prediction, where the model learns to predict if given a pair of sentences, the second one is the subsequent sentence in the original document. Pre-trained BERT can be fine-tuned …

Content truncated to 500 words