LISAC FSDM-USMBA Team at SemEval-2020 Task 12: Overcoming AraBERT’s pretrain-finetune discrepancy for Arabic offensive language identification

Proceedings of the 14th International Workshop on Semantic Evaluation, pages 2080–2085 Barcelona, Spain (Online), December 12, 2020. 2080 LISAC FSDM-USMBA Team at SemEval-2020 Task 12: Overcoming AraBERT’s pretrain-finetune discrepancy for Arabic offensive language identification Hamza Alami1, Said Ouatik El Alaoui1,2, Abdessamad Benlahbib1, Noureddine En-nahnahi1, 1 LISAC Laboratory, Faculty of Sciences Dhar EL Mehraz (F.S.D.M), Sidi Mohamed Ben Abdellah University (U.S.M.B.A) 2 Ibn Tofail University, National School of Applied Sciences, Kenitra, Morocco hamza.alami1@usmba.ac.ma, s ouatik@yahoo.com, abdessamad.benlahbib@usmba.ac.ma, noureddine.en-nahnahi@usmba.ac.ma Abstract AraBERT is an Arabic version of the state-of-the-art Bidirectional Encoder Representations from Transformers (BERT) model. The latter has achieved good performance in a variety of Natural Language Processing (NLP) tasks. In this paper, we propose an effective AraBERT embeddings- based method for dealing with offensive Arabic language in Twitter. First, we pre-process tweets by handling emojis and including their Arabic meanings. Next, to overcome the pretrain-finetune discrepancy, we substitute each detected emojis by the special token [MASK] into both fine tuning and inference phases. Then, we represent tweets tokens by applying AraBERT model. Finally, we feed the tweet representation into a sigmoid function to decide whether a tweet is offensive or not. The proposed method achieved the best results on OffensEval 2020: Arabic task and reached a macro F1 score equal to 90.17%. 1 Introduction Negative social media behaviors including cyberbullying, trolling and offensive language are intended to hurt or embarrass a victim. Prevention of antisocial internet use is then a protection for internet users that let them live a positive experience through the internet without any type of harassment. Zampier et al. (2019; 2020) introduced a shared task for identifying and categorizing offensive language in social media. The task is divided into three sub-tasks including Sub-task A: Detect if a post is offensive (OFF) or not (NOT); Sub-task B: Categorize the offense type of an offense post as targeted insult (TIN), targeted threat (TTH), or untargeted (UNT); and Sub-task C: Identify the offense target as individual (IND), group of people (GRP), organization or entity (ORG), or other (OTH). The three sub-tasks were run for English while only sub-task A was run for Arabic, Danish, Greek, and Turkish. The majority of OffensEval 2019 (Zampieri et al., 2019) participants used deep neural network models such as Bidirectional Encoder Representations from Transformers (BERT) (Devlin et al., 2019) to build their offensive language detector. These models have the capability to learn and automatically extract complex features from raw data, thus it has achieved some state of the art results in different Natural Language Processing (NLP) tasks. In this paper, we follow the same path by building an Arabic offensive language detector based on BERT more specifically AraBERT (Antoun et al., 2020). Our method overcomes AraBERT’s pretrain- finetune discrepancy by including the special token [MASK] during fine tuning and inference phases. We obtained promising results since we achieved 90.17% macro F1 score on test set and we ranked on top of the participants list. This paper is organised as follows: Section 2 describes the proposed method; Section …

Content truncated to 500 words