PGSG at SemEval-2020 Task 12: BERT-LSTM with Tweets’ Pretrained Model and Noisy Student Training Method

Proceedings of the 14th International Workshop on Semantic Evaluation, pages 2111–2116 Barcelona, Spain (Online), December 12, 2020. 2111 PGSG at SemEval-2020 Task 12: BERT-LSTM with Tweets’ pretrained model and Noisy Student training method Bao-Tran Pham-Hong PropertyGuru Group, Singapore phamtrancsek12@gmail.com Setu Chokshi PropertyGuru Group, Singapore setu@propertyguru.com.sg Abstract The paper presents a system developed for the SemEval-2020 competition Task 12 (OffensEval-2): Multilingual Offensive Language Identification in Social Media. We achieve the second place (2nd) in sub-task B: Automatic categorization of offense types and are ranked 55th with a macro F1-score of 90.59 in sub-task A: Offensive language identification. Our solution is using a stack of BERT and LSTM layers, training with the Noisy Student method. Since the tweets data contains a large number of noisy words and slang, we update the vocabulary of the BERT large model pre-trained by the Google AI Language team. We fine-tune the model with tweet sentences provided in the challenge. 1 Introduction Inappropriate and offensive online content has become a significant issue due to an exponential increase in the use of the Internet by people from different cultures and educational backgrounds. Twitter is one of the most popular social media platform, where people share their own opinions among various topics. Therefore, ‘tweets’ requires considerable resources to study offensive behaviors. The SemEval-2020 competition Task 12 is a multilingual challenge of 5 languages: Arabic, Danish, English, Greek, and Turkish (Zampieri et al., 2020). We participate in the English language track, sub-task A and sub-task B. In this language track, the organizers provide a data set of more than 9 million sentences of tweets along with their confidence measures produced by unsupervised learning methods (Rosenthal et al., 2020). To handle this semi-supervised learning task, we use the Noisy Student training method (Xie et al., 2019) to train the BERT-LSTM model. Our approach is more successful in sub-task B, where the standard deviation range of the provided label’s confidence is much larger, with 4.5 points better than the next system. We have publically released the code and our Tweet’s pre-trained model at https://github.com/phamtrancsek12/offensive-identification. The paper is organized as follows: Section 2 introduces the related works. Section 3 describes the data set and the preprocessing methods that we used. Section 4 describes our system architecture and training strategy. Experimental results are presented in Section 5. Finally, we conclude the paper in Section 6. 2 Related Work One of the most popular and successful methods in last year’s OffensEval challenge (Zampieri et al., 2019b) is transfer learning. Recently, transfer learning in NLP using transformer-like architecture has significantly improved on the state-of-the-art in natural language understanding. Despite their success on the variety of NLP benchmarks, such pre-trained models might fail to generalize to natural language tasks from a different distribution. BERT model pre-trained on specific domain data set presented a better performance compares to the model trained on Wikipedia corpus, such as SciBERT (Beltagy et al., 2019) and BioBERT (Lee et al., 2019). This work is licensed under a Creative Commons Attribution 4.0 International …

Content truncated to 500 words