Adaptation of Deep Bidirectional Transformers for Afrikaans Language

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 2475–2478 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 2475 Adaptation of Deep Bidirectional Transformers for Afrikaans Language Sello Ralethe Rand Merchant Bank Johannesburg, South Africa Ralethe.sello@gmail.com Abstract The recent success of pretrained language models in Natural Language Processing has sparked interest in training such models for languages other than English. Currently, training of these models can either be monolingual or multilingual based. In the case of multilingual models, such models are trained on concatenated data of multiple languages. We introduce AfriBERT, a language model for the Afrikaans language based on Bidirectional Encoder Representation from Transformers (BERT). We compare the performance of AfriBERT against multilingual BERT in multiple downstream tasks, namely part-of-speech tagging, named-entity recognition, and dependency parsing. Our results show that AfriBERT improves the current state-of-the-art in most of the tasks we considered, and that transfer learning from multilingual to monolingual model can have a significant performance improvement on downstream tasks. We release the pretrained model for AfriBERT. Keywords: Afrikaans, BERT, AfriBERT, Multilingual 1. Introduction Pretrained language models use large-scale unlabelled data to learn highly effective general language representations. These models have become ubiquitous in Natural Language Processing (NLP), pushing the state-of-the-art in a large variety of tasks involving natural language understanding and generation such as natural language inference, sentiment classification, and semantic textual similarity. A few prominent examples of these models include GPT (Radford et al., 2018), ELMO (Peters et al., 2018), BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019), and XLNET (Yang et al., 2019). An advantage that these pretrained language models have over traditional task-specific approaches is that they can be trained in an unsupervised manner. However, they can be difficult to implement because of the amount of data and computational resources needed for pretraining. The need for large-scale data has limited the availability of these pretrained models mostly to the English language. For other languages, these models are implemented as multilingual models. Lample and Conneau (2019) have demonstrated that multilingual models perform poorer than their monolingual models. In this paper, we show how a monolingual model can be trained using multilingual initialisation. We use multilingual BERT, with Afrikaans as a target language for transfer. To train a monolingual language model for Afrikaans, we used the following corpora: • OSCAR (Ortiz Su´arez et al., 2019) • Afrikaans News Article • NCHLT Afrikaans Text Corpora (Puttkammer et al., 2014) • Afrikaans Novels (Project Gutenberg) • Afrikaans Wikipedia • OPUS (Tiedemann, 2012) We train the model following the BERT architecture. We evaluate our trained model, called AfriBERT, on the following downstream tasks for Afrikaans: part-of-speech (POS) tagging, named-entity recognition (NER), and dependency parsing. AfriBERT improves the state-of-the- art for most of these tasks when compared with multilingual approach. In summary, we make three main contributions: • We use transfer learning to train a monolingual BERT model on the Afrikaans language using a combined corpus. • We evaluate AfriBERT …

Content truncated to 500 words