KinyaBERT: a Morphology-aware Kinyarwanda Language Model

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 5347 - 5363 May 22-27, 2022 c⃝2022 Association for Computational Linguistics KinyaBERT: a Morphology-aware Kinyarwanda Language Model Antoine Nzeyimana University of Massachusetts Amherst anthonzeyi@gmail.com Andre Niyongabo Rubungo Polytechnic University of Catalonia niyongabor.andre@gmail.com Abstract Pre-trained language models such as BERT have been successful at tackling many natural language processing tasks. However, the unsu- pervised sub-word tokenization methods com- monly used in these models (e.g., byte-pair encoding – BPE) are sub-optimal at handling morphologically rich languages. Even given a morphological analyzer, naive sequencing of morphemes into a standard BERT architec- ture is inefficient at capturing morphological compositionality and expressing word-relative syntactic regularities. We address these chal- lenges by proposing a simple yet effective two- tier BERT architecture that leverages a mor- phological analyzer and explicitly represents morphological compositionality. Despite the success of BERT, most of its evaluations have been conducted on high-resource languages, obscuring its applicability on low-resource lan- guages. We evaluate our proposed method on the low-resource morphologically rich Kinyarwanda language, naming the proposed model architecture KinyaBERT. A robust set of experimental results reveal that KinyaBERT outperforms solid baselines by 2% in F1 score on a named entity recognition task and by 4.3% in average score of a machine-translated GLUE benchmark. KinyaBERT fine-tuning has better convergence and achieves more ro- bust results on multiple tasks even in the pres- ence of translation noise.1 1 Introduction Recent advances in natural language processing (NLP) through deep learning have been largely en- abled by vector representations (or embeddings) learned through language model pre-training (Ben- gio et al., 2003; Mikolov et al., 2013; Penning- ton et al., 2014; Bojanowski et al., 2017; Peters et al., 2018; Devlin et al., 2019). Language mod- els such as BERT (Devlin et al., 2019) are pre- 1Code and data are released at https://github. com/anzeyimana/kinyabert-acl2022 trained on large text corpora and then fine-tuned on downstream tasks, resulting in better perfor- mance on many NLP tasks. Despite attempts to make multilingual BERT models (Conneau et al., 2020), research has shown that models pre-trained on high quality monolingual corpora outperform multilingual models pre-trained on large Inter- net data (Scheible et al., 2020; Virtanen et al., 2019). This has motivated many researchers to pre- train BERT models on individual languages rather than adopting the “language-agnostic” multilingual models. This work is partly motivated by the same findings, but also proposes an adaptation of the BERT architecture to address representational chal- lenges that are specific to morphologically rich languages such as Kinyarwanda. In order to handle rare words and reduce the vocabulary size, BERT-like models use statistical sub-word tokenization algorithms such as byte pair encoding (BPE) (Sennrich et al., 2016). While these techniques have been widely used in language modeling and machine translation, they are not op- timal for morphologically rich languages (Klein and Tsarfaty, 2020). In fact, sub-word tokenization methods that are solely based on surface forms, in- cluding BPE and character-based models, cannot capture all morphological …

Content truncated to 500 words