Improving Low-Resource Languages in Pre-Trained Multilingual Language Models

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11993 - 12006 December 7-11, 2022 ©2022 Association for Computational Linguistics Improving Low-Resource Languages in Pre-Trained Multilingual Language Models Viktor Hangya1,2, Hossain Shaikh Saadi3 and Alexander Fraser1,2 1Center for Information and Language Processing, LMU Munich, Germany 2Munich Center for Machine Learning, Germany 3Technical University of Munich, Germany {hangyav, fraser}@cis.lmu.de shaikh.saadi@tum.de Abstract Pre-trained multilingual language models are the foundation of many NLP approaches, in- cluding cross-lingual transfer solutions. How- ever, languages with small available mono- lingual corpora are often not well-supported by these models leading to poor performance. We propose an unsupervised approach to im- prove the cross-lingual representations of low- resource languages by bootstrapping word trans- lation pairs from monolingual corpora and us- ing them to improve language alignment in pre- trained language models. We perform experi- ments on nine languages, using contextual word retrieval and zero-shot named entity recognition to measure both intrinsic cross-lingual word representation quality and downstream task per- formance, showing improvements on both tasks. Our results show that it is possible to improve pre-trained multilingual language models by relying only on non-parallel resources. 1 Introduction Pre-trained language models (LMs) have replaced static word embeddings, such as word2vec, fast- Text or GloVe (Mikolov et al., 2013a; Bojanowski et al., 2017; Jameel and Schockaert, 2016), due to their superior contextualized representations. Ap- proaches such as mBERT (Devlin et al., 2019) or XLM-R (Conneau et al., 2020a) are trained on mul- tiple languages simultaneously resulting in mul- tilingual models which can be used for various cross-lingual transfer learning tasks (Conneau et al., 2018b; Schuster et al., 2019; Artetxe et al., 2020, inter alia). However, multilingual LMs mainly focus on high resource languages, e.g., mBERT supports the top 104 languages based on Wikipedia sizes, while XLM-R supports the top 100 based on Common- Crawl data. Additionally, many of these languages are underrepresented leading to low model perfor- mance in both monolingual and cross-lingual se- tups. Due to the small data sizes of low-resource languages, subword tokenizers trained jointly on multiple languages tend to over-split the tokens of such languages and LMs are not able to learn good quality representations for them. Recent work have shown that pre-trained LMs can be improved on low-resource languages using vocabulary ex- tension and model fine-tuning (Wang et al., 2020). The cross-lingual quality of LMs can further be improved by learning an additional alignment of language pairs (Aldarmaki and Diab, 2019; Wang et al., 2019) or fine-tuning the whole model (Cao et al., 2020). However these methods require cross- lingual data, often in the form of word aligned par- allel sentences, to improve cross-linguality which is often missing for low-resource languages. In this work we propose an unsupervised ap- proach to improve the language support of low- resource languages without any parallel data. Rely- ing on the initial cross-lingual quality of mBERT we mine word translation pairs from monolingual data of the source and target language pairs by lever- aging contextualized cross-lingual …

Content truncated to 500 words