Expanding Pretrained Models to Thousands More Languages via Lexicon-based Adaptation

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 863 - 877 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Expanding Pretrained Models to Thousands More Languages via Lexicon-based Adaptation Xinyi Wang1 Sebastian Ruder2 Graham Neubig1 1Language Technology Institute, Carnegie Mellon University 2Google Research xinyiw1@cs.cmu.edu,ruder@google.com,gneubig@cs.cmu.edu Abstract The performance of multilingual pretrained models is highly dependent on the availability of monolingual or parallel text present in a tar- get language. Thus, the majority of the world’s languages cannot benefit from recent progress in NLP as they have no or limited textual data. To expand possibilities of using NLP technol- ogy in these under-represented languages, we systematically study strategies that relax the reliance on conventional language resources through the use of bilingual lexicons, an al- ternative resource with much better language coverage. We analyze different strategies to synthesize textual or labeled data using lexi- cons, and how this data can be combined with monolingual or parallel text when available. For 19 under-represented languages across 3 tasks, our methods lead to consistent improve- ments of up to 5 and 15 points with and with- out extra monolingual text respectively. Over- all, our study highlights how NLP methods can be adapted to thousands more languages that are under-served by current technology.1 1 Introduction Multilingual pretrained models (Devlin et al., 2019; Conneau and Lample, 2019; Conneau et al., 2020) have become an essential method for cross-lingual transfer on a variety of NLP tasks (Pires et al., 2019; Wu and Dredze, 2019). These models can be finetuned on annotated data of a down-stream task in a high-resource language, often English, and then the resulting model is applied to other languages. This paradigm is supposed to benefit under-represented languages that do not have an- notated data. However, recent studies have found that the cross-lingual transfer performance of a language is highly contingent on the availability of monolingual data in the language during pre- training (Hu et al., 2020). Languages with more 1Code and data are available at: https: //github.com/cindyxinyiwang/ expand-via-lexicon-based-adaptation. mBERT/NMT Wikipedia/ CommonCrawl Bible Lexicons Covered Language Varieties 0 1750 3500 5250 7000 1% 4% 23% 70% Figure 1: The percentage of the world’s ≈7,000 languages covered by mBERT, monolingual data sources and lexicons. monolingual data tend to have better performance while languages not present during pretraining sig- nificantly lag behind. Several works propose methods to adapt the pretrained multilingual models to low-resource languages, but these generally involve continued training using monolingual text from these lan- guages (Wang et al., 2020; Chau et al., 2020; Pfeif- fer et al., 2020, 2021). Therefore, the performance of these methods is still constrained by the amount of monolingual or parallel text available, making it difficult for languages with little or no textual data to benefit from the progress in pretrained models. Joshi et al. (2020) indeed argue that unsupervised pretraining makes the ‘resource-poor poorer’. Fig. 1 plots the language coverage of multilin- gual BERT (mBERT; Devlin et al., 2019), a widely used …

Content truncated to 500 words