Adapting Monolingual Models: Data can be Scarce when Language Similarity is High

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 4901–4907 August 1–6, 2021. ©2021 Association for Computational Linguistics 4901 Adapting Monolingual Models: Data can be Scarce when Language Similarity is High Wietse de Vries∗, Martijn Bartelds∗, Malvina Nissim and Martijn Wieling University of Groningen The Netherlands {wietse.de.vries, m.bartelds, m.nissim, m.b.wieling}@rug.nl Abstract For many (minority) languages, the resources needed to train large models are not available. We investigate the performance of zero-shot transfer learning with as little data as possi- ble, and the influence of language similarity in this process. We retrain the lexical layers of four BERT-based models using data from two low-resource target language varieties, while the Transformer layers are independently fine- tuned on a POS-tagging task in the model’s source language. By combining the new lex- ical layers and fine-tuned Transformer layers, we achieve high task performance for both target languages. With high language sim- ilarity, 10MB of data appears sufficient to achieve substantial monolingual transfer per- formance. Monolingual BERT-based models generally achieve higher downstream task per- formance after retraining the lexical layer than multilingual BERT, even when the target lan- guage is included in the multilingual model. 1 Introduction Large pre-trained language models are the domi- nant approach for solving many tasks in natural language processing. These models represent lin- guistic structure on the basis of large corpora that exist for high-resource languages, such as English. However, for the majority of the world’s languages, these large corpora are not available. Past work on multilingual learning has found that multilingual BERT (mBERT; Devlin et al. 2019a) generalizes across languages with high zero-shot transfer performance on a variety of tasks (Pires et al., 2019; Wu and Dredze, 2019). However, it has also been observed that high-resource languages included in mBERT pre-training often have a better- performing monolingual model, and low-resource ∗These authors contributed equally. languages that are not included in mBERT pre- training usually show poor performance (Nozza et al., 2020; Wu and Dredze, 2020). An alternative to multilingual transfer learning is the adaptation of existing monolingual models to other languages. Zoph et al. (2016) introduce a method for transferring a pre-trained machine translation model to lower-resource languages by only fine-tuning the lexical layer. This method has also been applied to BERT (Artetxe et al., 2020) and GPT-2 (de Vries and Nissim, 2020). Artetxe et al. (2020) also show that BERT models with retrained lexical layers perform well in downstream tasks, but comparatively high performance has only been demonstrated for languages for which at least 400MB of data is available. To test if this procedure is also effective for low- to zero-resource languages, we consider two re- gional language varieties spoken in the North of the Netherlands, namely Gronings (Low Saxon lan- guage variant) and West Frisian. Figure 1: Geographical areas where Gronings (in green) and West Frisian (in red) are spoken. Im- age modified from https://en.wikipedia.org/ wiki/Low_German. 4902 Figure 1 visualizes the geographical areas where these regional language variants are spoken. The re- gional Low Saxon language …

Content truncated to 500 words