Phylogeny-Inspired Adaptation of Multilingual Models to New Languages

Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 434–452 November 20–23, 2022. ©2022 Association for Computational Linguistics 434 Phylogeny-Inspired Adaptation of Multilingual Models to New Languages Fahim Faisal, Antonios Anastasopoulos Department of Computer Science, George Mason University {ffaisal,antonis}@gmu.edu Abstract Large pretrained multilingual models, trained on dozens of languages, have delivered promis- ing results due to cross-lingual learning capa- bilities on a variety of language tasks. Further adapting these models to specific languages, especially ones unseen during pre-training, is an important goal toward expanding the cover- age of language technologies. In this study, we show how we can use language phylogenetic information to improve cross-lingual transfer leveraging closely related languages in a struc- tured, linguistically-informed manner. We per- form adapter-based training on languages from diverse language families (Germanic, Uralic, Tupian, Uto-Aztecan) and evaluate on both syn- tactic and semantic tasks, obtaining more than 20% relative performance improvements over strong commonly used baselines, especially on languages unseen during pre-training. 1 1 Introduction Language models have now become the standard for building state-of-the-art Natural Language Pro- cessing (NLP) systems. Beyond monolingual mod- els, large-scale multilingual models covering more than 100 languages are now available, such as XLM- R by Conneau et al. (2020) and mBERT by Devlin et al. (2019), achieving competitive performance across languages from a variety of families and using various scripts. Still, most of the 6500+ spoken languages in the world (Hammarström, 2016) are not covered –remaining unseen– by those models. Even lan- guages with millions of native speakers like Lin- gala (with 15-20 million speakers in central Africa, mostly D.R. Congo) or Bambara (spoken by around 5 million people in Mali and neighboring countries) are not covered by any available language models at the time of writing. 1Code and data are publicly available: https://github. com/ffaisal93/adapt_lang_phylogeny A recent line of work (see §2) has shown that these large multilingual language models (MLMs) can be finetuned on individual languages to further improve performance. Even better, they can be even adapted to languages unseen during the pre- training stage.2 This work focuses on using adapters, a popu- lar framework for such adaptation that has been proven successful for zero-shot and few-shot cross- lingual transfer. In particular, we significantly im- prove the adapter framework by drawing inspira- tion from a simple insight: that the adapters of related languages would likely need to perform the same function, and thus adapters could be trained leveraging multiple related languages. We impose a phylogenetically-inspired tree hierarchy for parameter-sharing between adapters and show empirically that our approach leads to large im- provements with experiments on three NLP tasks on several language families. 2 Background Adapting Large-Scale Models to Low-Resource Languages Multilingual language models (MLMs) can be used directly on unseen languages, or they can also be adapted using unsupervised methods. For example, Han and Eisenstein (2019) successfully used continued training with masked language modeling on unlabeled …

Content truncated to 500 words