MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 7654–7673, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 7654 MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer Jonas Pfeiffer1, Ivan Vuli´c2, Iryna Gurevych1, Sebastian Ruder3 1Ubiquitous Knowledge Processing Lab, Technical University of Darmstadt 2Language Technology Lab, University of Cambridge 3DeepMind pfeiffer@ukp.tu-darmstadt.de Abstract The main goal behind state-of-the-art pre- trained multilingual models such as multilin- gual BERT and XLM-R is enabling and boot- strapping NLP applications in low-resource languages through zero-shot or few-shot cross- lingual transfer. However, due to limited model capacity, their transfer performance is the weakest exactly on such low-resource lan- guages and languages unseen during pretrain- ing. We propose MAD-X, an adapter-based framework that enables high portability and parameter-efficient transfer to arbitrary tasks and languages by learning modular language and task representations. In addition, we in- troduce a novel invertible adapter architecture and a strong baseline method for adapting a pretrained multilingual model to a new lan- guage. MAD-X outperforms the state of the art in cross-lingual transfer across a represen- tative set of typologically diverse languages on named entity recognition and causal common- sense reasoning, and achieves competitive re- sults on question answering. Our code and adapters are available at AdapterHub.ml. 1 Introduction Current deep pretrained multilingual models (De- vlin et al., 2019; Conneau and Lample, 2019) achieve state-of-the-art results on cross-lingual transfer, but do not have enough capacity to repre- sent all languages. Evidence for this is the impor- tance of the vocabulary size (Artetxe et al., 2020) and the curse of multilinguality (Conneau et al., 2020), a trade-off between language coverage and model capacity. Scaling up a model to cover all of the world’s 7,000+ languages is prohibitive. At the same time, limited capacity is an issue even for high-resource languages where state-of-the-art multilingual models underperform their monolin- gual variants (Eisenschlos et al., 2019; Virtanen et al., 2019; Nozza et al., 2020), and performance decreases further with lower-resource languages covered by the pretrained models. Moreover, the model capacity issue is arguably most severe for languages that were not included in the training data at all, and pretrained models perform poorly on those languages (Ponti et al., 2020b). In this paper, we propose Multiple ADapters for Cross-lingual transfer (MAD-X), a modular framework that leverages a small number of extra parameters to address the fundamental capacity issue that limits pretrained multilingual models. Using a state-of-the-art multilingual model as the foundation, we adapt the model to arbitrary tasks and languages by learning modular language- and task-specific representations via adapters (Rebuffi et al., 2017; Houlsby et al., 2019), small bottleneck layers inserted between a model’s weights. In particular, using a recent efficient adapter vari- ant (Pfeiffer et al., 2020a; R¨uckl´e et al., 2020), we train 1) language-specific adapter modules via masked language modelling (MLM) on unlabelled target language data, and 2) task-specific adapter modules via optimising a target task on labelled data in any source language. Task and language adapters are stacked as in Figure …

Content truncated to 500 words