Lifting the Curse of Multilinguality by Pre-training Modular Transformers

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3479 - 3495 July 10-15, 2022 ©2022 Association for Computational Linguistics Lifting the Curse of Multilinguality by Pre-training Modular Transformers Jonas Pfeiffer∗1,2,3, Naman Goyal3, Xi Victoria Lin3, Xian Li3, James Cross3, Sebastian Riedel3, Mikel Artetxe3 1New York University, 2TU Darmstadt, 3Meta AI Abstract Multilingual pre-trained models are known to suffer from the curse of multilinguality, which causes per-language performance to drop as they cover more languages. We address this is- sue by introducing language-specific modules, which allows us to grow the total capacity of the model, while keeping the total number of trainable parameters per language constant. In contrast with prior work that learns language- specific components post-hoc, we pre-train the modules of our Cross-lingual Modular (X- MOD) models from the start. Our experiments on natural language inference, named entity recognition and question answering show that our approach not only mitigates the negative interference between languages, but also en- ables positive transfer, resulting in improved monolingual and cross-lingual performance. Furthermore, our approach enables adding lan- guages post-hoc with no measurable drop in performance, no longer limiting the model us- age to the set of pre-trained languages. 1 Introduction Recent work on multilingual NLP has focused on pre-training transformer-based models (Vaswani et al., 2017) on concatenated corpora of a large number of languages (Devlin et al., 2019; Conneau et al., 2020). These multilingual models have been shown to work surprisingly well in cross-lingual settings, despite the fact that they do not rely on direct cross-lingual supervision (e.g., parallel data or translation dictionaries; Pires et al., 2019; Wu and Dredze, 2019; Artetxe et al., 2020; Hu et al., 2020; K et al., 2020; Rust et al., 2021). However, recent work has uncovered fundamen- tal limitations of multilingual transformers. Con- neau et al. (2020) observe that pre-training a model with a fixed capacity on an increasing amount of languages only improves its cross-lingual perfor- mance up to a certain point, after which perfor- ∗ Work done while interning at Meta AI. Feed Forward Add & Norm Multi-Head Attention Add & Norm FF Down FF Up Add & Norm FF Down FF Up Lannguage n Lannguage 1 ... Figure 1: A transformer layer of our proposed modular architecture. The dark blue and green components illus- trate the modular layers, which are language specific. The Multi-Head Attention and Feed-Forward compo- nents are shared by all languages. mance drops can be measured—a phenomenon known as the curse of multilinguality (Figure 2). As such, prior work had to find a trade-off between supporting more languages and obtaining better performance on a smaller set of languages. In this work, we address this problem by in- troducing language-specific, modular components during pre-training (Figure 1). Our Cross-lingual, Modular (X-MOD) language model shares the ma- jority of the transformer parameters between all pre- training languages, while providing each language with individual capacity to learn idiosyncratic in- formation without increasing the total …

Content truncated to 500 words