On Negative Interference in Multilingual Models: Findings and A Meta-Learning Treatment

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 4438–4450, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 4438 On Negative Interference in Multilingual Models: Findings and A Meta-Learning Treatment Zirui Wang Zachary C. Lipton Yulia Tsvetkov Carnegie Mellon University, Pittsburgh, USA {ziruiw, zlipton, ytsvetko}@cs.cmu.edu Abstract Modern multilingual models are trained on concatenated text from multiple languages in hopes of conferring benefits to each (positive transfer), with the most pronounced benefits accruing to low-resource languages. How- ever, recent work has shown that this approach can degrade performance on high-resource lan- guages, a phenomenon known as negative in- terference. In this paper, we present the first systematic study of negative interference. We show that, contrary to previous belief, neg- ative interference also impacts low-resource languages. While parameters are maximally shared to learn language-universal structures, we demonstrate that language-specific param- eters do exist in multilingual models and they are a potential cause of negative interference. Motivated by these observations, we also present a meta-learning algorithm that obtains better cross-lingual transferability and allevi- ates negative interference, by adding language- specific layers as meta-parameters and train- ing them in a manner that explicitly improves shared layers’ generalization on all languages. Overall, our results show that negative interfer- ence is more common than previously known, suggesting new directions for improving mul- tilingual representations.1 1 Introduction Advances in pretraining language models (Devlin et al., 2018; Liu et al., 2019; Yang et al., 2019) as general-purpose representations have pushed the state of the art on a variety of natural language tasks. However, not all languages enjoy large pub- lic datasets for pretraining and/or downstream tasks. Multilingual language models such as mBERT (De- vlin et al., 2018) and XLM (Lample and Conneau, 2019) have been proven effective for cross-lingual 1Source code is available at https://github.com/ iedwardwangi/MetaAdapter. transfer learning by pretraining a single shared Transformer model (Vaswani et al., 2017) jointly on multiple languages. The goals of multilingual modeling are not limited to improving language modeling in low-resource languages (Lample and Conneau, 2019), but also include zero-shot cross- lingual transfer on downstream tasks—it has been shown that multilingual models can generalize to target languages even when labeled training data is only available in the source language (typically English) on a wide range of tasks (Pires et al., 2019; Wu and Dredze, 2019; Hu et al., 2020). However, multilingual models are not equally beneficial for all languages. Conneau et al. (2019) demonstrated that including more languages in a single model can improve performance for low- resource languages but hurt performance for high- resource languages. Similarly, recent work (John- son et al., 2017; Tan et al., 2019; Aharoni et al., 2019; Arivazhagan et al., 2019) in multilingual neu- ral machine translation (NMT) also observed per- formance degradation on high-resource language pairs. In multi-task learning (Ruder, 2017), this phenomenon is known as negative interference or negative transfer (Wang et al., 2019), where train- ing multiple tasks jointly hinders the performance on individual tasks. Despite these empirical observations, little …

Content truncated to 500 words