Balancing Training for Multilingual Neural Machine Translation

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8526–8537 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 8526 Balancing Training for Multilingual Neural Machine Translation Xinyi Wang Yulia Tsvetkov Graham Neubig Language Technology Institute, Carnegie Mellon University, Pittsburgh, PA 15213 {xinyiw1,ytsvetko,gneubig}@cs.cmu.edu Abstract When training multilingual machine transla- tion (MT) models that can translate to/from multiple languages, we are faced with im- balanced training sets: some languages have much more training data than others. Stan- dard practice is to up-sample less resourced languages to increase representation, and the degree of up-sampling has a large effect on the overall performance. In this paper, we propose a method that instead automatically learns how to weight training data through a data scorer that is optimized to maximize performance on all test languages. Experiments on two sets of languages under both one-to-many and many- to-one MT settings show our method not only consistently outperforms heuristic baselines in terms of average performance, but also offers flexible control over the performance of which languages are optimized.1 1 Introduction Multilingual models are trained to process differ- ent languages in a single model, and have been applied to a wide variety of NLP tasks such as text classification (Klementiev et al., 2012; Chen et al., 2018a), syntactic analysis (Plank et al., 2016; Ammar et al., 2016), named-entity recog- nition (Xie et al., 2018; Wu and Dredze, 2019), and machine translation (MT) (Dong et al., 2015; Johnson et al., 2016). These models have two par- ticularly concrete advantages over their monolin- gual counterparts. First, deploying a single mul- tilingual model is much more resource efficient than deploying one model for each language under consideration (Arivazhagan et al., 2019; Aharoni et al., 2019). Second, multilingual training makes it possible to transfer knowledge from high-resource languages (HRLs) to improve performance on low- resource languages (LRLs) (Zoph et al., 2016; 1The code is available at https://github.com/ cindyxinyiwang/fairseq/tree/multiDDS. Nguyen and Chiang, 2018; Neubig and Hu, 2018; Wang and Neubig, 2019; Aharoni et al., 2019). A common problem with multilingual training is that the data from different languages are both het- erogeneous (different languages may exhibit very different properties) and imbalanced (there may be wildly varying amounts of training data for each language). Thus, while LRLs will often benefit from transfer from other languages, for languages where sufficient monolingual data exists, perfor- mance will often decrease due to interference from the heterogeneous nature of the data. This is espe- cially the case for modestly-sized models that are conducive to efficient deployment (Arivazhagan et al., 2019; Conneau et al., 2019). To balance the performance on different lan- guages, the standard practice is to heuristically ad- just the distribution of data used in training, specifi- cally by over-sampling the training data from LRLs (Johnson et al., 2016; Neubig and Hu, 2018; Ari- vazhagan et al., 2019; Conneau et al., 2019). For example, Arivazhagan et al. (2019) sample training data from different languages based on the dataset size scaled by a …

Content truncated to 500 words