Distributionally Robust Multilingual Machine Translation

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5664–5674 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 5664 Distributionally Robust Multilingual Machine Translation Chunting Zhou∗,1, Daniel Levy∗,2, Xian Li3, Marjan Ghazvininejad3, Graham Neubig1 1Language Technologies Institute, Carnegie Mellon University 2Stanford University 3Facebook AI {chuntinz, gneubig}@cs.cmu.edu danilevy@stanford.edu {ghazvini,xianl}@fb.com Abstract Multilingual neural machine translation (MNMT) learns to translate multiple lan- guage pairs with a single model, potentially improving both the accuracy and the memory- efficiency of deployed models. However, the heavy data imbalance between languages hinders the model from performing uniformly across language pairs. In this paper, we propose a new learning objective for MNMT based on distributionally robust optimization, which minimizes the worst-case expected loss over the set of language pairs. We further show how to practically optimize this objective for large translation corpora using an iterated best response scheme, which is both effective and incurs negligible additional computational cost compared to standard empirical risk minimization. We perform ex- tensive experiments on three sets of languages from two datasets and show that our method consistently outperforms strong baseline methods in terms of average and per-language performance under both many-to-one and one-to-many translation settings.1 1 Introduction Multilingual methods that process multiple lan- guages with one single model have gained favor across a variety of NLP tasks (Firat et al., 2016; Ha et al., 2016; Johnson et al., 2017; Devlin et al., 2019; Aharoni et al., 2019; Conneau et al., 2020) because (1) training and deployment of one mul- tilingual model is more computationally efficient compared to maintaining one model for each lan- guage (Arivazhagan et al., 2019), (2) training mul- tilingually can improve accuracy, particularly for low-resource languages (LRLs) (Zoph et al., 2016; Neubig and Hu, 2018; Pires et al., 2019). However, in multilingual training, the amount and type of training data available varies greatly ∗Equal contribution. 1Our code is available at https://github.com/ violet-zct/fairseq-dro-mnmt fr zh en ptrain τ = 100 0.200 0.600 1.200 1.800 2.400 3.000 pτ, τ ∈ [1, 100] Figure 1: Illustration of different training distributions where the training distribution of the three languages fr, zh and en is (0.1, 0.3, 0.6). Contours represent different radii of the χ2-ball around ptrain. The blue points are the tempered distributions described in §2.1. across languages. Because most models are trained using empirical risk minimization (ERM), which minimizes the average training loss on the train- ing set, high-resource languages (HRLs) with large amounts of data contribute to the majority of the training objective. When model capacity is limited, this results in trade-offs or decreased performance on some languages, particularly LRLs (Arivazha- gan et al., 2019; Wang et al., 2020b, 2021). To better control this trade-off, a common practice is to balance the training distribution by heuristic oversampling of LRLs (Johnson et al., 2017; Neu- big and Hu, 2018; Arivazhagan et al., 2019). Although simple data balancing can improve the performance on LRLs significantly, it is far from optimal. First, the sampling hyperparameters need to be adjusted for …

Content truncated to 500 words