Towards Higher Pareto Frontier in Multilingual Machine Translation

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 3802–3818 July 9-14, 2023 ©2023 Association for Computational Linguistics Towards Higher Pareto Frontier in Multilingual Machine Translation Yichong Huang:, Xiaocheng Feng:;, Xinwei Geng:, Baohang Li:, Bing Qin:; :Harbin Institute of Technology ; Peng Cheng Laboratory {ychuang,xcfeng,xwgeng,baohangli,qinb}@ir.hit.edu.cn Abstract Multilingual neural machine translation has witnessed remarkable progress in recent years. However, the long-tailed distribution of multi- lingual corpora poses a challenge of Pareto op- timization, i.e., optimizing for some languages may come at the cost of degrading the perfor- mance of others. Existing balancing training strategies are equivalent to a series of Pareto optimal solutions, which trade off on a Pareto frontier1. In this work, we propose a new training framework, Pareto Mutual Distillation (Pareto-MD), towards pushing the Pareto fron- tier outwards rather than making trade-offs. Specifically, Pareto-MD collaboratively trains two Pareto optimal solutions that favor differ- ent languages and allows them to learn from the strengths of each other via knowledge distil- lation. Furthermore, we introduce a novel strat- egy to enable stronger communication between Pareto optimal solutions and broaden the appli- cability of our approach. Experimental results on the widely-used WMT and TED datasets show that our method significantly pushes the Pareto frontier and outperforms baselines by up to +2.46 BLEU2. 1 Introduction Multilingual neural machine translation (MNMT) is a popular paradigm that uses a unified model to handle the entire translation process for multiple language pairs (Ha et al., 2016; Firat et al., 2016; Johnson et al., 2017). This paradigm is particu- larly effective at improving the performance of low- resource languages through transfer learning (Aha- roni et al., 2019; Dabre et al., 2020; Siddhant et al., 2022). Besides, MNMT is highly deployable since only one model is required (Fan et al., 2021; Yang et al., 2021; NLLB Team et al., 2022). However, the severely imbalanced distribution of multilingual training data puts the MNMT in 1In Pareto optimization, Pareto optimal solutions refer to solutions in which none of the objectives can be improved without sacrificing at least one of the other objectives. The set HRLs LRLs A A' C B B' 5 7 9 13 10 12 16 18 14 11 Figure 1: Multilingual performance frontier shifts out- wards. X-axis and Y-axis indicate the performance of Low-Resource Languages and High-Resource Lan- guages, respectively. Existing methods reflect a trade- off on the Pareto frontier (i.e., the gray curve). Our work aims to push the original Pareto frontier i.e., the blue dotted curve. To this effect, we ameliorate each individ- ual model’s shortcoming while retaining their strengths, e.g., moving right the solution A to A1 and moving up the solution B to B1, via our Pareto Mutual Distillation. a situation of Pareto optimization (also known as multi-objective optimization). That is, when some languages are optimized, others degenerate. Ex- isting methods can be considered a set of Pareto optimal solutions that trade off on a Pareto fron- tier, which focus on balancing the performance across …

Content truncated to 500 words