Addressing Asymmetry in Multilingual Neural Machine Translation with Fuzzy Task Clustering

Proceedings of the 29th International Conference on Computational Linguistics, pages 5129–5141 October 12–17, 2022. 5129 Addressing Asymmetry in Multilingual Neural Machine Translation with Fuzzy Task Clustering Qian Wang1,2 and Jiajun Zhang1,2,3∗ 1National Laboratory of Pattern Recognition, Institute of Automation, CAS, Beijing, China 2School of Artificial Intelligence, University of Chinese Academy of Sciences , Beijing, China 3Beijing Academy of Artificial Intelligence, Beijing, China {qian.wang, jjzhang}@nlpr.ia.ac.cn Abstract Multilingual neural machine translation (NMT) enables positive knowledge transfer among multiple translation tasks with a shared underly- ing model, but a unified multilingual model usu- ally suffers from capacity bottleneck when tens or hundreds of languages are involved. A pos- sible solution is to cluster languages and train individual model for each cluster. However, the existing clustering methods based on lan- guage similarity cannot handle the asymmetric problem in multilingual NMT, i.e., one transla- tion task A can benefit from another translation task B but task B will be harmed by task A. To address this problem, we propose a fuzzy task clustering method for multilingual NMT. Specifically, we employ task affinity, defined as the loss change of one translation task caused by the training of another, as the clustering cri- terion. Next, we cluster the translation tasks based on the task affinity, such that tasks from the same cluster can benefit each other. For each cluster, we further find out a set of aux- iliary translation tasks that benefit the tasks in this cluster. In this way, the model for each clus- ter is trained not only on the tasks in the cluster but also on the auxiliary tasks. During training, we design a dynamic task sampling strategy that eliminate the negative influence of auxil- iary tasks while exploit the positive knowledge of them. We conduct extensive experiments for one-to-many, many-to-one, and many-to-many translation scenarios to verify the effectiveness of our method. 1 Introduction Neural machine translation (NMT) has achieved great success in recent years (Sutskever et al., 2014; Bahdanau et al., 2015; Vaswani et al., 2017). The conventional bilingual translation model well han- dles the translation task for a single language pair, but it is infeasible to train an individual model for each language pair since there are thousands of ∗Corresponding Author: Jiajun Zhang. A B∗ C∗ en-it en-nl en-de en-fr en-zh en-ja Indiv 32.6 31.6 28.1 38.8 22.2 13.6 Joint 34.3 34.3 25.9 37.9 25.3 10.4 Table 1: Three kinds of task relationship. (A): Tasks benefit from each other when jointly trained. (B): Tasks are negatively influenced by each other. (C): The task en-zh benefits from en-ja but en-zh is harmful to en-ja. The models are trained on the IWSLT’17 dataset and results with ∗ are reported in Xu and Yvon (2021). languages in the world. To improve the computa- tional efficiency, researchers propose multilingual NMT that enables one model to handle multiple translation tasks (Ha et al., 2016; Johnson et al., 2017). Apart from the benefits for training and de- ployment, the shared underlying neural network in multilingual NMT also brings …

Content truncated to 500 words