Uncertainty-Aware Balancing for Multilingual and Multi-Domain Neural Machine Translation Training

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7291–7305 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 7291 Uncertainty-Aware Balancing for Multilingual and Multi-Domain Neural Machine Translation Training Minghao Wu♥∗ Yitong Li♣ Meng Zhang♠ Liangyou Li♠ Gholamreza Haffari♥ Qun Liu♠ ♥Monash University ♣ Huawei Technologies Co., Ltd. ♠ Huawei Noah’s Ark Lab {minghao.wu, gholamreza.haffari}@monash.edu {liyitong3, zhangmeng92, liliangyou, qun.liu}@huawei.com Abstract Learning multilingual and multi-domain trans- lation model is challenging as the heteroge- neous and imbalanced data make the model converge inconsistently over different corpora in real world. One common practice is to ad- just the share of each corpus in the training, so that the learning process is balanced and low-resource cases can benefit from the high- resource ones. However, automatic balancing methods usually depend on the intra- and inter- dataset characteristics, which is usually agnos- tic or requires human priors. In this work, we propose an approach, MULTIUAT, that dy- namically adjusts the training data usage based on the model’s uncertainty on a small set of trusted clean data for multi-corpus machine translation. We experiment with two classes of uncertainty measures on multilingual (16 lan- guages with 4 settings) and multi-domain set- tings (4 for in-domain and 2 for out-of-domain on English-German translation) and demon- strate our approach MULTIUAT substantially outperforms its baselines, including both static and dynamic strategies. We analyze the cross- domain transfer and show the deficiency of static and similarity based methods.1 1 Introduction Text corpora are commonly collected from several different sources in different languages, raising the problem of learning a NLP system from the hetero- geneous corpora, such as multilingual models (Wu and Dredze, 2019; Arivazhagan et al., 2019; Aha- roni et al., 2019; Freitag and Firat, 2020; Arthur et al., 2021) and multi-domain models (Daumé III, 2007; Li et al., 2019; Deng et al., 2020; Jiang et al., 2020). A strong demand is to deploy a unified model for all the languages and domains, because Lab.∗Work done during the internship at Huawei Noah’s Ark 1Code available at https://github.com/ huawei-noah/noah-research/tree/master/ noahnmt/multiuat a unified model is much more resource-efficient, and knowledge learned from high-resource lan- guages/domains (HRLs/HRDs) can be transferred to low-resource languages/domains (LRLs/LRDs). One common issue on training models across corpora is that data from a variety of corpora are both heterogeneous (different corpora reveal dif- ferent linguistic properties) and imbalance (the ac- cessibility of training data varies across corpora). The standard practice to address this issue is to ad- just the training data distribution heuristically by up-sampling the training data from LRLs/LRDs (Arivazhagan et al., 2019; Conneau et al., 2020). Arivazhagan et al. (2019) rescale the training data distribution with a heuristic temperature term and demonstrate that the ideal temperature can sub- stantially improve the overall performance. How- ever, the optimal value for such heuristics is both hard to find and varies from one experimental set- ting to another (Wang and Neubig, 2019; Wang et al., 2020a,b). Wang et al. (2020a) and Wang et al. (2020b) hypothesize that the …

Content truncated to 500 words