Learning a Multi-Domain Curriculum for Neural Machine Translation

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7711–7723 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 7711 Learning a Multi-Domain Curriculum for Neural Machine Translation Wei Wang Google Research wangwe@google.com Ye Tian Google Research ytian@google.com Jiquan Ngiam Google Brain jngiam@google.com Yinfei Yang Google Research yinfeiy@google.com Isaac Caswell Google Research icaswell@google.com Zarana Parekh Google Research zarana@google.com Abstract Most data selection research in machine trans- lation focuses on improving a single domain. We perform data selection for multiple do- mains at once. This is achieved by carefully introducing instance-level domain-relevance features and automatically constructing a training curriculum to gradually concentrate on multi-domain relevant and noise-reduced data batches. Both the choice of features and the use of curriculum are crucial for balancing and improving all domains, including out-of- domain. In large-scale experiments, the multi- domain curriculum simultaneously reaches or outperforms the individual performance and brings solid gains over no-curriculum training. 1 Introduction In machine translation (MT), data selection, e.g., (Moore and Lewis, 2010; Axelrod et al., 2011), has remained as a fundamental and important re- search topic. It has played a crucial role in domain adaptation by selecting domain-matching training examples, or data cleaning (aka denoising) by se- lecting high-quality examples. So far, the most extensively studied scenario assumes a single do- main to improve. It becomes both technically challenging and practically appealing to build a large-scale multi- domain neural machine translation (NMT) model that performs simultaneously well on multiple do- mains at once. This requires addressing research challenges such as catastrophic forgetting (Good- fellow et al., 2014) at scale and data balancing. Such a model can easily find potential use cases, i.e., as a solid general service, for downstream transfer learning, for better deployment efficiency, or for transfer learning across datasets. Unfortunately, existing single-domain data- selection methods do not work well for multiple domains. For example, improving the translation Static Dynamic Single domain Y Y noise Y Y Multi domain Y noise N N (Our Work) Table 1: Data selection and data mixing research in NMT. ‘Y’: There is previous research that studies this case. ‘N’: No previous research has studied this case. accuracy of one domain will often hurt that of an- other (van der Wees et al., 2017; Britz et al., 2017), and improving model generalization across all do- mains by clean-data selection (Koehn et al., 2018) may not promise optimization of a particular do- main. Multiple aspects need to be considered for training a multi-domain model. This paper presents a dynamic data selection method to multi-domain NMT. Things we do dif- ferently from previous work in mixing data are the choice of instance-level features and the em- ployment of a multi-domain curriculum that is ad- ditionally able to denoise. These are crucial for mixing and improving all domains, including out- of-domain. We experiment with large datasets at different noise levels and show that the resulting models meet our requirements. 2 Related Work In MT, research that is most relevant to our …

Content truncated to 500 words