Effective Domain Mixing for Neural Machine Translation

Proceedings of the Conference on Machine Translation (WMT), Volume 1: Research Papers, pages 118–126 Copenhagen, Denmark, September 711, 2017. c⃝2017 Association for Computational Linguistics Effective Domain Mixing for Neural Machine Translation Reid Pryzant∗ Stanford University rpryzant@stanford.edu Denny Britz∗ Google Brain dennybritz@google.com Quoc V. Le Google Brain qvl@google.com Abstract Neural Machine Translation (NMT) models are often trained on hetero- geneous mixtures of domains, from news to parliamentary proceedings, each with unique distributions and lan- guage. In this work we show that train- ing NMT systems on naively mixed data can degrade performance versus models fit to each constituent domain. We demonstrate that this problem can be circumvented, and propose three models that do so by jointly learn- ing domain discrimination and transla- tion. We demonstrate the efficacy of these techniques by merging pairs of domains in three languages: Chinese, French, and Japanese. After training on composite data, each approach out- performs its domain-specific counter- parts, with a model based on a discrim- inator network doing so most reliably. We obtain consistent performance im- provements and an average increase of 1.1 BLEU. 1 Introduction Neural Machine Translation (NMT) (Kalch- brenner and Blunsom, 2013; Sutskever et al., 2014; Cho et al., 2014) is an end-to-end ap- proach for automated translation. NMT has shown impressive results (Bahdanau et al., 2015; Luong et al., 2015a; Wu et al., 2016) often surpassing those of phrase-based sys- tems while addressing shortcomings such as the need for hand-engineered features. In many translation settings (e.g. web translation, assistant translators), input may ∗Equal Contribution. come from more than one domain. Each do- main has unique properties that could con- found models not explicitly fitted to it. Thus, an important problem is to effectively mix a diversity of training data in a multi-domain setting. Our problem space is as follows: how can we train a translation model on multi-domain data to improve test-time performance in each constituent domain? This setting differs from the majority of work in domain adaptation, which explores how models trained on some source domain can be effectively applied to outside target domains. This setting is impor- tant, because previous research has shown that both standard NMT and adaptation methods degrade performance on the original source do- main(s) (Farajian et al., 2017; Haddow and Koehn, 2012). We seek to prove that this prob- lem can be overcome, and hypothesize that leveraging the heterogeneity of composite data rather than dampening it will allow us to do so. To this extent, we propose three new models for multi-domain machine translation. These models are based on discriminator networks, adversarial learning, and target-side domain tokens. We evaluate on pairs of linguisti- cally disparate corpora in three translation tasks (EN-JA, EN-ZH, EN-FR), and observe that unlike naively training on mixed data (as per current best practices), the proposed tech- niques consistently improve translation quality in each individual setting. The most signifi- cant of these tasks is EN-JA, where we obtain state-of-the-art performance in the process of examining the ASPEC corpus (Nakazawa et al., …

Content truncated to 500 words