Distilling Multiple Domains for Neural Machine Translation

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 4500–4511, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 4500 Distilling Multiple Domains for Neural Machine Translation Anna Currey Prashant Mathur Amazon AI {ancurrey,pramathu,gddinu}@amazon.com Georgiana Dinu Abstract Neural machine translation achieves impres- sive results in high-resource conditions, but performance often suffers when the input do- main is low-resource. The standard practice of adapting a separate model for each domain of interest does not scale well in practice from both a quality perspective (brittleness under domain shift) as well as a cost perspective (added maintenance and inference complex- ity). In this paper, we propose a framework for training a single multi-domain neural ma- chine translation model that is able to trans- late several domains without increasing infer- ence time or memory usage. We show that this model can improve translation on both high- and low-resource domains over strong multi- domain baselines. In addition, our proposed model is effective when domain labels are un- known during training, as well as robust under noisy data conditions. 1 Introduction Neural machine translation (NMT) (Bahdanau et al., 2015; Cho et al., 2014; Kalchbrenner and Blunsom, 2013; Sutskever et al., 2014) can achieve high quality when trained using deep architectures on large amounts of relevant data (Barrault et al., 2019). However, training data for generic trans- lation models is typically not balanced or diverse with respect to domain. As a result, translation qual- ity can be inconsistent across domains, with lower- quality outputs for low-resource domains such as chat compared to high-resource domains such as news (Koehn and Knowles, 2017). One way to address this is by building domain- adapted models (Freitag and Al-Onaizan, 2016; Chu and Wang, 2018) that target a specific domain. In this case, in-domain data is used to specialize the machine translation (MT) model for the target do- main, for example by fine-tuning a generic model Figure 1: Overview of the multi-domain knowledge distillation (MDKD) method. A single multi-domain model is trained on data that is distilled from high- performance deep teachers. MDKD trains multiple deep teachers, each an expert in a specific domain. on this data (Freitag and Al-Onaizan, 2016; Luong and Manning, 2015; Sennrich et al., 2016a; Servan et al., 2016). This yields “expert” models that are better than models trained on the in-domain data alone and more specialized than a generic transla- tion system. However, for an MT application that needs to translate multiple domains, this approach would require maintaining and running separate expert systems for each domain, which becomes prohibitively expensive as the number of domains increases. Additionally, in a real-world scenario, the domain of the input text might be unknown at inference time, adding the complexity of detecting which system should be used for a given input. One alternative to expert models are multi- domain MT systems (Britz et al., 2017; Farajian et al., 2017; Kobus et al., 2017; Pham et al., 2019; Sajjad et al., 2017). Specifically, the goal of a multi- domain …

Content truncated to 500 words