Improving the Quality Trade-Off for Neural Machine Translation Multi-Domain Adaptation

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 8470–8477 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 8470 Improving the Quality Trade-Off for Neural Machine Translation Multi-Domain Adaptation Eva Hasler∗ Tobias Domhan∗ Jonay Trenous Ke Tran Bill Byrne Felix Hieber Amazon AI Translate Berlin, Germany {ehasler,domhant,trenous,trnke,willbyrn,fhieber}@amazon.com Abstract Building neural machine translation systems to perform well on a specific target domain is a well-studied problem. Optimizing system per- formance for multiple, diverse target domains however remains a challenge. We study this problem in an adaptation setting where the goal is to preserve the existing system quality while incorporating data for domains that were not the focus of the original translation system. We find that we can improve over the performance trade-off offered by Elastic Weight Consolida- tion with a relatively simple data mixing strat- egy. At comparable performance on the new domains, catastrophic forgetting is mitigated significantly on strong WMT baselines. Com- bining both approaches improves the Pareto frontier on this task. 1 Introduction The quality of Neural Machine Translation (NMT) has improved considerably in recent years, mostly due to improvements in model architecture (Bah- danau et al., 2015; Cho et al., 2014; Vaswani et al., 2017; Chen et al., 2018). Training NMT models typically involves collecting parallel training data from multiple sources to achieve high translation quality and generalize well to unseen data (Barrault et al., 2019). However, translation quality depends strongly on the relevance of the training data to the input text, which is why performance varies across target domains (Koehn and Knowles, 2017a). A popular method for domain adaptation of NMT models is fine-tuning generic models on in- domain data to yield a domain-specific model (Lu- ong and Manning, 2015; Freitag and Al-Onaizan, 2016). When high quality output on more than one target domain is required, multi-domain adaptation methods aim to produce a single system that per- forms well on multiple domains (Britz et al., 2017; Pham et al., 2019; Currey et al., 2020). ∗Equal contributions. Our goal is to train a single NMT system per lan- guage pair that performs well across many different domains. This is motivated by simplified deploy- ment and maintenance in an industrial setting with hundreds of supported language pairs. At any point in the deployment cycle, new parallel training data – often significantly smaller than the original train- ing data – may become available for an additional domain that the system has not yet been optimized for. Depending on the size of this additional data, fully retraining the NMT system may not be prac- tical as it would require costly experimentation to find the right level of upsampling which might in turn lead to overfitting on that data. In addition, as system stability is desirable in an industrial setting, we want to maintain the status-quo performance – or generic domain performance – of our models which is easier to control in an adaptation setting. In this paper, we explore the following research question: given a strong general-purpose …

Content truncated to 500 words