Domain Adaptive Inference for Neural Machine Translation

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 222–228 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 222 Domain Adaptive Inference for Neural Machine Translation Danielle Saunders† and Felix Stahlberg† and Adri`a de Gispert‡ and Bill Byrne‡† †Department of Engineering, University of Cambridge, UK ‡SDL Research, Cambridge, UK {ds636, fs439, wjb31}@cam.ac.uk, {agispert, bbyrne}@sdl.com Abstract We investigate adaptive ensemble weighting for Neural Machine Translation, addressing the case of improving performance on a new and potentially unknown domain without sac- rificing performance on the original domain. We adapt sequentially across two Spanish- English and three English-German tasks, com- paring unregularized fine-tuning, L2 and Elas- tic Weight Consolidation. We then report a novel scheme for adaptive NMT ensemble de- coding by extending Bayesian Interpolation with source information, and show strong im- provements across test domains without access to the domain label. 1 Introduction Neural Machine Translation (NMT) models are ef- fective when trained on broad domains with large datasets, such as news translation (Bojar et al., 2017). However, test data may be drawn from a different domain, on which general models can perform poorly (Koehn and Knowles, 2017). We address the problem of adapting to one or more do- mains while maintaining good performance across all domains. Crucially, we assume the realistic scenario where the domain is unknown at infer- ence time. One solution is ensembling models trained on different domains (Freitag and Al-Onaizan, 2016). This approach has two main drawbacks. Firstly, obtaining models for each domain is challenging. Training from scratch on each new domain is im- practical, while continuing training on a new do- main can cause catastrophic forgetting of previous tasks (French, 1999), even in an ensemble (Fre- itag and Al-Onaizan, 2016). Secondly, ensemble weighting requires knowledge of the test domain. We address the model training problem with regularized fine-tuning, using an L2 regularizer (Barone et al., 2017) and Elastic Weight Consol- idation (EWC) (Kirkpatrick et al., 2017). We fine- tune sequentially to translate up to three domains with the same model. We then develop an adaptive inference scheme for NMT ensembles by extending Bayesian In- terpolation (BI) (Allauzen and Riley, 2011) to sequence-to-sequence models.1 This lets us calcu- late ensemble weights adaptively over time with- out needing the domain label, giving strong im- provements over uniform ensembling for baseline and fine-tuned models. 1.1 Adaptive training In NMT fine-tuning, a model is first trained on a task A, typically translating a large general- domain corpus (Luong and Manning, 2015). The optimized parameters θ∗ A are fine-tuned on task B, a new domain. Without regularization, catas- trophic forgetting can occur: performance on task A degrades as parameters adjust to the new objec- tive. A regularized objective is: L(θ) = LB(θ) + Λ � j Fj(θj − θ∗ A,j)2 (1) where LA(θ) and LB(θ) are the likelihood of tasks A and B. We compare three cases: • No-reg, where Λ = 0 • L2, where Fj = 1 for each parameter index …

Content truncated to 500 words