Fine-Tuning for Neural Machine Translation with Limited Degradation across In- and Out-of-Domain Data

Fine-Tuning for Neural Machine Translation with Limited Degradation across In- and Out-of-Domain Data Praveen Dakwale p.dakwale@uva.nl Informatics Institute, University of Amsterdam Christof Monz c.monz@uva.nl Informatics Institute, University of Amsterdam Abstract Neural machine translation is a recently proposed approach which has shown competitive re- sults to traditional MT approaches. Similar to other neural network based methods, NMT also suffers from low performance for the domains with less available training data. Domain adaptation deals with improving performance of a model trained on large general domain data over test instances from a new domain. Fine-tuning is a fast and simple domain adaptation method which has demonstrated substantial improvements for various neural network based tasks including NMT. However, it suffers from drastic performance degradation on the general or source domain test sentences, which is undesirable in real-time applications. To address this problem of drastic degradation, in this paper, we propose two simple modifications to the fine- tuning approach, namely multi-objective learning and multi-output learning which are based on the “Knowledge distillation” framework. Experiments on English-German translations demon- strate that our approaches achieve results comparable to simple fine-tuning on the target domain task with comparatively little loss on the general domain task. 1 Introduction Standard neural MT (Bahdanau et al., 2015) is an end-to-end neural network which allows for easy training of the NMT system without the need to separately train large phrase table and n-gram language models. However, neural methods including NMT, are known to be data- hungry and do not generalize well for rare events. As a result, NMT systems trained only on a specific domain with less available parallel data quickly overfit and do not perform better or even comparable to standard statistical MT approaches (Zoph et al., 2016). Research in domain adaptation deals with the problem of improving the performance of a model trained on a general domain data over test instances from a new domain. A simple solution to achieve better performance on both domains will be to train the network on the com- bined in-domain and general domain parallel data. However, as already discussed in (Freitag and Al-Onaizan, 2016) there are two problems with this approach. First, re-training the model from scratch on the combined data is time consuming and second due to the relatively small size of in-domain data, the learned model will be biased towards the general domain and may not perform comparatively on the in-domain test instances. Moreover, this solution is not efficient in real life applications because in situations where a general domain model is already deployed in an application, the original training data may not be available at the production time. Proceedings of MT Summit XVI, vol.1: Research Track Nagoya, Sep. 18-22, 2017 | p. 156 A fast and efficient method for domain adaptation for neural methods is “Fine-tuning” which has also been recently applied for Neural Machine translation (Freitag and Al-Onaizan, 2016; Chu et al., 2017). In fine-tuning, a neural network which is already trained on large general domain data is further trained …

Content truncated to 500 words