Neural Machine Translation Training in a Multi-Domain Scenario

Neural Machine Translation Training in a Multi-Domain Scenario Hassan Sajjad, Nadir Durrani, Fahim Dalvi, ⇤Yonatan Belinkov, Stephan Vogel Qatar Computing Research Institute – HBKU ⇤MIT Computer Science and Artificial Intelligence Laboratory {hsajjad,ndurrani,faimaduddin,svogel}@qf.org.qa, ⇤belinkov@mit.edu Abstract In this paper, we explore alternative ways to train a neu- ral machine translation system in a multi-domain scenario. We investigate data concatenation (with fine tuning), model stacking (multi-level fine tuning), data selection and multi- model ensemble. Our findings show that the best translation quality can be achieved by building an initial system on a concatenation of available out-of-domain data and then fine- tuning it on in-domain data. Model stacking works best when training begins with the furthest out-of-domain data and the model is incrementally fine-tuned with the next furthest do- main and so on. Data selection did not give the best results, but can be considered as a decent compromise between train- ing time and translation quality. A weighted ensemble of different individual models performed better than data selec- tion. It is beneficial in a scenario when there is no time for fine-tuning an already trained model. 1. Introduction Neural machine translation (NMT) systems are sensitive to the data they are trained on. The available parallel corpora come from various genres and have different stylistic vari- ations and semantic ambiguities. While such data is often beneficial for a general purpose machine translation system, a problem arises when building systems for specific domains such as lectures [1, 2], patents [3] or medical text [4], where either the in-domain bilingual text does not exist or is avail- able in small quantities. Domain adaptation aims to preserve the identity of the in-domain data while exploiting the out-of-domain data in favor of the in-domain data and avoiding possible drift to- wards out-of-domain jargon and style. The most commonly used approach to train a domain-specific neural MT system is to fine-tune an existing model (trained on generic data) with the new domain [5, 6, 7, 8] or to add domain-aware tags in building a concatenated system [9]. [10] proposed a gradual fine-tuning method that starts training with com- plete in- and out-of-domain data and gradually reduces the out-of-domain data for next epochs. Other approaches that have been recently proposed for domain adaptation of neural machine translation are instance weighting [11, 12] and data selection [13]. In this paper we explore NMT in a multi-domain sce- nario. Considering a small in-domain corpus and a number of out-of-domain corpora, we target questions like: • What are the different ways to combine multiple do- mains during a training process? • What is the best strategy to build an optimal in-domain system? • Which training strategy results in a robust system? • Which strategy should be used to build a decent in- domain system given limited time? To answer these, we try the following approaches: i) data concatenation: train a system by concatenating all the avail- able in-domain and out-of-domain data; ii) model stacking: build NMT in an online fashion starting from the most …

Content truncated to 500 words