Word-based Domain Adaptation for Neural Machine Translation

Word-based Domain Adaptation for Neural Machine Translation Shen Yan, Leonard Dahlmann, Pavel Petrushkov, Sanjika Hewavitharana, Shahram Khadivi eBay Inc. {shenyan,fdahlmann,ppetrushkov,shewavitharana,skhadivi}@ebay.com Abstract In this paper, we empirically investigate applying word-level weights to adapt neural machine translation to e-commerce domains, where small e-commerce datasets and large out- of-domain datasets are available. In order to mine in- domain like words in the out-of-domain datasets, we com- pute word weights by using a domain-specific and a non- domain-specific language model followed by smoothing and binary quantization. The baseline model is trained on mixed in-domain and out-of-domain datasets. Experimental re- sults on En → Zh e-commerce domain translation show that compared to continuing training without word weights, it improves MT quality by up to 3.11% BLEU absolute and 1.59% TER. We have also trained models using fine-tuning on the in-domain data. Pre-training a model with word weights improves fine-tuning up to 1.24% BLEU absolute and 1.64% TER, respectively. 1. Introduction Domain adaptation (DA) techniques in machine translation (MT) have been widely studied. For statistical machine translation (SMT), several DA methods have been proposed to overcome the lack of domain-specific data. For exam- ple, self-training [1, 2] uses a MT system trained on general corpus to translate in-domain monolingual data as additional training sentences. Topic-based DA [3, 4] employs topic- based translation models to adapt for different scenarios. Data selection approaches [5, 6, 7, 8] first score the out-of- domain data using language model trained on both domain- specific and non-domain-specific monolingual corpora, then rank and select the out-of-domain data that are similar to in-domain data. Instance weighting methods [9, 10] score each sentence/domain using statistical rules, then train the MT models by giving sentence/domain-level scores. Neural machine translation (NMT) has become state-of- the-art in recent years [11, 12, 13, 14, 15]. There are sev- eral research works on NMT domain adaptation. For exam- ple, back-translation methods [16] use a NMT model trained on the reverse direction to translate domain-specific mono- lingual data as additional training sentences. Fast DA ap- proaches [13, 17] train a base model using mixed in-domain and out-of-domain datasets, then fine-tuning on in-domain datasets. Mixed fine-tuning [18] combines fine-tuning and multi-domain NMT. Similar to instance weighting in SMT, sentence/domain weighting methods [19, 20] can also be used for NMT domain adaptation based on different objec- tives. DA with meta information [21] is proposed to train topic-aware models using domain-specific tags for the de- coder. Chunk weighting method [22] describes a way of se- lecting and integrating positive partial feedback from model- generated sentences into NMT training. In this paper, we propose word-level weighting for NMT domain adaptation. We compute the word weights in out-of- domain datasets based on the logarithm difference of prob- ability according to a domain-specific language model and non-domain-specific language model followed by smoothing and binary quantization. This gives the in-domain words in out-of-domain sentences higher weights and biases the NMT model to generate more in-domain-like words. Thus, the work presented in this paper can be viewed as a …

Content truncated to 500 words