Instance Weighting for Neural Machine Translation Domain Adaptation

Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1482–1488 Copenhagen, Denmark, September 7–11, 2017. c⃝2017 Association for Computational Linguistics Instance Weighting for Neural Machine Translation Domain Adaptation Rui Wang1, Masao Utiyama1, Lemao Liu2, Kehai Chen1,3 and Eiichiro Sumita1 1National Institute of Information and Communications Technology (NICT) 2Tencent AI Lab 3Harbin Institute of Technology {wangrui, mutiyama and eiichiro.sumita}@nict.go.jp lemaoliu@gmail.com and khchen@hit.edu.cn Abstract Instance weighting has been widely applied to phrase-based machine ittranslation domain adaptation. However, is challenging to be applied to Neural Machine Translation (NMT) directly, because NMT is not a linear model. In this paper, two instance weighting technologies, i.e., sentence weighting and domain weighting with a dynamic weight learning strategy, are proposed for NMT domain adaptation. Empirical results on the IWSLT English- German/French tasks show that the proposed methods can substantially improve NMT performance by up to 2.7-6.7 BLEU points, outperforming the existing baselines by up to 1.6-3.6 BLEU points. 1 Introduction In Statistical Machine Translation (SMT), unrelated additional corpora, known as out-of- domain corpora, have been shown not to benefit some domains and tasks, such as TED-talks and IWSLT tasks (Axelrod et al., 2011; Luong and Manning, 2015). Several Phrase-based SMT (PBSMT) domain adaptation methods have been proposed to overcome this problem of the lack of substantial data in some specific domains and languages: i) Data selection. The main idea is to score the out-of-domain data using models trained from the in-domain and out-of-domain data, respectively. Then select training data by using these ranked scores (Moore and Lewis, 2010; Axelrod et al., 2011; Duh et al., 2013; Hoang and Sima’an, 2014a,b; Durrani et al., 2015; Chen et al., 2016). ii) Model Linear Interpolation. Several PBSMT models, such as language models, translation models, and reordering models, individually corresponding to each corpus, are trained. These models are then combined to achieve the best performance (Sennrich, 2012; Sennrich et al., 2013; Durrani et al., 2015, 2016; Imamura and Sumita, 2016). iii) Instance Weighting. Instance Weighting has been applied to several NLP domain adaptation tasks (Jiang and Zhai, 2007), such as POS tagging, entity type classification and especially PBSMT (Matsoukas et al., 2009; Shah et al., 2010; Foster et al., 2010; Rousseau et al., 2011; Zhou et al., 2015; Wang et al., 2016; Imamura and Sumita, 2016). They firstly score each instance/domain by using rules or statistical methods as a weight, and then train PBSMT models by giving each instance/domain the weight. For Neural Machine Translation (NMT) domain adaptation, the sentence selection can also be used (Chen et al., 2016; Wang et al., 2017). Meanwhile, the model linear interpolation is not easily applied to NMT directly, because NMT is not a linear model. There are two methods for model combination of NMT: i) the in-domain model and out-of-domain model can be ensembled (Jean et al., 2015). ii) an NMT further training (fine-tuning) method (Luong and Manning, 2015). The training is performed in two steps: first, the NMT system is trained using out-of-domain data, and then further trained …

Content truncated to 500 words