Curriculum Learning for Domain Adaptation in Neural Machine Translation

Proceedings of NAACL-HLT 2019, pages 1903–1915 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 1903 Curriculum Learning for Domain Adaptation in Neural Machine Translation Xuan Zhang1, Pamela Shapiro1, Gaurav Kumar1, Paul McNamee1, Marine Carpuat2, Kevin Duh1 1Johns Hopkins University 2University of Maryland {xuanzhang, pshapiro, mcnamee}@jhu.edu, marine@cs.umd.edu, {gkumar, kevinduh}@cs.jhu.edu Abstract We introduce a curriculum learning approach to adapt generic neural machine translation models to a specific domain. Samples are grouped by their similarities to the domain of interest and each group is fed to the train- ing algorithm with a particular schedule. This approach is simple to implement on top of any neural framework or architecture, and consistently outperforms both unadapted and adapted baselines in experiments with two dis- tinct domains and two language pairs. 1 Introduction Neural machine translation (NMT) performance often drops when training and test domains do not match and when in-domain training data is scarce (Koehn and Knowles, 2017). Tailoring the NMT system to each domain could improve performance, but unfortunately high-quality par- allel data does not exist for all domains. Do- main adaptation techniques address this problem by exploiting diverse data sources to improve in- domain translation, including general domain data that does not match the domain of interest, and unlabeled domain data whose domain is unknown (e.g. webcrawl like Paracrawl). One approach to exploit unlabeled-domain bi- text is to apply data selection techniques (Moore and Lewis, 2010; Axelrod et al., 2011; Duh et al., 2013) to find bitext that are similar to in-domain data. This selected data can additionally be com- bined with in-domain bitext and trained in a con- tinued training framework, as shown in Figure 1. Continued training or fine-tuning (Luong et al., 2015; Freitag and Al-Onaizan, 2016; Chu et al., 2017) is an adaptation technique where a model is first trained on the large general domain data, then used as initialization of a new model which is further trained on in-domain bitext. In our Generic Model Domain Specific Model General Domain Data Unlabeled-Domain Data In-Domain Data Continued Training Initialization Figure 1: Workflow of our domain adaptation system. framework, the selected samples are concatenated with in-domain data, then used for continued train- ing. This effectively increases the in-domain train- ing size with “pseudo” in-domain samples, and is helpful in continued training (Koehn et al., 2018). A challenge with employing data selection in continued training is that there exists no clear-cut way to define whether a sample is sufficiently sim- ilar to in-domain data to be included. In practice, one has to define a threshold based on similarity scores, and even so the continued training algo- rithm may be faced with samples of diverse sim- ilarities. We introduce a new domain adaptation technique that addresses this challenge. Inspired by curriculum learning (Bengio et al., 2009), we use the similarity scores given by data selection to rearrange the order of training sam- ples, such that more similar examples are seen ear- lier and more frequently during training. To the best …

Content truncated to 500 words