Domain Adaptation of Neural Machine Translation by Lexicon Induction

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2989–3001 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 2989 Domain Adaptation of Neural Machine Translation by Lexicon Induction Junjie Hu, Mengzhou Xia, Graham Neubig, Jaime Carbonell Language Technologies Institute School of Computer Science Carnegie Mellon University {junjieh,gneubig,jgc}@cs.cmu.edu, mengzhox@andrew.cmu.edu Abstract It has been previously noted that neural ma- chine translation (NMT) is very sensitive to domain shift. In this paper, we argue that this is a dual effect of the highly lexicalized nature of NMT, resulting in failure for sen- tences with large numbers of unknown words, and lack of supervision for domain-specific words. To remedy this problem, we propose an unsupervised adaptation method which fine- tunes a pre-trained out-of-domain NMT model using a pseudo-in-domain corpus. Specifi- cally, we perform lexicon induction to ex- tract an in-domain lexicon, and construct a pseudo-parallel in-domain corpus by perform- ing word-for-word back-translation of mono- lingual in-domain target sentences. In five domains over twenty pairwise adaptation set- tings and two model architectures, our method achieves consistent improvements without us- ing any in-domain parallel sentences, improv- ing up to 14 BLEU over unadapted models, and up to 2 BLEU over strong back-translation baselines. 1 Introduction Neural machine translation (NMT) has demon- strated impressive performance when trained on large-scale corpora (Bojar et al., 2018). How- ever, it has also been noted that NMT models trained on corpora in a particular domain tend to perform poorly when translating sentences in a significantly different domain (Chu and Wang, 2018; Koehn and Knowles, 2017). Previous work in the context of phrase-based statistical machine translation (Daum´e III and Jagarlamudi, 2011) has noted that unseen (OOV) words account for a large portion of translation errors when switching to new domains. However this problem of OOV words in cross-domain transfer is under-examined Code/scripts are released at https://github.com/ junjiehu/dali. in the context of NMT, where both training meth- ods and experimental results will differ greatly. In this paper, we try to fill this gap, examining do- main adaptation methods for NMT specifically fo- cusing on correctly translating unknown words. As noted by Chu and Wang (2018), there are two important distinctions to make in adaptation methods for MT. The first is data requirements; supervised adaptation relies on in-domain paral- lel data, and unsupervised adaptation has no such requirement. There is also a distinction between model-based and data-based methods. Model- based methods make explicit changes to the model architecture such as jointly learning domain dis- crimination and translation (Britz et al., 2017), in- terpolation of language modeling and translation (Gulcehre et al., 2015; Domhan and Hieber, 2017), and domain control by adding tags and word fea- tures (Kobus et al., 2017). On the other hand, data-based methods perform adaptation either by combining in-domain and out-of-domain paral- lel corpora for supervised adaptation (Luong and Manning, 2015; Freitag and Al-Onaizan, 2016) or by generating pseudo-parallel corpora from in- domain monolingual data for unsupervised adap- tation (Sennrich et al., …

Content truncated to 500 words