Unsupervised Domain Adaptation for Neural Machine Translation with Domain-Aware Feature Embeddings

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 1417–1422, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 1417 Unsupervised Domain Adaptation for Neural Machine Translation with Domain-Aware Feature Embeddings Zi-Yi Dou, Junjie Hu, Antonios Anastasopoulos, Graham Neubig Language Technologies Institute, Carnegie Mellon University {zdou, junjieh, aanastas, gneubig}@cs.cmu.edu Abstract The recent success of neural machine transla- tion models relies on the availability of high quality, in-domain data. Domain adaptation is required when domain-specific data is scarce or nonexistent. Previous unsupervised do- main adaptation strategies include training the model with in-domain copied monolingual or back-translated data. However, these meth- ods use generic representations for text regard- less of domain shift, which makes it infeasible for translation models to control outputs con- ditional on a specific domain. In this work, we propose an approach that adapts models with domain-aware feature embeddings, which are learned via an auxiliary language modeling task. Our approach allows the model to assign domain-specific representations to words and output sentences in the desired domain. Our empirical results demonstrate the effectiveness of the proposed strategy, achieving consistent improvements in multiple experimental set- tings. In addition, we show that combining our method with back translation can further improve the performance of the model.1 1 Introduction While neural machine translation (NMT) sys- tems have proven to be effective in scenarios where large amounts of in-domain data are avail- able (Gehring et al., 2017; Vaswani et al., 2017; Chen et al., 2018), they have been demonstrated to perform poorly when the test domain does not match the training data (Koehn and Knowles, 2017). Collecting large amounts of parallel data in all possible domains we are interested in is costly, and in many cases impossible. Therefore, it is es- sential to explore effective methods to train mod- els that generalize well to new domains. 1Our code is publicly available at: https://github. com/zdou0830/DAFE. Input Word Input Domain + Word Embeddings Layer 1 Layer N … … θtask (0) + + … … θtask (1) θtask (N ) θbase (0) θbase (1) θbase (N ) θdomain (0) θdomain (1) θdomain (N ) Input Task Figure 1: Main architecture of DAFE. Embedding learners generate domain- and task-specific features at each layer, which are then integrated into the output of the base network. Domain adaptation for neural machine transla- tion has attracted much attention in the research community, with the majority of work focusing on the supervised setting where a small amount of in-domain data is available (Luong and Man- ning, 2015; Freitag and Al-Onaizan, 2016; Chu et al., 2017; Vilar, 2018). An established approach is to use domain tags as additional input, with the domain representations learned over parallel data (Kobus et al., 2017). In this work, we fo- cus on unsupervised adaptation, where there are no in-domain parallel data available. Within this paradigm, Currey et al. (2017) copy the in-domain monolingual data from the target side to the …

Content truncated to 500 words