Improving Domain Adaptation Translation with Domain Invariant and Specific Information

Proceedings of NAACL-HLT 2019, pages 3081–3091 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 3081 Improving Domain Adaptation Translation with Domain Invariant and Specific Information Shuhao Gu1,2 Yang Feng1,2* Qun Liu3 1Key Laboratory of Intelligent Information Processing Institute of Computing Technology, Chinese Academy of Sciences (ICT/CAS) 2University of Chinese Academy of Science 3Huawei Noah’s Ark Lab, Hong Kong, China 1,2{gushuhao17g, fengyang}@ict.ac.cn 3qun.liu@huawei.com Abstract In domain adaptation for neural machine trans- lation, translation performance can benefit from separating features into domain-specific features and common features. In this pa- per, we propose a method to explicitly model the two kinds of information in the encoder- decoder framework so as to exploit out-of- domain data in in-domain training. In our method, we maintain a private encoder and a private decoder for each domain which are used to model domain-specific information. In the meantime, we introduce a common en- coder and a common decoder shared by all the domains which can only have domain- independent information flow through. Be- sides, we add a discriminator to the shared en- coder and employ adversarial training for the whole model to reinforce the performance of information separation and machine transla- tion simultaneously. Experiment results show that our method can outperform competitive baselines greatly on multiple data sets. 1 Introduction Neural machine translation (NMT) (Kalchbrenner and Blunsom, 2013; Cho et al., 2014; Sutskever et al., 2014; Bahdanau et al., 2014; Gehring et al., 2017) has made great progress and drawn much attention recently. Most NMT models are based on the encoder-decoder architecture, where all the sentence pairs share the same set of param- eters for the encoder and decoder which makes NMT models have a tendency towards overfitting to frequent observations (e.g., words, word co- occurrences, translation patterns), but overlook- ing special cases that are not frequently observed. However, in practical applications, NMT mod- els usually need to perform translation for some specific domain with only a small quantity of in- *Corresponding Author domain training data but a large amount of out-of- domain data. Simply combining in-domain train- ing data with out-of-domain data will lead to over- fitting to the out-of-domain data. Therefore, some domain adaptation technique should be adopted to improve in-domain translation. Fortunately, out-of-domain data still embod- ies common knowledge shared between domains. And incorporating the common knowledge from out-of-domain data can help in-domain transla- tion. Britz et al. (2017) have done this kind of at- tempts and managed to improve in-domain trans- lation. The common architecture of this method is to share a single encoder and decoder among all the domains and add a discriminator to the encoder to distinguish the domains of the input sentences. The training is based on adversarial learning between the discriminator and the trans- lation , ensuring the encoder can learn common knowledge across domains that can help to gener- ate target translation. Zeng et al. (2018) extend this line of work by introducing a private encoder to learn some domain specific knowledge. They have …

Content truncated to 500 words