Iterative Dual Domain Adaptation for Neural Machine Translation

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 845–855, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 845 Iterative Dual Domain Adaptation for Neural Machine Translation Jiali Zeng1, Yang Liu2, Jinsong Su1∗, Yubin Ge3, Yaojie Lu4, Yongjing Yin1, Jiebo Luo5 1Xiamen University, Xiamen, China 2Tsinghua University, Beijing, China 3University of Illinois at Urbana-Champaign, Urbana, IL 61801, USA 4Institute of Software, Chinese Academy of Sciences, Beijing, China 5Department of Computer Science, University of Rochester, Rochester NY 14627, USA lemon@stu.xmu.edu.cn liuyang2011@tsinghua.edu.cn jssu@xmu.edu.cn Abstract Previous studies on the domain adaptation for neural machine translation (NMT) mainly focus on the one-pass transferring out-of- domain translation knowledge to in-domain NMT model. In this paper, we argue that such a strategy fails to fully extract the domain- shared translation knowledge, and repeatedly utilizing corpora of different domains can lead to better distillation of domain-shared trans- lation knowledge. To this end, we propose an iterative dual domain adaptation frame- work for NMT. Specifically, we first pre- train in-domain and out-of-domain NMT mod- els using their own training corpora respec- tively, and then iteratively perform bidirec- tional translation knowledge transfer (from in- domain to out-of-domain and then vice versa) based on knowledge distillation until the in- domain NMT model convergences. Further- more, we extend the proposed framework to the scenario of multiple out-of-domain train- ing corpora, where the above-mentioned trans- fer is performed sequentially between the in- domain and each out-of-domain NMT models in the ascending order of their domain similari- ties. Empirical results on Chinese-English and English-German translation tasks demonstrate the effectiveness of our framework. 1 Introduction Currently, neural machine translation (NMT) has become dominant in the community of machine translation due to its excellent performance (Bah- danau et al., 2015; Wu et al., 2016; Vaswani et al., 2017). With the development of NMT, prevail- ing NMT models become more and more com- plex with large numbers of parameters, which of- ten require abundant corpora for effective train- ing. However, for translation tasks in most do- mains, domain-specific parallel sentences are of- ∗Corresponding author. ten scarce. If we only use domain-specific data to train the NMT model for such a domain, the performance of resulting model is usually unsatis- fying. Therefore, NMT for low-resource domains becomes a challenge in its research and applica- tions. To deal with this issue, many researchers have conducted studies on the domain adaptation for NMT, which can be classified into two general categories. One is to transfer the rich-resource domain (out-of-domain) translation knowledge to benefit the low-resource (in-domain) NMT model. The other is to use the mixed-domain training cor- pus to construct a unified NMT model for all do- mains. Here, we mainly focus on the first type of research, of which typical methods include fine- tuning (Luong and Manning, 2015; Zoph et al., 2016; Servan et al., 2016), mixed fine-tuning (Chu et al., 2017), cost weighting (Chen et al., 2017), data selection (Wang …

Content truncated to 500 words