Multi-Domain Neural Machine Translation with Word-Level Adaptive Layer-wise Domain Mixing

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1823–1834 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 1823 Multi-Domain Neural Machine Translation with Word-Level Adaptive Layer-wise Domain Mixing Haoming Jiang Georgia Tech jianghm@gatech.edu Chen Liang Georgia Tech cliang73@gatech.edu Chong Wang ByteDance chong.wang@bytedance.com Tuo Zhao Georgia Tech tourzhao@gatech.edu Abstract Many multi-domain neural machine transla- tion (NMT) models achieve knowledge trans- fer by enforcing one encoder to learn shared embedding across domains. However, this de- sign lacks adaptation to individual domains. To overcome this limitation, we propose a novel multi-domain NMT model using individ- ual modules for each domain, on which we apply word-level, adaptive and layer-wise do- main mixing. We first observe that words in a sentence are often related to multiple do- mains. Hence, we assume each word has a domain proportion, which indicates its domain preference. Then word representations are ob- tained by mixing their embedding in individ- ual domains based on their domain propor- tions. We show this can be achieved by care- fully designing multi-head dot-product atten- tion modules for different domains, and even- tually taking weighted averages of their pa- rameters by word-level layer-wise domain pro- portions. Through this, we can achieve ef- fective domain knowledge sharing, and cap- ture fine-grained domain-specific knowledge as well. Our experiments show that our pro- posed model outperforms existing ones in sev- eral NMT tasks. 1 Introduction Neural Machine Translation (NMT) has made sig- nificant progress in various machine translation tasks (Kalchbrenner and Blunsom, 2013; Sutskever et al., 2014; Bahdanau et al., 2014; Luong et al., 2015; Wu et al., 2016). The success of NMT heav- ily relies on a huge amount of annotated parallel sentences as training data, which is often limited in certain domains, e.g., medical domain. One ap- proach to address this is to explore unparalleled corpora, such as unsupervised machine transla- tion (Lample et al., 2017, 2018). Another approach is to train a multi-domain NMT model and this is the focus of this paper. The simplest way is to build a unified model by directly pooling all training data from multiple domains together, as the languages from different domains often share some similar se- mantic traits, e.g., sentence structure, textual style and word usages. For domains with less training data, the unified model usually shows significant improvement. Researchers have proposed many methods for improving multi-domain NMT. Though certain se- mantic traits are shared across domains, there still exists significant heterogeneity among languages from different domains. For example, Haddow and Koehn (2012) show that for a domain with suffi- cient training data, a unified model may lead to weaker performance than the one trained solely over the domain; Farajian et al. (2017); Luong et al. (2015); Sennrich et al. (2015a); Servan et al. (2016) also show that to improve the translation perfor- mance over certain domains, fine-tuning the unified model is often needed, but at the expense of sacri- ficing the performance over other domains. This in- dicates that a …

Content truncated to 500 words