Factorized Transformer for Multi-Domain Neural Machine Translation

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4221–4230 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 4221 Factorized Transformer for Multi-Domain Neural Machine Translation Yongchao Deng1, Hongfei Yu2, Heng Yu1, Xiangyu Duan2, Weihua Luo1 1Machine Intelligence Technology Lab, Alibaba DAMO Academy 2Soochow University, Suzhou, China {yongchao.dyc, yuheng.yh, weihua.luowh}@alibaba-inc.com hfyutravel@gmail.com, xiangyuduan@suda.edu.cn Abstract Multi-Domain Neural Machine Translation (NMT) aims at building a single system that performs well on a range of target domains. However, along with the extreme diversity of cross-domain wording and phrasing style, the imperfections of training data distribution and the inherent defects of the current sequential learning process all contribute to making the task of multi-domain NMT very challenging. To mitigate these problems, we propose the Factorized Transformer, which consists of an in-depth factorization of the parameters of an NMT model, namely Transformer in this pa- per, into two categories: domain-shared ones that encode common cross-domain knowledge and domain-specific ones that are private for each constituent domain. We experiment with various designs of our model and conduct ex- tensive validations on English to French open multi-domain dataset. Our approach achieves state-of-the-art performance and opens up new perspectives for multi-domain and open- domain applications. 1 Introduction Recent advances in Neural Machine Transla- tion (NMT) (Bahdanau et al., 2015; Vaswani et al., 2017) have led to significant improvement in terms of translation quality (Wu et al., 2016; Hassan et al., 2018), opening new perspectives for Ma- chine Translation in real-world scenarios. In order to deliver trust-worthy translations for end users, an NMT system is often required to meet expert-level translation quality in one or multiple related target domains, while performing well enough on a range of generic subjects, just like human experts do. However, requiring a single NMT system to perform well on multiple distant domains simul- taneously is a very challenging task. First, lan- guages are highly polysemous: the same words or expressions may have different meanings in dif- ferent contexts. Also wording and syntactic style may significantly vary depending on the domains. Second, a multi-domain NMT system in general suffers from two major issues: Domain Bias and Catastrophic Forgetting (Mccloskey and Cohen, 1989; Kirkpatrick et al., 2016; Thompson et al., 2019) . While the former biases the model toward well-represented domains to the detriment of the low-resource ones, the latter makes the sequential learning process difficult as the model keeps forget- ting previously learned knowledge when exposed to the new training examples. Most of the existing NMT systems rely on the same network to model all domains, which means the same word embedding to represent all the mean- ings of a word and the same set of parameters to model its depending contexts. This type of config- uration in general maximizes the knowledge trans- fer, but overlooks the specificity of each domain (Koehn and Knowles, 2017). An obvious solution for this problem is to dedicate an individual model to each constituent domain, which is unrealistic in practice as it dramatically increases the number of …

Content truncated to 500 words