Pruning-then-Expanding Model for Domain Adaptation of Neural Machine Translation

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3942–3952 June 6–11, 2021. ©2021 Association for Computational Linguistics 3942 Pruning-then-Expanding Model for Domain Adaptation of Neural Machine Translation Shuhao Gu1,2, Yang Feng1,2∗, Wanying Xie1,3 1 Key Laboratory of Intelligent Information Processing, Institute of Computing Technology, Chinese Academy of Sciences (ICT/CAS) 2 University of Chinese Academy of Sciences 3 Beijing Language and Culture University, China {gushuhao19b,fengyang}@ict.ac.cn, xiewanying07@gmail.com Abstract Domain Adaptation is widely used in prac- tical applications of neural machine transla- tion, which aims to achieve good performance on both general domain and in-domain data. However, the existing methods for domain adaptation usually suffer from catastrophic for- getting, large domain divergence, and model explosion. To address these three problems, we propose a method of “divide and conquer” which is based on the importance of neurons or parameters for the translation model. In this method, we first prune the model and only keep the important neurons or parame- ters, making them responsible for both general- domain and in-domain translation. Then we further train the pruned model supervised by the original whole model with knowledge dis- tillation. Last we expand the model to the original size and fine-tune the added param- eters for the in-domain translation. We con- ducted experiments on different language pairs and domains and the results show that our method can achieve significant improvements compared with several strong baselines. 1 Introduction Neural machine translation (NMT) models (Kalch- brenner and Blunsom, 2013; Cho et al., 2014; Sutskever et al., 2014; Bahdanau et al., 2015; Gehring et al., 2017; Vaswani et al., 2017) are data- driven and hence require large-scale training data to achieve good performance (Zhang et al., 2019a). In practical applications, NMT models usually need to produce translation for some specific domains with only a small quantity of in-domain data avail- able, so domain adaptation is applied to address the problem. A typical domain adaptation scenario as discussed in Freitag and Al-Onaizan (2016) is that an NMT model have been trained with large-scale general-domain data and then is adapted to specific ∗Corresponding author: Yang Feng. Reproducible code: https://github.com/ictnlp/PTE-NMT. domains, hoping the model can fit in-domain data well meanwhile the performance will not degrade too much on the general domain. Towards this end, many researchers have made their attempts. The fine-tuning method (Luong and Manning, 2015) performs in-domain training based on the general-domain model by first training the model on general-domain data and then continuing to train on in-domain data. Despite its convenience for use and high-quality for in-domain translation, this method suffers from catastrophic forgetting which leads to poor performance in the previous domains. Regularization-based methods (Dakwale and Monz, 2017; Thompson et al., 2019; Barone et al., 2017; Khayrallah et al., 2018) instead in- troduce an additional loss to the original objec- tive so that the translation model can trade off be- tween general-domain and in-domain. This kind of methods usually has all the parameters shared by general-domain and …

Content truncated to 500 words