Two Parents, One Child: Dual Transfer for Low-Resource Neural Machine Translation

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 2726–2738 August 1–6, 2021. ©2021 Association for Computational Linguistics 2726 Two Parents, One Child: Dual Transfer for Low-Resource Neural Machine Translation Meng Zhang, Liangyou Li, Qun Liu Huawei Noah’s Ark Lab {zhangmeng92, liliangyou, qun.liu}@huawei.com Abstract Neural machine translation suffers when paral- lel data for training is scarce. Previous works have explored transfer learning to assist train- ing in low-resource scenarios. However, they transfer either from high-resource parallel data, or from monolingual data. In this work, we pro- pose a framework to transfer multiple sources of auxiliary data, including both high-resource parallel data and monolingual data of involved languages. Knowledge in those sources is re- spectively encoded in a high-resource trans- lation model and pretrained language models, and dually transferred to the low-resource trans- lation model by our approach. Extensive exper- iments show that our approach yields consis- tent improvements over strong competitors for multiple translation directions. Furthermore, our approach still exhibits benefit on top of back-translation, making it a useful addition to practitioners’ toolbox. 1 Introduction Neural machine translation (NMT) has achieved remarkable success in recent years, but its quality critically hinges on large-scale parallel data. In the low-resource scenarios for most world languages and many domains, its performance usually deteri- orates dramatically. Although parallel data for some translation tasks may be difficult to obtain, monolingual data is usu- ally within reach, and often comes in much larger quantity. Besides, parallel data for several high- resource languages is readily available. These cor- pora have been used in various methods to help training low-resource NMT. The most relevant method to our work is transfer learning. Transfer learning starts with training a source task and then initializes the target task with the pa- rameters. Recent advances in pretrained language models (PLM) like BERT (Devlin et al., 2019) can approach H L M P M P no transfer � (Zoph et al., 2016) � � (Kim et al., 2019) � � � BERT2RND � � BERT2BERT � � (Kocmi and Bojar, 2018) � � BBERT2BBERT � � BBERT transfer � � � � dual transfer (ours) � � � � Table 1: An overview of data usage by approaches considered in this work (Section 4.3). H/L: high/low- resource language pair; M: monolingual; P: parallel. BBERT transfer checks all the boxes but uses data in a different way from ours. be seen as transfer learning, where language mod- eling is the source task for downstream target tasks. In low-resource NMT, pretrained language models have also provided noticeable improvements (Clin- chant et al., 2019; Imamura and Sumita, 2019). As another source of transfer, high-resource NMT models have also been used for transfer learning low-resource NMT. Zoph et al. (2016) pioneered this direction with NMT based on recurrent neu- ral networks, and coined the high-resource and low-resource models as parent and child models, respectively. However, it is non-trivial to transfer from both PLMs and NMT models. This limitation constrains most existing transfer-learning-based low-resource NMT to a …

Content truncated to 500 words