Taking Actions Separately: A Bidirectionally-Adaptive Transfer Learning Method for Low-Resource Neural Machine Translation

Proceedings of the 29th International Conference on Computational Linguistics, pages 4481–4491 October 12–17, 2022. 4481 Taking Actions Separately: A Bidirectionally-Adaptive Transfer Learning Method for Low-Resource Neural Machine Translation Xiaolin Xing, Yu Hong∗, Minhan Xu, Jianmin Yao, Guodong Zhou School of Computer Science and Technology, Soochow University, China {actuallyxxl, tianxianer, cosmosbreak5712}@gmail.com {jyao, gdzhou}@suda.edu.cn Abstract Training Neural Machine Translation (NMT) models suffers from sparse parallel data, in the infrequent translation scenarios towards low- resource source languages. The existing solu- tions primarily concentrate on the utilization of Parent-Child (PC) transfer learning. It transfers well-trained NMT models on high-resource lan- guages (namely Parent NMT) to low-resource languages, so as to produce Child NMT models by fine-tuning. It has been carefully demon- strated that a variety of PC variants yield sig- nificant improvements for low-resource NMT. In this paper, we intend to enhance PC-based NMT by a bidirectionally-adaptive learning strategy. Specifically, we divide inner con- stituents (6 transformers) of Parent encoder into two “teams”, i.e., T1 and T2. During representation learning, T1 learns to encode low-resource languages conditioned on bilin- gual shareable latent space. Generative adver- sarial network and masked language modeling are used for space-shareable encoding. On the other hand, T2 is straightforwardly transferred to low-resource languages, and fine-tuned to- gether with T1 for low-resource translation. Briefly, T1 and T2 take actions separately for different goals. The former aims to adapt to characteristics of low-resource languages dur- ing encoding, while the latter adapts to trans- lation experiences learned from high-resource languages. We experiment on benchmark cor- pora SETIMES, conducting low-resource NMT for Albanian (Sq), Macedonian (Mk), Croat- ian (Hr) and Romanian (Ro). Experimental results show that our method yields substan- tial improvements, which allows the NMT per- formance to reach BLEU4-scores of 62.24%, 56.93%, 50.53% and 54.65% for Sq, Mk, Hr and Ro, respectively. 1 Introduction NMT has achieved significant improvements (Bah- danau et al., 2015; Vaswani et al., 2017) in recent ∗Corresponding author. years. Nevertheless, It heavily relies on large-scale observable parallel corpora. As a result, NMT gen- erally fails to perform perfectly in an infrequent translation scenario, where the available parallel data for training is sparse. For example, the size of training data for NMT between English (En) and Macedonian (Mk) is about 200K, which is signifi- cantly smaller than that (582M) between English (En) and German (De). The issue has been widely known as low-resource NMT. The existing studies attempt to overcome the issue primarily by 1) producing cross-language em- beddings, 2) constructing bilingual shareable latent space for encoding, and 3) transferring well-trained models to low-resource languages. We overview the studies in Section 2. Within the aforementioned arts, nowadays, Parent-Child (PC) transfer learn- ing (Zoph et al., 2016) represents a considerable advance in our knowledge. It allows a Parent NMT model to be fully trained and developed over high- resource languages (e.g., that for De→En), and transfers it to low-resource languages (e.g., that for Mk→En) for fine-tuning. This contributes to the construction of a Child NMT model that inherits the translation …

Content truncated to 500 words