ConsistTL: Modeling Consistency in Transfer Learning for Low-Resource Neural Machine Translation

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 8383 - 8394 December 7-11, 2022 ©2022 Association for Computational Linguistics ConsistTL: Modeling Consistency in Transfer Learning for Low-Resource Neural Machine Translation Zhaocong Li1 Xuebo Liu2∗ Derek F. Wong1∗ Lidia S. Chao1 Min Zhang2 1NLP2CT Lab, Department of Computer and Information Science, University of Macau nlp2ct.zhaocong@gmail.com, {derekfw,lidiasc}@um.edu.mo 2Institute of Computing and Intelligence, Harbin Institute of Technology, Shenzhen, China {liuxuebo,zhangmin2021}@hit.edu.cn Abstract Transfer learning is a simple and powerful method that can be used to boost model perfor- mance of low-resource neural machine trans- lation (NMT). Existing transfer learning meth- ods for NMT are static, which simply trans- fer knowledge from a parent model to a child model once via parameter initialization. In this paper, we propose a novel transfer learning method for NMT, namely ConsistTL, which can continuously transfer knowledge from the parent model during the training of the child model. Specifically, for each training instance of the child model, ConsistTL constructs the semantically-equivalent instance for the par- ent model and encourages prediction consis- tency between the parent and child for this in- stance, which is equivalent to the child model learning each instance under the guidance of the parent model. Experimental results on five low-resource NMT tasks demonstrate that ConsistTL results in significant improvements over strong transfer learning baselines, with a gain up to 1.7 BLEU over the existing back- translation model on the widely-used WMT17 Turkish-English benchmark. Further analysis reveals that ConsistTL can improve the infer- ence calibration of the child model. Code and scripts are freely available at https://github. com/NLP2CT/ConsistTL. 1 Introduction Neural machine translation (NMT) has achieved success on the high-resource language pairs (Vaswani et al., 2017). However, it achieves an inadequate performance on the low-resource lan- guage pairs with merely bilingual data due to the data sparsity (Koehn and Knowles, 2017; Sennrich and Zhang, 2019). Transfer learning is a simple and powerful method that can be used to boost the model performance of low-resource NMT, which can transfer knowledge from an off-the-shelf high- ∗Co-corresponding author (a) Vanilla TL (b) ConsistTL Figure 1: Comparison between vanilla TL and Con- sistTL. The dashed arrow represents the initialization of the parameters from the parent model. Apart from disposable parameter transfer, ConsistTL continuously transfers knowledge from the parent model during the child training. resource parent model to the low-resource child model (Zoph et al., 2016). The goal of transfer learning for low-resource NMT is to transfer knowledge sufficiently from the parent model. Prior works attempt to transfer more information from the embedding layer of the parent model, using methods such as building a joint dic- tionary (Kocmi and Bojar, 2018; Gheini and May, 2019), creating cross-lingual token mapping (Kim et al., 2019), and transferring the word of same form (Aji et al., 2020). These strategies achieve success in parameter transfer. The methods used in these works is summarized in Figure 1(a): the parent model transfers its parameters to the child model once and lacks continual guidance on …

Content truncated to 500 words