kNN-TL: k-Nearest-Neighbor Transfer Learning for Low-Resource Neural Machine Translation

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 1878–1891 July 9-14, 2023 ©2023 Association for Computational Linguistics kNN-TL: k-Nearest-Neighbor Transfer Learning for Low-Resource Neural Machine Translation Shudong Liu1 Xuebo Liu2∗ Derek F. Wong1∗ Zhaocong Li1 Wenxiang Jiao Lidia S. Chao1 Min Zhang2 1NLP2CT Lab, Department of Computer and Information Science, University of Macau 2Institute of Computing and Intelligence, Harbin Institute of Technology, Shenzhen, China nlp2ct.{shudong,zhaocong}@gmail.com, {liuxuebo,zhangmin2021}@hit.edu.cn {derekfw,lidiasc}@um.edu.mo, wenxiangjiaonju@gmail.com Abstract Transfer learning has been shown to be an effective technique for enhancing the perfor- mance of low-resource neural machine transla- tion (NMT). This is typically achieved through either fine-tuning a child model with a pre- trained parent model, or by utilizing the out- put of the parent model during the training of the child model. However, these methods do not make use of the parent knowledge dur- ing the child inference, which may limit the translation performance. In this paper, we pro- pose a k-Nearest-Neighbor Transfer Learning (kNN-TL) approach for low-resource NMT, which leverages the parent knowledge through- out the entire developing process of the child model. Our approach includes a parent-child representation alignment method, which en- sures consistency in the output representations between the two models, and a child-aware datastore construction method that improves inference efficiency by selectively distilling the parent datastore based on relevance to the child model. Experimental results on four low- resource translation tasks show that kNN-TL outperforms strong baselines. Extensive analy- ses further demonstrate the effectiveness of our approach. Code and scripts are freely available at https://github.com/NLP2CT/kNN-TL. 1 Introduction Although deep learning has significantly advanced the field of neural machine translation (NMT, Bah- danau et al., 2015; Vaswani et al., 2017; Liu et al., 2019, 2020), the standard training procedure of NMT is not well-suited for languages with only a small amount of bilingual data, leading to chal- lenges in developing NMT models for low-resource languages (Zhan et al., 2021; Wang et al., 2022d). To overcome this limitation, transfer learning has been proposed as an effective method to enhance low-resource NMT through the parent-child frame- work. This framework transfers knowledge from a ∗Co-corresponding author Method Init. Training Inference Vanilla TL ✓ ✗ ✗ ConsistTL ✓ ✓ ✗ kNN-TL ✓ ✓ ✓ Table 1: Comparison of three transfer learning frame- works for exploiting of parent knowledge throughout the developing process of a child model. “Init.” de- notes the initialization stage of the child model. Our proposed kNN-TL framework incorporates the use of parent knowledge throughout the entire process. high-resource parent model to a low-resource child model (Zoph et al., 2016). Previous works in transfer learning, such as Kim et al. (2019a) and Aji et al. (2020), have aimed to address the problem of vocabulary mismatch for more effective knowledge transfer. These works, re- ferred to as Vanilla TL, primarily focus on transfer- ring knowledge during the initialization stage of the child model and do not consider other stages of the development of the child model. Recently, Li et …

Content truncated to 500 words