Knowledge Transfer in Incremental Learning for Multilingual Neural Machine Translation

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 15286–15304 July 9-14, 2023 ©2023 Association for Computational Linguistics Knowledge Transfer in Incremental Learning for Multilingual Neural Machine Translation Kaiyu Huang1, Peng Li∗1,4, Jin Ma5,6, Ting Yao5, Yang Liu∗ 1,2,3,4 1Institute for AI Industry Research (AIR), Tsinghua University, Beijing, China 2Dept. of Comp. Sci. & Tech., Institute for AI, Tsinghua University, Beijing, China 3Beijing National Research Center for Information Science and Technology 4Shanghai Artificial Intelligence Laboratory, Shanghai, China 5Tencent 6Sch. of Comp. Sci. & Tech., University of Science and Technology of China {huangkaiyu,lipeng}@air.tsinghua.edu.cn; tessieyao@tencent.com majin01@mail.ustc.edu.cn; liuyang2011@tsinghua.edu.cn Abstract In the real-world scenario, a longstanding goal of multilingual neural machine transla- tion (MNMT) is that a single model can in- crementally adapt to new language pairs with- out accessing previous training data. In this scenario, previous studies concentrate on over- coming catastrophic forgetting while lacking encouragement to learn new knowledge from incremental language pairs, especially when the incremental language is not related to the set of original languages. To better acquire new knowledge, we propose a knowledge trans- fer method that can efficiently adapt original MNMT models to diverse incremental lan- guage pairs. The method flexibly introduces the knowledge from an external model into original models, which encourages the models to learn new language pairs, completing the procedure of knowledge transfer. Moreover, all original parameters are frozen to ensure that transla- tion qualities on original language pairs are not degraded. Experimental results show that our method can learn new knowledge from diverse language pairs incrementally meanwhile main- taining performance on original language pairs, outperforming various strong baselines in in- cremental learning for MNMT.1 1 Introduction Multilingual neural machine translation (MNMT) aims at handling multiple translation directions in a single model and achieves great success in recent years (Wenzek et al., 2021; Goyal et al., 2022; Cheng et al., 2022). However, the powerful MNMT models need to be retrained from scratch ∗Corresponding authors: Peng Li (lipeng@air.tsinghua. edu.cn) and Yang Liu (liuyang2011@tsinghua.edu.cn) 1Our code will be released at https://github.com/ THUNLP-MT/ktnmt Method en→uk en→bn From-scratch 23.70 15.54 Replay-Based 21.37 10.32 Regularization-Based 20.33 7.84 Adapter 19.92 9.46 Table 1: The BLEU scores of incremental learning meth- ods on new translation directions. The original model is mBART25 which does not support the languages of Ukrainian (uk) and Bengali (bn). using a mixture of original and incremental training data when new language pairs arrive (Tang et al., 2020). Considering that the original training data of MNMT models is often large-scale (Fan et al., 2021; Costa-jussà et al., 2022), and thus the method that utilizes original data to train incrementally is time-consuming and cumbersome (Ebrahimi and Kann, 2021). Therefore, a practical scenario is that these models can continually support new lan- guage pairs while preserving the previously learned knowledge without accessing previous training data, which belongs to incremental learning, and has drawn much attention recently (Dabre et al., 2020; Gu et al., 2022; Zhao et al., 2022). In this scenario, existing studies …

Content truncated to 500 words