Entropy-Based Vocabulary Substitution for Incremental Learning in Multilingual Neural Machine Translation

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10537 - 10550 December 7-11, 2022 ©2022 Association for Computational Linguistics Entropy-Based Vocabulary Substitution for Incremental Learning in Multilingual Neural Machine Translation Kaiyu Huang1, Peng Li∗1, Jin Ma5,6, Yang Liu∗ 1,2,3,4,7,8 1Institute for AI Industry Research, Tsinghua University, Beijing, China 2Dept. of Comp. Sci. & Tech., Institute for AI, Tsinghua University, Beijing, China 3Beijing National Research Center for Information Science and Technology 4Beijing Academy of Artificial Intelligence, Beijing, China 5Tencent 6Sch. of Comp. Sci. & Tech., University of Science and Technology of China 7International Innovation Center of Tsinghua University, Shanghai, China 8Quan Cheng Laboratory {huangkaiyu,lipeng}@air.tsinghua.edu.cn majin01@mail.ustc.edu.cn; liuyang2011@tsinghua.edu.cn Abstract In a practical real-world scenario, the long- standing goal is that a universal multilingual translation model can be incrementally updated when new language pairs arrive. Specifically, the initial vocabulary only covers some of the words in new languages, which hurts the trans- lation quality for incremental learning. Al- though existing approaches attempt to address this issue by replacing the original vocabu- lary with a rebuilt vocabulary or construct- ing independent language-specific vocabular- ies, these methods can not meet the following three demands simultaneously: (1) High trans- lation quality for original and incremental lan- guages, (2) low cost for model training, (3) low time overhead for preprocessing. In this work, we propose an entropy-based vocabulary substitution (EVS) method that just needs to walk through new language pairs for incre- mental learning in a large-scale multilingual data updating while remaining the size of the vocabulary. Our method has access to learn new knowledge from updated training samples incrementally while keeping high translation quality for original language pairs, alleviating the issue of catastrophic forgetting. Results of experiments show that EVS can achieve better performance and save excess overhead for in- cremental learning in the multilingual machine translation task.1 1 Introduction Multilingual neural machine translation (NMT) aims at performing multi-directional translation with a single model. Due to its effectiveness and ef- ficiency, it has attracted intensive attention in recent ∗Corresponding authors: Peng Li (lipeng@air.tsinghua. edu.cn) and Yang Liu (liuyang2011@tsinghua.edu.cn) 1https://github.com/koukaiu/evs years (Firat et al., 2016; Johnson et al., 2017; Gu et al., 2018; Wenzek et al., 2021; Goyal et al., 2022). Typically, a multilingual NMT system is trained on a multilingual parallel corpus covering all the interested language pairs. As a result, the work of preparing and optimizing the characteristics of the training corpus makes it cumbersome and time- consuming to extend an existing multilingual NMT system to support new language pairs (Dabre et al., 2020). The stream of data evolves over time by adding new language pairs in the real-world scenario. Due to the high-cost of GPU resources, an efficient way is to incrementally train the initial transla- tion model, instead of training the model from scratch every time when the new language pairs arrive (Neubig and Hu, 2018; Lakew et al., 2018; Chronopoulou et al., 2020; Garcia et al., 2021). In- cremental learning is a potential solution, which allows neural models …

Content truncated to 500 words