Continual Learning in Multilingual NMT via Language-Specific Embeddings

Proceedings of the Sixth Conference on Machine Translation (WMT), pages 542–565 November 10–11, 2021. ©2021 Association for Computational Linguistics 542 Continual Learning in Multilingual NMT via Language-Specific Embeddings Alexandre Bérard NAVER LABS Europe alexandre.berard@naverlabs.com Abstract This paper proposes a technique for adding a new source or target language to an ex- isting multilingual NMT model without re- training it on the initial set of languages. It consists in replacing the shared vocabulary with a small language-specific vocabulary and fine-tuning the new embeddings on the new language’s parallel data. Some additional language-specific components may be trained to improve performance (e.g., Transformer lay- ers or adapter modules). Because the param- eters of the original model are not modified, its performance on the initial languages does not degrade. We show on two sets of exper- iments (small-scale on TED Talks, and large- scale on ParaCrawl) that this approach per- forms as well or better as the more costly alter- natives; and that it has excellent zero-shot per- formance: training on English-centric data is enough to translate between the new language and any of the initial languages. 1 Introduction Multilingual Neural Machine Translation models are trained on multilingual data to translate from and/or into multiple languages (Firat et al., 2016; Johnson et al., 2017). Multilingual NMT is a com- pelling approach in production, as one only needs to train, deploy and maintain one model (instead of 2 × N ones, where N is the number of lan- guages). It has also been shown to improve MT quality for low-resource languages (at the cost of a slight degradation for high-resource languages) and it can allow translation between languages that have no aligned data (“zero-shot translation”). However, such models can be costly to train, as they usually involve larger architectures and large datasets. Moreover, because they are trained jointly on all the languages, they require to know in ad- vance the full set of languages. Adding a new language to an existing model usually means re- training the model on the full multilingual dataset. Naively fine-tuning the original model on the new language’s data is not an option because of vocab- ulary mismatch (the shared vocabulary needs to be modified to include the new language’s tokens) and catastrophic forgetting (the model will quickly forget how to translate in the other languages). In this paper, we study the problem of multilin- gual NMT incremental training or continual learn- ing and propose a novel way to efficiently add a new source or target language. Some desirable properties of an incremental training method are: • No degradation on the existing language pairs; • Efficient training (e.g., no re-training on the existing language pairs); • Minimal amount of added parameters: the approach should scale to many languages and the model fit on a single GPU; • Minimal degradation in inference speed; • Good zero-shot performance: when training with X-EN (or EN-X) data, where X is a new language, we would like the model to be able to translate from X …

Content truncated to 500 words