Efficiently Upgrading Multilingual Machine Translation Models to Support More Languages

Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 1513–1527 May 2-6, 2023 ©2023 Association for Computational Linguistics Efficiently Upgrading Multilingual Machine Translation Models to Support More Languages Simeng Sun1∗ Maha Elbayad2 Anna Sun2 James Cross2† University of Massachusetts Amherst1 Meta AI2 simengsun@umass.edu {elbayadm, annaysun, jcross}@meta.com Abstract With multilingual machine translation (MMT) models continuing to grow in size and number of supported languages, it is natural to reuse and upgrade existing models to save compu- tation as data becomes available in more lan- guages. However, adding new languages re- quires updating the vocabulary, which compli- cates the reuse of embeddings. The question of how to reuse existing models while also mak- ing architectural changes to provide capacity for both old and new languages has also not been closely studied. In this work, we intro- duce three techniques that help speed up effec- tive learning of the new languages and alleviate catastrophic forgetting despite vocabulary and architecture mismatches. Our results show that by (1) carefully initializing the network, (2) ap- plying learning rate scaling, and (3) performing data up-sampling, it is possible to exceed the performance of a same-sized baseline model with 30% computation and recover the perfor- mance of a larger model trained from scratch with over 50% reduction in computation. Fur- thermore, our analysis reveals that the intro- duced techniques help learn the new directions more effectively and alleviate catastrophic for- getting at the same time. We hope our work will guide research into more efficient approaches to growing languages for these MMT models and ultimately maximize the reuse of existing models. 1 Introduction Research into multilingual machine translation (MMT) (Aharoni et al., 2019; Fan et al., 2020) has shifted from a relatively small number of trans- lation directions (Dong et al., 2015; Firat et al., 2016; Ha et al., 2016) to much larger scale, recently reaching up to tens of thousands of translation di- rections (NLLB Team et al., 2022; Bapna et al., 2022). Despite the notable increase in the number ∗Work done during an internship at Meta AI. †Work done while working at Meta AI. of supported languages, these models still need to be upgraded as increasing amount of data in new languages are becoming available. The process of adding new languages to existing models is an in- stance of continual learning (Ring, 1994), in which the models need to effectively learn new tasks (new translation directions) while not catastrophically forgetting (French, 1993; McCloskey and Cohen, 1989) the knowledge about tasks from the previous training stage (original translation directions). Unlike the most conventional continual learning setup where the model remains the same as the initial learning stage, growing languages for MMT models needs to deal with new parameters. Adding new languages to existing training data shifts sub- word token distribution (e.g., tokens from unseen scripts are added), hence the need to re-train the tokenizer, which adds new embedding parameters to the MMT model. Previous studies have shown the effectiveness of …

Content truncated to 500 words