Towards Continual Learning for Multilingual Machine Translation via Vocabulary Substitution

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1184–1192 June 6–11, 2021. ©2021 Association for Computational Linguistics 1184 Towards Continual Learning for Multilingual Machine Translation via Vocabulary Substitution Xavier Garcia and Noah Constant and Ankur P. Parikh and Orhan Firat Google Research Mountain View California {xgarcia,nconstant,aparikh,orhanf}@google.com Abstract We propose a straightforward vocabulary adap- tation scheme to extend the language capac- ity of multilingual machine translation mod- els, paving the way towards efficient contin- ual learning for multilingual machine transla- tion. Our approach is suitable for large-scale datasets, applies to distant languages with un- seen scripts, incurs only minor degradation on the translation performance for the original lan- guage pairs and provides competitive perfor- mance even in the case where we only possess monolingual data for the new languages. 1 Introduction The longstanding goal of multilingual machine translation (Firat et al., 2016; Johnson et al., 2017; Aharoni et al., 2019; Gu et al., 2018) has been to develop a universal translation model, capable of providing high-quality translations between any pair of languages. Due to limitations on the data available, however, current approaches rely on first selecting a set of languages for which we have data and training an initial translation model on this data jointly for all languages in a multi-task setup. In an ideal setting, one would continually update the model once data for new language pairs arrives. This setting, dubbed in the literature as continual learning (Ring, 1994; Rebuffi et al., 2017; Kirk- patrick et al., 2017; Lopez-Paz and Ranzato, 2017), introduces new challenges not found in the tradi- tional multi-task setup, most famously catastrophic forgetting (McCloskey and Cohen, 1989), in which the model may lose its previously-learned knowl- edge as it learns new language pairs. This situation is further complicated by the training procedures of standard tokenizers, such as Byte-Pair Encoding (BPE) (Sennrich et al., 2015b) or Sentencepiece (Kudo and Richardson, 2018), which necessitate access to monolingual data for all the languages considered before producing the vocabulary. Fail- ing to comply with these requirements, one risks suboptimal segmentation rules which in the worst case could result in strings of entirely <UNK> to- kens for text in a previously-unseen alphabet. In this work, we investigate how vocabularies derived from BPE transform if they are rebuilt with the same settings but with additional data from a new language. We show in Section 3.1 that there is a large token overlap between the original and updated vocabularies. This large overlap allows us to retain the performance of a translation model after replacing its vocabulary with the updated vo- cabulary that additionally supports a new language. Past works have explored adapting translation models to new languages, typically focusing on related languages which share similar scripts (Gu et al., 2018; Neubig and Hu, 2018; Lakew et al., 2019; Chronopoulou et al., 2020). These works usually focus solely on learning the new language pair, with no consideration for catastrophic for- getting. Moreover, …

Content truncated to 500 words