Transfer Learning in Multilingual Neural Machine Translation with Dynamic Vocabulary

Transfer Learning in Multilingual Neural Machine Translation with Dynamic Vocabulary Surafel M. Lakew†+, Aliia Erofeeva†, Matteo Negri+, Marcello Federico⋆, Marco Turchi+ †University of Trento, +Fondazione Bruno Kessler, Trento, Italy ⋆Amazon AI, East Palo Alto, CA 94303, USA †name.surname@unitn.it, +surname@fbk.eu, ⋆marcfede@amazon.com Abstract We propose a method to transfer knowledge across neural machine translation (NMT) models by means of a shared dy- namic vocabulary. Our approach allows to extend an initial model for a given language pair to cover new languages by adapting its vocabulary as long as new data become avail- able (i.e., introducing new vocabulary items if they are not included in the initial model). The parameter transfer mecha- nism is evaluated in two scenarios: i) to adapt a trained single language NMT system to work with a new language pair and ii) to continuously add new language pairs to grow to a mul- tilingual NMT system. In both the scenarios our goal is to improve the translation performance, while minimizing the training convergence time. Preliminary experiments span- ning five languages with different training data sizes (i.e., 5k and 50k parallel sentences) show a significant performance gain ranging from +3.85 up to +13.63 BLEU in different lan- guage directions. Moreover, when compared with training an NMT model from scratch, our transfer-learning approach al- lows us to reach higher performance after training up to 4% of the total training steps. 1. Introduction Neural Machine Translation (NMT) has shown to sur- pass phrase based Machine Translation approaches not only in high-resource language settings, but also with low- resource [1] and zero-resource translation tasks [2, 3]. Al- though recent approaches yield promising results, training models in low-resource settings remains a challenge for MT research [4]. [2] have shown that a multilingual NMT (M- NMT) model that utilizes a concatenation of data covering multiple language pairs (including high-resourced ones) can result in better performance in the low-resource translation task. Alternatively, [5] proposed a transfer-learning approach from an NMT “parent-model” trained on a high-resource lan- guage to initialize a “child-model” in a low-resource setting showing consistent translation improvements on the latter task. Though effective, training models on a concatenation of data covering multiple language pairs or initializing them by (*) Work conducted while this author was at FBK. transferring knowledge from a parent model does not con- sider the dynamic nature of new language vocabularies. In relation to how and when model vocabularies are built, there can be two distinct scenarios. In the first one, all the training data for all the language pairs are available since the begin- ning. In this case, either separate or joint sub-word segmen- tation models can be applied on the training material to build vocabularies that represent all the data [6, 7]. In the second scenario, training data covering different language directions are not available at the same time (most real-world MT train- ing scenarios fall in this category, in which new data or new needs in terms of domains or language coverage emerge over time). In such cases, …

Content truncated to 500 words