Adapting Multilingual Neural Machine Translation to Unseen Languages

Adapting Multilingual Neural Machine Translation to Unseen Languages Surafel M. Lakew†+, Alina Karakanta†+, Marcello Federico+, Matteo Negri+, Marco Turchi+ †University of Trento, +Fondazione Bruno Kessler, Trento, Italy †name.surname@unitn.it, +surname@fbk.eu Abstract Multilingual Neural Machine Translation (MNMT) for low- resource languages (LRL) can be enhanced by the presence of related high-resource languages (HRL), but the related- ness of HRL usually relies on predefined linguistic assump- tions about language similarity. Recently, adapting MNMT to a LRL has shown to greatly improve performance. In this work, we explore the problem of adapting an MNMT model to an unseen LRL using data selection and model adapta- tion. In order to improve NMT for LRL, we employ per- plexity to select HRL data that are most similar to the LRL on the basis of language distance. We extensively explore data selection in popular multilingual NMT settings, namely in (zero-shot) translation, and in adaptation from a multilin- gual pre-trained model, for both directions (LRL↔en). We further show that dynamic adaptation of the model’s vocab- ulary results in a more favourable segmentation for the LRL in comparison with direct adaptation. Experiments show re- ductions in training time and significant performance gains over LRL baselines, even with zero LRL data (+13.0 BLEU), up to +17.0 BLEU for pre-trained multilingual model dy- namic adaptation with related data selection. Our method outperforms current approaches, such as massively multilin- gual models and data augmentation, on four LRL.1 1. Introduction Neural Machine Translation (NMT) has become prevalent in the past years, contributing to the flow of information across languages and facilitating communication around the world. However, NMT requires a large amount of “feature-label” aligned data for building high-quality and usable systems [1]. For the majority of the world’s languages, these resources are not available. Not benefiting from high quality MT (as it is usually the case with HRL) means that people’s access to different sources of information can be restricted. Multilingual Neural Machine Translation (MNMT) owes its success to cross-lingual knowledge transfer [2], which has been particularly beneficial for languages lacking large paral- lel data [3]. Previous works document further improvements when using languages from the same family, however they all rely on predefined linguistic assumptions about language 1Scripts to replicate the experiments and pre-trained models: https://github.com/surafelml/adapt-mnmt similarity. Another challenge for facilitating access to infor- mation through MNMT is that relevant LRL data might not be available at the time of training the initial seed model, or not available at all. In most real-life applications, new needs in terms of domains or language coverage arise continu- ously, making monolithic MNMT models susceptible to out- of-vocabulary words. Moreover, new relevant training data in several (related or not) languages might become available continuously. Taking advantage of relevant data for adapta- tion is crucial to the performance of the final models [4, 5]. Recently, building a large scale MNMT model was shown to be beneficial for LRL [6], even outperforming mod- els specifically fine-tuned on the LRL data [7]. Another approach optimizes embeddings through character n-grams …

Content truncated to 500 words