Adapting Multilingual NMT to Extremely Low Resource Languages FBK’s Participation in the Basque-English Low-Resource MT Task, IWSLT 2018

Adapting Multilingual NMT to Extremely Low Resource Languages FBK’s Participation in the Basque-English Low-Resource MT Task, IWSLT 2018 Surafel M. Lakew†+, Marcello Federico⋆ †University of Trento, +Fondazione Bruno Kessler, Trento, Italy ⋆Amazon AI, East Palo Alto, CA 94303, USA lastname@fbk.eu, ⋆marcfede@amazon.com Abstract Multilingual neural machine translation (M-NMT) has re- cently shown to improve performance of machine translation of low-resource languages. Thanks to its implicit transfer- learning mechanism, the availability of a highly resourced language pair can be leveraged to learn useful representa- tion for a lower resourced language. This work investigates how a low-resource translation task can be improved within a multilingual setting. First, we adapt a system trained on mul- tiple language directions to a specific language pair. Then, we utilize the adapted model to apply an iterative training- inference scheme [1] using monolingual data. In the exper- imental setting, an extremely low-resourced Basque-English language pair (i.e., ≈ 5.6K in-domain training data) is our target translation task, where we considered a closely re- lated French/Spanish-English parallel data to build the mul- tilingual model. Experimental results from an i) in-domain and ii) an out-of-domain setting with additional training data, show improvements with our approach. We report a transla- tion performance of 15.89 with the former and 23.99 BLEU with the latter on the official IWSLT 2018 Basque-English test set. 1. Introduction The amount and diversity of model training data have been shown to affect the performance of Neural Machine Transla- tion (NMT) system [2]. The direct relation between dataset size and performance of NMT [3], calls for alternative ap- proaches to improve low-resource language translation. Multilingual models that constitute more than one lan- guage pair has been shown to improve the translation perfor- mance of the low-resources language direction [4, 5]. In it’s simplified and most effective setting, building an M-NMT system requires only an additional “language-flag” on the data level. Then, the attentional encoder-decoder based NMT model can be trained with the aggregation of several lan- guage pairs. The flag functions as a mechanism to trigger and direct the generation of target tokens in a specific target language. Thus, when the training set is constructed with (*) Work conducted while this author was at FBK. the merge of several language directions, the latent transfer- learning across languages within the conventional NMT ar- chitecture showed to improve low-resourced language pairs. However, M-NMT training mechanism is biased towards generating the language pair with the largest portion of train- ing data [1]. This bias will consequently limit the expected level of improvement in translating low-resource language pairs. In this work, we propose a progressive adaptation of a multilingual model to a single language pair. We cast the adaptation stage in iterative training-inference operations that utilize monolingual data. Assuming, the availability of a low-resource language pair and a high resource re- lated/language pairs data, we specifically explore the follow- ing two mechanisms: • Adapting a multilingual model trained with several language directions to a specific low-resourced lan- guage pair, with the aim to …

Content truncated to 500 words