Training and Adapting Multilingual NMT for Less-resourced and Morphologically Rich Languages

Training and Adapting Multilingual NMT for Less-resourced and Morphologically Rich Languages Mat¯ıss Rikters, M¯arcis Pinnis, Rihards Kriˇslauks Tilde Vien¯ıbas gatve 75A, Riga, Latvia {matiss.rikters, marcis.pinnis, rihards.krislauks}@tilde.lv Abstract In this paper, we present results of employing multilingual and multi-way neural machine translation approaches for morphologically rich languages, such as Estonian and Russian. We experiment with different NMT architectures that allow achieving state-of-the-art translation quality and compare the multi-way model performance to one-way model performance. We report improvements of up to +3.27 BLEU points over our baseline results, when using a multi-way model trained using the transformer network architecture. We also provide open-source scripts used for shuffling and combining multiple parallel datasets for training of the multilingual systems. Keywords: neural machine translation, multilingual machine translation, morphologically rich languages 1. Introduction One of the major advantages of neural machine translation (NMT) is that unlike statistical machine translation (SMT), which was the previous industry standard (and is still ac- tively used in commercial applications), NMT is trained and used jointly as a single end-to-end system without the need to optimize multiple independent models and relations between the models. However, training NMT systems for individual language pairs has shown to take significantly more time (e.g., two to three weeks or up to a week with newer platforms, such as Marian (Junczys-Dowmunt et al., 2016) or Google’s Tensor2Tensor toolkit 1) than training of SMT systems (e.g., less than a day or up to several days for large systems). But even with this advantage, using the tra- ditional approaches, one would still need to train a separate model for each translation direction. Since running a high amount of GPU-intensive NMT models in a production en- vironment can quickly sum up to an enormous resource- usage cost, it has been natural (as shown by related work in Section 2) to look for solutions that allow compressing the models into an even more dense end-to-end solution that is able to handle multiple languages and language pairs si- multaneously. Another benefit of a single model for multiple translation directions could be the ability to learn not just from the training data of the language pair in question, but also from language pairs that include one of the languages. The ad- vantages of learning from multiple translation directions at the same time can be (1) the ability for a model to learn how to translate language specific attributes that are com- mon to multiple languages at the same time, and (2) to learn and generalize translations that may not occur in the paral- lel corpus of, e.g., A↔B, but do occur in parallel corpora of, e.g., A↔C and C↔B and therefore are deducible. This work has been driven by the need to identify the best neural network architectures for the development of one- way and multi-way NMT systems for low-resource lan- guage pairs that can be applied for low-resource NMT 1T2T: Tensor2Tensor Transformers - https://github. com/tensorflow/tensor2tensor system development (and/or system adaptation) within the project “Forest Industry Communication Technologies”. The structure of …

Content truncated to 500 words