Multilingual Machine Translation: Closing the Gap between Shared and Language-specific Encoder-Decoders

Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pages 944–948 April 19 - 23, 2021. ©2021 Association for Computational Linguistics 944 Multilingual Machine Translation: Closing the Gap between Shared and Language-specific Encoder-Decoders Carlos Escolano, Marta R. Costa-juss`a, Jos´e A. R. Fonollosa, Mikel Artetxe† TALP Research Center, Universitat Polit`ecnica de Catalunya, Barcelona †IXA NLP Group, University of the Basque Country (UPV/EHU) {carlos.escolano,marta.ruiz,jose.fonollosa}@upc.edu mikel.artetxe@ehu.eus Abstract State-of-the-art multilingual machine transla- tion relies on a universal encoder-decoder, which requires retraining the entire system to add new languages. In this paper, we pro- pose an alternative approach that is based on language-specific encoder-decoders, and can thus be more easily extended to new languages by learning their corresponding modules. So as to encourage a common interlingua repre- sentation, we simultaneously train the N ini- tial languages. Our experiments show that the proposed approach outperforms the universal encoder-decoder by 3.28 BLEU points on av- erage, while allowing to add new languages without the need to retrain the rest of the mod- ules. All in all, our work closes the gap be- tween shared and language-specific encoder- decoders, advancing toward modular multilin- gual machine translation systems that can be flexibly extended in lifelong learning settings. 1 Introduction Multilingual machine translation is the ability to generate translations automatically across a (large) number of languages. Research in this area has at- tracted a lot of attention in recent times both from the scientific and industrial community. Under the neural machine translation paradigm (Bahdanau et al., 2015), the opportunities for improving this area have dramatically expanded. Thanks to the encoder-decoder architecture, there are viable al- ternatives to expensive pairwise translation based on classic paradigms1. The main proposal in this direction is the uni- versal encoder-decoder (joh, 2017) with massive multilingual enhancements (Arivazhagan et al., 2019). While this approach enables zero-shot translation and is beneficial for low-resource lan- guages, it has multiple drawbacks: (i) the entire 1http://www.euromatrixplus.net system has to be retrained when adding new lan- guages or data; (ii) the quality of translation drops when adding too many languages or for those with the most resources (Arivazhagan et al., 2019); and (iii) the shared vocabulary grows dramatically when adding a large number of languages (espe- cially when they do not share alphabets). Other limitations include the incompatibility of adding multiple modalities such as image or speech. In this paper, we propose a new framework that can be incrementally extended to new languages without the aforementioned limitations (§3). Our proposal is based on language-specific encoders and decoders that rely on a common intermedi- ate representation space. For that purpose, we si- multaneously train the initial N languages in all translation directions. New languages are natu- rally added to the system by training a new mod- ule coupled with any of the existing ones, while new data can be easily added by retraining only the module for the corresponding language. We evaluate our proposal on three experimen- tal configurations: translation for the initial lan- guages, translation …

Content truncated to 500 words