Latest Development in the FoTran Project – Scaling Up Language Coverage in Neural Machine Translation Using Distributed Training with Language-Specific Components

Latest Development in the FoTran Project – Scaling Up Language Coverage in Neural Machine Translation Using Distributed Training with Language-Specific Components Ra´ul V´azquez♠ Michele Boggia♠ Alessandro Raganato♡ Niki A. Loppi♢ Stig-Arne Gr¨onroos♠♣ J¨org Tiedemann♠ ♠ University of Helsinki, Finland ♢ NVIDIA♡ University of Milano-Bicocca, Italy ♣ Silo.AI ♠{name.surname}@helsinki.fi, ♡{name.surname}@unimib.it, ♢nloppi@nvidia.com Abstract We give an update on the Found in Trans- lation (FoTran) project, focusing on the study of emerging language-agnostic rep- resentations from neural machine transla- tion (NMT). We describe our attention- bridge model, a modular NMT model which connects language-specific compo- nents through a shared network layer. Our latest implementation supports distributed training over many nodes and GPUs in or- der to substantially scale up the number of languages that can be included in a modern neural translation architecture. 1 Introduction The FoTran project aims at developing models for natural language understanding trained on implicit information given by large collections of human translations.1 It is funded by a European Research Council consolidation grant, running from 2018 to 2023 within the language technology research group at the University of Helsinki under coordi- nation of Prof. J¨org Tiedemann. Cross-lingual grounding, useful for resolving ambiguities through translation, is a guiding prin- ciple of the project. Consequently, we developed a model for multilingual NMT specifically designed to obtain meaning representations injected with multilingual data (V´azquez et al., 2020). Former project results pointed towards the improvement of both the translation quality and the abstractions acquired by our model when including more lan- guages (V´azquez et al., 2019; Raganato et al., © 2022 The authors. This article is licensed under a Creative Commons 3.0 licence, no derivative works, attribution, CC- BY-ND. 1http://www.helsinki.fi/fotran 2019). Due to the use of language-specific mod- ules, the overall model architecture grows when languages and translation directions are added. Doing this on a single device does not scale beyond the memory limits of that specific computing node. This is a limitation for testing the project hypothe- sis that training on increasing amounts of linguisti- cally diverse data improves the abstractions found by the model – eventually leading to language- independent meaning representations useful for machine translation and tasks that require semantic reasoning and inference. Here, we propose strate- gies to address those issues: (1) distribute mod- ules across several processing units, (2) efficiently train the network over many translation directions, and (3) reuse the trained modules without having to load the entire network. These together deliver a cost-effective multilingual NMT system that can further be used for extracting multilingual meaning representations.2 Despite the high computational resources needed to scale up the number of trans- lation directions when training the model, its mod- ularity allows to reuse the trained components on relatively small processing units, making multilin- gual models more affordable and increasing their availability. 2 Methodology The implementation follows an encoder–decoder architecture, incorporating language specific en- coders and decoders to enable multilingual train- ing. They are connected via a shared inner- attention layer that summarizes the encoder information in a fixed-size …

Content truncated to 500 words