An Empirical Investigation of Word Alignment Supervision for Zero-Shot Multilingual Neural Machine Translation

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 8449–8456 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 8449 An Empirical Investigation of Word Alignment Supervision for Zero-Shot Multilingual Neural Machine Translation Alessandro Raganato, Raúl Vázquez, Mathias Creutz and Jörg Tiedemann University of Helsinki, {name.surname}@helsinki.fi Abstract Zero-shot translations is a fascinating feature of Multilingual Neural Machine Translation (MNMT) systems. These MNMT models are usually trained on English-centric data, i.e. En- glish either as the source or target language, and with a language label prepended to the input indicating the target language. How- ever, recent work has highlighted several flaws of these models in zero-shot scenarios where language labels are ignored and the wrong language is generated or different runs show highly unstable results. In this paper, we inves- tigate the benefits of an explicit alignment to language labels in Transformer-based MNMT models in the zero-shot context, by jointly training one cross attention head with word alignment supervision to stress the focus on the target language label. We compare and evaluate several MNMT systems on three mul- tilingual MT benchmarks of different sizes, showing that simply supervising one cross at- tention head to focus both on word alignments and language labels reduces the bias towards translating into the wrong language, improv- ing the zero-shot performance overall. More- over, as an additional advantage, we find that our alignment supervision leads to more stable results across different training runs. 1 Introduction Multilingual Neural Machine Translation (MNMT) focuses on translation between multiple language pairs through a single optimized neural model, and has been explored from different angles witness- ing a rapid progress in recent years (Arivazha- gan et al., 2019b; Wang et al., 2020; Dabre et al., 2020; Lin et al., 2021). Besides the great flexibil- ity MNMT models offer, they are also highlighted by their so called zero-shot translation capabili- ties, i.e., translating between all combinations of languages available in the training data, including those with no parallel data seen at training time (Ha et al., 2016; Firat et al., 2016; Johnson et al., 2017). Many studies have investigated this feature, focusing on the impact of both, the model architec- ture design (Arivazhagan et al., 2019a; Pham et al., 2019) and data pre-processing (Lee et al., 2017; Wang et al., 2019; Rios et al., 2020; Wu et al., 2021). Broadly speaking, MNMT architectures are categorized according to their degree of parameter sharing, from fully shared (Johnson et al., 2017) to the use of language-specific components (Vázquez et al., 2020; Escolano et al., 2021; Zhang et al., 2021). The Johnson et al. (2017) MNMT model is widely used, due to its simplicity and good trans- lation quality. It uses the fully shared parameters setting, and relies on appending an artificial lan- guage label to each input sentence to indicate the target language. While this method allows for zero- shot translation, several works have highlighted two major flaws: i) its failure to reliably generalize to unseen language pairs, ending up with …

Content truncated to 500 words