Language Modeling, Lexical Translation, Reordering: The Training Process of NMT through the Lens of Classical SMT

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 8478–8491 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 8478 Language Modeling, Lexical Translation, Reordering: The Training Process of NMT through the Lens of Classical SMT Elena Voita1,2 Rico Sennrich3,1 Ivan Titov1,2 1University of Edinburgh, Scotland 2University of Amsterdam, Netherlands 3University of Zurich, Switzerland lena-voita@hotmail.com sennrich@cl.uzh.ch ititov@inf.ed.ac.uk Abstract Differently from the traditional statistical MT that decomposes the translation task into dis- tinct separately learned components, neural machine translation uses a single neural net- work to model the entire translation process. Despite neural machine translation being de- facto standard, it is still not clear how NMT models acquire different competences over the course of training, and how this mirrors the dif- ferent models in traditional SMT. In this work, we look at the competences related to three core SMT components and find that during training, NMT first focuses on learning target- side language modeling, then improves transla- tion quality approaching word-by-word trans- lation, and finally learns more complicated reordering patterns. We show that this be- havior holds for several models and language pairs. Additionally, we explain how such an understanding of the training process can be useful in practice and, as an example, show how it can be used to improve vanilla non- autoregressive neural machine translation by guiding teacher model selection. 1 Introduction In the last couple of decades, the two main ma- chine translation paradigms have been statistical and neural MT. Statistical MT (SMT) decomposes the translation task into several components (e.g., lexical translation probabilities, alignment proba- bilities, target-side language model, etc.) which are learned separately and then combined in a transla- tion model. Differently, neural MT (NMT) models the entire translation process with a single neural network that is trained end-to-end. Although joint training of all the components is one of the obvious NMT strengths, this is also one of its challenging aspects. While SMT models different competences with distinct model components and, therefore, can easily validate and/or improve each of them, NMT acquires these competences within the same network over the course of training. Even though previous work shows how to improve some of the competences in NMT, e.g., by using lexical translation proba- bilities, phrase memories, target-side LM, align- ment information (Arthur et al., 2016; He et al., 2016; Tang et al., 2016; Wang et al., 2017; Zhang et al., 2017a; Dahlmann et al., 2017; Gülçehre et al., 2015; Gülçehre et al., 2017; He et al., 2016; Sri- ram et al., 2017; Dahlmann et al., 2017; Stahlberg et al., 2018; Mi et al., 2016b; Liu et al., 2016; Chen et al., 2016; Alkhouli et al., 2016; Alkhouli and Ney, 2017; Park and Tsvetkov, 2019; Song et al., 2020a among others), it is still not clear how and when NMT acquires these competences during training. For example, are there any stages where NMT focuses on different aspects of translation, e.g., fluency (agreement on the target side) or ad- equacy (i.e. connection to the source), …

Content truncated to 500 words