Divide and Rule: Effective Pre-Training for Context-Aware Multi-Encoder Translation Models

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 4557 - 4572 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Divide and Rule: Effective Pre-Training for Context-Aware Multi-Encoder Translation Models Lorenzo Lupo1 Marco Dinarelli1 Laurent Besacier1,2 1Université Grenoble Alpes, France 2Naver Labs Europe, France lorenzo.lupo@univ-grenoble-alpes.fr marco.dinarelli@univ-grenoble-alpes.fr laurent.besacier@naverlabs.com Abstract Multi-encoder models are a broad family of context-aware neural machine translation sys- tems that aim to improve translation quality by encoding document-level contextual informa- tion alongside the current sentence. The con- text encoding is undertaken by contextual pa- rameters, trained on document-level data. In this work, we discuss the difficulty of training these parameters effectively, due to the spar- sity of the words in need of context (i.e., the training signal), and their relevant context. We propose to pre-train the contextual parameters over split sentence pairs, which makes an effi- cient use of the available data for two reasons. Firstly, it increases the contextual training sig- nal by breaking intra-sentential syntactic rela- tions, and thus pushing the model to search the context for disambiguating clues more fre- quently. Secondly, it eases the retrieval of relevant context, since context segments be- come shorter. We propose four different split- ting methods, and evaluate our approach with BLEU and contrastive test sets. Results show that it consistently improves learning of con- textual parameters, both in low and high re- source settings. 1 Introduction Neural machine translation (NMT) has seen sub- stantial improvements in recent years, fostered by the advent of the Transformer model (Vaswani et al., 2017). A remaining challenge for modern machine translation (MT) is the ability to contextu- alize translation of the current sentence with other sentences in the document (Läubli et al., 2018). For this reason, contextual NMT has recently trig- gered a lot of attention and many approaches have been proposed in the literature. A common tax- onomy (Kim et al., 2019; Li et al., 2020) divides them in two broad categories: single-encoder (con- catenation) approaches (Tiedemann and Scherrer, 2017; Agrawal et al., 2018; Ma et al., 2020; Zhang et al., 2020) and multi-encoder approaches (Jean et al., 2017; Tu et al., 2017; Bawden et al., 2018; Miculicich et al., 2018; Voita et al., 2018; Maruf et al., 2019; Zheng et al., 2020). Multi-encoder models are more flexible and can be more effi- cient than concatenation approaches, but they have been criticized as being mere regularization meth- ods (Kim et al., 2019; Li et al., 2020). In some cases, they have even been shown to perform worse than sentence-level systems on discourse-aware tar- geted test suites (Lopes et al., 2020). In this work, we address this criticism by show- ing that training multi-encoder models is difficult because of two reasons: (i) the sparsity of contex- tual training signal, i.e. the signal that pushes systems to translate in a context-aware fashion, which comes from the words that need context to be correctly translated; (ii) the sparsity of relevant context words, the ones needed to …

Content truncated to 500 words