Focused Concatenation for Context-Aware Neural Machine Translation

Proceedings of the Seventh Conference on Machine Translation (WMT), pages 830 - 842 December 7-8, 2022 ©2022 Association for Computational Linguistics Focused Concatenation for Context-Aware Neural Machine Translation Lorenzo Lupo1 Marco Dinarelli1 Laurent Besacier2 1Université Grenoble Alpes, France 2Naver Labs Europe, France lorenzo.lupo@univ-grenoble-alpes.fr marco.dinarelli@univ-grenoble-alpes.fr laurent.besacier@naverlabs.com Abstract A straightforward approach to context-aware neural machine translation consists in feeding the standard encoder-decoder architecture with a window of consecutive sentences, formed by the current sentence and a number of sen- tences from its context concatenated to it. In this work, we propose an improved concate- nation approach that encourages the model to focus on the translation of the current sentence, discounting the loss generated by target con- text. We also propose an additional improve- ment that strengthen the notion of sentence boundaries and of relative sentence distance, facilitating model compliance to the context- discounted objective. We evaluate our approach with both average-translation quality metrics and contrastive test sets for the translation of inter-sentential discourse phenomena, proving its superiority to the vanilla concatenation ap- proach and other sophisticated context-aware systems. 1 Introduction While current neural machine translation (NMT) systems have reached close-to-human quality in the translation of decontextualized sentences (Wu et al., 2016), they still have a wide margin of im- provement ahead when it comes to translating full documents (Läubli et al., 2018). Many works tried to reduce this margin, proposing various ap- proaches to context-aware NMT (CANMT)1. A common taxonomy (Kim et al., 2019; Li et al., 2020) divides them in two broad categories: multi- encoding approaches and concatenation (single- encoding) approaches. Despite its simplicity, the concatenation approaches have been shown to achieve competitive or superior performance to more sophisticated, multi-encoding systems (Lopes et al., 2020; Ma et al., 2021). Nonetheless, it 1Unless otherwise specified, we refer to context as the sentences that precede or follow a current sentence to be translated, within the same document. Figure 1: Example of the proposed approach applied over a window of 2 sentences, with context discount CD and segment-shifted positions by a factor of 10. has been shown that Transformer-based NMT sys- tems (Vaswani et al., 2017) struggle to learn locality properties (Hardmeier, 2012; Rizzi, 2013) of both the language itself and the source-target alignment when the input sequence grows in length, as in the case of concatenation (Bao et al., 2021). Unsur- prisingly, the presence of context makes learning harder for concatenation models by distracting at- tention. Moreover, we know from recent litera- ture that NMT systems require context for a sparse set of inter-sentential discourse phenomena only (Voita et al., 2019; Lupo et al., 2022). Therefore, it is desirable to make concatenation models more fo- cused on local linguistic phenomena, belonging to the current sentence, while also processing its con- text for enabling inter-sentential contextualization whenever it is needed. We propose an improved concatenation approach to CANMT that is more focused on the translation of the current sentence by means of two simple, parameter-free solutions: • Context-discounting: a simple modification of the NMT …

Content truncated to 500 words