What Works When in Context-aware Neural Machine Translation?

What Works When in Context-aware Neural Machine Translation? Harritxu Gete1,2 Thierry Etchegoyhen1 1Vicomtech Foundation, Basque Research and Technology Alliance (BRTA) 2University of the Basque Country UPV/EHU 3HiTZ Basque Center for Language Technologies - Ixa {hgete,tetchegoyhen}@vicomtech.org, gorka.labaka@ehu.eus Gorka Labaka2,3 Abstract Document-level Machine Translation has emerged as a promising means to enhance automated translation quality, but it is currently unclear how effectively context- aware models use the available context during translation. This paper aims to pro- vide insight into the current state of models based on input concatenation, with an in- depth evaluation on English–German and English–French standard datasets. We no- tably evaluate the impact of data bias, an- tecedent part-of-speech, context complex- ity, and the syntactic function of the el- ements involved in discourse phenomena. Our experimental results indicate that the selected models do improve the overall translation in context, with varying sensi- tivity to the different factors we examined. We notably show that the selected context- aware models operate markedly better on regular syntactic configurations involving subject antecedents and pronouns, with de- graded performance as the configurations become more dissimilar. 1 Introduction Neural Machine Translation (NMT) systems (Sutskever et al., 2014; Bahdanau et al., 2015; Vaswani et al., 2017) have traditionally translated sentences in isolation without considering rela- tions between discourse elements. This leads to translations lacking crucial textual properties such as cohesion, discourse coherence or intersentential © 2023 The authors. This article is licensed under a Creative Commons 4.0 licence, no derivative works, attribution, CC- BY-ND. anaphora resolution (Bawden et al., 2018; L¨aubli et al., 2018; Voita et al., 2019b; Lopes et al., 2020). Properly handling discourse-related phenomena requires extending the scope of the translation model beyond the sentence level. As a result, many methods have been developed to extend the modeling window beyond isolated sentences. These approaches range from extending the input of standard NMT models (Tiedemann and Scher- rer, 2017) to architectural variants (Tu et al., 2018; Miculicich et al., 2018; Li et al., 2020). Despite the promising results achieved by context-aware NMT, determining the precise use of context remains a significant challenge, leading to contradictory findings, including studies sug- gesting that context-aware models do not improve intersentential phenomena, but rather act as mere regularisers (Kim et al., 2019; Li et al., 2020; Rauf and Yvon, 2020). Standard translation metrics have limitations to measure document-level phe- nomena, whereas contrastive evaluations provide more precise measures but do not delve into how context information is actually used or ignored. We believe that in-depth analyses of context usage by context-aware models could help better under- stand their current strengths and limitations. In this paper, we analyse the performance of var- ious approaches based on context concatenation, a strong baseline for document-level NMT, examin- ing variations in the use of source and target con- text. We provide an in-depth analysis of the results achieved by the selected NMT models in terms of data bias, context complexity, as well as part of speech and syntactic functions of the relevant elements in contextual …

Content truncated to 500 words