A Case Study on Context Encoding in Multi-Encoder based Document-Level Neural Machine Translation

Proceedings of Machine Translation Summit XIX, Vol. 1: Research Track, pages 160–172 September 4–8, 2023, Macau SAR, China. ©2023 The authors. This article is licensed under a Creative Commons Attribution 4.0 International License (CC BY 4.0). 160 A Case Study on Context Encoding in Multi-Encoder based Document-Level Neural Machine Translation Ramakrishna Appicharla appicharla 2021cs01@iitp.ac.in Baban Gain gainbaban@gmail.com Department of Computer Science and Engineering, Indian Institute of Technology Patna, Patna, India Santanu Pal santanu.pal.ju@gmail.com Wipro AI, Lab45, London, UK Asif Ekbal asif@iitp.ac.in Department of Computer Science and Engineering, Indian Institute of Technology Patna, Patna, India Abstract Recent studies have shown that the multi-encoder models are agnostic to the choice of context, and the context encoder generates noise which helps improve the models in terms of BLEU score. In this paper, we further explore this idea by evaluating with context-aware pronoun translation test set by training multi-encoder models trained on three different context settings viz, previous two sentences, random two sentences, and a mix of both as context. Specifically, we evaluate the models on the ContraPro test set to study how different contexts affect pro- noun translation accuracy. The results show that the model can perform well on the ContraPro test set even when the context is random. We also analyze the source representations to study whether the context encoder generates noise. Our analysis shows that the context encoder pro- vides sufficient information to learn discourse-level information. Additionally, we observe that mixing the selected context (the previous two sentences in this case) and the random context is generally better than the other settings. 1 Introduction Document-level neural machine translation (DocNMT) has gained a lot of attention due to the ability to incorporate context through different paradigms such as single encoder (Tiedemann and Scherrer, 2017; Agrawal et al., 2018), multiple encoders (Zhang et al., 2018; Li et al., 2020; Huo et al., 2020), memory networks (Maruf and Haffari, 2018) and pre-trained language models (Donato et al., 2021). This additional context helps to produce more consistent trans- lations (Bawden et al., 2018; Voita et al., 2019) than sentence-level models. Two of the most followed approaches to incorporating context are the concatenation-based and multi-encoder- based approaches. In the concatenation-based method, by concatenating context and current input sentence, a context-aware input sentence is generated (Tiedemann and Scherrer, 2017; Agrawal et al., 2018; Junczys-Dowmunt, 2019; Zhang et al., 2020) and use it as the input to the 161 encoder. In the multi-encoder approach, to encode the source or target context, an additional encoder is used (Zhang et al., 2018; Voita et al., 2018; Kim et al., 2019; Ma et al., 2020) and the entire model is jointly optimized. Typically, the current sentence’s neighboring sentences (previous or next) are used as the context, whereas models consist of multiple encoders and a single decoder. Recent studies on Multi-Encoder (MultiEnc) based DocNMT models (Li et al., 2020; Wang et al., 2020; Gain et al., 2022) have shown that the context-encoder is acting as noise generator which improves the robustness of …

Content truncated to 500 words