CoDoNMT: Modeling Cohesion Devices for Document-Level Neural Machine Translation

Proceedings of the 29th International Conference on Computational Linguistics, pages 5205–5216 October 12–17, 2022. 5205 CoDoNMT: Modeling Cohesion Devices for Document-Level Neural Machine Translation Yikun Lei, Yuqi Ren, Deyi Xiong ∗ College of Intelligence and Computing, Tianjin University, Tianjin, China {yikunlei,ryq20,dyxiong}@tju.edu.cn Abstract Cohesion devices, e.g., reiteration, corefer- ence, are crucial for building cohesion links across sentences. In this paper, we propose a document-level neural machine translation framework, CoDoNMT, which models cohe- sion devices from two perspectives: Cohesion Device Masking (CoDM) and Cohesion Atten- tion Focusing (CoAF). In CoDM, we mask co- hesion devices in the current sentence and force NMT to predict them with inter-sentential con- text information. A prediction task is also in- troduced to be jointly trained with NMT. In CoAF, we attempt to guide the model to pay exclusive attention to relevant cohesion devices in the context when translating cohesion de- vices in the current sentence. Such a cohesion attention focusing strategy is softly applied to the self-attention layer. Experiments on three benchmark datasets demonstrate that our ap- proach outperforms state-of-the-art document- level neural machine translation baselines. Fur- ther linguistic evaluation validates the effec- tiveness of the proposed model in producing cohesive translations. 1 Introduction Neural Machine Translation (NMT) has become the dominant approach for machine translation and achieved substantial progress in comparison to sta- tistical machine translation. Some studies even claim that NMT has reached human parity (Hassan et al., 2018). Despite this, most NMT models are at the sentence level, which translate documents sen- tence by sentence, ignoring inter-sentential depen- dencies. Documents translated in this way are usu- ally incoherent and inconsistent across sentences. In order to address this issue, a wide range of efforts have been made to leverage inter- sentential context information for document-level NMT (Tiedemann and Scherrer, 2017; Zhang et al., ∗corresponding author 2018; Voita et al., 2019; Tan et al., 2019; Maruf et al., 2019; Xu et al., 2020b; Zhang et al., 2021). Most efforts have been dedicated to modeling local or global context via additional encoders, atten- tion, cache, concatenating inputs, etc (Maruf et al., 2021). However, these approaches normally focus on the way of integrating context into translation, rather than the context itself. The basic assumption behind this is that models are able to detect rele- vant contextual information. However, Kim et al. (2019) and Li et al. (2020) find that most of the im- provements obtained by these approaches cannot be explained as leveraging the right context. We suggest that it could be not a good choice to treat contextual words equally and rely on models to learn contextual clues in an implicit way. This is because it is usually difficult for NMT to capture key information from a long context through itself (Yin et al., 2021). Hence, pinpointing semantically or grammatically relevant context words in an ex- plicit way is desirable for document-level NMT. In this paper, different from the aforementioned context modeling schemes, we model contextual information for document-level NMT in an explicit way via cohesion …

Content truncated to 500 words