Document Flattening: Beyond Concatenating Context for Document-Level Neural Machine Translation

Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 448–462 May 2-6, 2023 ©2023 Association for Computational Linguistics Document Flattening: Beyond Concatenating Context for Document-Level Neural Machine Translation Minghao Wu♡ George Foster♠ Lizhen Qu♡ Gholamreza Haffari♡ ♡Monash University ♠Google Research {firstname.lastname}@monash.edu fosterg@google.com Abstract Existing work in document-level neural ma- chine translation commonly concatenates sev- eral consecutive sentences as a pseudo- document, and then learns inter-sentential de- pendencies. This strategy limits the model’s ability to leverage information from distant context. We overcome this limitation with a novel Document Flattening (DOCFLAT) tech- nique that integrates FLAT-BATCH ATTEN- TION (FBA) and NEURAL CONTEXT GATE (NCG) into Transformer model to utilize infor- mation beyond the pseudo-document bound- aries. FBA allows the model to attend to all the positions in the batch and learns the relation- ships between positions explicitly and NCG identifies the useful information from the dis- tant context. We conduct comprehensive ex- periments and analyses on three benchmark datasets for English-German translation, and validate the effectiveness of two variants of DOCFLAT. Empirical results show that our ap- proach outperforms strong baselines with sta- tistical significance on BLEU, COMET and accuracy on the contrastive test set. The analy- ses highlight that DOCFLAT is highly effective in capturing the long-range information. 1 Introduction Remarkable progress has been made in neural ma- chine translation (NMT) (Sutskever et al., 2014; Vaswani et al., 2017; Chen et al., 2018), yet human translation still clearly outperforms NMT at the document level (Läubli et al., 2018; Freitag et al., 2021), because current sentence-level NMT sys- tems ignore the inter-sentential relationships. To narrow this gap, numerous document-level NMT (DocNMT) approaches have been proposed in re- cent years to improve the context awareness by in- corporating the contextual information during the translation (Tiedemann and Scherrer, 2017; Maruf and Haffari, 2018; Wong et al., 2020). Existing DocNMT systems commonly concate- nate several consecutive sentences to form a 1 2 3 5 10 24.80 25.00 25.20 25.40 Context Size BLEU 1 2 3 5 10 0.29 0.30 0.31 Context Size COMET Figure 1: The change of BLEU (left) and COMET (right) given by DOC2DOC on TED with regard to the context size of pseudo-document (in sentences) based on the experimental setup described in Section 4.1. pseudo-document, instead of processing the entire document (Zhang et al., 2018b; Voita et al., 2019; Junczys-Dowmunt, 2019; Fernandes et al., 2021). One typical pseudo-document contains the current sentence to be translated and the surrronding con- text. Intuitively, larger context should result in better performance. In our preliminary study, the model performance does not always grow as the context size increases as shown in Figure 1. Liu et al. (2020) and Bao et al. (2021) also observe that Transformer’s performance declines with longer inputs. We refer to this phenomenon as the qual- ity saturation problem (Glaser and Strauss, 1967). Therefore, such formation of pseudo-document lim- its the DocNMT systems to leverage the informa- tion from a relatively small context. Consequently, once the …

Content truncated to 500 words