Rethinking Document-level Neural Machine Translation

Findings of the Association for Computational Linguistics: ACL 2022, pages 3537 - 3548 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Rethinking Document-level Neural Machine Translation Zewei Sun1,2,∗, Mingxuan Wang2, Hao Zhou2, Chengqi Zhao2 Shujian Huang1,3, Jiajun Chen1, Lei Li4,* 1 State Key Laboratory for Novel Software Technology, Nanjing University 2 ByteDance AI Lab, 3 Peng Cheng Laboratory, Shenzhen 4 University of California, Santa Barbara {sunzewei.v,wangmingxuan.89}@bytedance.com {zhouhao.nlp,zhaochengqi.d}@bytedance.com {huangsj,chenjj}@nju.edu.cn, leili@cs.ucsb.edu Abstract This paper does not aim at introducing a novel model for document-level neural ma- chine translation. Instead, we head back to the original Transformer model and hope to an- swer the following question: Is the capacity of current models strong enough for document- level translation? Interestingly, we observe that the original Transformer with appropriate training techniques can achieve strong results for document translation, even with a length of 2000 words. We evaluate this model and several recent approaches on nine document- level datasets and two sentence-level datasets across six languages. Experiments show that document-level Transformer models outper- forms sentence-level ones and many previous methods in a comprehensive set of metrics, including BLEU, four lexical indices, three newly proposed assistant linguistic indicators, and human evaluation. Our new datasets and evaluation scripts are in https://github. com/sunzewei2715/Doc2Doc_NMT. 1 Introduction Neural machine translation (Bahdanau et al., 2015; Wu et al., 2016; Vaswani et al., 2017) has achieved great progress and reached near human-level per- formance. However, most current sequence-to- sequence NMT models translate sentences individ- ually. In such cases, discourse phenomena, such as pronominal anaphora, lexical consistency, and doc- ument coherence that depend on long-range context going further than a few previous sentences, are ne- glected (Bawden et al., 2017). As a result, Läubli et al. (2018) find human raters still show a markedly stronger preference for human translations when evaluating at the level of documents. Many methods have been proposed to im- prove document-level neural machine translation (DNMT). Among them, the mainstream studies ∗* Work was done while at ByteDance focus on the model architecture modification, in- cluding hierarchical attention (Wang et al., 2017; Miculicich et al., 2018; Tan et al., 2019), additional context extraction encoders or query layers (Jean et al., 2017; Bawden et al., 2017; Zhang et al., 2018; Voita et al., 2018; Kuang and Xiong, 2018; Maruf et al., 2019; Yang et al., 2019; Jiang et al., 2019; Zheng et al., 2020; Yun et al., 2020; Xu et al., 2020), and cache-like memory network (Maruf and Haf- fari, 2018; Kuang et al., 2018; Tu et al., 2018). These studies come up with different structures in order to include discourse information, namely, introducing adjacent sentences into the encoder or decoder as document contexts. Experimental results show effective improvements on univer- sal translation metrics like BLEU (Papineni et al., 2002) and document-level linguistic indices (Tiede- mann and Scherrer, 2017; Bawden et al., 2017; Werlen and Popescu-Belis, 2017; Müller et al., 2018; Voita et al., 2018, 2019). Unlike previous work, this paper does not aim at introducing a novel model. Instead, we hope …

Content truncated to 500 words