Document Graph for Neural Machine Translation

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 8435–8448 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 8435 Document Graph for Neural Machine Translation Mingzhou Xu1∗, Liangyou Li2, Derek F. Wong1, Qun Liu2,Lidia S. Chao1, 1NLP2CT Lab, University of Macau 2Huawei Noah’s Ark Lab nlp2ct.mzxu@gmail.com, {derekfw,lidiasc}@um.edu.com {liliangyou,qun.liu}@huawei.com Abstract Previous works have shown that contextual information can improve the performance of neural machine translation (NMT). However, most existing document-level NMT methods only consider a few number of previous sen- tences. How to make use of the whole document as global contexts is still a chal- lenge. To address this issue, we hypothe- size that a document can be represented as a graph that connects relevant contexts re- gardless of their distances. We employ sev- eral types of relations, including adjacency, syntactic dependency, lexical consistency, and coreference, to construct the document graph. Then, we incorporate both source and target graphs into the conventional Transformer ar- chitecture with graph convolutional networks. Experiments on various NMT benchmarks, including IWSLT English–French, Chinese- English, WMT English–German and Opensub- title English–Russian, demonstrate that using document graphs can significantly improve the translation quality. Extensive analysis verifies that the document graph is beneficial for cap- turing discourse phenomena. 1 Introduction Although neural machine translation (NMT) has achieved great success on sentence-level translation tasks, many studies pointed out that translation mistakes become more noticeable at the document- level (Wang et al., 2017; Tiedemann and Scherrer, 2017; Zhang et al., 2018; Miculicich et al., 2018; Kuang et al., 2018; Voita et al., 2018; Läubli et al., 2018; Tu et al., 2018; Voita et al., 2019b; Kim et al., 2019; Yang et al., 2019). They proved that these mistakes can be alleviated by feeding the contexts into context-agnostic NMT models. Previous works have explored various methods to integrate context information into NMT models. ∗ Work was done when Mingzhou Xu was interning at Noah’s Ark Lab. S … … S S ⽶格尔 Miguel 其实 actually 他的 his 名字 name ⽶格尔 Miguel 看着 look at 我 I 我 I 说 say ⽶格尔 Miguel Figure 1: The structure of graph. Solid lines in blue de- pict adjacency relations. Dash lines in green denote de- pendency relations. Lexical consistency is represented as dashed lines in red. The brown line means a corefer- ence relation. S denotes Sentence node. We just show aspects of sentences for convenience.1 They usually take a limited number of previous sen- tences as contexts and learn context-aware repre- sentations using hierarchical networks (Miculicich et al., 2018; Wang et al., 2017; Tan et al., 2019) or extra context encoders (Jean et al., 2015; Zhang et al., 2018; Yang et al., 2019). Different from representation-based approaches, Tu et al. (2018) and Kuang et al. (2018) propose using a cache to memorize context information, which can be either history hidden states or lexicons. To keep track- ing of most recent contexts, the cache is updated when new translations are generated. Therefore, long-distance contexts would likely be erased. How to …

Content truncated to 500 words