Hierarchical Modeling of Global Context for Document-Level Neural Machine Translation

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 1576–1585, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 1576 Hierarchical Modeling of Global Context for Document-Level Neural Machine Translation Xin Tan, Longyin Zhang, Deyi Xiong, Guodong Zhou∗ School of Computer Science and Technology, Soochow University, Suzhou, China {annieT.x,zzlynx}@outlook.com {dyxiong,gdzhou}@suda.edu.cn Abstract Document-level machine translation (MT) re- mains challenging due to the difficulty in effi- ciently using document context for translation. In this paper, we propose a hierarchical model to learn the global context for document- level neural machine translation (NMT). This is done through a sentence encoder to cap- ture intra-sentence dependencies and a docu- ment encoder to model document-level inter- sentence consistency and coherence. With this hierarchical architecture, we feedback the ex- tracted global document context to each word in a top-down fashion to distinguish different translations of a word according to its specific surrounding context. In addition, since large- scale in-domain document-level parallel cor- pora are usually unavailable, we use a two- step training strategy to take advantage of a large-scale corpus with out-of-domain parallel sentence pairs and a small-scale corpus with in-domain parallel document pairs to achieve the domain adaptability. Experimental re- sults on several benchmark corpora show that our proposed model can significantly improve document-level translation performance over several strong NMT baselines. 1 Introduction Due to its flexibility and much less demand of manual efforts for feature engineering, neural ma- chine translation (NMT) has achieved remarkable progress and become the de-facto standard choice in machine translation. During the last few years, a variety of NMT models have been proposed to reduce the quality gap between human trans- lation and machine translation (Sutskever et al., 2014; Bahdanau et al., 2015; Gehring et al., 2017; Vaswani et al., 2017). Among them, the Trans- former model (Vaswani et al., 2017) has achieved the state-of-the-art performance in sentence-level ∗ Corresponding author. � � ? � � !"# !"$ ! !%$ !%& '()***+,-.)/. 0 Figure 1: An illustration of document-level translation under the guidance of context. translation and results on news benchmark test sets have shown its “translation quality at human par- ity when compared to professional human transla- tors” (Hassan et al., 2018). However, when turn- ing to document-level translation, even the Trans- former model yields a low performance as it trans- lates each sentence in the document independently and suffers from the problem of ignoring docu- ment context. To address above challenge, various extraction- based methods (Maruf and Haffari, 2018; Wang et al., 2017; Zhang et al., 2018; Miculicich et al., 2018) have been proposed to extract previous con- text (pre-context) to guide the translation of the current sentence si, as shown in Figure 1. How- ever, when there exists a huge gap between the pre-context and the context after the current sen- tence si, the guidance from pre-context is not suf- ficient for the NMT model to fully disambiguate the sentence si. …

Content truncated to 500 words