Exploiting Cross-Sentence Context for Neural Machine Translation

Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2826–2831 Copenhagen, Denmark, September 7–11, 2017. c⃝2017 Association for Computational Linguistics Exploiting Cross-Sentence Context for Neural Machine Translation Longyue Wang† Zhaopeng Tu‡∗ Andy Way† Qun Liu† †ADAPT Centre, School of Computing, Dublin City University, Ireland {longyue.wang, andy.way, qun.liu}@adaptcentre.ie ‡Tencent AI Lab, China tuzhaopeng@gmail.com Abstract In translation, considering the document as a whole can help to resolve ambiguities and inconsistencies. In this paper, we pro- pose a cross-sentence context-aware ap- proach and investigate the influence of his- torical contextual information on the per- formance of neural machine translation (NMT). First, this history is summarized in a hierarchical way. We then integrate the historical representation into NMT in two strategies: 1) a warm-start of en- coder and decoder states, and 2) an aux- iliary context source for updating decoder states. Experimental results on a large Chinese-English translation task show that our approach significantly improves upon a strong attention-based NMT system by up to +2.1 BLEU points. 1 Introduction Neural machine translation (NMT) has been rapidly developed in recent years (Kalchbrenner and Blunsom, 2013; Sutskever et al., 2014; Bah- danau et al., 2015; Tu et al., 2016). The encoder- decoder architecture is widely employed, in which the encoder summarizes the source sentence into a vector representation, and the decoder generates the target sentence word by word from the vector representation. Using the encoder-decoder frame- work as well as gating and attention techniques, it has been shown that the performance of NMT has surpassed the performance of traditional sta- tistical machine translation (SMT) on various lan- guage pairs (Luong et al., 2015). The continuous vector representation of a sym- bol encodes multiple dimensions of similarity, equivalent to encoding more than one meaning of ∗Corresponding Author: Zhaopeng Tu a word. Consequently, NMT needs to spend a substantial amount of its capacity in disambiguat- ing source and target words based on the context defined by a source sentence (Choi et al., 2016). Consistency is another critical issue in document- level translation, where a repeated term should keep the same translation throughout the whole document (Xiao et al., 2011; Carpuat and Simard, 2012). Nevertheless, current NMT models still process a documents by translating each sentence alone, suffering from inconsistency and ambigu- ity arising from a single source sentence. These problems are difficult to alleviate using only lim- ited intra-sentence context. The cross-sentence context, or global context, has proven helpful to better capture the meaning or intention in sequential tasks such as query sug- gestion (Sordoni et al., 2015) and dialogue model- ing (Vinyals and Le, 2015; Serban et al., 2016). The leverage of global context for NMT, how- ever, has received relatively little attention from the research community.1 In this paper, we pro- pose a cross-sentence context-aware NMT model, which considers the influence of previous source sentences in the same document.2 Specifically, we employ a hierarchy of Recur- rent Neural Networks (RNNs) to summarize the cross-sentence context from source-side previous sentences, which deploys an …

Content truncated to 500 words