Modeling Coherence for Neural Machine Translation with Dynamic and Topic Caches

Proceedings of the 27th International Conference on Computational Linguistics, pages 596–606 Santa Fe, New Mexico, USA, August 20-26, 2018. 596 Modeling Coherence for Neural Machine Translation with Dynamic and Topic Caches Shaohui Kuang† Deyi Xiong†∗ Weihua Luo†† Guodong Zhou† †School of Computer Science and Technology, Soochow University, Suzhou, China shkuang@stu.suda.edu.cn, {dyxiong, gdzhou}@suda.edu.cn ††Alibaba Group weihua.luowh@alibaba-inc.com Abstract Sentences in a well-formed text are connected to each other via various links to form the cohe- sive structure of the text. Current neural machine translation (NMT) systems translate a text in a conventional sentence-by-sentence fashion, ignoring such cross-sentence links and dependen- cies. This may lead to generate an incoherent target text for a coherent source text. In order to handle this issue, we propose a cache-based approach to modeling coherence for neural machine translation by capturing contextual information either from recently translated sentences or the entire document. Particularly, we explore two types of caches: a dynamic cache, which stores words from the best translation hypotheses of preceding sentences, and a topic cache, which maintains a set of target-side topical words that are semantically related to the document to be translated. On this basis, we build a new layer to score target words in these two caches with a cache-based neural model. Here the estimated probabilities from the cache-based neural model are combined with NMT probabilities into the final word prediction probabilities via a gating mechanism. Finally, the proposed cache-based neural model is trained jointly with NMT sys- tem in an end-to-end manner. Experiments and analysis presented in this paper demonstrate that the proposed cache-based model achieves substantial improvements over several state-of-the-art SMT and NMT baselines. 1 Introduction Neural machine translation (Sutskever et al., 2014; Bahdanau et al., 2015) as an emerging machine translation approach, quickly achieves the state-of-the-art translation performance on many language pairs, e.g., English-French (Jean et al., 2015; Luong et al., 2015b), English-German (Shen et al., 2015; Luong et al., 2015a) and so on. In principle, NMT is established on an encoder-decoder framework, where the encoder reads a source sentence and encodes it into a fixed-length semantic vector, and the decoder generates a translation according to this vector. In spite of its current success, NMT translates sentences of a text independently, ignoring document- level information during translation. This largely limits its success since document-level information imposes constraints on the translations of individual sentences of a text. And such document-level con- straints, at least, can be categorized into three aspects: consistency, disambiguation and coherence. First, the same phrases or terms should be translated consistently across the entire text as much as possible, no matter in which sentence they occur. If sentences of a text are translated independent of each other, it will be difficult to maintain the translation consistency across different sentences. Second, document-level information provides a global context that can help disambiguate words with multiple senses if sentence- level local context is not sufficient for disambiguation. Third, the topic of a document is able to keep individual sentences translated in a …

Content truncated to 500 words