Exploiting Sentential Context for Neural Machine Translation

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6197–6203 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 6197 Exploiting Sentential Context for Neural Machine Translation Xing Wang Tencent AI Lab brightxwang@tencent.com Zhaopeng Tu Tencent AI Lab zptu@tencent.com Longyue Wang Tencent AI Lab vinnylywang@tencent.com Shuming Shi Tencent AI Lab shumingshi@tencent.com Abstract In this work, we present novel approaches to exploit sentential context for neural machine translation (NMT). Specifically, we first show that a shallow sentential context extracted from the top encoder layer only, can improve translation performance via contextualizing the encoding representations of individual words. Next, we introduce a deep sentential context, which aggregates the sentential context representations from all the internal layers of the encoder to form a more compre- hensive context representation. Experimental results on the WMT14 English⇒German and English⇒French benchmarks show that our model consistently improves per- formance over the strong TRANSFORMER model (Vaswani et al., 2017), demonstrating the necessity and effectiveness of exploiting sentential context for NMT. 1 Introduction Sentential context, which involves deep syntactic and semantic structure of the source and target lan- guages (Nida, 1969), is crucial for machine trans- lation. In statistical machine translation (SMT), the sentential context has proven beneficial for predicting local translations (Meng et al., 2015; Zhang et al., 2015). The exploitation of sentential context in neural machine translation (NMT, Bah- danau et al., 2015), however, is not well studied. Recently, Lin et al. (2018) showed that the trans- lation at each time step should be conditioned on the whole target-side context. They introduced a deconvolution-based decoder to provide the global information from the target-side context for guid- ance of decoding. In this work, we propose simple yet effec- tive approaches to exploiting source-side global sentence-level context for NMT models. We use encoder representations to represent the source- side context, which are summarized into a senten- tial context vector. The source-side context vector is fed to the decoder, so that translation at each step is conditioned on the whole source-side con- text. Specifically, we propose two types of senten- tial context: 1) the shallow one that only exploits the top encoder layer, and 2) the deep one that ag- gregates the sentence representations of all the en- coder layers. The deep sentential context can be viewed as a more comprehensive global sentence representation, since different types of syntax and semantic information are encoded in different en- coder layers (Shi et al., 2016; Peters et al., 2018; Raganato and Tiedemann, 2018). We validate our approaches on top of the state- of-the-art TRANSFORMER model (Vaswani et al., 2017). Experimental results on the benchmarks WMT14 English⇒German and English⇒French translation tasks show that exploiting sentential context consistently improves translation perfor- mance across language pairs. Among the model variations, the deep strategies consistently outper- form their shallow counterparts, which confirms our claim. Linguistic analyses (Conneau et al., 2018) on the learned representations reveal that the proposed approach indeed provides richer linguis- tic information. The contributions of …

Content truncated to 500 words