Context Gates for Neural Machine Translation

Context Gates for Neural Machine Translation Zhaopeng Tu† Yang Liu‡ Zhengdong Lu† Xiaohua Liu† Hang Li† †Noah’s Ark Lab, Huawei Technologies, Hong Kong {tu.zhaopeng,lu.zhengdong,liuxiaohua3,hangli.hl}@huawei.com ‡Department of Computer Science and Technology, Tsinghua University, Beijing liuyang2011@tsinghua.edu.cn Abstract In neural machine translation (NMT), genera- tion of a target word depends on both source and target contexts. We find that source con- texts have a direct impact on the adequacy of a translation while target contexts affect the flu- ency. Intuitively, generation of a content word should rely more on the source context and generation of a functional word should rely more on the target context. Due to the lack of effective control over the influence from source and target contexts, conventional NMT tends to yield fluent but inadequate transla- tions. To address this problem, we propose context gates which dynamically control the ratios at which source and target contexts con- tribute to the generation of target words. In this way, we can enhance both the adequacy and fluency of NMT with more careful con- trol of the information flow from contexts. Experiments show that our approach signif- icantly improves upon a standard attention- based NMT system by +2.3 BLEU points. 1 Introduction Neural machine translation (NMT) (Kalchbrenner and Blunsom, 2013; Sutskever et al., 2014; Bah- danau et al., 2015) has made significant progress in the past several years. Its goal is to construct and utilize a single large neural network to accom- plish the entire translation task. One great advan- tage of NMT is that the translation system can be completely constructed by learning from data with- out human involvement (cf., feature engineering in statistical machine translation (SMT)). The encoder- decoder architecture is widely employed (Cho et al., input j¯ınni´an qi´an liˇang yu`e guˇangd¯ong g¯aox¯ın j`ısh`u chˇanpˇın ch¯ukˇou 37.6y`ı mˇeiyu´an NMT in the first two months of this year , the export of new high level technology product was UNK - billion us dollars ▽src china ’s guangdong hi - tech exports hit 58 billion dollars ▽tgt china ’s export of high and new hi - tech exports of the export of the export of the export of the export of the export of the export of the export of the export of · · · Table 1: Source and target contexts are highly cor- related to translation adequacy and fluency, respec- tively. ▽src and ▽tgt denote halving the contribu- tions from the source and target contexts when gen- erating the translation, respectively. 2014; Sutskever et al., 2014), in which the encoder summarizes the source sentence into a vector repre- sentation, and the decoder generates the target sen- tence word-by-word from the vector representation. The representation of the source sentence and the representation of the partially generated target sen- tence (translation) at each position are referred to as source context and target context, respectively. The generation of a target word is determined jointly by the source context and target context. Several techniques in NMT have proven to be very effective, including gating (Hochreiter …

Content truncated to 500 words