Incorporating Word Reordering Knowledge into Attention-based Neural Machine Translation

Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 1524–1534 Vancouver, Canada, July 30 - August 4, 2017. c⃝2017 Association for Computational Linguistics https://doi.org/10.18653/v1/P17-1140 Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 1524–1534 Vancouver, Canada, July 30 - August 4, 2017. c⃝2017 Association for Computational Linguistics https://doi.org/10.18653/v1/P17-1140 Incorporating Word Reordering Knowledge into Attention-based Neural Machine Translation Jinchao Zhang1 Mingxuan Wang1 Qun Liu3,1 Jie Zhou2 1Key Laboratory of Intelligent Information Processing, Institute of Computing Technology, Chinese Academy of Sciences {zhangjinchao,wangmingxuan,liuqun}@ict.ac.cn 2Baidu Research - Institute of Deep Learning, Baidu Inc.,Beijing,China {zhoujie01}@baidu.com 3ADAPT Centre, School of Computing, Dublin City University Abstract This paper proposes three distor- tion models to explicitly incorporate the word reordering knowledge into attention-based Neural Machine Trans- lation (NMT) for further improving translation performance. Our pro- posed models enable attention mech- anism to attend to source words re- garding both the semantic requirement and the word reordering penalty. Ex- periments on Chinese-English trans- lation show that the approaches can improve word alignment quality and achieve significant translation improve- ments over a basic attention-based N- MT by large margins. Compared with previous works on identical corpora, our system achieves the state-of-the-art performance on translation quality. 1 Introduction Word reordering model is one of the most crucial sub-components in Statistical Machine Translation (SMT) (Brown et al., 1993; Koehn et al., 2003; Chiang, 2005) which provides word reordering knowledge to ensure reason- able translation order of source words. It is separately trained and then incorporated into the SMT framework in a pipeline style. In recent years, end-to-end NMT (Kalch- brenner and Blunsom, 2013; Sutskever et al., 2014; Bahdanau et al., 2015) has made tremen- dous progress (Jean et al., 2015; Luong et al., 2015b; Shen et al., 2016; Sennrich et al., 2016; Tu et al., 2016; Zhou et al., 2016; Johnson et al., 2016). An encoder-decoder framework (Cho et al., 2014b; Sutskever et al., 2014) with attention mechanism (Bahdanau et al., 2015) is widely used, in which an encoder compresses the source sentence, an attention mechanism evaluates related source words and a decoder generates target words. The attention mechanism evaluates the dis- tribution of to-be-translated source words in a content-based addressing fashion (Graves et al., 2014) which tends to attend to the source words regarding the content relation with current translation status. Lack of ex- plicit models to exploit the word reordering knowledge may lead to attention faults and generate fluent but inaccurate or inadequate translations. Table 1 shows a translation in- stance and Figure 1 depicts the corresponding word alignment matrix that produced by the attention mechanism. In this example, even though the word “zuixin (latest)” is a common adjective in Chinese and its following word should be translated soon in Chinese to En- glish translation direction, the word “yiju (ev- idence)” does not obtain appropriate attention which leads to the incorrect translation. src youguan(related) baodao(report) shi(is) zhichi(support) tamen(their) lundian(arguments) de(’s) zuixin(latest) yiju(evidence) . ref the report is the latest evidence that supports …

Content truncated to 500 words