Dynamic Past and Future for Neural Machine Translation

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 931–941, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 931 Dynamic Past and Future for Neural Machine Translation Zaixiang Zheng Nanjing University zhengzx@smail.nju.edu.cn Shujian Huang Nanjing University huangsj@nju.edu.cn Zhaopeng Tu Tencent AI Lab zptu@tencent.com Xin-Yu Dai Nanjing University dxy@nju.edu.cn Jiajun Chen Nanjing University chenjj@nju.edu.cn Abstract Previous studies have shown that neural ma- chine translation (NMT) models can bene- fit from explicitly modeling translated (PAST) and untranslated (FUTURE) source contents as recurrent states (Zheng et al., 2018). How- ever, this less interpretable recurrent process hinders its power to model the dynamic up- dating of PAST and FUTURE contents during decoding. In this paper, we propose to model the dynamic principles by explicitly separating source words into groups of translated and un- translated contents through parts-to-wholes as- signment. The assignment is learned through a novel variant of routing-by-agreement mecha- nism (Sabour et al., 2017), namely Guided Dy- namic Routing, where the translating status at each decoding step guides the routing process to assign each source word to its associated group (i.e., translated or untranslated content) represented by a capsule, enabling translation to be made from holistic context. Experiments show that our approach achieves substantial improvements over both RNMT and Trans- former by producing more adequate transla- tions. Extensive analysis demonstrates that our method is highly interpretable, which is able to recognize the translated and untranslated con- tents as expected.1 1 Introduction Neural machine translation (NMT) generally adopts an attentive encoder-decoder frame- work (Sutskever et al., 2014; Vaswani et al., 2017), where the encoder maps a source sentence into a sequence of contextual representations (source contents), and the decoder generates a target sen- tence word-by-word based on part of the source content assigned by an attention model (Bahdanau 1Codes are released at https://github.com/ zhengzx-nlp/dynamic-nmt. et al., 2015). Like human translators, NMT sys- tems should have the ability to know the rel- evant source-side context for the current word (PRESENT), as well as recognize what parts in the source contents have been translated (PAST) and what parts have not (FUTURE), at each decoding step. Accordingly, the PAST, PRESENT and FU- TURE are three dynamically changing states dur- ing the whole translation process. Previous studies have shown that NMT models are likely to face the illness of inadequate transla- tion (Kong et al., 2019), which is usually embod- ied in over- and under-translation problems (Tu et al., 2016, 2017). This issue may be attributed to the poor ability of NMT of recognizing the dy- namic translated and untranslated contents. To remedy this, Zheng et al. (2018) first demonstrate that explicitly tracking PAST and FUTURE con- tents helps NMT models alleviate this issue and generate better translation. In their work, the run- ning PAST and FUTURE contents are modeled as recurrent states. However, the recurrent process is still non-trivial to determine which parts of the source words are the PAST …

Content truncated to 500 words