Synchronous Bidirectional Neural Machine Translation

Synchronous Bidirectional Neural Machine Translation Long Zhou1,2, Jiajun Zhang1,2∗, Chengqing Zong1,2,3 1National Laboratory of Pattern Recognition, CASIA, Beijing, China 2University of Chinese Academy of Sciences, Beijing, China 3CAS Center for Excellence in Brain Science and Intelligence Technology, Shanghai, China {long.zhou, jjzhang, cqzong}@nlpr.ia.ac.cn Abstract Existing approaches to neural machine trans- lation (NMT) generate the target language sequence token-by-token from left to right. However, this kind of unidirectional decod- ing framework cannot make full use of the target-side future contexts which can be pro- duced in a right-to-left decoding direction, and thus suffers from the issue of unbal- anced outputs. In this paper, we introduce a synchronous bidirectional–neural machine translation (SB-NMT) that predicts its outputs using left-to-right and right-to-left decoding simultaneously and interactively, in order to leverage both of the history and future in- formation at the same time. Specifically, we first propose a new algorithm that enables syn- chronous bidirectional decoding in a single model. Then, we present an interactive decod- ing model in which left-to-right (right-to-left) generation does not only depend on its pre- viously generated outputs, but also relies on future contexts predicted by right-to-left (left- to-right) decoding. We extensively evaluate the proposed SB-NMT model on large-scale NIST Chinese-English, WMT14 English- German, and WMT18 Russian-English trans- lation tasks. Experimental results demonstrate that our model achieves significant improve- ments over the strong Transformer model by 3.92, 1.49, and 1.04 BLEU points, respec- tively, and obtains the state-of-the-art per- ∗Corresponding author. formance on Chinese-English and English- German translation tasks.1 1 Introduction Neural machine translation has significantly im- proved the quality of machine translation in recent years (Sutskever et al., 2014; Bahdanau et al., 2015; Zhang and Zong, 2015; Wu et al., 2016; Gehring et al., 2017; Vaswani et al., 2017). Recent approaches to sequence-to-sequence learning typ- ically leverage recurrence (Sutskever et al., 2014), convolution (Gehring et al., 2017), or attention (Vaswani et al., 2017) as basic building blocks. Typically, NMT adopts the encoder-decoder architecture and generates the target translation from left to right. Despite their remarkable suc- cess, NMT models suffer from several weak- nesses (Koehn and Knowles, 2017). One of the most prominent issues is the problem of unbal- anced outputs in which the translation prefixes are better predicted than the suffixes (Liu et al., 2016). We analyze translation accuracy of the first and last 4 tokens for left-to-right (L2R) and right-to- left (R2L) directions, respectively. As shown in Table 1, the statistical results show that L2R per- forms better in the first 4 tokens, whereas R2L translates better in terms of the last 4 tokens. This problem is mainly caused by the left-to- right unidirectional decoding, which conditions each output word on previously generated out- puts only, but leaving the future information from target-side contexts unexploited during transla- tion. The future context is commonly used in reading and writing in human cognitive process 1The source code is available at https://github. com/wszlong/sb-nmt. 91 Transactions of the Association for Computational Linguistics, vol. 7, pp. 91−105, 2019. Action Editor: George Foster. Submission …

Content truncated to 500 words