Agreement on Target-bidirectional Neural Machine Translation

Proceedings of NAACL-HLT 2016, pages 411–416, San Diego, California, June 12-17, 2016. c⃝2016 Association for Computational Linguistics Agreement on Target-bidirectional Neural Machine Translation Lemao Liu, Masao Utiyama, Andrew Finch, Eiichiro Sumita National Institute of Information and Communications Technology (NICT) 3-5 Hikari-dai, Seika-cho, Soraku-gun, Kyoto, Japan {lmliu,first.last}@nict.go.jp Abstract Neural machine translation (NMT) with re- current neural networks, has proven to be an effective technique for end-to-end machine translation. However, in spite of its promis- ing advances over traditional translation meth- ods, it typically suffers from an issue of unbal- anced outputs, that arise from both the nature of recurrent neural networks themselves, and the challenges inherent in machine translation. To overcome this issue, we propose an agree- ment model for neural machine translation and show its effectiveness on large-scale Japanese- to-English and Chinese-to-English translation tasks. Our results show the model can achieve improvements of up to 1.4 BLEU over the strongest baseline NMT system. With the help of an ensemble technique, this new end-to-end NMT approach finally outperformed phrase- based and hierarchical phrase-based Moses baselines by up to 5.6 BLEU points. 1 Introduction Recurrent neural network (RNN) has achieved great successes on several structured prediction tasks (Graves, 2013; Watanabe and Sumita, 2015; Dyer et al., 2015), in which RNNs are required to make a sequence of dependent predictions. One of its ad- vantages is that an unbounded history is available to enrich the context for the prediction at the current time-step. Despite its successes, recently, (Liu et al., 2016) pointed out that the RNN suffers from a fundamental issue of generating unbalanced outputs: that is to say the suffixes of its outputs are typically worse than the prefixes. This is due to the fact that later predictions directly depend on the accuracy of previous pre- dictions. They empirically demonstrated this issue on two simple sequence-to-sequence learning tasks: machine transliteration and grapheme-to-phoneme conversion. On the more general sequence-to-sequence learn- ing task of machine translation (MT), neural ma- chine translation (NMT) based on RNNs has re- cently become an active research topic (Sutskever et al., 2014; Bahdanau et al., 2014). Compared to those two simple tasks, MT involves in much larger vocabulary and frequent reordering between input and output sequences. This makes the prediction at each time-step far more challenging. In addition, sequences in MT are much longer, with averaged length of 36.7 being about 5 times longer than that in grapheme-to-phoneme conversion. Therefore, we believe that the history is more likely to contain in- correct predictions and the issue of unbalanced out- puts may be more serious. This hypothesis is sup- ported later (see Table 1 in §4.1), by an analysis that shows the quality of the prefixes of translation hy- potheses is much higher than that of the suffixes. To address this issue for NMT, in this paper we extend the agreement model proposed in (Liu et al., 2016) to the task of machine translation. Its key idea is to encourage the agreement between a pair of target-directional (left-to-right and right-to-left) …

Content truncated to 500 words