Exploiting Pre-Ordering for Neural Machine Translation

Exploiting Pre-Ordering for Neural Machine Translation Yang Zhao, Jiajun Zhang and Chengqing Zong National Laboratory of Pattern Recognition, Institute of Automation, CAS University of Chinese Academy of Sciences {yang.zhao, jjzhang, cqzong}@nlpr.ia.ac.cn Abstract Neural Machine Translation (NMT) has drawn much attention due to its promising translation performance in recent years. However, the under-translation and over-translation problem still remain a big challenge. Through error analysis, we find that under-translation is much more prevalent than over-translation and the source words that need to be reordered during translation are more likely to be ignored. To address the under-translation problem, we explore the pre-ordering approach for NMT. Specifically, we pre-order the source sentences to approximate the target language word order. We then combine the pre-ordering model with position embedding to enhance the monotone translation. Finally, we augment our model with the coverage mechanism to tackle the over-translation problem. Experimental results on Chinese-to-English translation have shown that our method can significantly improve the translation quality by up to 2.43 BLEU points. Furthermore, the detailed analysis demonstrates that our approach can substantially reduce the number of under-translation cases by 30.4% (compared to 17.4% using the coverage model). Keywords: Neural Machine Translation, pre-ordering, under translation, over translation 1. Introduction The Past several years have witnessed a significant progress in Neural Machine Translation (NMT). Most NMT meth- ods are based on the encoder-decoder architecture proposed by (Kalchbrenner and Blunsom, 2013; Cho et al., 2014; Sutskever et al., 2014; Bahdanau et al., 2015) and can achieve promising translation performance in a variety of language pairs (Junczys-Dowmunt et al., 2016; Wu et al., 2016). However, previous studies have showed that NMT suffers from the problems that some source words are mistakenly translated for multiple times meanwhile some words are missed during translation (Tu et al., 2016; Tu et al., 2017; Mi et al., 2016; Feng et al., 2016), which can be called over-translation and under-translation, respectively1. Under-translation Over-translation Times No. Words Times No. Words 92 307 32 48 Table 1: Statistics on the under-translation and the over- translation in NMT. Under-translation Reorder No reorder Sub-sentence 48 26 18 Table 2: Statistics on different kinds of the under- translation. In order to figure out the distribution of over-translation and under-translation in NMT, we analyze 500 sentences trans- lated by the NMT system, which is trained by 2.1M parallel Chinese-English sentences pairs. Table 1 shows the statis- tical results. Specifically, in 500 sentences, NMT system 1 (Mi et al., 2016) calls this phenomenon as ”repeating and dropping translations”. Here, we adopt (Tu et al., 2016)’s expres- sions, i.e. over-translation and under-translation. produces 92 under-translations and 32 over-translations. Besides that, for the under-translation, the total number of missing words is 307, while the number of over-translated words is 48. From these statistics, we can see that the under-translation in NMT is more serious than the over- translation. Therefore, further analysis for the under-translation is made and Table 2 shows the results. In 92 under-translations, we find that the source words should to be reordered …

Content truncated to 500 words