Deconvolution-Based Global Decoding for Neural Machine Translation

Proceedings of the 27th International Conference on Computational Linguistics, pages 3260–3271 Santa Fe, New Mexico, USA, August 20-26, 2018. 3260 Deconvolution-Based Global Decoding for Neural Machine Translation Junyang Lin1,2, Xu Sun2, Xuancheng Ren2, Shuming Ma2, Jinsong Su3, Qi Su1 1School of Foreign Languages, Peking University 2MOE Key Lab of Computational Linguistics, School of EECS, Peking University 3School of Software, Xiamen University {linjunyang, xusun, renxc, shumingma, sukia}@pku.edu.cn jssu@xmu.edu.cn Abstract A great proportion of sequence-to-sequence (Seq2Seq) models for Neural Machine Translation (NMT) adopt Recurrent Neural Network (RNN) to generate translation word by word following a sequential order. As the studies of linguistics have proved that language is not linear word sequence but sequence of complex structure, translation at each step should be conditioned on the whole target-side context. To tackle the problem, we propose a new NMT model that decodes the sequence with the guidance of its structural prediction of the context of the target sequence. Our model generates translation based on the structural prediction of the target-side context so that the translation can be freed from the bind of sequential order. Experimental results demonstrate that our model is more competitive compared with the state-of-the-art methods, and the analysis reflects that our model is also robust to translating sentences of different lengths and it also reduces repetition with the instruction from the target-side context for decoding. 1 Introduction Deep learning has achieved tremendous success in machine translation, outperforming the traditional linguistic-rule-based and statistical methods. In recent studies of Neural Machine Translation (NMT), most models are based on the sequence-to-sequence (Seq2Seq) model based on the encoder-decoder framework (Kalchbrenner and Blunsom, 2013; Sutskever et al., 2014; Cho et al., 2014) with the attention mechanism (Bahdanau et al., 2014; Luong et al., 2015). While traditional linguistic-rule-based and sta- tistical methods of machine translation require much work of feature engineering, NMT can be trained in the end-to-end fashion. Besides, the attention mechanism can model the alignment relationship between the source text and translation (Bahdanau et al., 2014; Luong et al., 2015), and some recent improved versions of attention have proved successful in this task (Tu et al., 2016; Mi et al., 2016; Meng et al., 2016; Xiong et al., 2017; Vaswani et al., 2017). However, the decoding pattern of the recent Seq2Seq models is inconsistent with the linguistic analy- sis. As the conventional decoder translates words in a sequential order, the current generation is highly dependent on the previous generation and it is short of the knowledge about future generation. Nida (1969) pointed out that translation goes through a process of analysis, transfer and reconstruction, involv- ing the deep syntactic and semantic structure of the source and target languages. Language generation involves complex syntactic analysis and semantic integration, instead of a step-by step word generation (Frazier, 1987). Moreover, from the perspective of semantics and pragmatics, the syntactic analysis of utterance can be guided by the global lexical-semantic and discourse information (Altmann and Steed- man, 1988; Trueswell et al., 1994, 1993; Tyler and Marslen-Wilson, 1977). In brief, …

Content truncated to 500 words