Refining Source Representations with Relation Networks for Neural Machine Translation

Proceedings of the 27th International Conference on Computational Linguistics, pages 1292–1303 Santa Fe, New Mexico, USA, August 20-26, 2018. 1292 Refining Source Representations with Relation Networks for Neural Machine Translation Wen Zhang1,2 Jiawei Hu1,2 Yang Feng1,2∗ Qun Liu3,1 1Key Laboratory of Intelligent Information Processing, Institute of Computing Technology, CAS 2University of Chinese Academy of Sciences {zhangwen,hujiawei,fengyang}@ict.ac.cn 3ADAPT Centre, School of Computing, Dublin City University qun.liu@dcu.ie Abstract Although neural machine translation with the encoder-decoder framework has achieved great success recently, it still suffers drawbacks of forgetting distant information, which is an inherent disadvantage of recurrent neural network structure, and disregarding relationship between source words during encoding step. Whereas in practice, the former information and relationship are often useful in current step. We target on solving these problems and thus introduce relation networks to learn better representations of the source. The relation networks are able to facili- tate memorization capability of recurrent neural network via associating source words with each other, this would also help retain their relationships. Then the source representations and all the relations are fed into the attention component together while decoding, with the main encoder- decoder framework unchanged. Experiments on several datasets show that our method can im- prove the translation performance significantly over the conventional encoder-decoder model and even outperform the approach involving supervised syntactic knowledge. 1 Introduction In recent years, Neural Machine Translation (NMT) (Kalchbrenner and Blunsom, 2013; Sutskever et al., 2014; Bahdanau et al., 2015) has achieved great success in some language pairs, rivalling the state-of- the-art Statistical Machine Translation (SMT). The Recurrent Neural Network (RNN) encoder-decoder architecture is widely used framework for NMT, the principle behind which is that: encoding the mean- ing of the input bidirectionally into a concept space via RNNs and decoding into target words with RNNs based on this encoding (Sutskever et al., 2014; Bahdanau et al., 2015). This means that encoding prin- ciple leads to a deeper understanding and learning of the translation rules, and hence better translation than conventional SMT that considers only surface forms, e.g., words and phrases. The RNNs with gating, such as Gated Recurrent Unit (GRU) (Cho et al., 2014) or Long Short-Term Memory (LSTM) (Hochreiter and Schmidhuber, 1997), are designed to memorize useful history infor- mation and meanwhile forget irrelative information. Together with attention technique which makes the decoding process only focus on the most related source words, the RNN encoder-decoder framework is expected to be able to handle long sequences and consider the globally related information. However, the practical situation is that RNNs tend to forget old history information, especially the far older one. Sometimes the older information is indispensable for generating proper translation, e.g., for the source sentence “take the heavy box away”, when translating “away”, “take” should be considered together. In addition, it has been proven that using phrases rather than words in SMT (Koehn et al., 2003) brings performance improvement, while in NMT the attention is only modeled in the unit of words. In the same sense, improvement is expected …

Content truncated to 500 words