Memory-enhanced Decoder for Neural Machine Translation

Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 278–286, Austin, Texas, November 1-5, 2016. c⃝2016 Association for Computational Linguistics Memory-enhanced Decoder for Neural Machine Translation Mingxuan Wang1 Zhengdong Lu2 Hang Li2 Qun Liu3,1 1Key Laboratory of Intelligent Information Processing, Institute of Computing Technology, Chinese Academy of Sciences {wangmingxuan,liuqun}@ict.ac.cn 2Noah’s Ark Lab, Huawei Technologies {Lu.Zhengdong,HangLi.HL}@huawei.com 3ADAPT Centre, School of Computing, Dublin City University Abstract We propose to enhance the RNN decoder in a neural machine translator (NMT) with external memory, as a natural but power- ful extension to the state in the decoding RNN. This memory-enhanced RNN de- coder is called MEMDEC. At each time during decoding, MEMDEC will read from this memory and write to this memory once, both with content-based addressing. Unlike the unbounded memory in previ- ous work(Bahdanau et al., 2014) to store the representation of source sentence, the memory in MEMDEC is a matrix with pre- determined size designed to better cap- ture the information important for the de- coding process at each time step. Our empirical study on Chinese-English trans- lation shows that it can improve by 4.8 BLEU upon Groundhog and 5.3 BLEU upon on Moses, yielding the best perfor- mance achieved with the same training set. 1 Introduction The introduction of external memory has greatly expanded the representational capability of neu- ral network-based model on modeling se- quences(Graves et al., 2014), by providing flex- ible ways of storing and accessing information. More specifically, in neural machine translation, one great improvement came from using an array of vectors to represent the source in a sentence- level memory and dynamically accessing relevant segments of them (alignment) (Bahdanau et al., 2014) through content-based addressing (Graves et al., 2014). The success of RNNsearch demon- strated the advantage of saving the entire sen- tence of arbitrary length in an unbounded mem- ory for operations of next stage (e.g., decoding). In this paper, we show that an external memory can be used to facilitate the decoding/generation process thorough a memory-enhanced RNN de- coder, called MEMDEC. The memory in MEMDEC is a direct extension to the state in the decoding, therefore functionally closer to the memory cell in LSTM(Hochreiter and Schmid- huber, 1997). It takes the form of a matrix with pre-determined size, each column (“a memory cell”) can be accessed by the decoding RNN with content-based addressing for both reading and writing during the decoding process. This mem- ory is designed to provide a more flexible way to select, represent and synthesize the informa- tion of source sentence and previously generated words of target relevant to the decoding. This is in contrast to the set of hidden states of the entire source sentence (which can viewed as an- other form of memory) in (Bahdanau et al., 2014) for attentive read, but can be combined with it to greatly improve the performance of neural ma- chine translator. We apply our model on English- Chinese translation tasks, achieving performance superior to any published results, SMT or NMT, …

Content truncated to 500 words