Memory-Augmented Neural Networks for Machine Translation

Memory-Augmented Neural Networks for Machine Translation Mark Collier Trinity College Dublin School of Computer Science and Statistics, Dublin, Ireland mcollier@tcd.ie Joeran Beel Trinity College Dublin School of Computer Science and Statistics, ADAPT Centre, Dublin, Ireland joeran.beel@tcd.ie Abstract Memory-augmented neural networks (MANNs) have been shown to outper- form other recurrent neural network architectures on a series of artificial sequence learning tasks, yet they have had limited application to real-world tasks. We evaluate direct application of Neural Turing Machines (NTM) and Differentiable Neural Computers (DNC) to machine translation. We further propose and evaluate two models which extend the attentional encoder-decoder with capabili- ties inspired by memory augmented neural networks. We evaluate our proposed models on IWSLT Vietnamese→English and ACL Romanian→English datasets. Our proposed models and the memory augmented neural networks perform sim- ilarly to the attentional encoder-decoder on the Vietnamese→English translation task while have a 0.3-1.9 lower BLEU score for the Romanian→English task. Interestingly, our analysis shows that despite being equipped with additional flexibility and being randomly initialized memory augmented neural networks learn an algorithm for machine transla- tion almost identical to the attentional encoder-decoder. 1 Introduction Memory-Augmented Neural Networks (MANN) are a new class of recurrent neural network (RNN) c⃝ 2019 The authors. This article is licensed under a Creative Commons 4.0 licence, no derivative works, attribution, CC- BY-ND. that separate computation from memory. The key distinction between MANNs and other RNNs such as Long Short-Term Memory cells (LSTM) (Hochreiter and Schmidhuber, 1997) is the exis- tence of an external memory unit. A controller net- work in the MANN receives input, interacts with the external memory unit via read and write heads and produces output. MANNs have been shown to learn faster and generalize better than LSTMs on a range of artificial sequential learning tasks (Graves et al., 2014; Graves et al., 2016; Sukhbaatar et al., 2015). Despite their success on artificial tasks, LSTM based models remain the preferred choice for many commercially important sequence learn- ing tasks such as handwriting recognition (Graves et al., 2009), machine translation (Wu et al., 2016) and speech recognition (Graves and Jaitly, 2014). Attentional encoder-decoders (Bahdanau et al., 2014; Luong et al., 2015), where the encoder and decoder are often LSTMs or other gated RNNs such as the Gated Recurrent Unit (Cho et al., 2014b), are a class of neural network models that have achieved state-of-the-art performance on many language pairs for machine translation (Lu- ong and Manning, 2015; Sennrich et al., 2016a). An encoder RNN reads the source sentence one to- ken at a time. The encoder both maintains an inter- nal vector representing the full source sentence and it encodes each token in the source sentence into a vector often assumed to represent the meaning of that token in its surrounding context. The decoder receives the internal vector from the encoder and can read from the encoded source sentence when producing the target sentence. Attentional encoder-decoders can be seen as a basic form of MANN. The collection of vectors representing the encoded source …

Content truncated to 500 words