Improving End-to-End Memory Networks with Unified Weight Tying

Improving End-to-End Memory Networks with Unified Weight Tying Fei Liu Trevor Cohn Timothy Baldwin School of Computing and Information Systems The University of Melbourne Victoria, Australia fliu3@student.unimelb.edu.au t.cohn@unimelb.edu.au tb@ldwin.net Abstract Answering questions while reasoning over multiple supporting facts has long been a goal of artificial intelligence. Re- cently, remarkable advances have been made, focusing on reasoning over natu- ral language-based stories. In particu- lar, end-to-end memory networks (N2N), have achieved state-of-the-art results over such tasks. However, N2Ns are limited by the necessity to choose between two weight tying schemes, neither of which performs consistently well over all tasks. We propose a unified model generalis- ing weight tying and in doing so, make the model more expressive. The pro- posed model achieves uniformly high per- formance, improving on the best results for memory network-based models on the bAbI dataset, and competitive results on Dialog bAbI. 1 Introduction Deep neural network models have demonstrated strong performance on a number of challenging tasks, such as image classification (He et al., 2016), speech recognition (Graves et al., 2013), and various natural language processing tasks (Bahdanau et al., 2014; Kim, 2014; Xiong et al., 2016). Recently, the augmentation of neural net- works with external memory components has been shown to be a powerful means of capturing context of different types (Graves et al., 2014, 2016; Rae et al., 2016). Of particular interest to this work is the work by Sukhbaatar et al. (2015), on end-to- end memory networks (N2Ns), which exhibit re- markable reasoning capabilities, e.g. for reasoning (Weston et al., 2016) and goal-oriented dialogue tasks (Bordes and Weston, 2016). Typically, such tasks consist of three key components: a sequence of supporting facts (the story), a question, and its answer. An example task is given in Figure 1. Given the first two as input, it is the model’s job to reason over the supporting facts and predict the answer to the question. One drawback of N2Ns is the problem of choos- ing between two types of weight tying (adja- cent and layer-wise; see Section 2 for a techni- cal description). While N2Ns generally work well with either weight tying approach, as reported in Sukhbaatar et al. (2015), the performance is un- even on some difficult tasks. That is, for some tasks, one weight tying approach attains near- perfect accuracy and the other performs poorly, but for other tasks, this trend is reversed. In this paper, focusing on improving N2N, we propose a unified model, UN2N, capable of dy- namically determining the appropriate type of weight tying for a given task. This is realised through the use of a gating vector, inspired by Liu and Perez (2017). Our method achieves the best performance for a memory network-based model on the bAbI dataset, superior to both adjacent and layer-wise weight tying, and competitive results on Dialog bAbI. The paper is organised as follows: after we re- view N2N and related reasoning models in Sec- tion 2, we describe our motivation and detail the elements of our proposed …

Content truncated to 500 words