Gated End-to-End Memory Networks

Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 1–10, Valencia, Spain, April 3-7, 2017. c⃝2017 Association for Computational Linguistics Gated End-to-End Memory Networks Fei Liu ∗ † The University of Melbourne Victoria, Australia fliu3@student.unimelb.edu.au Julien Perez † Xerox Research Centre Europe Grenoble, France julien.perez@xrce.xerox.com Abstract Machine reading using differentiable rea- soning models has recently shown re- markable progress. In this context, End-to-End trainable Memory Networks (MemN2N) have demonstrated promising performance on simple natural language based reasoning tasks such as factual rea- soning and basic deduction. However, other tasks, namely multi-fact question- answering, positional reasoning or dialog related tasks, remain challenging particu- larly due to the necessity of more com- plex interactions between the memory and controller modules composing this family of models. In this paper, we introduce a novel end-to-end memory access regu- lation mechanism inspired by the current progress on the connection short-cutting principle in the field of computer vision. Concretely, we develop a Gated End-to- End trainable Memory Network architec- ture (GMemN2N). From the machine learn- ing perspective, this new capability is learned in an end-to-end fashion without the use of any additional supervision sig- nal which is, as far as our knowledge goes, the first of its kind. Our experi- ments show significant improvements on the most challenging tasks in the 20 bAbI dataset, without the use of any domain knowledge. Then, we show improvements on the Dialog bAbI tasks including the real human-bot conversion-based Di- alog State Tracking Challenge (DSTC-2) dataset. On these two datasets, our model sets the new state of the art. ∗Work carried out as an intern at XRCE †Equal contribution 1 Introduction Deeper Neural Network models are more diffi- cult to train and recurrency tends to complex- ify this optimization problem (Srivastava et al., 2015b). While Deep Neural Network architec- tures have shown superior performance in numer- ous areas, such as image, speech recognition and more recently text, the complexity of optimiz- ing such large and non-convex parameter sets re- mains a challenge. Indeed, the so-called vanish- ing/exploding gradient problem has been mainly addressed using: 1. algorithmic responses, e.g., normalized initialization strategies (LeCun et al., 1998; Glorot and Bengio, 2010); 2. architec- tural ones, e.g., intermediate normalization layers which facilitate the convergence of networks com- posed of tens of hidden layers (He et al., 2015; Saxe et al., 2014). Another problem of memory- enhanced neural models is the necessity of regulat- ing memory access at the controller level. Mem- ory access operations can be supervised (Kumar et al., 2016) and the number of times they are per- formed tends to be fixed apriori (Sukhbaatar et al., 2015), a design choice which tends to be based on the presumed degree of difficulty of the task in question. Inspired by the recent success of object recognition in the field of computer vision (Srivas- tava et al., 2015a; Srivastava et al., 2015b), we in- vestigate the use of a gating mechanism in the …

Content truncated to 500 words