Robust and Scalable Differentiable Neural Computer for Question Answering

Proceedings of the Workshop on Machine Reading for Question Answering, pages 47–59 Melbourne, Australia, July 19, 2018. c⃝2018 Association for Computational Linguistics 47 Robust and Scalable Differentiable Neural Computer for Question Answering J¨org Franke, Jan Niehues, Alex Waibel Institute for Anthropomatics and Robotics Karlsruhe Institute of Technology, Germany joerg.franke@student.kit.edu {jan.niehues,alex.waibel}@kit.edu Abstract Deep learning models are often not eas- ily adaptable to new tasks and require task-specific adjustments. The differen- tiable neural computer (DNC), a memory- augmented neural network, is designed as a general problem solver which can be used in a wide range of tasks. But in re- ality, it is hard to apply this model to new tasks. We analyze the DNC and identify possible improvements within the appli- cation of question answering. This mo- tivates a more robust and scalable DNC (rsDNC). The objective precondition is to keep the general character of this model intact while making its application more reliable and speeding up its required train- ing time. The rsDNC is distinguished by a more robust training, a slim memory unit and a bidirectional architecture. We not only achieve new state-of-the-art perfor- mance on the bAbI task, but also min- imize the performance variance between different initializations. Furthermore, we demonstrate the simplified applicability of the rsDNC to new tasks with passable re- sults on the CNN RC task without adap- tions. 1 Introduction In contrast to traditional statistical models, which often require a large amount of human effort on feature engineering and task-specific adjustments, a promise of deep learning is that little task- specific knowledge and minimal adaption is re- quired to achieve state-of-the-art performance on different tasks. But in reality, many deep learning solutions have to be adapted to a specific task to achieve good performance. However, there are more universal approaches for example the differentiable neural computer (DNC). It is introduced by Graves et al. (2016) as a general artificial neural network (ANN) model with an external memory “to solve complex, struc- tured tasks”. It can be seen as a generic memory- augmentation framework. Unlike a vanilla ANN, it separates computation and memorization with a computational controller and a memory unit, which are independently modifiable. This allows a more accurate model design. Due to its fully dif- ferentiable design, it can be learned in a supervised fashion. The original paper shows applications on the bAbI question answering (QA) task, graph exper- iments and a reinforcement learning block puzzle solver (Graves et al., 2016). But when applying this model to new QA tasks, no satisfying results are achieved. The issues of QA are the huge vo- cabulary, the length of the contexts and the re- quired model complexity to find the correct an- swer. In this work, we analyze the DNC in QA tasks and identify four main challenges: 1. High mem- ory consumption makes it hard to train large mod- els efficiently. 2. The large variance in training performance within different initializations. 3. A slow and unstable convergence requires long vary- ing training times. 4. …

Content truncated to 500 words