Exploiting Attention to Reveal Shortcomings in Memory Models

Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 378–380 Brussels, Belgium, November 1, 2018. c⃝2018 Association for Computational Linguistics 378 Exploiting Attention to Reveal Shortcomings in Memory Models Kaylee Burns UC Berkeley kayleeburns@berkeley.edu Aida Nematzadeh DeepMind nematzadeh@google.com Erin Grant UC Berkeley eringrant@berkeley.edu Alison Gopnik UC Berkeley gopnik@berkeley.edu Thomas L. Griffiths Princeton University tomg@princeton.edu Abstract The decision making processes of deep net- works are difficult to understand and while their accuracy often improves with increased architectural complexity, so too does their opacity. Practical use of machine learning models, especially for question and answering applications, demands a system that is inter- pretable. We analyze the attention of a mem- ory network model to reconcile contradic- tory performance on a challenging question- answering dataset that is inspired by theory- of-mind experiments. We equate success on questions to task classification, which explains not only test-time failures but also how well the model generalizes to new training condi- tions. 1 Reasoning about Beliefs Possessing a capacity similar to human reasoning has been argued to be necessary for the success of artificial intelligence systems (e.g., Levesque et al., 2011). One well-studied domain that re- quires reasoning is question answering, where simply memorizing and looking up information is often not enough to correctly answer a question. Recent research has focused on developing neural models that succeed in such scenarios (Sukhbaatar et al., 2015; Henaff et al., 2017). As a benchmark to evaluate these models, Weston et al. (2016) released a dataset – Facebook bAbi – that provides a set of toy tasks, each examining a specific type of reasoning. However, the bAbi tasks are already too simple for the current mod- els, which fail at only one or two (out of 20) tasks (Rae et al., 2016; Santoro et al., 2017). Considering humans’ reasoning abilities can provide inspiration for more complex tasks. Peo- ple reason not just about their own observations and beliefs but also about others’ mental states (such as beliefs and intentions). The capacity to recognize that others can have mental states dif- ferent than one’s own – theory of mind – marks an important milestone in the development of chil- dren and has been extensively studied by psychol- ogists (for a review, see Flavell, 2004). Recently, Nematzadeh et al. (2018) released a dataset in- spired by the theory-of-mind experiments from Baron-Cohen et al. (1985). The dataset is based on three tasks designed to capture increasingly com- plex theory-of-mind reasoning: true-, false-, and second-order false-belief tasks. Examples of each task type are given in Figure 1. In the true-belief task, Sally observes the world and as a result she has a first-order true-belief about the location of the milk – her belief matches reality. In the false- belief task, Sally’s first-order belief differs from reality (i.e., she has a false-belief) because she was absent when the state of the world changed. In the second-order false-belief task, Sally observes the new location of the milk; thus, she has a …

Content truncated to 500 words