Careful Selection of Knowledge to Solve Open Book Question Answering

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6120–6129 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 6120 Careful Selection of Knowledge to solve Open Book Question Answering Pratyay Banerjee∗and Kuntal Kumar Pal∗and Arindam Mitra∗and Chitta Baral Department of Computer Science, Arizona State University pbanerj6,kkpal,amitra7,chitta@asu.edu Abstract Open book question answering is a type of nat- ural language based QA (NLQA) where ques- tions are expected to be answered with respect to a given set of open book facts, and common knowledge about a topic. Recently a challenge involving such QA, OpenBookQA, has been proposed. Unlike most other NLQA tasks that focus on linguistic understanding, Open- BookQA requires deeper reasoning involving linguistic understanding as well as reasoning with common knowledge. In this paper we address QA with respect to the OpenBookQA dataset and combine state of the art language models with abductive information retrieval (IR), information gain based re-ranking, pas- sage selection and weighted scoring to achieve 72.0% accuracy, an 11.6% improvement over the current state of the art. 1 Introduction Natural language based question answering (NLQA) not only involves linguistic understand- ing, but often involves reasoning with various kinds of knowledge. In recent years, many NLQA datasets and challenges have been proposed, for example, SQuAD (Rajpurkar et al., 2016), Trivi- aQA (Joshi et al., 2017) and MultiRC (Khashabi et al., 2018), and each of them have their own focus, sometimes by design and other times by virtue of their development methodology. Many of these datasets and challenges try to mimic human question answering settings. One such setting is open book question answering where humans are asked to answer questions in a setup where they can refer to books and other materials related to their questions. In such a setting, the focus is not on memorization but, as mentioned in Mihaylov et al. (2018), on “deeper understanding of the materials and its application ∗ These authors contributed equally to this work. to new situations (Jenkins, 1995; Landsberger, 1996).” In Mihaylov et al. (2018), they propose the OpenBookQA dataset mimicking this setting. Question: A tool used to identify the percent chance of a trait being passed down has how many squares ? (A) Two squares (B) Four squares (C) Six squares (D) Eight squares Extracted from OpenBook: a punnett square is used to identify the percent chance of a trait being passed down from a parent to its offspring. Retrieved Missing Knowledge: Two squares is four. The Punnett square is made up of 4 squares and 2 of them are blue and 2 of them are brown, this means you have a 50% chance of having blue or brown eyes. Table 1: An example of distracting retrieved knowl- edge The OpenBookQA dataset has a collection of questions and four answer choices for each ques- tion. The dataset comes with 1326 facts represent- ing an open book. It is expected that answering each question requires at least one of these facts. In addition …

Content truncated to 500 words