Combining Knowledge Hunting and Neural Language Models to Solve the Winograd Schema Challenge

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6110–6119 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 6110 Combining Knowledge Hunting and Neural Language Models to Solve the Winograd Schema Challenge Ashok Prakash, Arpit Sharma, Arindam Mitra, Chitta Baral Arizona State University Tempe, USA {apraka23,asharm73,amitra7,chitta}@asu.edu Abstract Winograd Schema Challenge (WSC) is a pro- noun resolution task which seems to require reasoning with commonsense knowledge. The needed knowledge is not present in the given text. Automatic extraction of the needed knowledge is a bottleneck in solving the chal- lenge. The existing state-of-the-art approach uses the knowledge embedded in their pre- trained language model. However, the lan- guage models only embed part of the knowl- edge, the ones related to frequently co-existing concepts. This limits the performance of such models on the WSC problems. In this work, we build-up on the language model based methods and augment them with a common- sense knowledge hunting (using automatic ex- traction from text) module and an explicit rea- soning module. Our end-to-end system built in such a manner improves on the accuracy of two of the available language model based approaches by 5.53% and 7.7% respectively. Overall our system achieves the state-of-the- art accuracy of 71.06% on the WSC dataset, an improvement of 7.36% over the previous best. 1 Introduction Reasoning with commonsense knowledge is an in- tegral component of human behavior. It is due to this capability that people know that they should dodge a stone that is thrown towards them. It has been a long standing goal of the Artificial Intelli- gence community to simulate such commonsense reasoning abilities in machines. Over the years, many advances have been made and various chal- lenges have been proposed to test their abilities (Clark et al., 2018; Mihaylov et al., 2018; Mishra et al., 2018). The Winograd Schema Challenge (WSC) (Levesque et al., 2011) is one such natu- ral language understanding challenge. It is made up of pronoun resolution problems of a particu- lar kind. The main part of each WSC problem is a set of sentences containing a pronoun. In ad- dition, two definite noun phrases, called “answer choices” are also given. The answer choices are part of the input set of sentences. The goal is to determine which answer provides the most natural resolution for the pronoun. Below is an example problem from the WSC. Sentences (S1): The fish ate the worm. It was tasty. Pronoun to resolve: It Answer Choices: a) fish b) worm A WSC problem also specifies a “special word” that occurs in the sentences, and an “alternate word.” Replacing the former by the latter changes the resolution of the pronoun. In the example above, the special word is tasty and the alternate word is hungry. The resolution of the pronoun is difficult be- cause the commonsense knowledge that is re- quired to perform the resolution is not explicitly present in the input text. The above example re- quires the commonsense …

Content truncated to 500 words