Resolving Complex Cases of Definite Pronouns: The Winograd Schema Challenge

Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 777–789, Jeju Island, Korea, 12–14 July 2012. c⃝2012 Association for Computational Linguistics Resolving Complex Cases of Definite Pronouns: The Winograd Schema Challenge Altaf Rahman and Vincent Ng Human Language Technology Research Institute University of Texas at Dallas Richardson, TX 75083-0688 {altaf,vince}@hlt.utdallas.edu Abstract We examine the task of resolving complex cases of definite pronouns, specifically those for which traditional linguistic constraints on coreference (e.g., Binding Constraints, gender and number agreement) as well as commonly-used resolution heuristics (e.g., string-matching facilities, syntactic salience) are not useful. Being able to solve this task has broader implications in artificial intelligence: a restricted version of it, sometimes referred to as the Winograd Schema Challenge, has been suggested as a conceptually and practi- cally appealing alternative to the Turing Test. We employ a knowledge-rich approach to this task, which yields a pronoun resolver that out- performs state-of-the-art resolvers by nearly 18 points in accuracy on our dataset. 1 Introduction Despite the significant amount of work on pronoun resolution in the natural language processing com- munity in the past forty years, the problem is still far from being solved. Its difficulty stems in part from its reliance on sophisticated knowledge sources and inference mechanisms. The sentence pair below, which we will subsequently refer to as the shout ex- ample, illustrates how difficult the problem can be: (1a) Ed shouted at Tim because he crashed the car. (1b) Ed shouted at Tim because he was angry. The pronoun he refers to Tim in 1a and Ed in 1b. Humans can resolve the pronoun easily, but state- of-the-art coreference resolvers cannot. The reason is that humans have the kind of world knowledge needed to resolve the pronouns that machines do not. Our world knowledge tells us that if someone is an- gry, he may shout at other people. Since Ed shouted, he should be the one who was angry. Our world knowledge also tells us that we may shout at some- one who made a mistake and that crashing a car is a mistake. Combining these two pieces of evidence, we can easily infer that Tim crashed the car. Our goal in this paper is to examine the resolu- tion of complex cases of definite pronouns that ap- pear in sentences exemplified by the shout example. Specifically, each sentence (1) has two clauses sepa- rated by a discourse connective (i.e., the connective appears between the two clauses, just like because in the shout example), where the first clause con- tains two or more candidate antecedents (e.g., Ed and Tim), and the second clause contains the tar- get pronoun (e.g., he); and (2) the target pronoun agrees in gender, number, and semantic class with each candidate antecedent, but does not have any overlap in content words with any of them. For con- venience, we will refer to the target pronoun that ap- pears in this kind of sentences as a difficult pronoun. …

Content truncated to 500 words