How Reasonable are Common-Sense Reasoning Tasks: A Case-Study on the Winograd Schema Challenge and SWAG

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 3382–3387, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 3382 How Reasonable are Common-Sense Reasoning Tasks: A Case-Study on the Winograd Schema Challenge and SWAG Paul Trichelair*1, Ali Emami*1, Adam Trischler2, Kaheer Suleman2, and Jackie Chi Kit Cheung1 1School of Computer Science, Mila/McGill University 2Microsoft Research Montreal {paul.trichelair, ali.emami}@mail.mcgill.ca {adam.trischler, kasulema}@microsoft.com jcheung@cs.mcgill.ca Abstract Recent studies have significantly improved the state-of-the-art on common-sense reasoning (CSR) benchmarks like the Winograd Schema Challenge (WSC) and SWAG. The question we ask in this paper is whether improved per- formance on these benchmarks represents gen- uine progress towards common-sense-enabled systems. We make case studies of both bench- marks and design protocols that clarify and qualify the results of previous work by ana- lyzing threats to the validity of previous ex- perimental designs. Our protocols account for several properties prevalent in common-sense benchmarks including size limitations, struc- tural regularities, and variable instance diffi- culty. 1 Introduction The proliferation of artificial-intelligence tech- nologies that interact with human users (e.g., di- alogue systems, recommendation systems, infor- mation retrieval tools) has led to renewed in- terest in common-sense reasoning (CSR). The progress of these technologies and the general societal reaction toward them greatly depend on advances in CSR, since systems can seem glar- ingly unintelligent when they lack common sense. Common sense is vital for resolving ambiguity that arises from implicit knowledge and under- specification. Consider the following sentence: (1) The delivery truck zoomed by the school bus because it was going so fast. Humans resolve the pronoun it to the delivery truck with no difficulty, whereas a system with- out common sense might be unable to distin- guish the truck from the otherwise viable can- didate, the school bus. The above sentence is *Equal contribution. an example from a popular binary-choice pro- noun co-reference problem called the Winograd Schema Challenge (WSC) (Levesque et al., 2011), designed to directly test a machine’s grasp of common sense. What makes sentences like (1) especially challenging for machine learning ap- proaches is that they are formulated such that simple word co-occurrence statistics cannot re- solve them at a rate above chance (i.e., the de- livery truck is unlikely to co-occur with going so fast much more frequently than the school bus does in large text corpora). In the same vein, a recently proposed common-sense inference task called SWAG (Zellers et al., 2018) further chal- lenges co-occurrence-based approaches. SWAG’s problem instances comprise a partial description, along with four candidate succeeding sentences designed to be distributionally similar. Among these, one successor is the most plausible. An ex- ample SWAG instance follows. (2) Someone is lifting the pinata. The pinata a) drops from the swings. b) bounces bigger than a third. c) slumps across his shoulder back. d) falls on the ground. Recently, a number of systems have attained new state-of-the-art results on WSC and SWAG by querying a …

Content truncated to 500 words