Back to Square One: Artifact Detection, Training and Commonsense Disentanglement in the Winograd Schema

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10486–10500 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 10486 Back to Square One: Artifact Detection, Training and Commonsense Disentanglement in the Winograd Schema Yanai Elazar1,2 Hongming Zhang3,4 Yoav Goldberg1,2 Dan Roth4 1Bar Ilan University, 2AI2, 3HKUST, 4UPenn {yanaiela,yoav.goldberg}@gmail.com hzhangal@cse.ust.hk, danroth@seas.upenn.edu Abstract The Winograd Schema (WS) has been pro- posed as a test for measuring commonsense ca- pabilities of models. Recently, pre-trained lan- guage model-based approaches have boosted performance on some WS benchmarks but the source of improvement is still not clear. This paper suggests that the apparent progress on WS may not necessarily reflect progress in commonsense reasoning. To support this claim, we first show that the current evaluation method of WS is sub-optimal and propose a modification that uses twin sentences for eval- uation. We also propose two new baselines that indicate the existence of artifacts in WS benchmarks. We then develop a method for evaluating WS-like sentences in a zero-shot setting to account for the commonsense rea- soning abilities acquired during the pretrain- ing and observe that popular language mod- els perform randomly in this setting when us- ing our more strict evaluation. We conclude that the observed progress is mostly due to the use of supervision in training WS mod- els, which is not likely to successfully support all the required commonsense reasoning skills and knowledge.1 1 Introduction The Winograd Schema (WS) (Levesque et al., 2012) was proposed as an alternative to the Tur- ing test, by virtue of evaluating progress on com- monsense reasoning. The task is a multi-choice question akin to coreference resolution. Given a text snippet with two entities and a pronoun that refers to one of the entities, select the entity re- ferred to by the pronoun.2 Consider the following example: 1. The trophy doesn’t fit into the brown suitcase because it is too large. 1The code and evaluation are available at: https://gi thub.com/yanaiela/winograd_square_one 2It can also be a possessive adjective, but for simplicity, we refer these as pronouns. Setup Example Answer Original twin-1 The trophy doesn’t fit into the brown suitcase because it is too large. trophy twin-2 The trophy doesn’t fit into the brown suitcase because it is too small. suitcase Baselines no-cands doesn’t fit into because it is too large. ? part-sent because it is too large. ? Zero-shot twin-1 The trophy doesn’t fit into the brown suitcase because the trophy is too [MASK]. large twin-2 The trophy doesn’t fit into the brown suitcase because the brown suitcase is too [MASK]. small Figure 1: Examples from the Winograd Schema Chal- lenge (top), our proposed modification to these sen- tences that we use as novel baselines (middle) and the new formulation of the WS task which allows us to test LMs in a zero-shot setting (bottom). The entities are marked in italics, the pronoun in bold, and the special word3 is underlined. In this case, it refers to The trophy, since smaller objects typically fit into larger …

Content truncated to 500 words