WinoWhy: A Deep Diagnosis of Essential Commonsense Knowledge for Answering Winograd Schema Challenge

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5736–5745 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 5736 WinoWhy: A Deep Diagnosis of Essential Commonsense Knowledge for Answering Winograd Schema Challenge Hongming Zhang∗, Xinran Zhao∗, and Yangqiu Song Department of CSE, HKUST hzhangal@cse.ust.hk, xzhaoar@connect.ust.hk, yqsong@cse.ust.hk Abstract In this paper, we present the first compre- hensive categorization of essential common- sense knowledge for answering the Winograd Schema Challenge (WSC). For each of the questions, we invite annotators to first pro- vide reasons for making correct decisions and then categorize them into six major knowl- edge categories. By doing so, we better un- derstand the limitation of existing methods (i.e., what kind of knowledge cannot be ef- fectively represented or inferred with existing methods) and shed some light on the com- monsense knowledge that we need to acquire in the future for better commonsense reason- ing. Moreover, to investigate whether cur- rent WSC models can understand the com- monsense or they simply solve the WSC ques- tions based on the statistical bias of the dataset, we leverage the collected reasons to develop a new task called WinoWhy, which requires models to distinguish plausible reasons from very similar but wrong reasons for all WSC questions. Experimental results prove that even though pre-trained language representa- tion models have achieved promising progress on the original WSC dataset, they are still struggling at WinoWhy. Further experiments show that even though supervised models can achieve better performance, the perfor- mance of these models can be sensitive to the dataset distribution. WinoWhy and all codes are available at: https://github.com/ HKUST-KnowComp/WinoWhy. 1 Introduction Commonsense reasoning, as an important prob- lem of natural language understanding, has at- tracted much more attention in the NLP com- munity recently (Levesque et al., 2012; Zhou et al., 2018; Ostermann et al., 2018; Talmor et al., ∗Equal contribution. Figure 1: A pair of questions in WSC. 2019). Among all developed commonsense rea- soning tasks, the Winograd Schema Challenge (WSC) (Levesque et al., 2012), which is a hard pronoun coreference resolution task, is one of the most influential ones. All questions in WSC are grouped into pairs such that paired questions have minor differences (mostly one-word difference), but reversed answers. For each question, we de- note the other question in the same pair as its reverse question. One pair of the WSC task is shown in Figure 1. Based on the design guide- line of WSC, all commonly used features (e.g., gender, plurality, and co-occurrence frequency) do not have any effect. Human beings can solve these questions because of their shared common- sense knowledge. For example, ordinary people can know that the pronoun ‘it’ in the first sentence refers to ‘fish’ while the one in the second sentence refers to ‘worm’ because ‘hungry’ is a common property of something eating things while ‘tasty’ is a common property of something being eaten. Conventionally, people tried to leverage crowd- sourced commonsense knowledge bases (Liu et al., 2017) or search engines …

Content truncated to 500 words