WinoLogic: A Zero-Shot Logic-based Diagnostic Dataset for Winograd Schema Challenge

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3779–3789 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 3779 WINOLOGIC: A Zero-Shot Logic-based Diagnostic Dataset for Winograd Schema Challenge Weinan He1, Canming Huang1, Yongmei Liu1, Xiaodan Zhu2 1Dept. of Computer Science, Sun Yat-sen University, Guangzhou 510006, China 2ECE & Ingenuity Labs Research Institute, Queen’s University, Canada {heweinan, huangcm}@mail2.sysu.edu.cn, ymliu@mail.sysu.edu.cn, xiaodan.zhu@queensu.ca Abstract The recent success of neural language mod- els (NLMs) on the Winograd Schema Chal- lenge has called for further investigation of the commonsense reasoning ability of these models. Previous diagnostic datasets rely on crowd-sourcing which fails to provide coherent commonsense crucial for solving WSC prob- lems. To better evaluate NLMs, we propose a logic-based framework that focuses on high- quality commonsense knowledge. Specifically, we identify and collect formal knowledge for- mulas verified by theorem provers and translate such formulas into natural language sentences. Based on these true knowledge sentences, ad- versarial false ones are generated. We propose a new dataset named WINOLOGIC with these sentences. Given a problem in WINOLOGIC, NLMs need to decide whether the plausible knowledge sentences could correctly solve the corresponding WSC problems in a zero-shot setting. We also ask human annotators to validate WINOLOGIC to ensure it is human- agreeable. Experiments show that NLMs still struggle to comprehend commonsense knowl- edge as humans do, indicating that their reason- ing ability could have been overestimated. 1 Introduction Recently, large-scale neural language models (NLMs) have shown promising results on many challenging tasks, including the Winograd Schema Challenge (WSC), a multiple-choice coreference resolution problem that is designed to test nat- ural language understanding and reasoning with commonsense knowledge (Levesque et al., 2012; Levesque, 2014). Each problem depicts a daily sit- uation with an ambiguous pronoun to be resolved. For example, in the WSC sentence “the trophy doesn’t fit into the brown suitcase because it is too large”, the pronoun “it” could plausibly re- fer to either “the trophy” or “the suitcase”. The lack of training examples in WSC has driven re- searchers to fine-tune NLMs on similar datasets, which achieved near-human performance (Sak- aguchi et al., 2020). Still, the opacity of the NLMs has raised ques- tions about whether they truly capture common sense or merely exploit biases. Such concerns were confirmed for the natural language inference task, as McCoy et al. (2019) discovered an LM could provide correct answers using fallible heuristics. To investigate if NLMs understand the reasons for solving WSC, Zhang et al. (2020) crowd-sourced reasons for the WSC problems and built a new dataset WinoWhy. While in the correct direction, crowd-sourcing is far from perfect for collecting reliable explanations. Consider the following WinoWhy example. Example 1 (WinoWhy). The trophy doesn’t fit into the brown suitcase because it is too large. The it is more likely to refer to the trophy than the brown suitcase because (a) the brown suitcase is too large. (b) it is a game. (c) The trophy is not fit into the suitcase. Given a WSC sentence …

Content truncated to 500 words