Towards Zero-shot Commonsense Reasoning with Self-supervised Refinement of Language Models

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 8737–8743 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 8737 Towards Zero-shot Commonsense Reasoning with Self-supervised Refinement of Language Models Tassilo Klein SAP AI Research tassilo.klein@sap.com Moin Nabi SAP AI Research m.nabi@sap.com Abstract Can we get existing language models and re- fine them for zero-shot commonsense reason- ing? This paper presents an initial study ex- ploring the feasibility of zero-shot common- sense reasoning for the Winograd Schema Challenge by formulating the task as self- supervised refinement of a pre-trained lan- guage model. In contrast to previous studies that rely on fine-tuning annotated datasets, we seek to boost conceptualization via loss land- scape refinement. To this end, we propose a novel self-supervised learning approach that refines the language model utilizing a set of linguistic perturbations of similar concept re- lationships. Empirical analysis of our concep- tually simple framework demonstrates the via- bility of zero-shot commonsense reasoning on multiple benchmarks.1 1 Introduction Natural language processing has recently experi- enced unprecedented progress, boosting the perfor- mance of many applications to new levels. How- ever, this gain in performance does not equally transfer to applications requiring commonsense rea- soning capabilities, which has largely remained an unsolved problem (Marcus, 2020; Kocijan et al., 2020). In order to assess the commonsense reason- ing capabilities of automatic systems, several tasks have been devised. Among them is the popular Winograd Schema Challenge (WSC), which frames commonsense reasoning as a pronoun resolution task (Levesque et al., 2012). Although appearing evident and natural to the human mind, modern machine learning methods still struggle to solve this challenge. Lately, the research community has experienced an abundance of methods proposing utilization of 1The source code can be found at: https://github.com/SAP-samples/ emnlp2021-contrastive-refinement/ “The trophy does not fit into the suitcase, because it is too big.” “The trophy does not fit into the suitcase, because it is too small.” “The medal does not fit into the box, because it is too big.” “The medal does not fit into the box, because it is too small.” a) b) Figure 1: WSC sample: a) original sentence, b) pertur- bation (noun synonym). Task: resolve pronoun with a candidate. The trigger-word induces an answer flip. language models (LM) to tackle commonsense rea- soning in a two-stage learning pipeline. Starting from an initial self-supervised learned model, com- monsense enhanced LMs are obtained in a subse- quent fine-tuning (ft) phase. Fine-tuning enforces the LM to solve the downstream WSC task as a plain co-reference resolution task. However, such supervised approaches are prone to leverage statis- tical data artifacts for reasoning, giving rise to the “Clever Hans” effect (Lapuschkin et al., 2019). As such, instead of truly featuring reasoning capabili- ties, approaches become very good in faking. On the other hand, the lack of commonsense reason- ing capabilities of LMs can be partially attributed to the training corpora itself, as the commonsense knowledge is often not incorporated into the train- ing text due to the assumed triviality (Trichelair …

Content truncated to 500 words