Contrastive Self-Supervised Learning for Commonsense Reasoning

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7517–7523 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 7517 Contrastive Self-Supervised Learning for Commonsense Reasoning Tassilo Klein Moin Nabi SAP AI Research, Berlin, Germany {tassilo.klein, m.nabi}@sap.com Abstract We propose a self-supervised method to solve Pronoun Disambiguation and Winograd Schema Challenge problems. Our approach exploits the characteristic structure of training corpora related to so-called “trigger” words, which are responsible for flipping the answer in pronoun disambiguation. We achieve such commonsense reasoning by constructing pair- wise contrastive auxiliary predictions. To this end, we leverage a mutual exclusive loss regu- larized by a contrastive margin. Our architec- ture is based on the recently introduced trans- former networks, BERT, that exhibits strong performance on many NLP benchmarks. Em- pirical results show that our method allevi- ates the limitation of current supervised ap- proaches for commonsense reasoning. This study opens up avenues for exploiting inexpen- sive self-supervision to achieve performance gain in commonsense reasoning tasks. 1 1 Introduction Natural language representation learning (e.g., BERT (Devlin et al., 2018), etc.) can capture rich semantics from text and consistently improve the performance of downstream natural language pro- cessing (NLP) tasks. However, despite the recent progress, the task of commonsense reasoning is still far from being solved. Among many factors, this can be attributed to the strong correlation between attainable accuracy and training corpora size and quality. A particular case in point is the Winograd Schema Challenge (WSC) (Levesque et al., 2012). Despite its seeming simplicity for humans, it is still not solved by current algorithms. Below is a popular example of a question-answer pair from the binary-choice pronoun coreference problem (Lee et al., 2017) of WSC: 1Code available at https://github.com/ SAP-samples/acl2020-commonsense/ Sentence-1: The trophy doesn't fit in the suitcase because it is too small. Answers: A) the trophy B) the suitcase Sentence-2: The trophy doesn't fit in the suitcase because it is too big. Answers: A) the trophy B) the suitcase For humans resolving the pronoun “it” to “the suitcase” is straightforward. However, a system without the capacity of commonsense reasoning is unable to conceptualize the inherent relationship and, therefore, unable to distinguish “the suitcase” from the alternative “the trophy”. Recently, the research community has experi- enced an abundance in methods proposing to utilize latest word embedding and language model (LM) technologies for commonsense reasoning (Kocijan et al., 2019; He et al., 2019; Ye et al., 2019; Ruan et al., 2019; Trinh and Le, 2018; Klein and Nabi, 2019). The underlying assumption of these meth- ods is that, since such models are learned on large text corpora (such as Wikipedia), they implicitly capture to a certain degree commonsense knowl- edge. As a result, models permit reasoning about complex relationships between entities at inference time. Most of the methods proposed a two-stage learning pipeline. They are starting from an initial self-supervised model, commonsense-aware word embeddings are then obtained in a subsequent fine- tuning (ft) phase. Fine-tuning enforces the learned embedding to solve …

Content truncated to 500 words