Unsupervised Commonsense Question Answering with Self-Talk

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 4615–4629, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 4615 Unsupervised Commonsense Question Answering with Self-Talk Vered Shwartz1,2, Peter West1,2, Ronan Le Bras1, Chandra Bhagavatula1, and Yejin Choi1,2 1Allen Institute for Artificial Intelligence 2Paul G. Allen School of Computer Science & Engineering, University of Washington {vereds,peterw,ronanlb,chandrab,yejinc}@allenai.org Abstract Natural language understanding involves read- ing between the lines with implicit background knowledge. Current systems either rely on pre- trained language models as the sole implicit source of world knowledge, or resort to ex- ternal knowledge bases (KBs) to incorporate additional relevant knowledge. We propose an unsupervised framework based on self-talk as a novel alternative to multiple-choice com- monsense tasks. Inspired by inquiry-based dis- covery learning (Bruner, 1961), our approach inquires language models with a number of information seeking questions such as “what is the definition of ...” to discover additional background knowledge. Empirical results demonstrate that the self-talk procedure sub- stantially improves the performance of zero- shot language model baselines on four out of six commonsense benchmarks, and competes with models that obtain knowledge from ex- ternal KBs. While our approach improves performance on several benchmarks, the self- talk induced knowledge even when leading to correct answers is not always seen as helpful by human judges, raising interesting questions about the inner-workings of pre-trained lan- guage models for commonsense reasoning. 1 Introduction Human level natural language understanding in- volves reading between the lines and relying on implicit background knowledge. Consider the sen- tence: Alice let Bob stand in front of her at the con- cert. Using physical and social commonsense – (i) Bob and Alice want to see the stage, and (ii) If Bob is taller, they would block Alice’s view – one can infer that Alice is taller than Bob. Such examples are ubiquitous across natural language understand- ing (NLU) tasks such as reading comprehension (Hirschman et al., 1999) and recognizing textual entailment (Dagan et al., 2013), and even more so in tasks dedicated to commonsense reasoning such as the Winograd schema challenge (Levesque et al., 2012). Most current NLU models rely on pre- trained language models (LMs; e.g. Radford et al., 2019; Devlin et al., 2019; Raffel et al., 2020). The standard practice is to fine-tune a pre-trained LM in a supervised manner on task-specific data. Alterna- tively, LM score is used to rank answer choices in a zero-shot setup (Wang et al., 2019; Bosselut and Choi, 2019). In both setups, pre-trained LMs yield improved performance upon prior methods, greatly due to the world knowledge that such LMs capture, having been trained on massive texts (Petroni et al., 2019; Davison et al., 2019). Despite the performance boost, LMs as knowl- edge providers suffer from various shortcomings: (i) insufficient coverage: due to reporting bias, many trivial facts might not be captured by LMs because they are rarely written about (Gordon and Van Durme, 2013). (ii) insufficient precision: the distributional training objective increases the prob- ability of non-facts that …

Content truncated to 500 words