Thinking Like a Skeptic: Defeasible Inference in Natural Language

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4661–4675 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 4661 Thinking Like a Skeptic: Defeasible Inference in Natural Language Rachel Rudinger†‡§, Vered Shwartz†‡, Jena D. Hwang†, Chandra Bhagavatula†, Maxwell Forbes†‡, Ronan Le Bras†, Noah A. Smith†‡, Yejin Choi†‡ †Allen Institute for AI ‡Paul G. Allen School of Computer Science & Engineering, University of Washington §University of Maryland, College Park, MD rudinger@umd.edu {vereds,jenah,chandrab,maxf,ronanlb,noah,yejinc}@allenai.org Abstract Defeasible inference is a mode of reasoning in which an inference (X is a bird, therefore X flies) may be weakened or overturned in light of new evidence (X is a penguin). Though long recognized in classical AI and philoso- phy, defeasible inference has not been exten- sively studied in the context of contemporary data-driven research on natural language infer- ence and commonsense reasoning. We intro- duce Defeasible NLI (abbreviated δ-NLI), a dataset for defeasible inference in natural lan- guage. δ-NLI contains extensions to three existing inference datasets covering diverse modes of reasoning: common sense, natural language inference, and social norms. From δ-NLI, we develop both a classification and generation task for defeasible inference, and demonstrate that the generation task is much more challenging. Despite lagging human per- formance, however, generative models trained on this data are capable of writing sentences that weaken or strengthen a specified inference up to 68% of the time. 1 Introduction Commonsense reasoning tasks are frequently for- mulated in terms of soft inferences: what is likely or plausibly true given some context, rather than (or in addition to) what is necessarily true. Given a context such as “The drinking glass fell”, it is common sense to infer that what likely happened next is “The drinking glass broke”. However, with the addition of new information, this inference may be blocked or weakened. If, for example, we sub- sequently learn that “The glass fell onto a pile of laundry” or that “The glass was made of durable material”, our original expectation that the glass will break is greatly diminished. This pattern of reasoning, in which an initially supported inference may subsequently be weakened or retracted in light Two men and a dog are standing among rolling green hills. They are wearing backpacks. One man is using his binoculars. The men are studying a tour map. The men are holding pitchforks. The men are facing their granary. The dog is a sheep dog. The men are farmers. Premise: Hypothesis: Figure 1: Examples from the δ-SNLI portion of the δ-NLI dataset. A neutral premise-hypothesis pair from SNLI is augmented with three update sentences that weaken the hypothesis (left, red) and three update sen- tences that strengthen it (right, blue). of new evidence, is known as defeasible reasoning (Koons, 2017). To the extent, then, that commonsense and nat- ural language inference systems must be able to reason about plausible or likely inferences, they must also be able to reason about the defeasibil- ity of those inferences. While most contemporary resources and datasets for these tasks …

Content truncated to 500 words