Penguins Don’t Fly: Reasoning about Generics through Instantiations and Exceptions

Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 2618–2635 May 2-6, 2023 ©2023 Association for Computational Linguistics Penguins Don’t Fly: Reasoning about Generics through Instantiations and Exceptions Emily Allaway⋆ Jena D. Hwang† Chandra Bhagavatula† Kathleen McKeown⋆ Doug Downey†¶ Yejin Choi†‡ ⋆Columbia University, USA †Allen Institute for Artificial Intelligence, USA ¶Northwestern University, USA ‡Paul G. Allen School of Computer Science & Engineering, University of Washington, USA eallaway@cs.columbia.edu Abstract Generics express generalizations about the world (e.g., birds can fly) that are not univer- sally true (e.g., newborn birds and penguins cannot fly). Commonsense knowledge bases, used extensively in NLP, encode some generic knowledge but rarely enumerate such excep- tions and knowing when a generic statement holds or does not hold true is crucial for devel- oping a comprehensive understanding of gener- ics. We present a novel framework informed by linguistic theory to generate EXEMPLARS— specific cases when a generic holds true or false. We generate ∼19k exemplars for ∼650 gener- ics and show that our framework outperforms a strong GPT-3 baseline by 12.8 precision points. Our analysis highlights the importance of lin- guistic theory-based controllability for generat- ing exemplars, the insufficiency of knowledge bases as a source of exemplars, and the chal- lenges exemplars pose for the task of natural language inference. 1 Introduction Generics express generalizations (e.g., birds can fly) that allow humans to reason and act with in- complete world knowledge (Asher and Morreau, 1995). Generics allow us to draw plausible infer- ences about individuals (e.g., Polly is a bird, so Polly can fly) even when we know of counterexam- ples (e.g., penguins cannot fly). Despite the utility of generalizations, knowledge of counterexamples is necessary for modeling generics and effectively reasoning with them in computational systems. Recent studies of generics (e.g., Bhagavatula et al., 2022; Bhakthavatsalam et al., 2020) and commonsense KBs (e.g., Speer et al., 2017) pro- vide repositories of generic knowledge. However, these resources rarely mention EXCEPTIONS (i.e., counterexamples) or INSTANTIATIONS (i.e., cases where the generic holds); collectively EXEMPLARS. For systems using these resources as a source of world knowledge, such incomplete information can xp(1): Penguins can’t xp(2): Sparrows can’t xp(i): Sparrows can xp(j): Birds can … “Sparrows can fly.” “Penguins can fly.” ✅ CONCEPTNET bird fly penguin sparrow ❌ Generic: “Birds can fly” EXEMPLARS INSTANTIATIONS EXCEPTIONS Quality Filters Prompts (Speer et al., 2017) Sparrows can fly. Birds can fly long distances. Penguins can’t fly. INSTANTIATION INSTANTIATION EXCEPTION Sparrows can fly Birds can wings Penguins can’t take off Baby birds can’t fly … Candidate Exemplars capable of property(soar ∨ take off ∨ glide ∨ migrate ∨ . . . ) # GPT2-XL Neurologic A*esque (Lu et al., 2022) & Constraints is a is a GENERATOR Figure 1: We present EXEMPLARS generator: given a generic like “Birds can fly” it generates truthful state- ments where the generic does (INSTANTIATIONS) and does not (EXCEPTIONS) hold. We extract commonsense knowledge (e.g., from ConceptNet (Speer et al., 2017)) in linguistically-informed prompts and constraints for …

Content truncated to 500 words