Flexible Generation of Natural Language Deductions

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6266–6278 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 6266 Flexible Generation of Natural Language Deductions Kaj Bostrom Xinyu Zhao Swarat Chaudhuri Greg Durrett Department of Computer Science The University of Texas at Austin kaj@cs.utexas.edu Abstract An interpretable system for open-domain reasoning needs to express its reasoning process in a transparent form. Natural language is an attractive representation for this purpose — it is both highly expressive and easy for humans to understand. However, manipulating natural language statements in logically consistent ways is hard: models must cope with variation in how meaning is expressed while remaining precise. In this paper, we describe PARAPATTERN, a method for building models to generate deductive inferences from diverse natural language inputs without direct human supervision. We train BART-based models (Lewis et al., 2020) to generate the result of applying a particular logical operation to one or more premise statements. Crucially, we develop a largely automated pipeline for constructing suitable training examples from Wikipedia. We evaluate our models using out-of-domain sentence compositions from the QASC (Khot et al., 2020) and EntailmentBank (Dalvi et al., 2021) datasets as well as targeted perturbation sets. Our results show that our models are substantially more accurate and flexible than baseline systems. PARAPATTERN achieves 85% validity on examples of the ‘substitution’ operation from EntailmentBank without the use of any in-domain training data, matching the performance of a model fine-tuned for EntailmentBank. The full source code for our method is publicly available.1. 1 Introduction Developing models that can make useful inferences from natural language premises has been a core goal in artificial intelligence since the field’s early days (Bobrow, 1964; Winograd, 1971). Since then, there has been massive progress in automated formal reasoning (De Moura and Bjørner, 2011); in contrast, progress in automated 1https://github.com/alephic/ParaPattern The Green Bay Packers play in the NFL. NFL teams play from September to January. Substitution model The Green Bay Packers play from September to January. Conditions lacking environmental causes are strictly hereditary. Contraposition model Conditions that aren’t strictly hereditary have environmental causes. Figure 1: Examples of the natural deduction operations for which we construct models. Note that conclusions involve both lexical inferences (X plays in the NFL → X is an NFL team, ¬[X lacks Y ] → X has Y ) and logical transformations. natural language reasoning has been comparatively slow. Today, ‘natural language inference’ usually means recognizing textual entailment (RTE), a pairwise sentence classification task. Models have saturated RTE benchmarks (Bowman et al., 2015; Williams et al., 2018) largely through surface- level heuristics (Gururangan et al., 2018; Poliak et al., 2018); hill-climbing on these benchmarks has failed to yield robust models (Naik et al., 2018) or systems capable of more complex reasoning. Following a line of work on multi-hop question answering (Welbl et al., 2018; Yang et al., 2018; Chen and Durrett, 2019; Min et al., 2019), the reading comprehension community has started to make inroads in the area of reasoning. …

Content truncated to 500 words