Are Natural Language Inference Models IMPPRESsive? Learning IMPlicature and PRESupposition

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8690–8705 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 8690 Are Natural Language Inference Models IMPPRESsive? Learning IMPlicature and PRESupposition Paloma Jeretiˇc∗1, Alex Warstadt∗1, Suvrat Bhooshan2, Adina Williams2 1Department of Linguistics, New York University 2Facebook AI Research {paloma,warstadt}@nyu.edu, {sbh,adinawilliams}@fb.com Abstract Natural language inference (NLI) is an in- creasingly important task for natural language understanding, which requires one to infer whether a sentence entails another. However, the ability of NLI models to make pragmatic inferences remains understudied. We create an IMPlicature and PRESupposition diagnostic dataset (IMPPRES), consisting of >25k semi- automatically generated sentence pairs illus- trating well-studied pragmatic inference types. We use IMPPRES to evaluate whether BERT, InferSent, and BOW NLI models trained on MultiNLI (Williams et al., 2018) learn to make pragmatic inferences. Although MultiNLI appears to contain very few pairs illustrat- ing these inference types, we find that BERT learns to draw pragmatic inferences. It re- liably treats scalar implicatures triggered by “some” as entailments. For some presuppo- sition triggers like only, BERT reliably recog- nizes the presupposition as an entailment, even when the trigger is embedded under an entail- ment canceling operator like negation. BOW and InferSent show weaker evidence of prag- matic reasoning. We conclude that NLI train- ing encourages models to learn some, but not all, pragmatic inferences. 1 Introduction One of the most foundational semantic discover- ies is that systematic rules govern the inferential relationships between pairs of natural language sentences (Aristotle, De Interpretatione, Ch. 6). In natural language processing, Natural Language Inference (NLI)—a task whereby a system de- termines whether a pair of sentences instantiates in an entailment, a contradiction, or a neutral relation—has been useful for training and evaluat- ing models on sentential reasoning. However, lin- guists and philosophers now recognize that there ∗Equal Contribution Figure 1: Illustration of key properties of classical en- tailments, implicatures, and presuppositions. Solid ar- rows indicate valid commonsense entailments, and ar- rows with X’s indicate lack of entailment. Dashed ar- rows indicate follow up statements with the addition of in fact, which can either be acceptable (marked with ‘’) or unacceptable (marked with ‘’). are separate semantic and pragmatic modes of rea- soning (Grice, 1975; Clark, 1996; Beaver, 1997; Horn and Ward, 2004; Potts, 2015), and it is not clear which of these modes, if either, NLI mod- els learn. We investigate two pragmatic inference types that are known to differ from classical en- tailment: scalar implicatures and presuppositions. As shown in Figure 1, implicatures differ from en- tailments in that they can be denied, and presuppo- sitions differ from entailments in that they are not canceled when placed in entailment-cancelling en- vironments (e.g., negation, questions). To enable research into the relationship be- tween NLI and pragmatic reasoning, we introduce IMPPRES, a fine-grained NLI-style diagnostic test dataset for probing how well NLI models perform implicature and presupposition. Containing 25.5K sentence pairs illustrating key properties of these pragmatic inference types, IMPPRES is automati- …

Content truncated to 500 words