How well do NLI models capture verb veridicality?

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 2230–2240, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 2230 How well do NLI models capture verb veridicality? Alexis Ross Harvard University alexis ross@college.harvard.edu Ellie Pavlick Brown University ellie pavlick@brown.edu Abstract In natural language inference (NLI), contexts are considered veridical if they allow us to infer that their underlying propositions make true claims about the real world. We inves- tigate whether a state-of-the-art natural lan- guage inference model (BERT) learns to make correct inferences about veridicality in verb- complement constructions. We introduce an NLI dataset for veridicality evaluation con- sisting of 1,500 sentence pairs, covering 137 unique verbs. We find that both human and model inferences generally follow theoretical patterns, but exhibit a systematic bias towards assuming that verbs are veridical–a bias which is amplified in BERT. We further show that, encouragingly, BERT’s inferences are sensi- tive not only to the presence of individual verb types, but also to the syntactic role of the verb, the form of the complement clause (to- vs. that-complements), and negation. 1 Introduction A context is veridical when the propositions it contains are taken to be true, even if not explic- itly asserted. For example, in the sentence “He does not know that the answer is 5”, “know” is veridical with respect to “The answer is 5”, since a speaker cannot felicitously say the former sen- tence unless they believe the latter proposition to be true. In contrast, “think” would not be veridical here, since “He does not think that the answer is 5” is felicitous whether or not it is taken to be true that “The answer is 5”. Understanding veridical- ity requires semantic subtlety and is still an open problem for computational models of natural lan- guage inference (NLI) (Rudinger et al., 2018). This paper deals specifically with veridicality in verb-complement constructions. Prior work in this area has focused on characterizing verb classes–e.g. factives like “know that” (Kiparsky and Kiparsky, 1968) and implicatives like “man- age to” (Karttunen, 1971)–and on incorporating such lexical semantic information into computa- tional models (MacCartney and Manning, 2009). However, increasingly, linguistic evidence sug- gests that inferences involving veridicality rely heavily on non-lexical information and are better understood as a graded, pragmatic phenomenon (de Marneffe et al., 2012; Tonhauser et al., 2018). Thus, in this paper, we revisit the question of whether neural models of natural language inference–which are not explicitly endowed with knowledge of verbs’ lexical semantic categories– learn to make inferences about veridicality consis- tent with those made by humans. We solicit hu- man judgements on 1,500 sentence pairs involv- ing 137 verb-complement constructions. Analy- sis of these annotations provides new evidence of the importance of pragmatic inference in model- ing veridicality judgements. We use our collected annotations to analyze how well a state-of-the-art NLI model (BERT, Devlin et al., 2018) is able to mimic human behavior on such inferences. The …

Content truncated to 500 words