Investigating the Performance of Transformer-Based NLI Models on Presuppositional Inferences

Proceedings of the 29th International Conference on Computational Linguistics, pages 779–785 October 12–17, 2022. 779 Investigating the Performance of Transformer-Based NLI Models on Presuppositional Inferences Jad Kabbara and Jackie Chi Kit Cheung† School of Computer Science, McGill University, Montreal, QC, Canada Montreal Institute for Learning Algorithms (Mila), Montreal, QC, Canada {jad, jcheung}@cs.mcgill.ca † Canada CIFAR AI Chair Abstract Presuppositions are assumptions that are taken for granted by an utterance, and identifying them is key to a pragmatic interpretation of lan- guage. In this paper, we investigate the capa- bilities of transformer models to perform NLI on cases involving presupposition. First, we present simple heuristics to create alternative “contrastive” test cases based on the ImpPres dataset and investigate the model performance on those test cases. Second, to better under- stand how the model is making its predictions, we analyze samples from sub-datasets of Imp- Pres and examine model performance on them. Overall, our findings suggest that NLI-trained transformer models seem to be exploiting spe- cific structural and lexical cues as opposed to performing some kind of pragmatic reasoning. 1 Introduction Natural language inference (NLI) is the task of pre- dicting whether a sentence entails, contradicts or is neutral with respect to another sentence. While NLI is presented as a task with the goal of push- ing the frontier of language understanding, one question that remains elusive is the extent to which learning models trained on NLI datasets acquire competence in pragmatic reasoning. For example, “Rose’s drawing stunned the audience" presupposes that “Rose has a drawing." In terms of the NLI task, the relationship between these two statements is one of entailment. However, “Rose stunned the audience" does not trigger the same presupposition. If an NLI system relies on an incorrect heuristic to predict entailment, such as the fact that both state- ments have “Rose" in them, the system would fail on the modified version. While there exists a plethora of NLI datasets (e.g. SNLI (Bowman et al., 2015), MNLI (Williams et al., 2018), HANS (McCoy et al., 2019)), is- sues related to pragmatics remain under-explored. One exception is the ImpPres dataset (Jeretic et al., 2020), which contains English NLI instances in- volving two pragmatic phenomena: implicature and presupposition. In linguistics and philosophy of language, pre- suppositions are assumptions and beliefs that are shared and taken for granted by discourse partici- pants without an explicit mention in the discourse context. Presuppositions are prevalent in language and understanding them facilitates smooth commu- nication and is crucial for a proper understanding of the meaning being conveyed in a given context. In this work, we investigate how well trans- former models perform on cases from ImpPres involving presupposition. We ask the following question: In cases where these models seem to be doing well in terms of (NLI) accuracy performance, are they exploiting any patterns or correlations be- tween certain words and decision labels which hap- pen to lead to a good performance on ImpPres, but which might not generalize beyond it? Our contribution is two-fold: …

Content truncated to 500 words