Partial-input baselines show that NLI models can ignore context, but they don’t.

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4753 - 4763 July 10-15, 2022 ©2022 Association for Computational Linguistics Partial-input baselines show that NLI models can ignore context, but they don’t. Neha Srikanth Department of Computer Science University of Maryland, College Park nehasrik@umd.edu Rachel Rudinger Department of Computer Science University of Maryland, College Park rudinger@umd.edu Abstract When strong partial-input baselines reveal ar- tifacts in crowdsourced NLI datasets, the per- formance of full-input models trained on such datasets is often dismissed as reliance on spuri- ous correlations. We investigate whether state- of-the-art NLI models are capable of overrid- ing default inferences made by a partial-input baseline. We introduce an evaluation set of 600 examples consisting of perturbed premises to examine a RoBERTa model’s sensitivity to edited contexts. Our results indicate that NLI models are still capable of learning to condi- tion on context—a necessary component of in- ferential reasoning—despite being trained on artifact-ridden datasets. 1 Introduction Natural language inference (NLI) is integral to building systems that are broadly capable of lan- guage understanding (White et al., 2017). In a traditional NLI setup, models are provided with a premise as context and a corresponding hypothe- sis. They must then determine whether the premise entails, contradicts, or is neutral in relation to the hypothesis (Giampiccolo et al., 2007). Researchers have shown that many NLI datasets contain statistical biases, or “annotation artifacts” (Gururangan et al., 2018; Herlihy and Rudinger, 2021) that systems leverage to correctly predict entailment. To diagnose such artifacts in datasets and provide a stronger alternative to majority-class baselines, Poliak et al. (2018) introduced partial- input baselines, a setting in which models are pro- vided parts of NLI instances to predict an entail- ment relation. Poliak et al. (2018), Gururangan et al. (2018), and others posit that datasets contain- ing artifacts may in turn produce models that are incapable of learning to perform true reasoning. In this paper, we re-examine such dismissal of the reasoning capabilities of models trained on datasets containing artifacts. While a competi- tive partial-input baseline is sufficient to cast doubt P: A dog plays with a stuffed animal. P’: A dog angrily growls at his owner. P: Female gymnasts warm up before a competition. H: Gymnasts get ready for the biggest competition of their life. H’: Gymnasts compete against others from their own country. H: The dog is enjoying himself. U: The gymnasts are intimidated by the Olympic committee. Context (C ) Target (T ) Neutral Contradiction Edited Context (C’) Strengthener Weakener SNLI δ-NLI ✎ ✎ P: Female gymnasts warm up before a competition. Figure 1: Given NLI and δ-NLI instances that partial- input baselines correctly label, we edit their context to induce a different gold label. We use the edited instances to probe a full-input model’s ability to leverage context to deviate from the predicted partial-input label. on the inferential capabilities of full-input models trained on the same data, it is premature to con- clude …

Content truncated to 500 words