Don’t Take the Premise for Granted: Mitigating Artifacts in Natural Language Inference

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 877–891 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 877 Don’t Take the Premise for Granted: Mitigating Artifacts in Natural Language Inference Yonatan Belinkov13∗ Adam Poliak2∗ Stuart M. Shieber1 Benjamin Van Durme2 Alexander M. Rush1 1Harvard University 2Johns Hopkins University 3Massachusetts Institute of Technology {belinkov,shieber,srush}@seas.harvard.edu {azpoliak,vandurme}@cs.jhu.edu Abstract Natural Language Inference (NLI) datasets of- ten contain hypothesis-only biases—artifacts that allow models to achieve non-trivial per- formance without learning whether a premise entails a hypothesis. We propose two prob- abilistic methods to build models that are more robust to such biases and better trans- fer across datasets. In contrast to standard approaches to NLI, our methods predict the probability of a premise given a hypothesis and NLI label, discouraging models from ig- noring the premise. We evaluate our meth- ods on synthetic and existing NLI datasets by training on datasets containing biases and test- ing on datasets containing no (or different) hypothesis-only biases. Our results indicate that these methods can make NLI models more robust to dataset-specific artifacts, transferring better than a baseline architecture in 9 out of 12 NLI datasets. Additionally, we provide an extensive analysis of the interplay of our meth- ods with known biases in NLI datasets, as well as the effects of encouraging models to ignore biases and fine-tuning on target datasets.1 1 Introduction Natural Language Inference (NLI) is often used to gauge a model’s ability to understand a relation- ship between two texts (Cooper et al., 1996; Dagan et al., 2006). In NLI, a model is tasked with deter- mining whether a hypothesis (a woman is sleep- ing) would likely be inferred from a premise (a woman is talking on the phone).2 The develop- ment of new large-scale datasets has led to a flurry of various neural network architectures for solv- ing NLI. However, recent work has found that ∗ Equal contribution 1Our code is available at https://github.com/ azpoliak/robust-nli. 2This hypothesis contradicts the premise and would likely not be inferred. many NLI datasets contain biases, or annotation artifacts, i.e., features present in hypotheses that enable models to perform surprisingly well using only the hypothesis, without learning the relation- ship between two texts (Gururangan et al., 2018; Poliak et al., 2018b; Tsuchiya, 2018).3 For in- stance, in some datasets, negation words like “not” and “nobody” are often associated with a relation- ship of contradiction. As a ramification of such biases, models may not generalize well to other datasets that contain different or no such biases. Recent studies have tried to create new NLI datasets that do not contain such artifacts, but many approaches to dealing with this issue remain unsatisfactory: constructing new datasets (Sharma et al., 2018) is costly and may still result in other artifacts; filtering “easy” examples and defin- ing a harder subset is useful for evaluation pur- poses (Gururangan et al., 2018), but difficult to do on a large scale that enables training; and compil- ing adversarial …

Content truncated to 500 words