Hypothesis Only Baselines in Natural Language Inference

Proceedings of the 7th Joint Conference on Lexical and Computational Semantics (*SEM), pages 180–191 New Orleans, June 5-6, 2018. c⃝2018 Association for Computational Linguistics Hypothesis Only Baselines in Natural Language Inference Adam Poliak1 Jason Naradowsky1 Aparajita Haldar1,2 Rachel Rudinger1 Benjamin Van Durme1 1Johns Hopkins University 2BITS Pilani, Goa Campus, India {azpoliak,vandurme}@cs.jhu.edu {narad,ahaldar1,rudinger}@jhu.edu Abstract We propose a hypothesis only baseline for di- agnosing Natural Language Inference (NLI). Especially when an NLI dataset assumes infer- ence is occurring based purely on the relation- ship between a context and a hypothesis, it fol- lows that assessing entailment relations while ignoring the provided context is a degenerate solution. Yet, through experiments on ten dis- tinct NLI datasets, we find that this approach, which we refer to as a hypothesis-only model, is able to significantly outperform a majority- class baseline across a number of NLI datasets. Our analysis suggests that statistical irregular- ities may allow a model to perform NLI in some datasets beyond what should be achiev- able without access to the context. 1 Introduction Though datasets for the task of Natural Language Inference (NLI) may vary in just about every as- pect (size, construction, genre, label classes), they generally share a common structure: each instance consists of two fragments of natural language text (a context, also known as a premise, and a hypoth- esis), and a label indicating the entailment relation between the two fragments (e.g., ENTAILMENT, NEUTRAL, CONTRADICTION). Computationally, the task of NLI is to predict an entailment rela- tion label (output) given a premise-hypothesis pair (input), i.e., to determine whether the truth of the hypothesis follows from the truth of the premise (Dagan et al., 2006, 2013). When these NLI datasets are constructed to facilitate the training and evaluation of natural language understanding (NLU) systems (Nangia et al., 2017), it is tempting to claim that systems achieving high accuracy on such datasets have successfully “understood” natural language or at least a logical relationship between a premise and hypothesis. While this paper does not attempt to (a) (b) Figure 1: (1a) shows a typical NLI model that en- codes the premise and hypothesis sentences into a vec- tor space to classify the sentence pair. (1b) shows our hypothesis-only baseline method that ignores the premise and only encodes the hypothesis sentence. prescribe the sufficient conditions of such a claim, we argue for an obvious necessary, or at least de- sired condition: that interesting natural language inference should depend on both premise and hy- pothesis. In other words, a baseline system with access only to hypotheses (Figure 1b) can be said to perform NLI only in the sense that it is un- derstanding language based on prior background knowledge. If this background knowledge is about the world, this may be justifiable as an aspect of natural language understanding, if not in keep- ing with the spirit of NLI. But if the “background knowledge” consists of learned statistical irregu- larities in the data, this may not be ideal. Here we explore the question: do NLI …

Content truncated to 500 words