Stress Test Evaluation for Natural Language Inference

Proceedings of the 27th International Conference on Computational Linguistics, pages 2340–2353 Santa Fe, New Mexico, USA, August 20-26, 2018. 2340 Stress Test Evaluation for Natural Language Inference Aakanksha Naik1∗, Abhilasha Ravichander1∗, Norman Sadeh2, Carolyn Rose1, Graham Neubig1 1Language Technologies Institute, Carnegie Mellon University 2Institute of Software Research, Carnegie Mellon University {anaik, aravicha, sadeh, cprose, gneubig}@cs.cmu.edu Abstract Natural language inference (NLI) is the task of determining if a natural language hypothesis can be inferred from a given premise in a justifiable manner. NLI was proposed as a benchmark task for natural language understanding. Existing models perform well at standard datasets for NLI, achieving impressive results across different genres of text. However, the extent to which these models understand the semantic content of sentences is unclear. In this work, we propose an evaluation methodology consisting of automatically constructed “stress tests” that allow us to examine whether systems have the ability to make real inferential decisions. Our evaluation of six sentence-encoder models on these stress tests reveals strengths and weaknesses of these models with respect to challenging linguistic phenomena, and suggests important directions for future work in this area. 1 Introduction Natural language inference (NLI), also known as recognizing textual entailment (RTE), is concerned with determining the relationship between a premise sentence and an associated hypothesis. This requires a model to make the 3-way decision of whether a hypothesis is true given the premise (entailment), false given the premise (contradiction), or whether the truth value cannot be determined (neutral). NLI has been proposed as a benchmark task for natural language understanding research (Cooper et al., 1996; Dagan et al., 2006; Giampiccolo et al., 2007; Dagan et al., 2013; Bowman et al., 2015; Nangia et al., 2017), due to the requirement for models to reason about several difficult linguistic phenomena1 to perform well at this task (Bowman et al., 2015; Williams et al., 2017). Due to its importance, significant prior work (Dagan et al., 2006; Dagan et al., 2009; Dagan et al., 2013; Marelli et al., 2014a) has focused on developing datasets and models for this benchmark task. Most recently, this task has been concretely implemented in the Stanford NLI (SNLI; Bowman et al. (2015)), and Multi-genre NLI (MultiNLI; Williams et al. (2017)) datasets, where crowdworkers are given a premise sentence and asked to generate novel sentences representing the three categories of entailment relations. Within the scope of these benchmark datasets, state-of-the-art deep learning-based sentence encoder models2 (Nie and Bansal (2017), Chen et al. (2017), Conneau et al. (2017), Balazs et al. (2017), inter alia) have been shown to consistently achieve high accuracies, which may lead us to believe that these models excel at NLI across genres of text. However, machine learning models are known to exploit idiosyncrasies of how the data was produced, allowing them to imitate desired behavior using tricks such as pattern matching (Levesque, 2014; Rimell et al., 2009; Papernot et al., 2017). In NLI this arises when the test set contains many easy examples, and the extent to which difficult …

Content truncated to 500 words