A large annotated corpus for learning natural language inference

Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632–642, Lisbon, Portugal, 17-21 September 2015. c⃝2015 Association for Computational Linguistics. A large annotated corpus for learning natural language inference Samuel R. Bowman∗† sbowman@stanford.edu Gabor Angeli†‡ angeli@stanford.edu Christopher Potts∗ cgpotts@stanford.edu Christopher D. Manning∗†‡ manning@stanford.edu ∗Stanford Linguistics †Stanford NLP Group ‡Stanford Computer Science Abstract Understanding entailment and contradic- tion is fundamental to understanding nat- ural language, and inference about entail- ment and contradiction is a valuable test- ing ground for the development of seman- tic representations. However, machine learning research in this area has been dra- matically limited by the lack of large-scale resources. To address this, we introduce the Stanford Natural Language Inference corpus, a new, freely available collection of labeled sentence pairs, written by hu- mans doing a novel grounded task based on image captioning. At 570K pairs, it is two orders of magnitude larger than all other resources of its type. This in- crease in scale allows lexicalized classi- fiers to outperform some sophisticated ex- isting entailment models, and it allows a neural network-based model to perform competitively on natural language infer- ence benchmarks for the first time. 1 Introduction The semantic concepts of entailment and contra- diction are central to all aspects of natural lan- guage meaning (Katz, 1972; van Benthem, 2008), from the lexicon to the content of entire texts. Thus, natural language inference (NLI) — charac- terizing and using these relations in computational systems (Fyodorov et al., 2000; Condoravdi et al., 2003; Bos and Markert, 2005; Dagan et al., 2006; MacCartney and Manning, 2009) — is essential in tasks ranging from information retrieval to seman- tic parsing to commonsense reasoning. NLI has been addressed using a variety of tech- niques, including those based on symbolic logic, knowledge bases, and neural networks. In recent years, it has become an important testing ground for approaches employing distributed word and phrase representations. Distributed representa- tions excel at capturing relations based in similar- ity, and have proven effective at modeling simple dimensions of meaning like evaluative sentiment (e.g., Socher et al. 2013), but it is less clear that they can be trained to support the full range of logical and commonsense inferences required for NLI (Bowman et al., 2015; Weston et al., 2015b; Weston et al., 2015a). In a SemEval 2014 task aimed at evaluating distributed representations for NLI, the best-performing systems relied heavily on additional features and reasoning capabilities (Marelli et al., 2014a). Our ultimate objective is to provide an empiri- cal evaluation of learning-centered approaches to NLI, advancing the case for NLI as a tool for the evaluation of domain-general approaches to semantic representation. However, in our view, existing NLI corpora do not permit such an as- sessment. They are generally too small for train- ing modern data-intensive, wide-coverage models, many contain sentences that were algorithmically generated, and they are often beset with indeter- minacies of event and entity coreference that sig- nificantly impact annotation quality. To address this, this paper introduces the Stan- …

Content truncated to 500 words