A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference

Proceedings of NAACL-HLT 2018, pages 1112–1122 New Orleans, Louisiana, June 1 - 6, 2018. c⃝2018 Association for Computational Linguistics A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference Adina Williams1 adinawilliams@nyu.edu Nikita Nangia2 nikitanangia@nyu.edu Samuel R. Bowman1,2,3 bowman@nyu.edu 1Department of Linguistics New York University 2Center for Data Science New York University 3Department of Computer Science New York University Abstract This paper introduces the Multi-Genre Natu- ral Language Inference (MultiNLI) corpus, a dataset designed for use in the development and evaluation of machine learning models for sentence understanding. At 433k examples, this resource is one of the largest corpora avail- able for natural language inference (a.k.a. rec- ognizing textual entailment), improving upon available resources in both its coverage and difficulty. MultiNLI accomplishes this by of- fering data from ten distinct genres of written and spoken English, making it possible to eval- uate systems on nearly the full complexity of the language, while supplying an explicit set- ting for evaluating cross-genre domain adap- tation. In addition, an evaluation using exist- ing machine learning models designed for the Stanford NLI corpus shows that it represents a substantially more difficult task than does that corpus, despite the two showing similar levels of inter-annotator agreement. 1 Introduction Many of the most actively studied problems in NLP, including question answering, translation, and dialog, depend in large part on natural lan- guage understanding (NLU) for success. While there has been a great deal of work that uses rep- resentation learning techniques to pursue progress on these applied NLU problems directly, in or- der for a representation learning model to fully succeed at one of these problems, it must simul- taneously succeed both at NLU, and at one or more additional hard machine learning problems like structured prediction or memory access. This makes it difficult to accurately judge the degree to which current models extract reasonable represen- tations of language meaning in these settings. The task of natural language inference (NLI) is well positioned to serve as a benchmark task for research on NLU. In this task, also known as recognizing textual entailment (Cooper et al., 1996; Fyodorov et al., 2000; Condoravdi et al., 2003; Bos and Markert, 2005; Dagan et al., 2006; MacCartney and Manning, 2009), a model is pre- sented with a pair of sentences—like one of those in Figure 1—and asked to judge the relationship between their meanings by picking a label from a small set: typically ENTAILMENT, NEUTRAL, and CONTRADICTION. Succeeding at NLI does not require a system to solve any difficult machine learning problems except, crucially, that of ex- tracting effective and thorough representations for the meanings of sentences (i.e., their lexical and compositional semantics). In particular, a model must handle phenomena like lexical entailment, quantification, coreference, tense, belief, modal- ity, and lexical and syntactic ambiguity. As the only large human-annotated corpus for NLI currently available, the Stanford NLI Cor- pus (SNLI; Bowman et al., 2015) has enabled a good deal of progress on NLU, serving as a ma- jor benchmark for machine …

Content truncated to 500 words