Collecting Diverse Natural Language Inference Problems for Sentence Representation Evaluation

Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 337–340 Brussels, Belgium, November 1, 2018. c⃝2018 Association for Computational Linguistics 337 Collecting Diverse Natural Language Inference Problems for Sentence Representation Evaluation Adam Poliak1 Aparajita Haldar1,2 Rachel Rudinger1 J. Edward Hu1 Ellie Pavlick3 Aaron Steven White4 Benjamin Van Durme1 1Johns Hopkins University, 2 BITS Pilani, Goa Campus, India 3Brown University, 4University of Rochester Abstract We present a large scale collection of diverse natural language inference (NLI) datasets that help provide insight into how well a sen- tence representation encoded by a neural net- work captures distinct types of reasoning. The collection results from recasting 13 existing datasets from 7 semantic phenomena into a common NLI structure, resulting in over half a million labeled context-hypothesis pairs in to- tal. Our collection of diverse datasets is avail- able at http://www.decomp.net/, and will grow over time as additional resources are recast and added from novel sources. 1 Introduction A plethora of new natural language inference (NLI)1 datasets has been created in recent years (Bowman et al., 2015; Williams et al., 2017; Lai et al., 2017; Khot et al., 2018). However, these datasets do not provide clear insight into what type of reasoning or inference a model may be perform- ing. For example, these datasets cannot be used to evaluate whether competitive NLI models can determine if an event occurred, correctly differ- entiate between figurative and literal language, or accurately identify and categorize named entities. Consequently, these datasets cannot answer how well sentence representation learning models cap- ture distinct semantic phenomena necessary for general natural language understanding (NLU). To answer these questions, we introduce the Diverse NLI Collection (DNC), a large-scale NLI dataset that tests a model’s ability to perform di- verse types of reasoning. DNC is a collection of NLI problems, each requiring a model to perform a unique type of reasoning. Each NLI dataset con- tains labeled context-hypothesis pairs that we re- 1The task of determining if a hypothesis would likely be inferred from a context, or premise; also known as Recogniz- ing Textual Entailment (RTE) (Dagan et al., 2006, 2013). ▶ Find him before he finds the dog food Event The finding did not happen  Factuality ▶ I’ll need to ponder The pondering happened  ▶ Ward joined Tom in their native Perth Relation Ward was born in Perth  Extraction ▶ Stefan had visited his son in Bulgaria Stefan was born in Bulgaria  ▶ Kim heard masks have no face value Kim heard a pun  ▶ Tod heard that thrift is better than annuity Puns Tod heard a pun  Table 1: Example sentence pairs for different semantic phe- nomena. ▶ indicates the line is a context and the following line is its corresponding hypothesis.  and  respectively in- dicate that the context entails, or does not entail the hypothe- sis. cast from semantic annotations for specific struc- tured prediction tasks. Table 1 includes a sample of NLI pairs that test …

Content truncated to 500 words