Improving the Precision of Natural Textual Entailment Problem Datasets

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 6835–6840 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 6835 Improving the precision of natural textual entailment problem datasets Jean-Philippe Bernardy, Stergios Chatzikyriakidis University of Gothenburg Centre for Linguistic Theory and Studies in Probability jean-philippe.bernardy,stergios.chatzikyriakidis@gu.se Abstract In this paper, we propose a method to modify natural textual entailment problem datasets so that they better reflect a more precise notion of entailment. We apply this method to a subset of the Recognizing Textual Entailment datasets. We thus obtain a new corpus of entailment problems, which has the following three characteristics: 1. it is precise (does not leave out implicit hypotheses) 2. it is based on “real-world” texts (i.e. most of the premises were written for purposes other than testing textual entailment). 3. its size is 150. Broadly, the method that we employ is to make any missing hypotheses explicit using a crowd of experts. We discuss the relevance of our method in improving existing NLI datasets to be more fit for precise reasoning and we argue that this corpus can be the basis a first step towards wide-coverage testing of precise natural-language inference systems. Keywords: Natural Language Inference, Textual Entailment, RTE, SNLI, SICK 1. Introduction and Background Reasoning is part of our every day routine: we hear Natural Language (NL) sentences, we participate in dialogues, we read books or legal documents. Successfully understand- ing, participating or communicating with others in these situations presupposes some form of reasoning: about in- dividual sentences, whole paragraphs of legal documents, small or bigger pieces of dialogue and so on. Depending on the domain, and in general the situation, it appears that the reasoning performed can be more or less precise. Con- sider the following example: (1) Three representatives are needed. If a human reasoner with expert knowledge was to interpret the above utterance in a legal context, s/he will most proba- bly judge that a situation where more than three references are provided could be compatible with the semantics of the utterance. To the contrary, if the same reasoner was to in- terpret the above as part of a casual, everyday conversation, then three would most likely be interpreted as exactly three, making the same situation incompatible with the utterance. We aim to examine to what extent existing corpora for NLI capture precise reasoning. We call “precise reasoning”, in- ference which is either performed by experts or normal peo- ple after taking some time to consider the inferences that follow or not from a set of premises. There have been claims that large corpora (>1000 inference problems) like SNLI and MultiNLI (Bowman et al., 2015; Williams et al., 2017) are not suited to test systems of this type. Only the FraCaS test suite is intended to capture such reasoning, but it only involves 346 hand-constructed prob- lems. The next candidates for precise larger-scale corpora would be the Recognizing Textual Entailment (RTE) datasets. While the SICK dataset (Marelli …

Content truncated to 500 words