Inference is Everything: Recasting Semantic Resources into a Unified Evaluation Framework

Proceedings of the The 8th International Joint Conference on Natural Language Processing, pages 996–1005, Taipei, Taiwan, November 27 – December 1, 2017 c⃝2017 AFNLP Inference is Everything: Recasting Semantic Resources into a Unified Evaluation Framework Aaron Steven White University of Rochester Pushpendre Rastogi Johns Hopkins University Kevin Duh Johns Hopkins University Benjamin Van Durme Johns Hopkins University Abstract We propose to unify a variety of existing se- mantic classification tasks, such as seman- tic role labeling, anaphora resolution, and paraphrase detection, under the heading of Recognizing Textual Entailment (RTE). We present a general strategy to automat- ically generate one or more sentential hy- potheses based on an input sentence and pre-existing manual semantic annotations. The resulting suite of datasets enables us to probe a statistical RTE model’s perfor- mance on different aspects of semantics. We demonstrate the value of this approach by investigating the behavior of a popular neural network RTE model. 1 Introduction The Recognizing Textual Entailment (RTE) task aims to assess a system’s ability to do textual inference—i.e. derive valid conclusions from tex- tual clues (Dagan et al., 2006, 2013; Bar-Haim et al., 2006; Giampiccolo et al., 2007, 2009; Ben- tivogli et al., 2009, 2010, 2011). In this task, a system judges whether “typically, a human read- ing [the sentential context, or text] T would infer that [the sentential hypothesis] H is most likely true” (Dagan et al., 2006). Recent efforts in textual inference have focused on the Stanford Natural Language Inference (SNLI) dataset. SNLI is made up of hundreds of thou- sands of text-hypothesis pairs, wherein the texts are image captions drawn from the Flickr30k corpus (Young et al., 2014) and the hypotheses are elicited from crowdsourcing workers based on those cap- tions (but not the corresponding image). While SNLI has led to significant methodological im- provements, its collection protocol does not lend itself to understanding the types of semantic knowl- edge necessary for properly understanding a partic- ular example. Researchers compete on which sys- tem achieves the highest score on a test set, but this itself does not lead to an understanding of which linguistic properties are better captured by a quan- titatively superior system. In contrast, datasets such as FraCaS (Cooper et al., 1996) are precisely designed to illustrate a range of semantic phenomenon that a text un- derstanding system should handle. But though this careful design enables fine-grained probes into a system’s semantic capabilities, FraCaS-like datasets tend not to be large-scale enough for recent work in data-driven computational semantics. Ask- ing experts, such as those who constructed FraCaS, to author hundreds of thousands of examples is not practical, just as the existing elicitation protocol behind SNLI will not lead to cleanly partitioned sets of examples that focus specifically on certain kinds of semantic inference. Our proposal is to leverage existing large-scale semantic annotation collections as a source of tar- geted textual inference examples. This strategy requires only minor effort in developing dataset- specific generation capabilities to recast annota- tions into a shared universal representation: …

Content truncated to 500 words