Grounded Textual Entailment

Proceedings of the 27th International Conference on Computational Linguistics, pages 2354–2368 Santa Fe, New Mexico, USA, August 20-26, 2018. 2354 Grounded Textual Entailment Hoa Trong Vu∗+, Claudio Greco†, Aliia Erofeeva†+, Somayeh Jafaritazehjan∗+ Guido Linders†+, Marc Tanti∗, Alberto Testoni†, Raffaella Bernardi†, Albert Gatt∗ +Erasmus Mundus European Program in Language and Communication Technology ∗University of Malta, †University of Trento ∗name.surname@um.edu.mt, †name.surname@unitn.it Abstract Capturing semantic relations between sentences, such as entailment, is a long-standing challenge for computational semantics. Logic-based models analyse entailment in terms of possible worlds (interpretations, or situations) where a premise P entails a hypothesis H iff in all worlds where P is true, H is also true. Statistical models view this relationship probabilistically, addressing it in terms of whether a human would likely infer H from P. In this paper, we wish to bridge these two perspectives, by arguing for a visually-grounded version of the Textual Entailment task. Specifically, we ask whether models can perform better if, in addition to P and H, there is also an image (corresponding to the relevant “world” or “situation”). We use a multimodal version of the SNLI dataset (Bowman et al., 2015) and we compare “blind” and visually-augmented models of textual entailment. We show that visual information is beneficial, but we also conduct an in-depth error analysis that reveals that current multimodal models are not performing “grounding” in an optimal fashion. 1 Introduction Evaluating the ability to infer information from a text is a crucial test of the capability of models to grasp meaning. As a result, the computational linguistics community has invested huge efforts into developing textual entailment (TE) datasets. After formal semanticists developed FraCas in the mid ’90 (Cooper et al., 1996), an increase in statisti- cal approaches to computational semantics gave rise to the need for suitable evaluation datasets. Hence, Recognizing Textual Entailment (RTE) shared tasks have been organized regularly (Sammons et al., 2012). Recent work on compositional distributional models has motivated the development of the SICK dataset of sentence pairs in entailment relations for evaluating such models (Marelli et al., 2014). Fur- ther advances with Neural Networks (NNs) have once more motivated efforts to develop a large natural language inference dataset, SNLI (Bowman et al., 2015), since NNs need to be trained on big data. However, meaning is not something we obtain just from text and the ability to reason is not unimodal either. The importance of enriching meaning representations with other modalities has been advocated by cognitive scientists, (e.g., (Andrews et al., 2009; Barsalou, 2010)) and computational linguists (e.g., (Glavaˇs et al., 2017)). While efforts have been put into developing multimodal datasets for the task of checking Semantic Text Similarity Text (Agirre et al., 2017), we are not aware of any available datasets to tackle the problem of Grounded Textual Entailment (GTE). Our paper is a first effort in this direction. Textual Entailment is defined in terms of the likelihood of two sentences (a premise P and an hypoth- esis H) to be in a certain relation: P entails, …

Content truncated to 500 words