Corpora for Learning the Mutual Relationship between Semantic Relatedness and Textual Entailment

Corpora for Learning the Mutual Relationship between Semantic Relatedness and Textual Entailment Ngoc Phuoc An Vo1,2,3, Octavian Popescu4 Xerox Research Centre Europe1, Fondazione Bruno Kessler2, University of Trento3, IBM Research4 an.vo@xrce.xerox.com, o.popescu@us.ibm.com Abstract In this paper we present the creation of a corpora annotated with both semantic relatedness (SR) scores and textual entailment (TE) judgments. In building this corpus we aimed at discovering, if any, the relationship between these two tasks for the mutual benefit of resolving one of them by relying on the insights gained from the other. We considered a corpora already annotated with TE judgments and we proceed to the manual annotation with SR scores. The RTE 1-4 corpora used in the PASCAL competition fit our need. The annotators worked independently of one each other and they did not have access to the TE judgment during annotation. The intuition that the two annotations are correlated received major support from this experiment and this finding led to a system that uses this information to revise the initial estimates of SR scores. As semantic relatedness is one of the most general and difficult task in natural language processing we expect that future systems will combine different sources of information in order to solve it. Our work suggests that textual entailment plays a quantifiable role in addressing it. Keywords: Semantic Relatedness, Textual Entailment, RTE Corpora 1. Introduction In the last years, two tasks that involve meaning processing, namely Semantic Relatedness (SR) (Marelli et al., 2014b) and Textual Entailment (TE) (Dagan et al., 2006), have re- ceived particular attention. In various academic competi- tions, both the TE and SR tasks have been proposed, and useful benchmarks have been created. The SR scores are usually given in the [1-5] interval, while the TE values are discrete signaling an entailment, a contradiction or a non- related relationship between candidate sentences. Interestingly, the core approach seems to be similar for both of these tasks (Agirre et al., 2012; Dagan et al., 2013). Yet, till 2014, no corpus annotated with both SR and TE labels was available. In the last SemEval 2014, the SICK cor- pus containing both SR and TE annotations for the same pairs have been released (Marelli et al., 2014a). This cor- pus allows a direct comparison between the systems that addresses both tasks. We have developed a technique that uses both SR and TE scores in order to improve the accuracy on each of these tasks. We show that when we take into account the esti- mates of one task in resolving the other better results are obtained. The strategy we propose is based on the corre- lation between SR scores and TE values. The main idea is that, in the first phase, we determine an estimation of the SR scores which are taken into account in deciding TE values at a second phase, which are used to re-adjust the SR scores at a third phase. The main intuition is that high/low SR scores signal ENTAILMENT/NEUTRAL TE values, and from correct …

Content truncated to 500 words