The Meaning Factory: Formal Semantics for Recognizing Textual Entailment and Determining Semantic Similarity

Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014), pages 642–646, Dublin, Ireland, August 23-24, 2014. The Meaning Factory: Formal Semantics for Recognizing Textual Entailment and Determining Semantic Similarity Johannes Bjerva Univ. of Groningen j.bjerva@rug.nl Johan Bos Univ. of Groningen johan.bos@rug.nl Rob van der Goot Univ. of Groningen r.van.der.goot@rug.nl Malvina Nissim Univ. of Bologna malvina.nissim@unibo.it Abstract Shared Task 1 of SemEval-2014 com- prised two subtasks on the same dataset of sentence pairs: recognizing textual en- tailment and determining textual similar- ity. We used an existing system based on formal semantics and logical inference to participate in the first subtask, reaching an accuracy of 82%, ranking in the top 5 of more than twenty participating sys- tems. For determining semantic similar- ity we took a supervised approach using a variety of features, the majority of which was produced by our system for recogniz- ing textual entailment. In this subtask our system achieved a mean squared error of 0.322, the best of all participating systems. 1 Introduction The recent popularity of employing distributional approaches to semantic interpretation has also lead to interesting questions about the relationship be- tween classic formal semantics (including its com- putational adaptations) and statistical semantics. A promising way to provide insight into these questions was brought forward as Shared Task 1 in the SemEval-2014 campaign for semantic evalua- tion (Marelli et al., 2014). In this task, a system is given a set of sentence pairs, and has to predict for each pair whether the sentences are somehow re- lated in meaning. Interestingly, this is done using two different metrics: the first stemming from the formal tradition (contradiction, entailed, neutral), and the second in a distributional fashion (a simi- larity score between 1 and 5). We participated in this shared task with a system rooted in formal se- mantics. In particular, we were interested in find- ing out whether paraphrasing techniques could in- crease the accuracy of our system, whether mean- ing representations used for textual entailment are This work is licensed under a Creative Commons Attribution 4.0 International Licence. Page numbers and proceedings footer are added by the organisers. Licence details: http: //creativecommons.org/licenses/by/4.0/ useful for predicting semantic similarity, and con- versely, whether similarity features could be used to boost accuracy of recognizing textual entail- ment. In this paper we outline our method and present the results for both the textual entailment and the semantic similarity task.1 2 Recognizing Textual Entailment 2.1 Overview The core of our system for recognizing textual en- tailment works as follows: (i) produce a formal se- mantic representation for each sentence for a given sentence pair; (ii) translate these semantic repre- sentations into first-order logic; (iii) use off-the- shelf theorem provers and model builders to check whether the first sentence entails the second, or whether the sentences are contradictory. This is essentially an improved version of the framework introduced by Bos & Markert (2006). To generate background knowledge that could assist in finding a proof we used the lexical database WordNet (Fellbaum, …

Content truncated to 500 words