SemEval-2014 Task 1: Evaluation of Compositional Distributional Semantic Models on Full Sentences through Semantic Relatedness and Textual Entailment

Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014), pages 1–8, Dublin, Ireland, August 23-24, 2014. SemEval-2014 Task 1: Evaluation of Compositional Distributional Semantic Models on Full Sentences through Semantic Relatedness and Textual Entailment Marco Marelli(1) Luisa Bentivogli(2) Marco Baroni(1) Raffaella Bernardi(1) Stefano Menini(1,2) Roberto Zamparelli(1) (1) University of Trento, Italy (2) FBK - Fondazione Bruno Kessler, Trento, Italy {name.surname}@unitn.it, {bentivo,menini}@fbk.eu Abstract This paper presents the task on the evalu- ation of Compositional Distributional Se- mantics Models on full sentences orga- nized for the first time within SemEval- 2014. Participation was open to systems based on any approach. Systems were pre- sented with pairs of sentences and were evaluated on their ability to predict hu- man judgments on (i) semantic relatedness and (ii) entailment. The task attracted 21 teams, most of which participated in both subtasks. We received 17 submissions in the relatedness subtask (for a total of 66 runs) and 18 in the entailment subtask (65 runs). 1 Introduction Distributional Semantic Models (DSMs) approx- imate the meaning of words with vectors sum- marizing their patterns of co-occurrence in cor- pora. Recently, several compositional extensions of DSMs (CDSMs) have been proposed, with the purpose of representing the meaning of phrases and sentences by composing the distributional rep- resentations of the words they contain (Baroni and Zamparelli, 2010; Grefenstette and Sadrzadeh, 2011; Mitchell and Lapata, 2010; Socher et al., 2012). Despite the ever increasing interest in the field, the development of adequate benchmarks for CDSMs, especially at the sentence level, is still lagging. Existing data sets, such as those intro- duced by Mitchell and Lapata (2008) and Grefen- stette and Sadrzadeh (2011), are limited to a few hundred instances of very short sentences with a fixed structure. In the last ten years, several large This work is licensed under a Creative Commons At- tribution 4.0 International Licence. Page numbers and pro- ceedings footer are added by the organisers. Licence details: http://creativecommons.org/licenses/by/4.0/ data sets have been developed for various com- putational semantics tasks, such as Semantic Text Similarity (STS)(Agirre et al., 2012) or Recogniz- ing Textual Entailment (RTE) (Dagan et al., 2006). Working with such data sets, however, requires dealing with issues, such as identifying multiword expressions, recognizing named entities or access- ing encyclopedic knowledge, which have little to do with compositionality per se. CDSMs should instead be evaluated on data that are challenging for reasons due to semantic compositionality (e.g. context-cued synonymy resolution and other lexi- cal variation phenomena, active/passive and other syntactic alternations, impact of negation at vari- ous levels, operator scope, and other effects linked to the functional lexicon). These issues do not oc- cur frequently in, e.g., the STS and RTE data sets. With these considerations in mind, we devel- oped SICK (Sentences Involving Compositional Knowledge), a data set aimed at filling the void, including a large number of sentence pairs that are rich in the lexical, syntactic and semantic phe- nomena that CDSMs are expected to account for, but do not require dealing with other …

Content truncated to 500 words