SemEval-2014 Task 3: Cross-Level Semantic Similarity

Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014), pages 17–26, Dublin, Ireland, August 23-24, 2014. SemEval-2014 Task 3: Cross-Level Semantic Similarity David Jurgens, Mohammad Taher Pilehvar, and Roberto Navigli Department of Computer Science Sapienza University of Rome {jurgens,pilehvar,navigli}@di.uniroma1.it Abstract This paper introduces a new SemEval task on Cross-Level Semantic Similarity (CLSS), which measures the degree to which the meaning of a larger linguistic item, such as a paragraph, is captured by a smaller item, such as a sentence. High- quality data sets were constructed for four comparison types using multi-stage an- notation procedures with a graded scale of similarity. Nineteen teams submitted 38 systems. Most systems surpassed the baseline performance, with several attain- ing high performance for multiple com- parison types. Further, our results show that comparisons of semantic representa- tion increase performance beyond what is possible with text alone. 1 Introduction Given two linguistic items, semantic similarity measures the degree to which the two items have the same meaning. Semantic similarity is an es- sential component of many applications in Nat- ural Language Processing (NLP), and similarity measurements between all types of text as well as between word senses lend themselves to a va- riety of NLP tasks such as information retrieval (Hliaoutakis et al., 2006) or paraphrasing (Glick- man and Dagan, 2003). Semantic similarity evaluations have largely fo- cused on comparing similar types of lexical items. Most recently, tasks in SemEval (Agirre et al., 2012) and *SEM (Agirre et al., 2013) have intro- duced benchmarks for measuring Semantic Tex- tual Similarity (STS) between similar-sized sen- tences and phrases. Other data sets such as that This work is licensed under a Creative Commons Attribution 4.0 International License. Page numbers and proceedings footer are added by the organizers. License details: http: //creativecommons.org/licenses/by/4.0/ of Rubenstein and Goodenough (1965) measure similarity between word pairs, while the data sets of Navigli (2006) and Kilgarriff (2001) offer a bi- nary similar-dissimilar distinction between senses. Notably, all of these evaluations have focused on comparisons between a single type, in contrast to application-based evaluations such as summariza- tion and compositionality which incorporate tex- tual items of different sizes, e.g., measuring the quality of a paragraph’s sentence summarization. Task 3 introduces a new evaluation where sim- ilarity is measured between items of different types: paragraphs, sentences, phrases, words and senses. Given an item of the lexically-larger type, a system measures the degree to which the mean- ing of the larger item is captured in the smaller type, e.g., comparing a paragraph to a sentence. We refer to this task as Cross-Level Semantic Sim- ilarity (CLSS). A major motivation of this task is to produce semantic similarity systems that are able to compare all types of text, thereby free- ing downstream NLP applications from needing to consider the type of text being compared. Task 3 enables assessing the extent to which the mean- ing of the sentence “do u know where i can watch free older movies online without download?” is captured in the phrase …

Content truncated to 500 words