*SEM 2013 shared task: Semantic Textual Similarity

Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 1: Proceedings of the Main Conference and the Shared Task, pages 32–43, Atlanta, Georgia, June 13-14, 2013. c⃝2013 Association for Computational Linguistics *SEM 2013 shared task: Semantic Textual Similarity Eneko Agirre University of the Basque Country e.agirre@ehu.es Daniel Cer Stanford University danielcer@stanford.edu Mona Diab George Washington University mtdiab@gwu.edu Aitor Gonzalez-Agirre University of the Basque Country agonzalez278@ikasle.ehu.es Weiwei Guo Columbia University weiwei@cs.columbia.edu Abstract In Semantic Textual Similarity (STS), sys- tems rate the degree of semantic equivalence, on a graded scale from 0 to 5, with 5 be- ing the most similar. This year we set up two tasks: (i) a core task (CORE), and (ii) a typed-similarity task (TYPED). CORE is similar in set up to SemEval STS 2012 task with pairs of sentences from sources related to those of 2012, yet different in genre from the 2012 set, namely, this year we included newswire headlines, machine translation eval- uation datasets and multiple lexical resource glossed sets. TYPED, on the other hand, is novel and tries to characterize why two items are deemed similar, using cultural heritage items which are described with metadata such as title, author or description. Several types of similarity have been defined, including simi- lar author, similar time period or similar lo- cation. The annotation for both tasks lever- ages crowdsourcing, with relative high inter- annotator correlation, ranging from 62% to 87%. The CORE task attracted 34 participants with 89 runs, and the TYPED task attracted 6 teams with 14 runs. 1 Introduction Given two snippets of text, Semantic Textual Simi- larity (STS) captures the notion that some texts are more similar than others, measuring the degree of semantic equivalence. Textual similarity can range from exact semantic equivalence to complete un- relatedness, corresponding to quantified values be- tween 5 and 0. The graded similarity intuitively cap- tures the notion of intermediate shades of similarity such as pairs of text differ only in some minor nu- anced aspects of meaning only, to relatively impor- tant differences in meaning, to sharing only some details, or to simply being related to the same topic, as shown in Figure 1. One of the goals of the STS task is to create a unified framework for combining several semantic components that otherwise have historically tended to be evaluated independently and without character- ization of impact on NLP applications. By providing such a framework, STS will allow for an extrinsic evaluation for these modules. Moreover, this STS framework itself could in turn be evaluated intrin- sically and extrinsically as a grey/black box within various NLP applications such as Machine Trans- lation (MT), Summarization, Generation, Question Answering (QA), etc. STS is related to both Textual Entailment (TE) and Paraphrasing, but differs in a number of ways and it is more directly applicable to a number of NLP tasks. STS is different from TE inasmuch as it as- sumes bidirectional graded equivalence between the pair of textual snippets. In the case of TE the equiv- …

Content truncated to 500 words