SemEval-2014 Task 10: Multilingual Semantic Textual Similarity

Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014), pages 81–91, Dublin, Ireland, August 23-24, 2014. SemEval-2014 Task 10: Multilingual Semantic Textual Similarity Eneko Agirrea, Carmen Baneab∗, Claire Cardiec, Daniel Cerd, Mona Diabe∗ Aitor Gonzalez-Agirrea, Weiwei Guof, Rada Mihalceab, German Rigaua, Janyce Wiebeg aUniversity of the Basque Country Basque Country, Spain bUniversity of Michigan Ann Arbor, MI cCornell University Ithaca, NY dGoogle Inc. Mountain View, CA eGeorge Washington University Washington, DC fColumbia University New York, NY gUniversity of Pittsburgh Pittsburgh, PA Abstract In Semantic Textual Similarity, systems rate the degree of semantic equivalence between two text snippets. This year, the participants were challenged with new data sets for English, as well as the in- troduction of Spanish, as a new language in which to assess semantic similarity. For the English subtask, we exposed the systems to a diversity of testing scenar- ios, by preparing additional OntoNotes- WordNet sense mappings and news head- lines, as well as introducing new gen- res, including image descriptions, DEFT discussion forums, DEFT newswire, and tweet-newswire headline mappings. For Spanish, since, to our knowledge, this is the first time that official evaluations are conducted, we used well-formed text, by featuring sentences extracted from ency- clopedic content and newswire. The an- notations for both tasks leveraged crowd- sourcing. The Spanish subtask engaged 9 teams participating with 22 system runs, and the English subtask attracted 15 teams with 38 system runs. 1 Introduction and motivation Given two snippets of text, Semantic Textual Sim- ilarity (STS) captures the notion that some texts are more similar than others, measuring their de- gree of semantic equivalence. Textual similar- ity can range from complete unrelatedness to ex- act semantic equivalence, and a graded similar- ity intuitively captures the notion of intermediate ∗carmennb@umich.edu, mtdiab@gwu.edu This work is licensed under a Creative Commons At- tribution 4.0 International Licence. Page numbers and pro- ceedings footer are added by the organisers. Licence details: http://creativecommons.org/licenses/by/4.0/ shades of similarity, as pairs of text may differ from some minor nuanced aspects of meaning, to relatively important semantic differences, to shar- ing only some details, or to simply being related to the same topic (cf. Section 2). One of the goals of the STS task is to create a unified framework for combining several seman- tic components that otherwise have historically tended to be evaluated independently and with- out characterization of impact on NLP applica- tions. By providing such a framework, STS al- lows for an extrinsic evaluation of these modules. Moreover, such an STS framework itself could in turn be evaluated intrinsically and extrinsically as a grey/black box within various NLP applications such as Machine Translation (MT), Summariza- tion, Generation, Question Answering (QA), etc. STS is related to both Textual Entailment (TE) and Paraphrasing, but differs in a number of ways and it is more directly applicable to a number of NLP tasks. STS is different from TE inasmuch as it assumes bidirectional graded equivalence be- tween the pair of textual snippets. In the case of TE …

Content truncated to 500 words