SemEval-2015 Task 2: Semantic Textual Similarity, English, Spanish and Pilot on Interpretability

Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015), pages 252–263, Denver, Colorado, June 4-5, 2015. c⃝2015 Association for Computational Linguistics SemEval-2015 Task 2: Semantic Textual Similarity, English, Spanish and Pilot on Interpretability Eneko Agirrea∗, Carmen Baneab∗, Claire Cardiec, Daniel Cerd, Mona Diabe∗, Aitor Gonzalez-Agirrea, Weiwei Guof, I˜nigo Lopez-Gazpioa, Montse Maritxalara∗, Rada Mihalceab, German Rigaua, Larraitz Uriaa, Janyce Wiebeg aUniversity of the Basque Country Donostia, Basque Country bUniversity of Michigan Ann Arbor, MI cCornell University Ithaca, NY dGoogle Inc. Mountain View, CA eGeorge Washington University Washington, DC fColumbia University New York, NY gUniversity of Pittsburgh Pittsburgh, PA Abstract In semantic textual similarity (STS), systems rate the degree of semantic equivalence be- tween two text snippets. This year, the par- ticipants were challenged with new datasets in English and Spanish. The annotations for both subtasks leveraged crowdsourcing. The En- glish subtask attracted 29 teams with 74 sys- tem runs, and the Spanish subtask engaged 7 teams participating with 16 system runs. In addition, this year we ran a pilot task on in- terpretable STS, where the systems needed to add an explanatory layer, that is, they had to align the chunks in the sentence pair, explicitly annotating the kind of relation and the score of the chunk pair. The train and test data were manually annotated by an expert, and included headline and image sentence pairs from previ- ous years. 7 teams participated with 29 runs. 1 Introduction and Motivation Given two snippets of text, semantic textual sim- ilarity (STS) captures the notion that some texts are more similar than others, measuring their de- gree of semantic equivalence. Textual similarity can range from complete unrelatedness to exact seman- tic equivalence, and a graded similarity score intu- itively captures the notion of intermediate shades of similarity, as pairs of text may differ from some mi- nor nuanced aspects of meaning to relatively impor- ∗Coordinators: e.agirre@ehu.eus, carmennb@umich.edu, mtdiab@gwu.edu, montse.maritxalar@ehu.eus tant semantic differences, to sharing only some de- tails, or to simply unrelated in meaning (cf. Sect. 2). One of the goals of the STS task is to create a unified framework for combining several semantic components that otherwise have historically tended to be evaluated independently and without character- ization of impact on NLP applications. By providing such a framework, STS allows for an extrinsic eval- uation of these modules. Moreover, such an STS framework could itself be in turn evaluated intrin- sically and extrinsically as a grey/black box within various NLP applications. STS is related to both textual entailment (TE) and paraphrasing, but it differs in a number of ways and it is more directly applicable to a number of NLP tasks. STS is different from TE inasmuch as it as- sumes bidirectional graded equivalence between a pair of textual snippets. In the case of TE the equiv- alence is directional, e.g. a car is a vehicle, but a ve- hicle is not necessarily a car. STS also differs from both TE and paraphrasing (in as far as both tasks have …

Content truncated to 500 words