SemEval-2012 Task 6: A Pilot on Semantic Textual Similarity

First Joint Conference on Lexical and Computational Semantics (*SEM), pages 385–393, Montr´eal, Canada, June 7-8, 2012. c⃝2012 Association for Computational Linguistics SemEval-2012 Task 6: A Pilot on Semantic Textual Similarity Eneko Agirre University of the Basque Country Donostia, 20018, Basque Country e.agirre@ehu.es Daniel Cer Stanford University Stanford, CA 94305, USA danielcer@stanford.edu Mona Diab Center for Computational Learning Systems Columbia University mdiab@ccls.columbia.edu Aitor Gonzalez-Agirre University of the Basque Country Donostia, 20018, Basque Country agonzalez278@ikasle.ehu.es Abstract Semantic Textual Similarity (STS) measures the degree of semantic equivalence between two texts. This paper presents the results of the STS pilot task in Semeval. The training data contained 2000 sentence pairs from pre- viously existing paraphrase datasets and ma- chine translation evaluation resources. The test data also comprised 2000 sentences pairs for those datasets, plus two surprise datasets with 400 pairs from a different machine trans- lation evaluation corpus and 750 pairs from a lexical resource mapping exercise. The sim- ilarity of pairs of sentences was rated on a 0-5 scale (low to high similarity) by human judges using Amazon Mechanical Turk, with high Pearson correlation scores, around 90%. 35 teams participated in the task, submitting 88 runs. The best results scored a Pearson correlation >80%, well above a simple lexical baseline that only scored a 31% correlation. This pilot task opens an exciting way ahead, although there are still open issues, specially the evaluation metric. 1 Introduction Semantic Textual Similarity (STS) measures the degree of semantic equivalence between two sen- tences. STS is related to both Textual Entailment (TE) and Paraphrase (PARA). STS is more directly applicable in a number of NLP tasks than TE and PARA such as Machine Translation and evaluation, Summarization, Machine Reading, Deep Question Answering, etc. STS differs from TE in as much as it assumes symmetric graded equivalence between the pair of textual snippets. In the case of TE the equivalence is directional, e.g. a car is a vehicle, but a vehicle is not necessarily a car. Additionally, STS differs from both TE and PARA in that, rather than being a binary yes/no decision (e.g. a vehicle is not a car), STS incorporates the notion of graded semantic similarity (e.g. a vehicle and a car are more similar than a wave and a car). STS provides a unified framework that allows for an extrinsic evaluation of multiple semantic compo- nents that otherwise have tended to be evaluated in- dependently and without broad characterization of their impact on NLP applications. Such components include word sense disambiguation and induction, lexical substitution, semantic role labeling, multi- word expression detection and handling, anaphora and coreference resolution, time and date resolution, named-entity handling, underspecification, hedging, semantic scoping and discourse analysis. Though not in the scope of the current pilot task, we plan to explore building an open source toolkit for integrat- ing and applying diverse linguistic analysis modules to the STS task. While the characterization of STS is still prelim- inary, we observed that there was no comparable existing dataset extensively annotated for pairwise …

Content truncated to 500 words