SemEval-2016 Task 1: Semantic Textual Similarity, Monolingual and Cross-Lingual Evaluation

Proceedings of SemEval-2016, pages 497–511, San Diego, California, June 16-17, 2016. c⃝2016 Association for Computational Linguistics SemEval-2016 Task 1: Semantic Textual Similarity, Monolingual and Cross-Lingual Evaluation Eneko Agirrea, Carmen Baneab, Daniel Cerd, Mona Diabe, Aitor Gonzalez-Agirrea, Rada Mihalceab, German Rigaua, Janyce Wiebef aUniversity of the Basque Country Donostia, Basque Country bUniversity of Michigan Ann Arbor, MI dGoogle Inc. Mountain View, CA eGeorge Washington University Washington, DC fUniversity of Pittsburgh Pittsburgh, PA Abstract Semantic Textual Similarity (STS) seeks to measure the degree of semantic equivalence between two snippets of text. Similarity is ex- pressed on an ordinal scale that spans from semantic equivalence to complete unrelated- ness. Intermediate values capture specifically defined levels of partial similarity. While prior evaluations constrained themselves to just monolingual snippets of text, the 2016 shared task includes a pilot subtask on com- puting semantic similarity on cross-lingual text snippets. This year’s traditional mono- lingual subtask involves the evaluation of En- glish text snippets from the following four do- mains: Plagiarism Detection, Post-Edited Ma- chine Translations, Question-Answering and News Article Headlines. From the question- answering domain, we include both question- question and answer-answer pairs. The cross-lingual subtask provides paired Spanish- English text snippets drawn from the same sources as the English data as well as indepen- dently sampled news data. The English sub- task attracted 43 participating teams produc- ing 119 system submissions, while the cross- lingual Spanish-English pilot subtask attracted 10 teams resulting in 26 systems. 1 Introduction Semantic Textual Similarity (STS) assesses the de- gree to which the underlying semantics of two seg- ments of text are equivalent to each other. This as- sessment is performed using an ordinal scale that The authors of this paper are listed in alphabetic order. ranges from complete semantic equivalence to com- plete semantic dissimilarity. The intermediate levels capture specifically defined degrees of partial sim- ilarity, such as topicality or rough equivalence, but with differing details. The snippets being scored are approximately one sentence in length, with their as- sessment being performed outside of any contextu- alizing text. While STS has previously just involved judging text snippets that are written in the same lan- guage, this year’s evaluation includes a pilot subtask on the evaluation of cross-lingual sentence pairs. The systems and techniques explored as a part of STS have a broad range of applications including Machine Translation (MT), Summarization, Gener- ation and Question Answering (QA). STS allows for the independent evaluation of methods for com- puting semantic similarity drawn from a diverse set of domains that would otherwise be only studied within a particular subfield of computational linguis- tics. Existing methods from a subfield that are found to perform well in a more general setting as well as novel techniques created specifically for STS may improve any natural language processing or lan- guage understanding application where knowing the similarity in meaning between two pieces of text is relevant to the behavior of the system. Paraphrase detection and textual entailment are both highly related to STS. However, STS is …

Content truncated to 500 words