SemEval-2017 Task 1: Semantic Textual Similarity Multilingual and Crosslingual Focused Evaluation

Proceedings of the 11th International Workshop on Semantic Evaluations (SemEval-2017), pages 1–14, Vancouver, Canada, August 3 - 4, 2017. c⃝2017 Association for Computational Linguistics SemEval-2017 Task 1: Semantic Textual Similarity Multilingual and Cross-lingual Focused Evaluation Daniel Cera, Mona Diabb, Eneko Agirrec, I˜nigo Lopez-Gazpioc, and Lucia Speciad aGoogle Research Mountain View, CA bGeorge Washington University Washington, DC cUniversity of the Basque Country Donostia, Basque Country dUniversity of Sheffield Sheffield, UK Abstract Semantic Textual Similarity (STS) mea- sures the meaning similarity of sentences. Applications include machine translation (MT), summarization, generation, question answering (QA), short answer grading, se- mantic search, dialog and conversational systems. The STS shared task is a venue for assessing the current state-of-the-art. The 2017 task focuses on multilingual and cross-lingual pairs with one sub-track ex- ploring MT quality estimation (MTQE) data. The task obtained strong participa- tion from 31 teams, with 17 participating in all language tracks. We summarize per- formance and review a selection of well performing methods. Analysis highlights common errors, providing insight into the limitations of existing models. To support ongoing work on semantic representations, the STS Benchmark is introduced as a new shared training and evaluation set carefully selected from the corpus of English STS shared task data (2012-2017). 1 Introduction Semantic Textual Similarity (STS) assesses the degree to which two sentences are semantically equivalent to each other. The STS task is moti- vated by the observation that accurately modeling the meaning similarity of sentences is a founda- tional language understanding problem relevant to numerous applications including: machine trans- lation (MT), summarization, generation, question answering (QA), short answer grading, semantic search, dialog and conversational systems. STS en- ables the evaluation of techniques from a diverse set of domains against a shared interpretable perfor- mance criteria. Semantic inference tasks related to STS include textual entailment (Bentivogli et al., 2016; Bowman et al., 2015; Dagan et al., 2010), semantic relatedness (Bentivogli et al., 2016) and paraphrase detection (Xu et al., 2015; Ganitkevitch et al., 2013; Dolan et al., 2004). STS differs from both textual entailment and paraphrase detection in that it captures gradations of meaning overlap rather than making binary classifications of par- ticular relationships. While semantic relatedness expresses a graded semantic relationship as well, it is non-specific about the nature of the relationship with contradictory material still being a candidate for a high score (e.g., “night” and “day” are highly related but not particularly similar). To encourage and support research in this area, the STS shared task has been held annually since 2012, providing a venue for evaluation of state-of- the-art algorithms and models (Agirre et al., 2012, 2013, 2014, 2015, 2016). During this time, di- verse similarity methods and data sets1 have been explored. Early methods focused on lexical se- mantics, surface form matching and basic syntac- tic similarity (B¨ar et al., 2012; ˇSari´c et al., 2012a; Jimenez et al., 2012a). During subsequent evalua- tions, strong new similarity signals emerged, such as Sultan et al. (2015)’s alignment based method. More recently, deep learning became competitive …

Content truncated to 500 words