SemEval-2017 Task 2: Multilingual and Cross-lingual Semantic Word Similarity

Proceedings of the 11th International Workshop on Semantic Evaluations (SemEval-2017), pages 15–26, Vancouver, Canada, August 3 - 4, 2017. c⃝2017 Association for Computational Linguistics SemEval-2017 Task 2: Multilingual and Cross-lingual Semantic Word Similarity Jose Camacho-Collados*1, Mohammad Taher Pilehvar*2, Nigel Collier2 and Roberto Navigli1 1Department of Computer Science, Sapienza University of Rome 2Department of Theoretical and Applied Linguistics, University of Cambridge 1{collados,navigli}@di.uniroma1.it 2{mp792,nhc30}@cam.ac.uk Abstract This paper introduces a new task on Multi- lingual and Cross-lingual Semantic Word Similarity which measures the semantic similarity of word pairs within and across five languages: English, Farsi, German, Italian and Spanish. High quality datasets were manually curated for the five lan- guages with high inter-annotator agree- ments (consistently in the 0.9 ballpark). These were used for semi-automatic con- struction of ten cross-lingual datasets. 17 teams participated in the task, submitting 24 systems in subtask 1 and 14 systems in subtask 2. Results show that systems that combine statistical knowledge from text corpora, in the form of word embeddings, and external knowledge from lexical re- sources are best performers in both sub- tasks. More information can be found on the task website: http://alt.qcri. org/semeval2017/task2/ . 1 Introduction Measuring the extent to which two words are se- mantically similar is one of the most popular re- search fields in lexical semantics, with a wide range of Natural Language Processing (NLP) ap- plications. Examples include Word Sense Disam- biguation (Miller et al., 2012), Information Re- trieval (Hliaoutakis et al., 2006), Machine Trans- lation (Lavie and Denkowski, 2009), Lexical Sub- stitution (McCarthy and Navigli, 2009), Question Answering (Mohler et al., 2011), Text Summa- rization (Mohammad and Hirst, 2012), and On- tology Alignment (Pilehvar and Navigli, 2014). Moreover, word similarity is generally accepted as the most direct in-vitro evaluation framework for Authors marked with * contributed equally. word representation, a research field that has re- cently received massive research attention mainly as a result of the advancements in the use of neural networks for learning dense low-dimensional se- mantic representations, often referred to as word embeddings (Mikolov et al., 2013; Pennington et al., 2014). Almost any application in NLP that deals with semantics can benefit from efficient se- mantic representation of words (Turney and Pan- tel, 2010). However, research in semantic representation has in the main focused on the English language only. This is partly due to the limited availabil- ity of word similarity benchmarks in languages other than English. Given the central role of similarity datasets in lexical semantics, and given the importance of moving beyond the barriers of the English language and developing language- independent and multilingual techniques, we felt that this was an appropriate time to conduct a task that provides a reliable framework for evaluating multilingual and cross-lingual semantic represen- tation and similarity techniques. The task has two related subtasks: multilingual semantic sim- ilarity (Section 1.1), which focuses on representa- tion learning for individual languages, and cross- lingual semantic similarity (Section 1.2), which provides a benchmark for multilingual research that learns unified representations for multiple lan- …

Content truncated to 500 words