SemEval-2010 Task 2: Cross-Lingual Lexical Substitution

Proceedings of the 5th International Workshop on Semantic Evaluation, ACL 2010, pages 9–14, Uppsala, Sweden, 15-16 July 2010. c⃝2010 Association for Computational Linguistics SemEval-2010 Task 2: Cross-Lingual Lexical Substitution Rada Mihalcea University of North Texas rada@cs.unt.edu Ravi Sinha University of North Texas ravisinha@unt.edu Diana McCarthy Lexical Computing Ltd. diana@dianamccarthy.co.uk Abstract In this paper we describe the SemEval- 2010 Cross-Lingual Lexical Substitution task, where given an English target word in context, participating systems had to find an alternative substitute word or phrase in Spanish. The task is based on the English Lexical Substitution task run at SemEval-2007. In this paper we pro- vide background and motivation for the task, we describe the data annotation pro- cess and the scoring system, and present the results of the participating systems. 1 Introduction In the Cross-Lingual Lexical Substitution task, an- notators and systems had to find an alternative substitute word or phrase in Spanish for an En- glish target word in context. The task is based on the English Lexical Substitution task run at SemEval-2007, where both target words and sub- stitutes were in English. An automatic system for cross-lingual lexical substitution would be useful for a number of ap- plications. For instance, such a system could be used to assist human translators in their work, by providing a number of correct translations that the human translator can choose from. Similarly, the system could be used to assist language learners, by providing them with the interpretation of the unknown words in a text written in the language they are learning. Last but not least, the output of a cross-lingual lexical substitution system could be used as input to existing systems for cross- language information retrieval or automatic ma- chine translation. 2 Motivation and Related Work While there has been a lot of discussion on the rel- evant sense distinctions for monolingual WSD sys- tems, for machine translation applications there is a consensus that the relevant sense distinctions are those that reflect different translations. One early and notable work was the SENSEVAL-2 Japanese Translation task (Kurohashi, 2001) that obtained alternative translation records of typical usages of a test word, also referred to as a translation mem- ory. Systems could either select the most appro- priate translation memory record for each instance and were scored against a gold-standard set of an- notations, or they could provide a translation that was scored by translation experts after the results were submitted. In contrast to this work, in our task we provided actual translations for target in- stances in advance, rather than predetermine trans- lations using lexicographers or rely on post-hoc evaluation, which does not permit evaluation of new systems after the competition. Previous standalone WSD tasks based on par- allel data have obtained distinct translations for senses as listed in a dictionary (Ng and Chan, 2007). In this way fine-grained senses with the same translations can be lumped together, how- ever this does not fully allow for the fact that some senses for the same words may have …

Content truncated to 500 words