SemEval-2007 Task 10: English Lexical Substitution Task

Proceedings of the 4th International Workshop on Semantic Evaluations (SemEval-2007), pages 48–53, Prague, June 2007. c⃝2007 Association for Computational Linguistics SemEval-2007 Task 10: English Lexical Substitution Task Diana McCarthy University of Sussex Falmer, East Sussex BN1 9QH, UK dianam@sussex.ac.uk Roberto Navigli University of Rome “La Sapienza” Via Salaria, 113 00198 Roma, Italy navigli@di.uniroma1.it Abstract In this paper we describe the English Lexical Substitution task for SemEval. In the task, annotators and systems find an alternative substitute word or phrase for a target word in context. The task involves both finding the synonyms and disambiguating the context. Participating systems are free to use any lex- ical resource. There is a subtask which re- quires identifying cases where the word is functioning as part of a multiword in the sen- tence and detecting what that multiword is. 1 Introduction Word sense disambiguation (WSD) has been de- scribed as a task in need of an application. Whilst researchers believe that it will ultimately prove use- ful for applications which need some degree of se- mantic interpretation, the jury is still out on this point. One problem is that WSD systems have been tested on fine-grained inventories, rendering the task harder than it need be for many applications (Ide and Wilks, 2006). Another significant problem is that there is no clear choice of inventory for any given task (other than the use of a parallel corpus for a specific language pair for a machine translation application). The lexical substitution task follows on from some previous ideas (McCarthy, 2002) to exam- ine the capabilities of WSD systems built by re- searchers on a task which has potential for NLP applications. Finding alternative words that can occur in given contexts would potentially be use- ful to many applications such as question answer- ing, summarisation, paraphrase acquisition (Dagan et al., 2006), text simplification and lexical acquisi- tion (McCarthy, 2002). Crucially this task does not specify the inventory for use beforehand to avoid bias to one predefined inventory and makes it eas- ier for those using automatically acquired resources to enter the arena. Indeed, since the systems in SemEval did not know the candidate substitutes for a word before hand, the lexical resource is evaluated as much as the context based disambiguation com- ponent. 2 Task set up The task involves a lexical sample of nouns, verbs, adjectives and adverbs. Both annotators and sys- tems select one or more substitutes for the target word in the context of a sentence. The data was selected from the English Internet Corpus of En- glish produced by Sharoff (2006) from the Inter- net (http://corpus.leeds.ac.uk/internet.html). This is a balanced corpus similar in flavour to the BNC, though with less bias to British English, obtained by sampling data from the web. Annotators are not provided with the PoS (noun, verb, adjective or ad- verb) but the systems are. Annotators can provide up to three substitutes but all should be equally as good. They are instructed that they can provide a phrase if they …

Content truncated to 500 words