SemEval-2007 Task-17: English Lexical Sample, SRL and All Words

Proceedings of the 4th International Workshop on Semantic Evaluations (SemEval-2007), pages 87–92, Prague, June 2007. c⃝2007 Association for Computational Linguistics SemEval-2007 Task 17: English Lexical Sample, SRL and All Words Sameer S. Pradhan BBN Technologies, Cambridge, MA 02138 Edward Loper University of Pennsylvania, Philadelphia, PA 19104 Dmitriy Dligach and Martha Palmer University of Colorado, Boulder, CO 80303 Abstract This paper describes our experience in preparing the data and evaluating the results for three subtasks of SemEval-2007 Task-17 – Lexical Sample, Semantic Role Labeling (SRL) and All-Words respectively. We tab- ulate and analyze the results of participating systems. 1 Introduction Correctly disambiguating words (WSD), and cor- rectly identifying the semantic relationships be- tween those words (SRL), is an important step for building successful natural language processing ap- plications, such as text summarization, question an- swering, and machine translation. SemEval-2007 Task-17 (English Lexical Sample, SRL and All- Words) focuses on both of these challenges, WSD and SRL, using annotated English text taken from the Wall Street Journal and the Brown Corpus. It includes three subtasks: i) the traditional All- Words task comprising fine-grained word sense dis- ambiguation using a 3,500 word section of the Wall Street Journal, annotated with WordNet 2.1 sense tags, ii) a Lexical Sample task for coarse-grained word sense disambiguation on a selected set of lex- emes, and iii) Semantic Role Labeling, using two different types of arguments, on the same subset of lexemes. 2 Word Sense Disambiguation 2.1 English fine-grained All-Words In this task we measure the ability of systems to identify the correct fine-grained WordNet 2.1 word sense for all the verbs and head words of their argu- ments. 2.1.1 WeData Preparation began by selecting three articles wsj 0105.mrg (on homelessness), wsj 0186.mrg (about a book on corruption), and wsj 0239.mrg (about hot-air ballooning) from a section of the WSJ corpus that has been Treebanked and PropBanked. All instances of verbs were identified using the Treebank part-of-speech tags, and also the head- words of their noun arguments (using the PropBank and standard headword rules). The locations of the sentences containing them as well as the locations of the verbs and the nouns within these sentences were recorded for subsequent sense-annotation. A total of 465 lemmas were selected from about 3500 words of text. We use a tool called STAMP written by Ben- jamin Snyder for sense-annotation of these in- stances. STAMP accepts a list of pointers to the in- stances that need to be annotated. These pointers consist of the name of the file where the instance is located, the sentence number of the instance, and finally, the word number of the ambiguous word within that sentence. These pointers were obtained as described in the previous paragraph. STAMP also requires a sense inventory, which must be stored in XML format. This sense inventory was obtained by querying WordNet 2.1 and storing the output as a 87 set of XML files (one for each word to be anno- tated) prior to tagging. STAMP works by displaying to the …

Content truncated to 500 words