The Senseval-3 English lexical sample task

The SENSEVAL–3 English Lexical Sample Task Rada Mihalcea Department of Computer Science University of North Texas Dallas, TX, USA rada@cs.unt.edu Timothy Chklovski Information Sciences Institute University of Southern California Marina del Rey, CA, USA timc@isi.edu Adam Kilgarriff Information Technology Research Institute University of Brighton Brighton, UK Adam.Kilgarriff@itri.brighton.ac.uk Abstract This paper presents the task definition, resources, participating systems, and comparative results for the English lexical sample task, which was orga- nized as part of the SENSEVAL-3 evaluation exer- cise. The task drew the participation of 27 teams from around the world, with a total of 47 systems. 1 Introduction We describe in this paper the task definition, re- sources, participating systems, and comparative re- sults for the English lexical sample task, which was organized as part of the SENSEVAL-3 evaluation ex- ercise. The goal of this task was to create a frame- work for evaluation of systems that perform targeted Word Sense Disambiguation. This task is a follow-up to similar tasks organized during the SENSEVAL-1 (Kilgarriff and Palmer, 2000) and SENSEVAL-2 (Preiss and Yarowsky, 2001) evaluations. The main changes in this year’s evaluation consist of a new methodology for collecting annotated data (with contributions from Web users, as opposed to trained lexicographers), and a new sense inventory used for verb entries (Wordsmyth). 2 Building a Sense Tagged Corpus with Volunteer Contributions over the Web The sense annotated corpus required for this task was built using the Open Mind Word Expert system (Chklovski and Mihalcea, 2002) 1. To overcome the current lack of sense tagged data and the limitations imposed by the creation of such data using trained lexicographers, the OMWE system enables the col- lection of semantically annotated corpora over the Web. Sense tagged examples are collected using 1Open Mind Word Expert can be accessed at http://teach- computers.org/ a Web-based application that allows contributors to annotate words with their meanings. The tagging exercise proceeds as follows. For each target word the system extracts a set of sen- tences from a large textual corpus. These examples are presented to the contributors, who are asked to select the most appropriate sense for the target word in each sentence. The selection is made using check- boxes, which list all possible senses of the current target word, plus two additional choices, “unclear” and “none of the above.” Although users are encour- aged to select only one meaning per word, the se- lection of two or more senses is also possible. The results of the classification submitted by other users are not presented to avoid artificial biases. Similar to the annotation scheme used for the En- glish lexical sample at SENSEVAL-2, we use a “tag until two agree” scheme, with an upper bound on the number of annotations collected for each item set to four. 2.1 Source Corpora The data set used for the SENSEVAL-3 English lexical sample task consists of examples extracted from the British National Corpus (BNC). Ear- lier versions of OMWE also included data from the Penn Treebank corpus, the Los Angeles Times …

Content truncated to 500 words