An evaluation exercise for Romanian Word Sense Disambiguation

An Evaluation Exercise for Romanian Word Sense Disambiguation Rada Mihalcea Department of Computer Science University of North Texas Dallas, TX, USA rada@cs.unt.edu Vivi N˘astase School of Computer Science University of Ottawa Ottawa, ON, Canada vnastase@site.uottawa.ca Timothy Chklovski Information Sciences Institute University of Southern California Marina del Rey, CA, USA timc@isi.edu Doina T˘atar Department of Computer Science Babes¸-Bolyai University Cluj-Napoca, Romania dtatar@ubb.ro Dan Tufis¸ Romanian Academy Center for Artificial Intelligence Bucharest, Romania tufis@racai.ro Florentina Hristea Department of Computer Science University of Bucharest Bucharest, Romania fhristea@mailbox.ro Abstract This paper presents the task definition, resources, participating systems, and comparative results for a Romanian Word Sense Disambiguation task, which was organized as part of the SENSEVAL-3 evaluation exercise. Five teams with a total of seven systems were drawn to this task. 1 Introduction SENSEVAL is an evaluation exercise of the lat- est word-sense disambiguation (WSD) systems. It serves as a forum that brings together researchers in WSD and domains that use WSD for various tasks. It allows researchers to discuss modifications that improve the performance of their systems, and an- alyze combinations that are optimal. Since the first edition of the SENSEVAL competi- tions, a number of languages were added to the orig- inal set of tasks. Having the WSD task prepared for several languages provides the opportunity to test the generality of WSD systems, and to detect dif- ferences with respect to word senses in various lan- guages. This year we have proposed a Romanian WSD task. Five teams with a total of seven systems have tackled this task. We present in this paper the data used and how it was obtained, and the performance of the participating systems. 2 Open Mind Word Expert The sense annotated corpus required for this task was built using the Open Mind Word Expert system (Chklovski and Mihalcea, 2002), adapted to Roma- nian1. To overcome the current lack of sense tagged data and the limitations imposed by the creation of such data using trained lexicographers, the Open Mind Word Expert system enables the collection of semantically annotated corpora over the Web. Sense tagged examples are collected using a Web- based application that allows contributors to anno- tate words with their meanings. The tagging exercise proceeds as follows. For each target word the system extracts a set of sen- tences from a large textual corpus. These examples are presented to the contributors, who are asked to select the most appropriate sense for the target word in each sentence. The selection is made using check- boxes, which list all possible senses of the current target word, plus two additional choices, “unclear” and “none of the above.” Although users are en- couraged to select only one meaning per word, the selection of two or more senses is also possible. The results of the classification submitted by other users are not presented to avoid artificial biases. 3 Sense inventory For the Romanian WSD task, we have chosen a set of words from three parts of speech - nouns, verbs and adjectives. …

Content truncated to 500 words