The Basque Task: Did Systems Perform in the Upperbound?

The Basque task: did systems perform in the upperbound? Eneko Agirre, Elena Garcia, Mikel Lersundi, David Martinez, Eli Pociello IxA NLP group, Basque Country University 649 pk. 20.080 Donostia, Spain eneko@ si.ehu.es Abstract In this paper we describe the Senseval 2 Basque lexical-sample task. The task comprised 40 words (15 nouns, 15 verbs and 10 adjectives) selected from Euskal Hiztegia, the main Basque dictionary. Most examples were taken from the Egunkaria newspaper. The method used to hand-tag the examples produced low inter-tagger agreement (75%) before arbitration. The four competing systems attained results well above the most frequent baseline and the best system scored 75% precision at 100% coverage. The paper includes an analysis of the tagging procedure used, as well as the performance of the competing systems. In particular, we argue that inter-tagger agreement is not a real upperbound for the B,asque WSD task. 1 Introduction This paper reviews the design of the lexical- sample task for Basque. The following steps were taken in order to build the hand-tagged corpus: 1. set the exercise a. choose sense inventory b. choose target corpus c. choose target words d. select examples from the corpus 2. hand-tagging a. define procedure b. tag c. analysis of inter-tagger agreement d. arbitration The following section presents the setting of the exercise. Section 3 reviews the hand-tagging, and section 4 the results of the participant systems. Section 5 discusses the design of the task, as well 9 as the results, and section 6 presents some future work. 2 Setting of the exercise In this section we present the setting of the Basque lexical-sample exercise. 2.1 Basque Basque is an agglutinative language, that is, for the formation of words, the dictionary entry independently takes each of the elements necessary for the different functions (syntactic case included). More specifically, the affixes corresponding to the determinant, number and declension case are taken in this order and independently of each other (deep morphological structure). One of the main characteristics of Basque is its declension system with numerous cases, which differentiates it from the languages spoken in the surrounding countries. An example follows (the order of the lemmas is the reverse): etxekoari emaiozu [Give it] [to the one in the house] 2.2 Sense inventory We chose a published dictionary, Euskal Hiztegia (Sarasola, 1996), for the sense inventory. It is a monolingual dictionary of Basque. It is normative and repository of standard Basque. It was produced based mainly on literary tradition. The dictionary has 30,715 entries and 41,699 main senses (see comment on nuances below). The TEl version with all the information for each entry was included in the distribution. As the format was quite complex, another version was also included, which listed a plain list of word senses and multiword terms. This dictionary has the particularity that word senses can have very specific sub-senses, called nuances which sometimes are illustrated with just an example and other times have a full definition. · These nuances were also included in the set of word …

Content truncated to 500 words