Cro36WSD: A Lexical Sample for Croatian Word Sense Disambiguation

Cro36WSD: A Lexical Sample for Croatian Word Sense Disambiguation Domagoj Alagi´c and Jan ˇSnajder University of Zagreb, Faculty of Electrical Engineering and Computing Text Analysis and Knowledge Engineering Lab Unska 3, 10000 Zagreb, Croatia {domagoj.alagic, jan.snajder}@fer.hr Abstract We introduce Cro36WSD, a freely-available medium-sized lexical sample for Croatian word sense disambiguation (WSD). Cro36WSD comprises 36 words: 12 adjectives, 12 nouns, and 12 verbs, balanced across both frequency bands and polysemy levels. We adopt the multi-label annotation scheme in the hope of lessening the drawbacks of discrete sense inventories and obtaining more realistic annotations from human experts. Sense-annotated data is collected through multiple annotation rounds to ensure high-quality annotations: with a 115 person-hours effort we reached an inter-annotator agreement score of 0.877. We analyze the obtained data and perform a correlation analysis between several relevant variables, including word frequency, number of senses, sense distribution skewness, average annotation time, and the observed inter-annotator agreement (IAA). Using the obtained data, we compile multi- and single-labeled dataset variants using different label aggregation schemes. Finally, we evaluate three different baseline WSD models on both dataset variants and report on the insights gained. We make both dataset variants freely available. Keywords: word sense disambiguation, lexical sample, multi-label annotation, Croatian language 1. Introduction Word sense disambiguation (WSD), a task of computation- ally determining the meaning of a word in its context (Nav- igli, 2009), is one of the longest-standing and most crucial tasks of natural language processing (NLP). Knowing the right sense of a word can be beneficial in various NLP appli- cations, such as machine translation (Carpuat and Wu, 2007), information retrieval (Stokoe et al., 2003), and information extraction (Ciaramita and Altun, 2006). An indispensable ingredient in the development and testing of WSD systems are sense-annotated corpora. Unfortu- nately, such corpora are extremely costly to produce, mainly because a sufficient number of contexts has to be manually labeled for each polysemous word. Moreover, most work on WSD has focused on English and has relied on WSD datasets built as part of the SemEval (Senseval) evaluation exercises (Navigli et al., 2007; Agirre et al., 2009; Manand- har et al., 2010; Moro and Navigli, 2015). Consequently, WSD datasets for languages other than English are compa- rably rare. In this paper, we present Cro36WSD – a medium-sized lex- ical sample dataset for Croatian WSD, which extends our Cro6WSD dataset introduced in (Alagi´c and ˇSnajder, 2015). Extensions include a larger number of words in the lexical sample, adoption of the multi-label annotation scheme, and the sense inventory more apt to the task. The latter two exten- sions are motivated by the often-discussed inadequateness of discrete sense inventories (Erk and McCarthy, 2009) and their granularities (Edmonds and Kilgarriff, 2002; Snyder and Palmer, 2004). We construct two different variants of the dataset according to different label aggregation schemes. We also carry out a correlation analysis of the collected sense-annotated data. Lastly, we evaluate three baseline WSD models and report on the insights gained. We make both dataset variants publicly available,1 in the hope …

Content truncated to 500 words