String Distance-Based Stemming of the Highly Inflected Croatian Language

International Conference RANLP 2009 - Borovets, Bulgaria, pages 411–415 String Distance-Based Stemming of the Highly Inflected Croatian Language Jan ˇSnajder and Bojana Dalbelo Baˇsi´c Faculty of Electrical Engineering and Computing, University of Zagreb Unska 3, 10000 Zagreb, Croatia {jan.snajder, bojana.dalbelo}@fer.hr Abstract Stemming refers to the grouping of morphologi- cally related words into so-called stem classes for the purpose of improving information retrieval performance. Traditional approaches to stem- ming are language-specific and require a sub- stantial amount of linguistic knowledge. A vi- able alternative is string distance-based stem- ming, in which stem classes are obtained by clus- tering word-forms from a corpus. In this pa- per, we apply string distance-based stemming to the highly inflected Croatian language using a number of string distance measures proposed in the literature. We focus on evaluating the stemming performance at both inflectional and derivational level, and investigate how this per- formance relates to the choice of the distance threshold value. Although our focus is on the Croatian language, we believe our results trans- fer well to languages of similar morphological complexity. Keywords Stemming, morphology, string distance,Croatian language 1 Introduction Most information retrieval (IR) systems represent doc- uments simply as a collection of words. The perfor- mance of such systems is negatively affected by the fact that words in texts appear in various morphologi- cal forms, either as the result of inflection (transforma- tion of a word into various word-forms) or derivation (transformation of a word into new, but semantically related words). To reduce morphological variation, IR systems typically rely upon some sort of morphologi- cal normalization to conflate the various morphologi- cal forms into a single representative form. Numerous studies have shown morphological normalization to be beneficial for IR; this has been shown for English [4], as well as for other, more morphologically complex lan- guages [13, 15]. The most common morphological normalization technique is stemming. Stemming refers to the removal of affixes from word-forms, yielding a stem common to all word-forms. The well-known Porter algorithm [11] is an example of such a rule-based approach to stem- ming. More generally, stemming refers to the pro- cess of grouping morphologically related word-forms into the so-called stem classes. Traditional rule- and dictionary-based stemming requires significant linguis- tic expertise and resources, which is why, for resource- poor languages, language-independent approaches are gaining popularity. Among others, string distance- based stemming, in which stem classes are derived by clustering word-forms based on their character struc- ture, has been shown to be a viable alternative. String distance-based clustering was first proposed by Adam- son and Boreham [1] for the English language, and sim- ilar approaches were later employed for Arabic [12] and Turkish [3]. More recently, Majumder et al. [9] pro- posed string distance measures for stemming in Ben- gali, as well as in Hungarian and Czech [8]. The per- formance of their stemming procedure has been shown to be comparable to traditional rule-based approaches. In this paper, we investigate the applicability of string distance-based stemming to the Croatian lan- guage. …

Content truncated to 500 words