Leveraging Inflection Tables for Stemming and Lemmatization.

Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 1138–1147, Berlin, Germany, August 7-12, 2016. c⃝2016 Association for Computational Linguistics Leveraging Inflection Tables for Stemming and Lemmatization Garrett Nicolai and Grzegorz Kondrak Department of Computing Science University of Alberta {nicolai,gkondrak}@ualberta.ca Abstract We present several methods for stemming and lemmatization based on discrimina- tive string transduction. We exploit the paradigmatic regularity of semi-structured inflection tables to identify stems in an un- supervised manner with over 85% accu- racy. Experiments on English, Dutch and German show that our stemmers substan- tially outperform Snowball and Morfes- sor, and approach the accuracy of a super- vised model. Furthermore, the generated stems are more consistent than those an- notated by experts. Our direct lemmatiza- tion model is more accurate than Morfette and Lemming on most datasets. Finally, we test our methods on the data from the shared task on morphological reinflection. 1 Introduction Many languages contain multiple inflected forms that correspond to the same dictionary word. In- flection is a grammatical procedure that has little impact on the meaning of the word. For example, the German words in Table 1 all refer to the action of giving. When working with these languages, it is often beneficial to establish a consistent rep- resentation across a set of inflections. This is the task that we address here. There are two principal approaches to inflec- tional simplification: stemming and lemmatiza- tion. Stemming aims at removing inflectional af- fixes from a word form. It can be viewed as a kind of word segmentation, in which the boundaries of the stem are identified within the word; no attempt is made to restore stem changes that may occur as part of the inflection process. The goal of lemma- tization is to map any inflected form to its unique lemma, which is typically the word form that rep- Word form Meaning Tag Stem geben “to give” INF geb gibt “gives” 3SIE gib gab “gave” 1SIA gab gegeben “given” PP geb Table 1: Examples of German word-forms corre- sponding to the lemma geben. resents a set of related inflections in a dictionary. Unlike stemming, lemmatization must always pro- duce an actual word form. In this paper, we present a discriminative string transduction approach to both stemming and lemmatization. Supervised stemmers require mor- phologically annotated corpora, which are expen- sive to build. We remove this constraint by ex- tracting stems from semi-structured inflection ta- bles, such as the one shown in Table 2, in an un- supervised manner. We design two transduction models that are trained on such stems, and eval- uate them on unseen forms against a supervised model. We then extend our stemming models to perform the lemmatization task, and to incorporate an unannotated corpus. We evaluate them on sev- eral datasets.Our best system improves the state of the art for Dutch, German, and Spanish. Finally, we test our methods on the data from the shared task on morphological reinflection. This paper is organized as follows. In Section …

Content truncated to 500 words