Automatic training of lemmatization rules that handle morphological changes in pre-, in- and suffixes alike

Proceedings of the 47th Annual Meeting of the ACL and the 4th IJCNLP of the AFNLP, pages 145–153, Suntec, Singapore, 2-7 August 2009. c⃝2009 ACL and AFNLP Automatic training of lemmatization rules that handle morphological changes in pre-, in- and suffixes alike Bart Jongejan CST-University of Copenhagen Njalsgade 140-142 2300 København S Denmark bartj@hum.ku.dk Hercules Dalianis† ‡ †DSV, KTH - Stockholm University Forum 100, 164 40 Kista, Sweden ‡Euroling AB, SiteSeeker Igeldammsgatan 22c 112 49 Stockholm, Sweden hercules@dsv.su.se Abstract We propose a method to automatically train lemmatization rules that handle prefix, infix and suffix changes to generate the lemma from the full form of a word. We explain how the lemmatization rules are created and how the lemmatizer works. We trained this lemmatizer on Danish, Dutch, English, German, Greek, Icelandic, Norwegian, Polish, Slovene and Swedish full form-lemma pairs respectively. We obtained significant improvements of 24 percent for Polish, 2.3 percent for Dutch, 1.5 percent for English, 1.2 percent for German and 1.0 percent for Swedish compared to plain suffix lemmatization using a suffix-only lem- matizer. Icelandic deteriorated with 1.9 per- cent. We also made an observation regarding the number of produced lemmatization rules as a function of the number of training pairs. 1 Introduction Lemmatizers and stemmers are valuable human language technology tools to improve precision and recall in an information retrieval setting. For example, stemming and lemmatization make it possible to match a query in one morphological form with a word in a document in another mor- phological form. Lemmatizers can also be used in lexicography to find new words in text mate- rial, including the words’ frequency of use. Other applications are creation of index lists for book indexes as well as key word lists Lemmatization is the process of reducing a word to its base form, normally the dictionary look-up form (lemma) of the word. A trivial way to do this is by dictionary look-up. More ad- vanced systems use hand crafted or automatically generated transformation rules that look at the surface form of the word and attempt to produce the correct base form by replacing all or parts of the word. Stemming conflates a word to its stem. A stem does not have to be the lemma of the word, but can be any trait that is shared between a group of words, so that even the group membership itself can be regarded as the group’s stem. The most famous stemmer is the Porter Stem- mer for English (Porter 1980). This stemmer re- moves around 60 different suffixes, using rewrit- ing rules in two steps. The paper is structured as follows: section 2 discusses related work, section 3 explains what the new algorithm is supposed to do, section 4 describes some details of the new algorithm, sec- tion 5 evaluates the results, conclusions are drawn in section 6, and finally in section 7 we mention plans for further tests and improve- ments. 2 Related work There have been some attempts in creating stemmers or lemmatizers automatically. …

Content truncated to 500 words