A Self-Learning Context-Aware Lemmatizer for German

Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing (HLT/EMNLP), pages 636–643, Vancouver, October 2005. c⃝2005 Association for Computational Linguistics A Self-Learning Context-Aware Lemmatizer for German Praharshana Perera and Ren´e Witte Institute for Program Structures and Data Organization (IPD) Universit¨at Karlsruhe, Germany perera|witte@ipd.uka.de Abstract Accurate lemmatization of German nouns mandates the use of a lexicon. Compre- hensive lexicons, however, are expensive to build and maintain. We present a self- learning lemmatizer capable of automati- cally creating a full-form lexicon by pro- cessing German documents. 1 Introduction Lemmatization is the process of deriving the base form, or lemma, of a word from one of its inflected forms. For morphologically complex languages like German this is not a simple task that can be solved solely through a rule-based algorithm: Performing an accurate lemmatization for German requires a lexicon. This can be either a lexicon containing all inflected forms of a word together with its base form (full-form lexicon), or just the lemma together with a set of rules for creating its inflected forms (base- form lexicon) (Hausser, 2000). Creating such a lexicon by hand, however, is ex- pensive and time-consuming. Perhaps because of this there are currently no freely available lexical resources for German that include full case and in- flection information.1 Moreover, even a full-scale commercial lexicon can fail when encountering spe- cialized terminology. As a consequence, most systems processing Ger- man texts currently perform the much simpler task of stemming, which often generates stem forms of words that might not actually exist in the language (so-called overstemming). Stemming is frequently used for information retrieval (IR) tasks, an example being the German stemmer contained in the full-text 1The free online dictionary Wiktionary (http://de. wiktionary.org/) had at the time of writing (May 2005) less than 5000 entries for German. search engine Lucene,2 which is based on the algo- rithm described in (Caumanns, 1999). While over- stemming is a feasible approach for text retrieval, a text mining system often needs to obtain a more precise lemma, for example, in order to perform a gazetteer lookup to identify named entities or for de- scription logic (DL) queries within an ontology. The goal of our work, therefore, is to allow the semi-automatic generation of a lexicon by mining full-text documents. Since there are currently no free lemmatization systems for German available,3 all components have been developed for release as free, open-source software. 2 Lemmatization Algorithm Our lemmatization system has two main compo- nents, an algorithm and a lexicon. The algorithm lemmatizes German nouns depending on morpho- logical classes. The lexicon, which is described in Section 3, is generated from the nouns that have been processed by this algorithm, with some addi- tional capabilities for self-correction. The lemmatization algorithm considers the con- text and grammatical features of the language to lemmatize German words. It requires an additional POS tagger and an NP chunker, which are used as resources to extract the features of words and their surrounding context. It has …

Content truncated to 500 words