Combining Finite State and Corpus-based Techniques for Unknown Word Prediction

International Conference RANLP 2009 - Borovets, Bulgaria, pages 60–64 Combining Finite State and Corpus-based Techniques for Unknown Word Prediction Kostadin Cholakov and Gertjan van Noord University of Groningen PO Box 716 9700 AS Groningen, The Netherlands k.cholakov@rug.nl, g.j.m.van.noord@rug.nl Abstract Many NLP systems make use of various lex- icons and dictionaries. However, unknown words are a major problem for such resources when applied to real-life data. We propose a method that combines nite state techniques and web queries to deliver possible analyses for a given unknown word and to generate its paradigm. We ensure the general applicability of our approach by applying it to a test set of Dutch words. 1 Introduction Unknown words are a major hindrance for the per- formance of NLP tools that make use of lexicons and dictionaries. To overcome this problem, most applica- tions try to extract (partially) the necessary morpho- logical knowledge by implementing various heuristics and unknown word guessers. In this paper, we present a two-phase method which delivers an accurate morphological analysis for a given unknown word and generates its paradigm. It deals with open-class words nouns, adjectives and verbs based on the assumption that the other word classes are already covered by most lexicons. We test its e ciency and accuracy by applying it to real-life Dutch data. Dutch is a language with a productive in ectional morphology that exhibits quite a few in- teresting phenomena and thus poses a challenge for morphological processing. In the rst phase we use nite state techniques which are one of the most common approaches for mor- phological processing (Beesley and Karttunen, 2003; Petitpierre and Russel, 1995) since they can conveni- ently be used for both analysis and generation. We em- ploy a small set of non-deterministic `unweighted' - nite state transducers (FSTs) whose manually encoded rules cover regular morphological phenomena. Since our method deals with unknown words, it does not have access to any additional information apart from the limited knowledge provided by the word structure. Restricted only by that limited knowledge the FSTs, in analysis mode, identify all possible forms and roots allowed by the word structure. For example, the word schnabbel (a gig) is analysed as a singu- lar noun, a base adjective and a rst person singular present verb. As a consequence, three possible roots are produced. Since there is no way to know which of them is the correct one, in generation mode, all pos- sible paradigms for each of these roots are generated. The problem of disambiguating the output of the FSTs is dealt with in the second phase. We use Ya- hoo to search the web for each root and generated paradigm form and, based on the number of occur- rences found, we try to identify the correct root and paradigm of the unknown word. Commercial search engines have already been successfully used for vari- ous NLP tasks (Keller and Lapata, 2003) and it is our claim that they are su cient for ours as well. Since the whole …

Content truncated to 500 words