Unsupervised Segmentation of Words Using Prior Distributions of Morph Length and Frequency

Unsupervised Segmentation of Words Using Prior Distributions of Morph Length and Frequency Mathias Creutz Neural Networks Research Centre, Helsinki University of Technology P.O.Box 9800, FIN-02015 HUT, Finland Mathias.Creutz@hut.fi Abstract We present a language-independent and unsupervised algorithm for the segmenta- tion of words into morphs. The algorithm is based on a new generative probabilis- tic model, which makes use of relevant prior information on the length and fre- quency distributions of morphs in a lan- guage. Our algorithm is shown to out- perform two competing algorithms, when evaluated on data from a language with agglutinative morphology (Finnish), and to perform well also on English data. 1 Introduction In order to artificially “understand” or produce nat- ural language, a system presumably has to know the elementary building blocks, i.e., the lexicon, of the language. Additionally, the system needs to model the relations between these lexical units. Many ex- isting NLP (natural language processing) applica- tions make use of words as such units. For in- stance, in statistical language modelling, probabil- ities of word sequences are typically estimated, and bag-of-word models are common in information re- trieval. However, for some languages it is infeasible to construct lexicons for NLP applications, if the lexi- cons contain entire words. In especially agglutina- tive languages,1 such as Finnish and Turkish, the 1In agglutinative languages words are formed by the con- catenation of morphemes. number of possible different word forms is simply too high. For example, in Finnish, a single verb may appear in thousands of different forms (Karls- son, 1987). According to linguistic theory, words are built from smaller units, morphemes. Morphemes are the smallest meaning-bearing elements of language and could be used as lexical units instead of entire words. However, the construction of a comprehensive mor- phological lexicon or analyzer based on linguistic theory requires a considerable amount of work by experts. This is both time-consuming and expen- sive and hardly applicable to all languages. Further- more, as language evolves the lexicon must be up- dated continuously in order to remain up-to-date. Alternatively, an interesting field of research lies open: Minimally supervised algorithms can be de- signed that automatically discover morphemes or morpheme-like units from data. There exist a num- ber of such algorithms, some of which are entirely unsupervised and others that use some knowledge of the language. In the following, we discuss recent un- supervised algorithms and refer the reader to (Gold- smith, 2001) for a comprehensive survey of previous research in the whole field. Many algorithms proceed by segmenting (i.e., splitting) words into smaller components. Often the limiting assumption is made that words con- sist of only one stem followed by one (possibly empty) suffix (D´ejean, 1998; Snover and Brent, 2001; Snover et al., 2002). This limitation is reduced in (Goldsmith, 2001) by allowing a recursive struc- ture, where stems can have inner structure, so that they in turn consist of a substem and a suffix. Also prefixes are possible. However, for languages with agglutinative morphology this may not be enough. In …

Content truncated to 500 words