Combining Distributional and Morphological Information for Part of Speech Induction

Combining Distributional and Morphological Information for Part of Speech Induction Alexander Clark ISSCO / TIM University of Geneva UNI-MAIL, Boulevard du Pont-d'Arve, CH-1211 Geneve 4, Switzerland Alex.Clark@issco.unige.ch Abstract In this paper we discuss algorithms for clustering words into classes from un- labelled text using unsupervised algo- rithms, based on distributional and mor- phological information. We show how the use of morphological information can improve the performance on rare words, and that this is robust across a wide range of languages. 1 Introduction The task studied in this paper is the unsupervised learning of parts-of-speech, that is to say lexical categories corresponding to traditional notions of, for example, nouns and verbs. As is often the case in machine learning of natural language, there are two parallel motivations: first a simple engineer- ing one — the induction of these categories can help in smoothing and generalising other mod- els, particularly in language modelling for speech recognition as explored by (Ney et al., 1994) and secondly a cognitive science motivation — explor- ing how evidence in the primary linguistic data can account for first language acquisition by in- fant children (Finch and Chater, 1992a; Finch and Chater, 1992b; Redington et al., 1998). At this early phase of learning, only limited sources of information can be used: primarily distributional evidence, about the contexts in which words oc- cur, and morphological evidence, (more strictly phonotactic or orthotactic evidence) about the se- quence of symbols (letters or phonemes) of which each word is formed. A number of different ap- proaches have been presented for this task using exclusively distributional evidence to cluster the words together, starting with (Lamb, 1961) and these have been shown to produce good results in English, Japanese and Chinese. These languages have however rather simple morphology and thus words will tend to have higher frequency than in more morphologically complex languages. In this paper we will address two issues: first, whether the existing algorithms work adequately on a range of languages and secondly how we can incorporate morphological information. We are particularly interested in rare words: as (Rosen- feld, 2000, pp.1313-1314) points out, it is most important to cluster the infrequent words, as we will have reliable information about the frequent words; and yet it is these words that are most dif- ficult to cluster. We accordingly focus both in our algorithms and our evaluation on how to cluster words effectively that occur only a few times (or not at all) in the training data. In addition we are interested primarily in inducing small numbers of clusters (at most 128) from comparatively small amounts of data using limited or no sources of external knowledge, and in approaches that will work across a wide range of languages, rather than inducing large numbers (say 1000) from hundreds of millions of words. Note this is different from the common task of guessing the word category of an unknown word given a pre-existing set of parts-of-speech, a task which has been studied ex- tensively (Mikheev, 1997). …

Content truncated to 500 words